Problem

A build host in my fleet needed a full system closure built for a specific machine. Eleven builds failed. Each attempt was long, each failure looked different, and the build was blocking work on the rest of the fleet. What should have been a routine task turned into a fourteen-hour debugging session.

Diagnosis

The failures traced to two stacked problems. First, a microarchitecture mismatch: the build was targeting the wrong CPU generation for the machine, so compiled output failed in ways that looked random build flakiness rather than a configuration error. Second, the storage path the build ran over was slow for the calculation phases, dragging every retry out and making the diagnosis process itself expensive. The two problems masked each other: slow builds plus intermittent failures read like a compiler fault.

Action

• Worked the mismatch iteratively, eleven failed closures deep, until the CPU targeting was the proven cause
• Applied a live NFS fix, an attribute-cache bind-mount on the store path, which sped up the calculation phases without a reboot
• Wrote the whole saga up as a 505-line handoff: live state per host, runbook commands, and a lessons-learned section
• Followed up later when /proc/cpuinfo showed the machine is Excavator generation (bdver4), not the btver2 originally documented, and retuned the build cache and CI accordingly

Outcome

• The build path was fixed and the fleet unblocked
• The storage bottleneck was removed for every subsequent build
• A documented, honest failure record exists instead of tribal memory
• The follow-up caught a wrong assumption in my own documentation and corrected it

What I'd do differently