Hermetic builds are not an optimisation
Remote caching on a non-reproducible build does not make it faster. It makes it confidently wrong, and I learned that the expensive way.
I enabled remote caching in the second week of a CI overhaul, because it was the change with the most obvious payoff. It produced two genuinely wrong builds before I turned it off — artefacts served from cache that did not match what the source would have produced. That cost me a fortnight and a measurable amount of credibility.
A cache key is a claim about inputs
When a build system hashes a task and serves a cached artefact, it is asserting that the hash fully describes everything that could affect the output. If the toolchain version, a system library, an environment variable or the machine architecture can vary without changing the hash, that assertion is false — and the cache will confidently hand you an artefact built from different inputs.
Without a cache, that non-determinism shows up as flakiness. Annoying, visible, and someone eventually investigates. With a cache, it shows up as an artefact that is simply wrong, deterministically, until the key changes. That is much harder to find.
Hermeticity first, caching second
Pinning the toolchain and system dependencies — Nix in my case, though the tool matters less than the property — took two of the four months on that project. It produced no visible speedup on its own. It was also the prerequisite for every gain that followed, because it made the task hash genuinely describe its inputs.
- Pin the compiler, the runtime, and the system libraries. All of them, by content hash.
- Declare inputs explicitly; a task that reads an undeclared file is a bug waiting for a cache.
- Strip timestamps and paths from artefacts so identical inputs produce byte-identical output.
- Verify by building the same commit on two different machines and diffing the hashes.
Hermetic inputs are not a refinement you add to a cache. They are the thing that makes a cache safe.
The payoff, once it is safe
With reproducibility in place, the rest arrived quickly: an 89% cache hit rate, median pipeline time from 47 minutes to 6, and a 64% lower bill despite running four times as often. None of that was possible in week two, and all of it was inevitable by month three.
The general lesson is one I keep relearning in different clothes. When a change makes something faster, ask what it is assuming. If the assumption is not enforced, you have not made it faster — you have made it fast and occasionally wrong, and you will pay for the second half later.