Hermetic builds are not an optimisation

Remote caching is usually the first thing reached for in a CI overhaul, because it is the change with the most obvious payoff. On a build that is not reproducible, it is also the change most likely to serve an artefact that does not match what the source would have produced — silently, and with the full confidence of a cache hit. Finding that costs far more time and trust than the cache ever saved.

A cache key is a claim about inputs

When a build system hashes a task and serves a cached artefact, it is asserting that the hash fully describes everything that could affect the output. If the toolchain version, a system library, an environment variable or the machine architecture can vary without changing the hash, that assertion is false — and the cache will confidently hand you an artefact built from different inputs.

Without a cache, that non-determinism shows up as flakiness. Annoying, visible, and someone eventually investigates. With a cache, it shows up as an artefact that is simply wrong, deterministically, until the key changes. That is much harder to find.

Hermeticity first, caching second

Pinning the toolchain and system dependencies — with Nix, or anything that gives the same guarantee; the property matters more than the tool — is slow, unglamorous work. It produces no visible speedup on its own. It is also the prerequisite for every gain that follows, because it makes the task hash genuinely describe its inputs.

• Pin the compiler, the runtime, and the system libraries. All of them, by content hash.
• Declare inputs explicitly; a task that reads an undeclared file is a bug waiting for a cache.
• Strip timestamps and paths from artefacts so identical inputs produce byte-identical output.
• Verify by building the same commit on two different machines and diffing the hashes.

Hermetic inputs are not a refinement you add to a cache. They are the thing that makes a cache safe.

The payoff, once it is safe

With reproducibility in place, the rest arrives quickly: a cache that actually hits, pipelines that finish in a fraction of the time, and a smaller bill even while they run more often. None of that is safely available on day one. All of it follows once the inputs are honest.

The general lesson keeps coming back in different clothes. When a change makes something faster, ask what it is assuming. If the assumption is not enforced, you have not made it faster — you have made it fast and occasionally wrong, and you will pay for the second half later.