Degrade in latency, not in correctness
When a system saturates it has to give something up. Almost every default gives up the wrong thing — and users only find out much later.
Every system has a point where demand exceeds what it can serve. The interesting question is not whether you reach that point — you will — but what the system chooses to sacrifice when it gets there. That choice is usually made by accident, buried in a queue depth or a timeout that someone set to a round number three years ago.
There are only two things to give up: time, or truth. A system can get slower, or it can get wrong. The overwhelming majority of defaults choose wrong, because dropping a message looks like recovery on a dashboard while a growing queue looks like a problem.
Dropping is not recovery, it is deferred cost
When an ingestion tier drops messages under load, the graph recovers immediately. Latency returns to normal, error rates fall, the page resolves. It looks like the system healed itself. What actually happened is that the cost moved somewhere it will not be noticed for weeks — into a report that is quietly missing an hour, or a reconciliation that will not balance at month end.
On the edge telemetry work I did, the previous managed service treated every network disconnection as data loss. The dashboards looked healthy. The holes appeared exactly where the interesting failures were, because a device having a bad time is also a device on a bad link.
Back-pressure means telling the truth upstream
The alternative is to push the constraint back to the producer. When the ingestion tier saturates, it says so, and producers slow down or buffer. Nothing is lost; things simply arrive later. The system degrades along the axis you can afford.
- The producer holds its own data, so the buffer is distributed rather than concentrated.
- Lateness is visible and measurable; loss is neither until it is too late.
- Recovery drains oldest-first with a cap, so a recovering fleet cannot stampede the tier it just overwhelmed.
A system that gets later is a system you can reason about. A system that gets wrong is one you find out about from a customer.
Where this actually costs you
This is not free, and it is worth being honest about the trade. Back-pressure requires bounded buffers at the edge, which means the producer needs somewhere to put things and a policy for what happens when even that fills. It requires a signal path back from consumer to producer, which is real protocol work. And it makes your latency graphs look worse under load, which means you have to explain to people that the worse-looking graph is the healthier system.
There are also cases where dropping is genuinely correct. If you are sampling metrics for a dashboard, losing a data point costs nothing and holding the whole fleet back to preserve it is absurd. The rule is not "never drop" — it is "know which one you are choosing, and choose it deliberately".
The question to ask
Pick any component in your system and ask what it does at twice the expected load. If the answer is "it drops things", ask who finds out, and when. If the answer is "a customer, next month", you have not built a resilient system. You have built one that fails quietly, which is considerably worse than one that fails loudly.