Retries are the first resilience pattern most people add and the one most likely to cause the outage it was meant to prevent.
The logic seems unimpeachable. Networks are unreliable. Some failures are transient. Trying again converts a transient failure into a success, and the user never notices. All of that is true, and it’s true right up until the failure stops being transient — at which point the same mechanism becomes an amplifier pointed at a system that is already struggling.
The mechanics of a retry storm#
Consider a service calling a dependency that starts responding slowly — not failing, just degraded. Maybe a database is doing more work than usual, maybe an instance is being replaced.
Callers time out. Callers retry. The dependency now receives its normal traffic plus retry traffic, which makes it slower. More calls time out. Those retry too.
If your retry policy is three attempts, a degradation that would have caused a partial failure now produces four times the normal request volume aimed at the component least able to handle it. The dependency was struggling at 1x. It has no chance at 4x.
Worse, this composes. If service A calls B calls C, and each retries three times, a failure at C sees up to sixteen attempts for one user request. Retry multiplication through a call chain is how a small dependency problem becomes a platform-wide outage, and it’s essentially invisible in code review because each service’s retry policy looks reasonable in isolation.
Then the failure clears — and every client retries simultaneously, because they all backed off by the same amount and are now synchronized. The recovering system takes a thundering herd on its first breath and falls over again.
Four things that make retries safe#
Exponential backoff with jitter#
Backoff alone isn’t enough, and the reason is the synchronization above. If every client waits exactly 1s, then 2s, then 4s, clients that failed together retry together forever.
Jitter — randomizing the delay — breaks the synchronization. It’s a one-line change and it’s the difference between a recovering system getting a smooth ramp and getting a wall.
The variant worth knowing is decorrelated jitter, where each delay is sampled from a range based on the previous delay rather than the attempt number. It spreads retries more evenly than adding randomness to a fixed schedule.
Retry budgets#
Per-request retry limits don’t bound aggregate load. Three retries per request sounds conservative, and if every request is failing, it’s still 3x traffic.
A retry budget caps retries as a fraction of total requests — for example, retries may not exceed 10% of successful request volume. When failures are isolated, retries proceed normally. When everything is failing, the budget exhausts and retries stop, which is exactly the behavior you want: retry when the failure is likely transient, stop when it clearly isn’t.
This single mechanism prevents most retry-induced outages, and it’s absent from most retry implementations.
Retry only what’s safe to retry#
Not every failed request should be retried, and the decision isn’t only about idempotency.
A 500 might mean the request was never processed, or it was processed and the response was lost. Retrying the second case duplicates the work. If the operation isn’t idempotent, that’s a correctness bug that appears only under failure conditions — the worst kind to debug.
Some failures should never be retried at all. A 400 will be a 400 next time. Authentication failures don’t fix themselves. Retrying deterministic failures is pure waste and it consumes budget that transient failures need.
The practical guidance: retry on timeouts, connection failures, 429s and 503s with a retry-after honored, and 5xx only where the operation is idempotent or you have an idempotency key.
Circuit breakers#
Retries make sense when a failure is likely transient. Once a dependency is clearly unhealthy, continued retrying is both futile and harmful.
A circuit breaker tracks failure rate and, past a threshold, stops sending traffic entirely for a period — failing fast instead. This protects the caller (threads aren’t tied up waiting on something that won’t respond) and the dependency (it gets room to recover without load).
It’s a genuine tradeoff, and worth stating plainly: an open breaker fails requests that might have succeeded. Tuning is difficult, and a mistuned breaker causes outages of its own — usually by opening too eagerly under a brief blip. Half-open probing, where a small number of requests are allowed through to test recovery, is what keeps it from being a manual reset.
Timeouts are part of the retry story#
Retry behavior is meaningless without correct timeouts, and timeouts are usually wrong in a specific way: a dependency timeout longer than the caller’s own timeout.
If your caller gives up after 2 seconds but your call to the database has a 5-second timeout, the caller has already returned an error while the database work continues, consuming a connection and CPU for a result nobody will read. Under load this is how you exhaust a connection pool while appearing idle.
Timeout budgets have to be reasoned about across the whole call chain: each hop gets a fraction of the remaining budget, and the sum must be less than the caller’s patience. This requires knowing your call chain, which is one of the quieter arguments for distributed tracing.
Server-side defenses#
Everything above is client-side, and client-side discipline is insufficient when you have many clients — some of which you may not control.
Load shedding. Under overload, rejecting a fraction of requests immediately is better than accepting everything and serving all of it slowly. Fast rejection lets clients back off; uniform slowness times everyone out and generates retries.
Explicit backpressure. A 429 with a Retry-After tells clients precisely what
to do. Well-behaved clients honor it, and that’s dramatically more effective
than each client guessing.
Concurrency limits per client. Prevents one misbehaving caller from consuming shared capacity.
The general principle: a server should communicate its state clearly rather than degrading silently. Silent degradation is what triggers retry storms, because clients can’t distinguish “slow” from “broken” and default to trying again.
When failing fast is the right answer#
The instinct that every failure should be retried is worth interrogating.
If a dependency is down and you retry for thirty seconds, you’ve converted a fast error into a slow error. The user waits half a minute for the same failure. Meanwhile you’re holding a connection, a thread, and memory — resources your service needs to serve requests that could succeed.
Often the better behavior is failing immediately and degrading gracefully: return partial results, serve stale cache, or disable the non-essential feature. That requires having decided in advance what’s non-essential, which is a product conversation most teams postpone until an incident forces it at the worst possible moment.
Test it, because assumptions rot#
Retry and circuit breaker configuration is written once, under assumptions about latency and failure rate that stop being true within months. Then it sits unexercised until the incident where it matters.
Fault injection is the only reliable way to know it behaves as intended. Inject latency and failures into dependencies deliberately, in a controlled way, and watch what your retry logic does. The failures found this way are consistently surprising — timeouts that were never configured, breakers that never open, retries multiplying across a chain nobody had traced end to end.
What to check tomorrow#
If you want a concrete starting point:
- Find your retry multiplication. Map one critical path and multiply the retry counts. If the number is above about 5, you have an amplifier.
- Verify jitter is present. Backoff without jitter is extremely common and trivially fixed.
- Check timeout ordering. Any dependency timeout longer than its caller’s is a bug.
- Add a retry budget if you don’t have one. It’s the highest-value single change available here.
- Confirm non-idempotent operations aren’t being retried blindly.
None of this is exotic, and all of it is commonly missing. Retries aren’t a bad pattern — they’re an unbounded one by default, and bounding them is the entire job.