When Retries Make an Outage Worse: Idempotency Keys, Backoff with Jitter, and Circuit Breakers

Key takeaways

Retry amplification is real arithmetic. Three retries per request means a dependency at its breaking point receives up to four times its normal load at the exact moment it has the least capacity.

Only retry failures that are safe to repeat. Connection errors, HTTP 502/503/504, and HTTP 429 are retryable; HTTP 400, 401, 403, and 422 are not, because repeating a request the server understood and rejected will never produce a different answer.

An idempotency key is a client-generated unique string sent in the Idempotency-Key header so the server can recognise a retry of a write it already processed and return the original response instead of performing the work twice.

Full jitter beats fixed backoff. Sleeping a random duration between zero and base multiplied by two to the power of the attempt number spreads a fleet of retrying clients across the window instead of synchronising them into repeated thundering herds.

A circuit breaker fails fast on a dependency that is already down, converting a 30-second timeout into an immediate error and giving the dependency room to recover. Scope it per dependency, never globally.

Why do retries make an outage worse instead of better?

Retries multiply load precisely when a dependency has the least capacity to absorb it. A service that normally handles 1,000 requests per second, configured with three retries across every caller, can see 4,000 requests per second the moment it starts failing. The failure causes the load, and the load sustains the failure. This feedback loop is called retry amplification, and it is the single most common way a partial degradation becomes a total outage.

Synchronisation makes it worse. If every worker retries after exactly one second, then two, then four, the whole fleet arrives at the dependency at the same three instants. The recovering service gets a spike, tips over again, and resets everyone's retry clock to the same value — the classic thundering herd. Fixed backoff does not spread load; it schedules a collision.

The cost our team underestimated is that retries consume your own capacity too. A worker sitting in a retry sleep is a worker not processing the queue. With no per-attempt timeout, four attempts at 30 seconds each means a single message occupies a worker for two minutes. Throughput falls before the dependency ever fully fails.

Which failures are actually safe to retry?

Retry only when the request either never reached the server or definitely failed without side effects. In practice that means classifying by status code and error type, not by wrapping everything in a generic try/catch.

Retryable: TCP connection errors such as ECONNRESET, ECONNREFUSED and ETIMEDOUT, DNS failures, HTTP 502, 503 and 504, and HTTP 429 with the delay taken from the Retry-After header. These indicate the server was unavailable or explicitly asked you to wait.

Not retryable: HTTP 400, 401, 403, 404 and 422. The server received the request, understood it, and rejected it. Retrying a malformed payload produces the same rejection with extra load, and retrying a 401 can trip an authentication provider's own rate limiting.

The genuinely hard case is HTTP 500 and the request timeout. Both are ambiguous: the write may have committed before the response was lost. This is exactly the situation idempotency keys exist to resolve. Without a key, retrying an ambiguous POST to a charges endpoint risks a duplicate charge, and not retrying it risks losing a legitimate payment. With a key, the ambiguity disappears and the retry is safe.

What is an idempotency key, and how do we implement one server-side?

An idempotency key is a client-generated unique string, typically a UUID v4, sent in the Idempotency-Key HTTP header so that repeating a request produces the original result rather than a second side effect. Stripe popularised the header, and it is now the de facto convention for payment and provisioning APIs.

The critical detail is that the key must be generated once per logical operation, not once per HTTP attempt. Generating a fresh UUID inside the retry loop defeats the entire mechanism, because each attempt then looks like a brand-new operation to the server. We generate the key when the job is enqueued and store it on the job record, so every retry — including a retry after a process crash and restart — reuses the same value.

On the receiving side, the implementation is a table with a unique constraint doing the concurrency control, not application code checking before inserting. The table stores the key as its primary key, a hash of the request body, a status of in_progress or completed, and the response status and body to replay.

The handler inserts the key with status in_progress inside the same transaction as the business write. A unique-violation error on insert means another attempt is already handling this key: if the stored record is completed, replay the stored response status and body verbatim; if it is still in_progress, return HTTP 409 so the client backs off rather than racing. Relying on a SELECT followed by an INSERT reintroduces the race the key was meant to close, because two concurrent retries can both read not-found.

Store a hash of the request body alongside the key. If the same key arrives with a different payload, that is a client bug, and returning HTTP 422 surfaces it immediately instead of silently replaying an unrelated response. Expire records after a fixed window — 24 hours is a common choice — because keys are a retry-safety mechanism, not permanent storage.

How should we configure backoff, jitter, and timeouts?

Use exponential backoff with full jitter, cap the delay, and bound the whole operation with a deadline. Full jitter means sleeping a uniformly random duration between zero and the smaller of the cap and base multiplied by two to the power of the attempt number, rather than the deterministic value. The randomness is the point: it decorrelates a fleet of clients so the recovering dependency sees a smooth arrival rate instead of synchronised spikes.

In a plain fetch wrapper that is a loop over attempts: treat a thrown error as retryable unless it is an AbortError, treat a response as retryable only when its status is 429, 502, 503 or 504, and otherwise return immediately. When the response carries a Retry-After header, honour that value; otherwise sleep a random duration up to the exponential ceiling.

Three configuration rules we now treat as non-negotiable. First, every attempt gets its own timeout: AbortSignal.timeout is one line and it is what stops a hung socket from holding a worker indefinitely. Second, honour Retry-After when the server sends it, because a 429 with an explicit delay is the dependency telling you exactly what it needs. Third, keep retries shallow — two or three — and never retry inside a layer that is itself being retried, or the counts multiply: three retries at each of three layers is 27 requests from one logical call.

Set a deadline for the whole operation, not just a retry count. If the caller is an HTTP request that will time out in 30 seconds anyway, a retry schedule that can run for 60 seconds is pure waste: the client has already given up and the work is discarded.

When does a circuit breaker help, and when is it overkill?

A circuit breaker is a wrapper that tracks recent failures for one dependency and, past a threshold, short-circuits subsequent calls by failing immediately instead of attempting them. It has three states: closed, where calls pass through; open, where calls fail instantly for a cooldown period; and half-open, where a small number of trial calls decide whether to close again or re-open.

The breaker solves a problem retries cannot. When a dependency is fully down, the correct behaviour is to stop calling it — both to let it recover and to stop burning your own latency budget on calls that are certain to fail. A 10-second timeout becomes an instant error, which means the API returns a fast, honest degraded response instead of queueing requests behind dead connections.

Scope the breaker per dependency, and ideally per endpoint group. One global breaker means a broken analytics endpoint can block checkout calls. Choose thresholds from a failure rate over a rolling window rather than a raw count, so low-traffic services do not trip on two unlucky requests. And make sure the fallback path is genuinely useful: serving stale cached data or queueing the work for later is worth building a breaker for, while an error page you would have shown anyway is not.

Overkill looks like a single-instance internal service with predictable traffic, or a dependency whose failures are already rare and fast. A breaker adds state, configuration, and a new way to be wrong — we have seen a misconfigured breaker stay open long after the dependency recovered. Timeouts and bounded retries come first; add the breaker when a dependency has actually demonstrated that it fails hard.

Which mechanism fixes which failure?

These four mechanisms are not alternatives. Each addresses a different failure, and using one without the others leaves a hole.

A per-attempt timeout solves a hung request holding a worker or connection forever. It says nothing about whether the work succeeded.

A bounded retry with jitter solves transient failures and synchronised client herds. It does nothing about duplicate side effects on ambiguous writes.

An idempotency key solves duplicate writes when a retry follows a lost response. It does nothing about load amplification or a dependency that is fully down.

A circuit breaker solves wasted calls and latency against a dependency that is down. It does nothing for individual transient blips or the correctness of a single write.

Ordered by return on effort: add per-attempt timeouts first, because they are one line and they bound the worst case. Add idempotency keys to every non-idempotent write next, because without them you cannot retry ambiguous failures safely at all. Fix the retry schedule with jitter and a budget third. Add breakers last, on the dependencies that have earned one.

FAQ

Q: Can I use a hash of the request body as the idempotency key?

A: No, because two legitimate identical requests — a customer intentionally buying the same item twice — would collide and the second one would be silently swallowed. Generate a UUID per logical operation and store the body hash separately to detect key reuse with a different payload.

Q: Is it safe to retry a GET request without an idempotency key?

A: Yes. GET, HEAD, PUT and DELETE are defined as idempotent in the HTTP specification, so repeating them produces the same server state. POST and PATCH are the methods that need a key, and a PUT implemented with side effects such as counters needs one in practice too.

Q: How many retries should I configure?

A: Two or three at a single layer. More attempts rarely convert a failure into a success, and retries at multiple nested layers multiply: three retries in an HTTP client wrapped by three retries in a job queue is up to sixteen requests from one logical call.

Q: Does full jitter make retries slower on average?

A: Slightly, for a single client, since the expected delay is half the ceiling rather than the full value. Across a fleet it is faster overall, because decorrelated arrivals let the dependency recover instead of being knocked over by each synchronised wave.

Q: Where should the idempotency key live for a queued background job?

A: On the job record, written when the job is enqueued. Generating it at call time means a crashed and re-delivered job produces a new key, which is exactly the duplicate-write scenario the key was supposed to prevent.