Quad9 DNS Resolver to exit France
Concept of the day · distributed systems
Exponential Backoff with Jitter
Retries without jitter re-synchronise your clients into a self-inflicted DDoS.
The mechanism. On failure, wait base * 2^attempt before retrying, capped at some maximum, then add randomness. The exponential part gives an overloaded dependency room to recover. The random part — jitter — is the half everyone skips, and it is the half that matters.
Why jitter is not optional. A dependency blips and fails a thousand requests at once. Every client computes the same backoff and retries at the same instant. The dependency, still recovering, falls over again. The clients retry again, still in lockstep. You have built a retry storm, and it is periodic. Jitter spreads the herd out.
Which jitter. AWS measured the variants: full jitter (sleep = random(0, min(cap, base * 2^n))) wins on both completion time and total work in most cases. "Equal jitter" (half fixed, half random) is marginally more predictable. Decorrelated jitter is competitive. Plain exponential with no jitter is always worst.
The parts that are not the sleep. Only retry idempotent operations, or carry an idempotency key. Cap total attempts and total elapsed time, not just the interval. Never retry a 4xx — the request is wrong, and retrying it is pure load. Use a retry budget: a token bucket that caps retries at, say, 10% of normal traffic, so a widespread failure cannot multiply your load. And do not retry at every layer: three layers each retrying three times is twenty-seven requests.
Takeaway. retry(3) with no jitter, no budget and no idempotency is how a brief blip becomes an outage.