A retry is a local decision with a global cost. Each layer that retries independently multiplies the load the bottom of the stack sees, and the layers rarely know about each other.
They multiply, they do not add
Three attempts in an application client, sitting on top of a service mesh that defaults to two retries, is nine requests at the bottom for one at the top:
app client: 3 attempts × sidecar: 1 + 2 retries = 9 requests downstreamIstio applies a default retry policy whether or not anyone configures one, so adding a mesh to a service that already retries silently triples its worst-case load. Nothing in either configuration mentions the other, and neither team’s config review can see the product.
Synchronised retries
Retries without jitter arrive together. Every client that failed at t=0 retries at t=1s, then t=2s, in lockstep, so a recovering service is hit by a spike rather than a ramp — and fails again, resynchronising everyone for the next round. Exponential backoff alone does not fix this; backoff decides when, jitter decides whether they all pick the same when.
Why the outage outlives its cause
This is the part that surprises people, and it has a name — metastable failure. A brief trigger raises latency. Higher latency produces retries. Retries raise load. Higher load keeps latency high, which produces more retries.
Once that loop closes, the original trigger is irrelevant. Rolling back the bad deploy, ending the GC pause, or restoring the slow disk changes nothing, because the load sustaining the outage is now generated by the outage itself. The system has settled into a second stable state that it will not leave on its own — which is how a 40-second blip becomes a 25-minute outage.
Breaking it requires removing load, not fixing the trigger: shed requests, cut retries, or take clients away.
What actually bounds it
- Retry budgets — cap retries as a percentage of active requests (Envoy’s default budget is 20%) rather than as a per-request count. A count-based policy amplifies exactly when amplification is most harmful; a budget cannot.
- Retry at one layer only. Decide which hop owns it and disable the others. Two layers is a product, not a backup.
- Jitter, always. Full jitter over the backoff window, not a fixed multiplier.
- Circuit breaking on the caller, so a persistently failing dependency stops receiving traffic instead of receiving more of it.
The measurement that names this failure in one number is the ratio of requests arriving at a dependency to requests arriving at its caller. At rest it is a little over 1. During amplification it is 4, and no other failure mode produces that.