A 40-second downstream blip became a 25-minute outage that continued after the downstream recovered

Example note — the format applied to a well-documented failure mode, not a specific engagement.

Symptom

• Since when: 14:02, coinciding with a 40-second latency excursion on the pricing service. That excursion ended at 14:02:40. The outage ran until 14:27.
• Reproduces when: any latency excursion on a dependency long enough for callers to start retrying, at any traffic level above roughly half of peak. It has almost certainly happened before at smaller scale and been recorded as “a blip”.
• Does not reproduce when: the same excursion happens off-peak. Below about half peak traffic the amplified load still fits, the loop never closes, and it genuinely is a blip.

The shape of this incident is in one comparison, and it is the reason nothing anyone tried worked:

14:00  user-facing requests   1,020 /s     pricing inbound   1,070 /s    ratio 1.05
14:05  user-facing requests   1,010 /s     pricing inbound   3,940 /s    ratio 3.90
14:20  user-facing requests     990 /s     pricing inbound   3,880 /s    ratio 3.92

User traffic never moved. The load that kept the pricing service saturated for twenty-five minutes was generated by the callers responding to the saturation.

At 14:09 the pricing service’s most recent deploy was rolled back on the assumption that it was the trigger. Nothing changed. That is not a sign the diagnosis was wrong — it is the defining property of the failure.

Environment

Product / version	Istio 1.21, Envoy sidecars; Go and Java callers
Deployment	40 caller pods across 6 services, one pricing service behind them
Scale	~1k user requests/s at peak; pricing sized for ~1.5k/s
Last change	none that day. The application client’s retry policy — 3 attempts, fixed 1s delay, no jitter — has been in the shared HTTP wrapper for three years. The mesh, with its own default of 2 retries, was adopted four months ago.

Neither retry policy was introduced by anyone recently, and neither is unreasonable read on its own. What changed four months ago is that there are now two of them, and no configuration file mentions both.

Investigation

Hypothesis	How it was checked	Verdict
The pricing deploy at 13:58 is the cause	Rolled back at 14:09. No improvement over the following ten minutes	rejected as the cause — it was plausibly the trigger, and removing a trigger does not help once the loop is closed
A traffic spike from users or a bot	Ingress request rate flat within 3% across the whole window	rejected — and this is the discriminator
The pricing service’s database is saturated	Database CPU under 30%, query latency flat. Pricing’s own CPU was pegged in request handling	rejected
Pricing pods are crash-looping	Zero restarts; the pods were up and serving, just far too slowly	rejected
Callers are amplifying the load through retries	Inbound/outbound ratio at 3.9 against a resting 1.05. Disabling sidecar retries on one caller dropped that caller’s contribution immediately	accepted

The ratio is the whole diagnosis and it is one division. It was not computed until 14:24, because every dashboard shows each service’s request rate on its own panel and nothing shows the relationship between two of them — see retry amplification.

Root cause

The application HTTP wrapper retries 3 times. Istio applies its own default retry policy of 2 retries whether or not anyone asks for it. The two compose:

3 application attempts  ×  (1 + 2 sidecar attempts)  =  up to 9 requests downstream

The 40-second excursion pushed enough requests past their timeout to start that multiplication. The amplified load kept pricing saturated, saturation produced more timeouts, and more timeouts produced more retries. Within about ninety seconds the system was in a state that sustained itself: the trigger was gone and the load causing the outage was the outage’s own output.

Fixed 1-second delays with no jitter made it worse in a specific way — retries arrived in synchronised waves rather than spread out, so each recovery attempt met a spike instead of a ramp and re-synchronised everyone for the next round.

Fix

What ended it, at 14:27 — remove load, not the trigger:

kubectl annotate deployment/orders app.kubernetes.io/retries=disabled
kubectl set env deployment/orders HTTP_RETRY_ATTEMPTS=1

The ratio fell to 1.06 and pricing recovered in about ninety seconds without any change to pricing itself.

The durable changes, in order of how much they matter:

• Retries at one layer only. The mesh owns them now; the shared HTTP wrapper defaults to a single attempt. Two layers is a product, not redundancy.
• Jitter on the backoff, so a recovering dependency sees a ramp:

  attempt N delay = random(0, min(cap, base × 2^N))

• A budget rather than a count. Envoy caps retries as a percentage of active requests (20% by default), which cannot amplify at exactly the moment amplification is most harmful. This is the control that would have prevented the incident regardless of what the application did.
• Circuit breaking via outlierDetection, so a dependency that is failing persistently stops receiving traffic rather than receiving more of it.

Prevention

• Detection: alert on the amplification ratio — requests arriving at a dependency divided by requests arriving at its caller. It sits near 1 in normal operation, no other failure mode drives it to 4, and it is a single division over two metrics that already existed. It is now a panel on the same dashboard as both rates, which is the actual fix to the twenty-two minutes.
• Prevention: document which hop owns retries, per dependency. The failure here is not that someone chose badly; it is that two reasonable choices were composed and nothing displays the product.
• Remaining debt: five other services still have both layers enabled. They were left alone because changing retry behaviour across the fleet during the same week as an outage is its own risk, and there is now a scheduled change and a list. The list was assembled by grep and is not trusted to be complete.

Open questions

• Would it have recovered on its own? Unknown, and unknowable from this incident — it was broken manually at 14:27. Metastable states can persist indefinitely by definition, but nothing here proves this one would have.
• The original 40-second excursion was never explained. It is attributed to a GC pause on the strength of one heap graph, and the investigation stopped once the amplification was understood. The trigger is still out there and will fire again; only its consequences have been bounded.
• Whether load shedding at the pricing service would have been sufficient on its own. It has none, adding it was proposed, and nobody could say whether shedding at 1.5k/s would have broken the loop or merely relocated the failure.

Related

Parent concept: retry amplification. Filed under Config in the incident map — nothing was broken, two independently sensible settings were composed.

Sibling: the intermittent 503s on quiet endpoints, where retries were added as a safety net. That was the right call there and it is worth reading the two together rather than treating this note as a correction of it: there, retries were added at one hop, for one idempotent path, with the reason stated. Here the same instinct is applied at two layers with no budget and no jitter, on the assumption that a retry is free. The difference between the two is not whether to retry — it is whether anyone can say how many requests one user request can become.

Contrast with the Kafka rebalance loop. Both are systems that make no progress while every component is healthy, and both loop by reprocessing their own output. But that loop is internal to one service and stops when its config changes, while this one runs between services and is sustained by the callers — so the fix has to be applied at the layer that is not visibly broken. In an amplification failure the saturated service is the symptom, and it is also the only place anyone is looking.