senior loop
Deep Dives/Resilience Patterns

Resilience Patterns

Retries alone make a struggling downstream worse, not better — every retry is another request piled onto a service that is already falling over. Circuit breakers stop calling a dependency that's failing, bulkheads stop one slow dependency from starving every other request, and backpressure makes overload visible instead of silently queuing forever. This note builds all three from the timeouts-and-retries baseline, shows how they compose, and covers the ways they fail too.

Distributed SystemsReliabilitySenior IC~40 min · 10 sections

Prerequisites: Timeouts, retries, and thread/connection pool basics.

Cover these firstFailures, Timeouts & Retries

After this: Combine circuit breakers, bulkheads, and backpressure to contain a failing dependency instead of amplifying it.

Suggested first pass: Read sections 1–5, answer each section in your own words, then use the remaining failure modes and exercises as the advanced pass.

Resilience patterns in three lines

1. A timeout bounds how long you wait; a retry tries again after a transient failure — but naive retries during a real outage just multiply load on an already-struggling dependency.

2. A circuit breaker stops calling a dependency once it's clearly failing, giving it room to recover instead of piling on. A bulkhead isolates the resources (threads, connections) used for one dependency so its failure can't starve every other request.

3. Backpressure makes overload visible and bounded — reject or shed load explicitly, with a signal the caller can act on, instead of an unbounded queue that turns overload into an unrecoverable latency spiral.

Numbers to reason with — tune every threshold from measurement, not instinct
50%
A typical circuit breaker failure-rate threshold before it trips open
N=20
A typical rolling window size (calls) the breaker evaluates the failure rate over
10–60s
Typical open-state duration before the breaker probes with a half-open trial
1
Trial calls allowed through in half-open state before deciding close vs re-open
O(retries²)
Roughly how fast naive retry-without-backoff amplifies load under a real outage
bounded
The one property every queue in the request path must have — unbounded queues convert overload into permanent latency
Why timeouts and retries aren't enough on their own

Timeouts and retries (covered in Failures, Timeouts & Retries) answer "what do I do about one slow or failed call." Resilience patterns answer the harder question: what happens when a dependency is failing systemically, for seconds or minutes, under load from hundreds of callers simultaneously. At that point, more retries is more load on the thing that's already down — the fix has to change behavior at the caller-population level, not the single-call level. That's what circuit breakers, bulkheads, and backpressure are for.

Next deep dive
Kafka Internals & Production Operations →
~45 min