Resilience Patterns
Retries alone make a struggling downstream worse, not better — every retry is another request piled onto a service that is already falling over. Circuit breakers stop calling a dependency that's failing, bulkheads stop one slow dependency from starving every other request, and backpressure makes overload visible instead of silently queuing forever. This note builds all three from the timeouts-and-retries baseline, shows how they compose, and covers the ways they fail too.
Prerequisites: Timeouts, retries, and thread/connection pool basics.
Cover these firstFailures, Timeouts & Retries
After this: Combine circuit breakers, bulkheads, and backpressure to contain a failing dependency instead of amplifying it.
Suggested first pass: Read sections 1–5, answer each section in your own words, then use the remaining failure modes and exercises as the advanced pass.
Content reviewed 21 June 2026 · References: Microsoft: Circuit Breaker pattern · Microsoft: Bulkhead pattern
1. A timeout bounds how long you wait; a retry tries again after a transient failure — but naive retries during a real outage just multiply load on an already-struggling dependency.
2. A circuit breaker stops calling a dependency once it's clearly failing, giving it room to recover instead of piling on. A bulkhead isolates the resources (threads, connections) used for one dependency so its failure can't starve every other request.
3. Backpressure makes overload visible and bounded — reject or shed load explicitly, with a signal the caller can act on, instead of an unbounded queue that turns overload into an unrecoverable latency spiral.
Timeouts and retries (covered in Failures, Timeouts & Retries) answer "what do I do about one slow or failed call." Resilience patterns answer the harder question: what happens when a dependency is failing systemically, for seconds or minutes, under load from hundreds of callers simultaneously. At that point, more retries is more load on the thing that's already down — the fix has to change behavior at the caller-population level, not the single-call level. That's what circuit breakers, bulkheads, and backpressure are for.
A single client retrying a single failed call is harmless. The failure mode appears at population scale: when a downstream service degrades, every caller's requests start timing out and retrying at roughly the same moment, and that additional retry traffic is exactly what a struggling service can least afford.
- 1Downstream service slows down (GC pause, DB lock contention, a bad deploy) — call latency creeps past the timeout for a growing fraction of requests.
- 2Every caller independently times out and retries. Retry traffic adds to the load the struggling service is already failing to handle — it now has less capacity per request, not more.
- 3More requests time out, triggering more retries. Without backoff, this is roughly exponential in the number of retry layers (a retries in your service, called by a service the caller itself retries b times, compounds to a×b calls per original request).
- 4The downstream service, now serving several multiples of its normal load, cannot recover even after the original cause clears — the retry traffic is the outage now. This is a metastable failure: stable at normal load, stable at zero load, unstable and self-sustaining in between.
- →Fix requires cutting load, not adding more attempts: exponential backoff with jitter slows the retry rate, and a circuit breaker stops retrying at all once failure is the norm rather than the exception.
Backoff with jitter smooths when retries happen but doesn't cap how many total retries the whole fleet issues. A retry budget (e.g. "retries may not exceed 10% of the request rate over the last 10 seconds," per client or per service) caps total amplification directly — once the budget is spent, calls fail fast instead of retrying. This is the piece most systems skip, and it's the one that actually bounds worst-case load during a real outage.
Retrying is only correct if the call is idempotent, or the caller can distinguish "definitely didn't happen" from "unknown" (e.g. connection refused vs. a timeout after the request was sent). Retrying a non-idempotent write on an ambiguous failure risks a duplicate side effect, which is a correctness bug dressed up as a reliability improvement. See Idempotency & Exactly-Once Payment Processing for the idempotency-key pattern that makes retries safe for writes.
A circuit breaker wraps a call to a dependency and tracks its recent success/failure rate. It has three states, and the entire pattern is the state machine between them — the electrical-circuit metaphor is apt: when the breaker is open, no current (calls) flows through.
| State | Behavior | Transition out |
|---|---|---|
| Closed | Calls pass through normally; failures are counted in a rolling window | Failure rate crosses threshold → Open |
| Open | Calls fail immediately (fast-fail), no request reaches the dependency | After a cooldown timer → Half-Open |
| Half-Open | A small number of trial calls are allowed through to test recovery | Trials succeed → Closed; trials fail → back to Open |
Without a breaker, every caller during an outage still pays the full timeout on every call before giving up — a slow, expensive way to fail that also keeps sending load downstream. An open breaker fails immediately, with no network call at all: callers get a fast, cheap failure they can handle (fallback, cached value, error to the user) instead of blocking a thread for the full timeout. Fast-fail is the mechanism; giving the downstream room to recover is the side effect that makes the whole system heal.
A time-based window ("failure rate over the last 10 seconds") behaves oddly at low request rates — 1 call in 10 seconds that fails looks identical to a 100% failure rate, tripping the breaker on a single data point. A count-based rolling window (last N calls, e.g. N=20) is more robust to traffic bursts and quiet periods alike, which is why resilience4j and most modern implementations default to it.
Every threshold on a circuit breaker is a trade-off between tripping too eagerly (false positives that cut off a dependency that was only briefly slow) and too reluctantly (real outages pile on load for too long before the breaker acts). Play with the simulator below to see the state machine react to a failure spike and recovery.
If half-open let all traffic through to test recovery, a dependency that's still fragile gets hit with the full traffic spike the instant it shows the first sign of life — which can knock it back down immediately, creating a trip/probe/re-trip oscillation. Limiting the trial to one or a handful of calls tests recovery cheaply, at a cost the dependency can absorb even if the test fails.
A single breaker guarding "all downstream calls" ties an unrelated dependency's health to another's — if the payments API is failing, a shared breaker can trip and cut off the (perfectly healthy) inventory API too. Each dependency (ideally each distinct endpoint, not just each service) gets its own breaker with its own window and threshold, so failures stay isolated to the thing that's actually failing.
Named for a ship's bulkheads — watertight compartments that keep one breached section from sinking the whole hull. Applied to services: if every downstream call shares one thread pool or one connection pool, a single slow dependency can exhaust that shared resource and starve calls to every other, perfectly healthy dependency. A bulkhead gives each dependency its own bounded slice of resources.
- 1Service calls three downstreams (A, B, C) using one shared thread pool of 100 threads for all outbound HTTP calls.
- 2Dependency A starts responding slowly (not failing — just slow). Threads calling A pile up waiting on responses, holding their thread the whole time.
- 3Within seconds, all 100 threads are blocked waiting on A. Calls to B and C — completely healthy — now have zero threads available and start failing or queuing too.
- →One slow dependency has taken down the whole service. With separate bounded pools per dependency (e.g. 30 threads for A, 30 for B, 30 for C), A's slowdown exhausts only its own 30 — B and C keep serving normally.
| Isolation style | Mechanism | Cost | Used by |
|---|---|---|---|
| Thread pool isolation | Dedicated thread pool per dependency; calls run on that pool | Thread overhead per pool; more memory | Original Hystrix default |
| Semaphore isolation | Shared threads, but a counting semaphore caps concurrent calls per dependency | No thread-count overhead, but a slow call still blocks the calling thread | Hystrix (lighter mode), resilience4j Bulkhead |
| Connection pool isolation | Separate DB/HTTP connection pool per downstream | More idle connections held open | HikariCP per-datasource, HTTP client pools per host |
| Process/container isolation | Separate deployable per dependency's call path | Operational overhead of more services | Extreme cases — usually overkill |
Thread pool isolation gives the strongest guarantee — a slow dependency literally cannot touch threads reserved for another — at the cost of context-switch overhead and memory for extra thread pools, which adds up with dozens of dependencies. Semaphore isolation is cheaper (no extra threads) but the calling thread itself still blocks on the slow call up to the semaphore limit, so it only bounds concurrency, not the calling thread's own exposure to latency. In practice: thread pool isolation for a small number of critical, failure-prone dependencies; semaphore isolation (or async I/O, which sidesteps the whole trade-off) for everything else.
A bulkhead's job is entirely "stop the bleeding from spreading" — it does nothing to make the slow dependency healthy again, and by itself it doesn't even stop calls to that dependency from continuing to queue up to its own pool's limit. Pair it with a circuit breaker (stop calling once it's clearly failing) and a timeout (bound how long each call can hold a bulkhead slot) — the three are complementary, not substitutes for each other.
Circuit breakers and bulkheads protect a caller from a downstream dependency. Backpressure protects a service from its own callers — when incoming request volume exceeds what the service can actually process, something has to give: either the service explicitly rejects the excess (backpressure) or an unbounded queue absorbs it silently until latency and memory both blow up.
An unbounded in-memory queue in front of a slow consumer feels safe — nothing is ever rejected — but it converts a capacity problem into a latency and eventually a memory problem. Every request added to a growing queue waits behind everything already in it, so tail latency grows without bound even after the load spike ends, because the queue has to drain the entire backlog first. The system doesn't "fail" in an obvious way; it just gets slower and slower until it OOMs or every caller's own timeout fires anyway — the rejection happens, just later, more chaotically, and after wasting the resources on requests nobody will wait for.
| Technique | Idea | Signal to caller |
|---|---|---|
| Bounded queue + reject | Cap queue depth; reject new work once full | HTTP 503 / 429 immediately, not after a long wait |
| Load shedding | Drop low-priority work first under pressure (e.g. by request class or client tier) | 429 for shed requests; unaffected for priority traffic |
| Rate limiting | Cap requests per caller/key over a window (see Distributed Rate Limiter) | 429 with Retry-After |
| Adaptive concurrency limits | Measure actual latency, adjust the allowed in-flight request count (AIMD, like TCP congestion control) | 503 once the adaptive limit is hit |
| Reactive streams backpressure | Consumer signals producer how much it can accept (pull-based, not push) | Producer slows emission — no rejection needed |
A fixed "max 500 concurrent requests" limit has to be guessed and re-tuned every time the service's capacity changes (new hardware, a slower downstream, a code change). Netflix's concurrency-limits library instead measures actual observed latency continuously and adjusts the allowed concurrency using an AIMD scheme (additive increase while latency is healthy, multiplicative decrease when it isn't) — the same idea TCP congestion control uses to find available bandwidth without being told it in advance. The limit tracks the service's real, current capacity rather than a number someone guessed six months ago.
Explicitly rejecting the 5% of requests that exceed capacity, fast and with a clear signal, preserves good service for the other 95% and lets the caller retry with backoff or degrade gracefully. Silently accepting 100% of requests and serving all of them slowly and eventually erroring out anyway is strictly worse for every caller, including the ones that would have been fine. Backpressure isn't a compromise on reliability — it's what makes the healthy majority of traffic reliable at all under real load.
None of these patterns is a substitute for the others — a production call path layers all of them, and the order matters, because each layer's job is to catch what the layer before it couldn't.
- 1Bulkhead — acquire a slot from this dependency's dedicated pool/semaphore. If the pool is full, fail immediately; a healthy dependency elsewhere is unaffected.
- 2Circuit breaker — if open, fail immediately without attempting the call at all. This is checked before the network call, so an outage costs nothing but a state check.
- 3Timeout — bound how long this specific attempt is allowed to take, so it can't hold its bulkhead slot forever.
- 4Retry with backoff + jitter, inside a budget — on a transient, idempotency-safe failure, retry a bounded number of times with increasing, jittered delay, capped by the fleet-wide retry budget.
- →Outcome recorded back into the circuit breaker's window either way — this is what lets the breaker trip on sustained failure and later probe for recovery.
Meanwhile, on the receiving side of every service in this chain, backpressure caps how much of this (retried, bulkheaded, breaker-wrapped) traffic it will accept before rejecting the rest outright — protecting it from every caller upstream, symmetric to how the caller is protecting itself from every dependency downstream.
Checking the circuit breaker before acquiring a bulkhead slot would waste a slot on a call that's about to fast-fail anyway — so bulkhead-then-breaker, or breaker-then-bulkhead, is mostly a wash, but most implementations (resilience4j included) check bulkhead capacity first since it's cheaper. What's non-negotiable: timeout must wrap every individual attempt (not the whole retry loop, or one hung attempt blocks all remaining retries too), and retries must happen inside the per-attempt timeout/circuit-breaker check, not around it — otherwise a breaker trip mid-retry-loop doesn't actually stop further attempts.
A brief trigger (a deploy blip, a GC pause, a network glitch) pushes the system into a self-sustaining bad state — full queues, tripped-open breakers everywhere, retry storms — that persists after the original trigger is gone, because the recovery attempts themselves (retries, reconnects, cache-refill stampedes) are now the load keeping it down. Fixing the root cause is necessary but not sufficient; recovering from a metastable failure usually requires actively shedding load below the level that would let the system heal, which can mean deliberately rejecting traffic that could otherwise be served, until backlogs drain.
If every instance of a service trips its breaker for the same dependency at roughly the same time (likely, since they're all seeing the same outage), they all enter half-open and send trial calls at roughly the same moment too — a mini thundering herd hitting a dependency that's mid-recovery, potentially knocking it back down and re-tripping every breaker in lockstep. Jittering the cooldown duration per instance (not a fixed 30s for everyone) spreads the probes out.
Bulkheads and breakers stop failure from spreading through direct call paths, but a shared resource underneath everything — a connection pool to a shared database, a shared cache cluster, a shared message broker — can still let one dependency's failure starve every service that touches that shared resource, bulkheads notwithstanding. This is why the resilience story has to include the data layer: connection pool limits per caller, cache stampede protection, broker-level backpressure — not just the application-level call-wrapping patterns in this note.
A circuit breaker's fallback (return a cached value, a default, a degraded response) is good practice, but a fallback that masks a real, ongoing outage — returning stale or wrong data with a 200 instead of surfacing that something is broken — can turn a visible, alertable failure into an invisible correctness bug. Fallbacks need their own monitoring: alert on fallback rate, not just on raw error rate, or a permanently-open breaker silently serving degraded data for weeks goes unnoticed.
A load balancer's shallow health check (TCP connect, or a trivial /health endpoint that doesn't touch the database) can report "healthy" for an instance whose actual request-serving path is completely broken (DB connection pool exhausted, but the process is still up and answering pings). Health checks need to exercise enough of the real path to catch the failure modes that matter, without being so deep that the health check itself becomes a source of load or false negatives.
| Symptom | Pattern | What it changes |
|---|---|---|
| One call occasionally hangs | Timeout | Bounds worst-case latency of a single attempt |
| Transient, isolated failures (one bad node behind an LB) | Retry with backoff + jitter | Recovers from failures that clear on their own, without herding |
| A dependency is systemically failing, not just occasionally | Circuit breaker | Stops sending load to something that can't currently serve it; fails fast instead |
| One slow dependency degrades calls to unrelated dependencies | Bulkhead | Caps blast radius to the dependency that's actually failing |
| Your own service is overloaded by legitimate traffic volume | Backpressure / load shedding | Protects the majority of requests by explicitly rejecting the excess |
Timeouts and idempotency-safe, budgeted, jittered retries are close to mandatory on every outbound call. Circuit breakers and bulkheads earn their complexity on calls to dependencies that can fail systemically and that other, healthier traffic shares infrastructure with. Backpressure earns its complexity the moment a service has ever been paged for an overload incident, or serves traffic whose volume it doesn't fully control. Reach for all of them by default on a payments-adjacent or otherwise correctness-and-availability-critical path; be more selective on low-stakes internal tooling, where the operational overhead of four extra failure modes to reason about may not be worth it.
resilience4j (JVM), Polly (.NET), and the retry/circuit-breaker support built into service meshes (Envoy's outlier detection, Istio) implement all of this correctly, with sane defaults and battle-tested edge cases already handled. The value of understanding the mechanics is choosing and tuning thresholds correctly and reasoning about failure modes in an interview or an incident — not reimplementing a circuit breaker in production.
Follow-up questions that probe the design beyond the happy path. Click each to expand the answer.
A single caller retrying a single failed call is harmless. The problem is population scale: when a dependency degrades, every caller sees failures at roughly the same time and retries at roughly the same time, and that retry traffic adds directly to the load on a service that's already failing to keep up. Without backoff and jitter, this compounds across retry layers (my retries times my caller's retries times their caller's retries) roughly exponentially, and the dependency ends up serving several multiples of its normal load — it now can't recover even after whatever originally caused the slowdown clears, because the retry storm itself has become the load.
The fix isn't "don't retry" — it's exponential backoff with jitter to spread retries out instead of spiking them in lockstep, a retry budget to cap total fleet-wide amplification, and a circuit breaker to stop retrying entirely once failure is the norm rather than the transient exception.
Closed is the normal state: calls pass through, and outcomes are recorded in a rolling window (typically count-based, e.g. last 20 calls, since time-based windows misbehave at low traffic volume). If the failure rate in that window crosses a threshold (commonly ~50%), the breaker trips to open.
In open, calls fail immediately with no network call at all — fast-fail instead of waiting out a timeout, which both protects the caller's own latency and stops sending load to a struggling dependency. After a cooldown timer, the breaker moves to half-open and allows one (or a small number of) trial calls through. If the trial succeeds, back to closed with a fresh window; if it fails, back to open for another cooldown. The point of limiting half-open to a small number of calls is that a still-fragile dependency shouldn't get the full traffic spike the instant it shows the first sign of recovery.
Thread pool isolation gives each dependency its own dedicated pool of threads; calls to that dependency can only ever consume threads from its own pool, so a slow dependency literally cannot touch the threads reserved for another. That's the strongest guarantee, but it costs memory and context-switch overhead per extra pool, which adds up with many dependencies.
Semaphore isolation uses the caller's existing thread but caps how many concurrent calls to a given dependency are allowed via a counting semaphore. It's cheaper — no extra threads — but the calling thread itself still blocks for the duration of a slow call, up to the semaphore's limit; it bounds concurrency but not the calling thread's own exposure to latency. In practice: thread pool isolation for a small number of critical or historically failure-prone dependencies, semaphore isolation (or fully async I/O, which sidesteps this trade-off entirely) for the rest.
It feels safe because nothing is explicitly rejected, but it converts a capacity problem into a latency and then a memory problem. Every new request queues behind the entire existing backlog, so tail latency grows without bound and keeps growing even after the load spike that caused it ends, because the queue has to drain the whole backlog first. Eventually the process either OOMs, or every caller's own client-side timeout fires anyway — so the effective rejection still happens, just later, more chaotically, and after burning resources on requests nobody is still waiting for.
The fix is a bounded queue that rejects explicitly (503/429) once full, ideally paired with load shedding that drops lower-priority work first and an adaptive concurrency limit that adjusts the accepted in-flight count to the service's actual current capacity rather than a static guessed number.
A metastable failure is a state the system falls into after a brief trigger — a deploy blip, a GC pause, a short network glitch — that then sustains itself even after the trigger is gone, because the system's own recovery behavior (retries, reconnects, cache-refill stampedes, queued backlog) is now generating enough load to keep it down. The system is stable at normal load and stable at zero load, but unstable in between, and the outage traffic is self-reinforcing.
That's why fixing the root cause (rolling back the bad deploy, letting the GC pause pass) is necessary but often not sufficient — the retry storm and backlog it triggered can keep the system down on their own. Recovery usually needs active load shedding, deliberately below what the system could otherwise serve, to let backlogs drain and breakers close, before traffic is allowed back up to normal levels.
A fallback that returns a cached or default value with a 200 status can mask a real, ongoing outage — from the outside, the system looks like it's working, but it's silently serving stale or degraded data. Without dedicated monitoring on fallback rate (not just raw error rate, which the fallback is specifically designed to suppress), a breaker can sit open for days or weeks serving degraded responses with nobody noticing, which turns a visible, alertable failure into an invisible correctness problem instead.
The fix is treating "fallback engaged" as its own signal worth alerting on, separate from "request failed," so a permanently tripped breaker gets investigated even though it's technically "handling" every request without erroring.
It's a trade-off in both directions, same shape as picking a lock TTL. Too sensitive a threshold (e.g. tripping at 20% failures) or too short a window means normal, brief blips trip the breaker and cut off a dependency that would have recovered on its own — unnecessary fast-failing of traffic that could have succeeded. Too lax (80%+ threshold, huge window) means the breaker doesn't trip until the dependency is already deeply unhealthy and has absorbed a lot of pointless load first.
In practice I'd start from the dependency's normal baseline error rate (measured, not guessed) and set the threshold meaningfully above that — enough to not fire on routine noise, low enough to fire well before a real outage compounds. Cooldown duration should roughly match how long the dependency typically takes to recover from the failure modes it actually has (a restart, a failover, a GC pause clearing) — and should be jittered per-instance so every caller's breaker doesn't probe at the exact same moment and re-trip the dependency in a herd.
Per-call: a timeout bounding each individual attempt (not the whole retry loop). A bulkhead (thread pool or semaphore, dedicated to this dependency) so a slow fraud-check doesn't starve calls to, say, the payment-processor API running through the same service. A circuit breaker wrapping the call, tripping on sustained failure so a real outage fast-fails instead of every payment attempt eating the full timeout. Retries only if the failure is clearly transient and the call is safe to repeat — and for a fraud check specifically, that needs care: a duplicate fraud-check call is usually harmless (it's a read), but if it has any side effect (logging a flagged attempt), that needs an idempotency key too.
Critically, this needs an explicit fallback decision for what happens when the breaker is open: does the payment proceed without a fraud check (risk), or does it fail closed and reject the payment (availability cost)? That's a product/risk decision, not a purely technical one, and it's the kind of judgment call that's worth stating explicitly rather than assuming.
Build a circuit breaker class wrapping an arbitrary function call: closed state tracks the last N outcomes in a rolling window and trips to open once the failure rate crosses a configurable threshold; open state fails immediately without invoking the wrapped function; after a cooldown, transition to half-open and allow exactly one trial call to decide close vs. re-open.
Write a test harness that simulates a dependency going from healthy to 100% failing to recovered, and verify the breaker trips at the right point, fast-fails while open (assert the wrapped function is never actually called), and correctly closes again once the dependency recovers.
Your service calls: a database (critical, every request), a cache (optional — a miss just means slower response), a fraud-check API (critical for payment requests only), a recommendations service (optional, degrades gracefully to a static fallback), and an email-notification service (fire-and-forget, failure is fine to ignore). For each, decide: does it need a bulkhead, a circuit breaker, retries, a fallback? What isolation level (thread pool vs. semaphore) for the bulkhead, if any?
Then: your service itself is starting to get overloaded during a traffic spike from an unrelated marketing campaign. Design the backpressure story — what gets shed first, what never gets shed, and what signal do rejected callers get back?
Postmortem summary: a 90-second network blip between two regions caused a spike in timeouts. Retries (fixed 1-second interval, no jitter, no budget) kicked in across ~800 service instances simultaneously. The downstream service, now receiving roughly 4x normal load from retry traffic alone, started timing out on requests that would otherwise have succeeded, triggering more retries. The network blip resolved after 90 seconds, but the outage continued for another 40 minutes. Explain exactly why the outage outlived its trigger, and what should have prevented the retry storm from compounding in the first place. Then explain why the on-call engineer's first instinct — "just restart the downstream service" — didn't work, and what they had to do instead to actually recover.
Covers retry storms and jitter, retry budgets, circuit breakers, and the metastable-failure recovery pattern of deliberately shedding load below normal capacity before ramping back up.