Resilience Patterns
Retries alone make a struggling downstream worse, not better. Every retry is another request added to a service that is already failing. Circuit breakers stop calling a dependency that's failing. Bulkheads stop one slow dependency from starving every other request. Backpressure makes overload visible instead of silently queuing forever. This note builds all three from the timeouts-and-retries baseline, shows how they compose, and covers the ways they fail too.
Prerequisites: Timeouts, retries, and thread/connection pool basics.
Cover these firstFailures, Timeouts & Retries
After this: Combine circuit breakers, bulkheads, and backpressure to contain a failing dependency instead of amplifying it.
Suggested first pass: Read sections 1–5, answer each section in your own words, then use the remaining failure modes and exercises as the advanced pass.
Content reviewed 21 June 2026 · References: Microsoft: Circuit Breaker pattern · Microsoft: Bulkhead pattern
1. A timeout bounds how long you wait. A retry tries again after a transient failure. But during a real outage, naive retries just multiply load on an already-struggling dependency.
2. A circuit breaker stops calling a dependency once it's clearly failing, giving it room to recover instead of piling on. A bulkhead isolates the resources (threads, connections) used for one dependency so its failure can't starve every other request.
3. Backpressure makes overload visible and bounded. It rejects or sheds load explicitly, with a signal the caller can act on. The alternative is an unbounded queue that turns overload into an unrecoverable latency spiral.
Timeouts and retries (covered in Failures, Timeouts & Retries) answer "what do I do about one slow or failed call." Resilience patterns answer a harder question. What happens when a dependency is failing systemically, for seconds or minutes, under load from hundreds of callers simultaneously? At that point, more retries is more load on the thing that's already down. The fix has to change behavior at the caller-population level, not the single-call level. That's what circuit breakers, bulkheads, and backpressure are for.
A single client retrying a single failed call is harmless. The failure mode appears at population scale. When a downstream service degrades, every caller's requests start timing out and retrying at roughly the same moment. That additional retry traffic is exactly what a struggling service can least afford.
- 1Downstream service slows down (GC pause, DB lock contention, a bad deploy). Call latency creeps past the timeout for a growing fraction of requests.
- 2Every caller independently times out and retries. Retry traffic adds to the load the struggling service is already failing to handle. It now has less capacity per request, not more.
- 3More requests time out, triggering more retries. Without backoff, this is roughly exponential in the number of retry layers. Your service retries a times, and its caller retries b times, so each original request compounds to a×b calls.
- 4The downstream service now serves several multiples of its normal load. It cannot recover even after the original cause clears, because the retry traffic is the outage now. This is a metastable failure: stable at normal load, stable at zero load, unstable and self-sustaining in between.
- →The fix requires cutting load, not adding more attempts. Exponential backoff with jitter slows the retry rate. A circuit breaker stops retrying at all once failure is the norm rather than the exception.
Backoff with jitter smooths when retries happen but doesn't cap how many total retries the whole fleet issues. A retry budget caps total amplification directly, for example "retries may not exceed 10% of the request rate over the last 10 seconds," per client or per service. Once the budget is spent, calls fail fast instead of retrying. Most systems skip this piece, and it's the one that actually bounds worst-case load during a real outage.
Retrying is only correct if the call is idempotent, or the caller can distinguish "definitely didn't happen" from "unknown" (e.g. connection refused vs. a timeout after the request was sent). Retrying a non-idempotent write on an ambiguous failure risks a duplicate side effect, which is a correctness bug dressed up as a reliability improvement. See Idempotency & Exactly-Once Payment Processing for the idempotency-key pattern that makes retries safe for writes.
A circuit breaker wraps a call to a dependency and tracks its recent success/failure rate. It has three states, and the entire pattern is the state machine between them. The electrical-circuit metaphor is apt: when the breaker is open, no current (calls) flows through.
| State | Behavior | Transition out |
|---|---|---|
| Closed | Calls pass through normally. Failures are counted in a rolling window | Failure rate crosses threshold → Open |
| Open | Calls fail immediately (fast-fail), no request reaches the dependency | After a cooldown timer → Half-Open |
| Half-Open | A small number of trial calls are allowed through to test recovery | Trials succeed → Closed. Trials fail → back to Open |
Without a breaker, every caller during an outage still pays the full timeout on every call before it stops. That is a slow, expensive way to fail, and it also keeps sending load downstream. An open breaker fails immediately, with no network call at all. Callers get a fast, cheap failure they can handle (fallback, cached value, error to the user) instead of blocking a thread for the full timeout. Fast-fail is the mechanism. Giving the downstream room to recover is the side effect that makes the whole system heal.
A time-based window ("failure rate over the last 10 seconds") behaves oddly at low request rates. 1 call in 10 seconds that fails looks identical to a 100% failure rate, which trips the breaker on a single data point. A count-based rolling window (last N calls, e.g. N=20) is more resilient to traffic bursts and quiet periods alike. That is why resilience4j and most modern implementations default to it.
Every threshold on a circuit breaker is a trade-off. Too eager, and false positives block a dependency that was only briefly slow. Too reluctant, and real outages add load for too long before the breaker acts. Play with the simulator below to see the state machine react to a failure spike and recovery.
Suppose half-open let all traffic through to test recovery. A dependency that's still fragile would get the full traffic spike the instant it shows the first sign of life. That can knock it back down immediately, creating a trip/probe/re-trip oscillation. Limiting the trial to one or a handful of calls tests recovery cheaply, at a cost the dependency can absorb even if the test fails.
A single breaker guarding "all downstream calls" ties one dependency's health to an unrelated one. If the payments API is failing, a shared breaker can trip and block the (perfectly healthy) inventory API too. Each dependency gets its own breaker with its own window and threshold, ideally each distinct endpoint, not just each service. Then failures stay isolated to the thing that's actually failing.
The name comes from a ship's bulkheads: watertight compartments that keep one breached section from sinking the whole hull. Applied to services: every downstream call may share one thread pool or one connection pool. Then a single slow dependency can exhaust that shared resource and starve calls to every other, perfectly healthy dependency. A bulkhead gives each dependency its own bounded slice of resources.
- 1Service calls three downstreams (A, B, C) using one shared thread pool of 100 threads for all outbound HTTP calls.
- 2Dependency A starts responding slowly (not failing, just slow). Threads calling A accumulate as they wait on responses, holding their thread the whole time.
- 3Within seconds, all 100 threads are blocked waiting on A. Calls to B and C (completely healthy) now have zero threads available and start failing or queuing too.
- →One slow dependency has taken down the whole service. With separate bounded pools per dependency (e.g. 30 threads for A, 30 for B, 30 for C), A's slowdown exhausts only its own 30. B and C keep serving normally.
| Isolation style | Mechanism | Cost | Used by |
|---|---|---|---|
| Thread pool isolation | Dedicated thread pool per dependency. Calls run on that pool | Thread overhead per pool, more memory | Original Hystrix default |
| Semaphore isolation | Shared threads, but a counting semaphore caps concurrent calls per dependency | No thread-count overhead, but a slow call still blocks the calling thread | Hystrix (lighter mode), resilience4j Bulkhead |
| Connection pool isolation | Separate DB/HTTP connection pool per downstream | More idle connections held open | HikariCP per-datasource, HTTP client pools per host |
| Process/container isolation | Separate deployable per dependency's call path | Operational overhead of more services | Extreme cases, usually overkill |
Thread pool isolation gives the strongest guarantee: a slow dependency literally cannot touch threads reserved for another. The cost is context-switch overhead and memory for extra thread pools, which adds up with dozens of dependencies. Semaphore isolation is cheaper (no extra threads). But the calling thread itself still blocks on the slow call up to the semaphore limit. So it only bounds concurrency, not the calling thread's own exposure to latency. In practice, use thread pool isolation for a small number of critical, failure-prone dependencies. Use semaphore isolation (or async I/O, which sidesteps the whole trade-off) for everything else.
A bulkhead's job is entirely "stop the bleeding from spreading". It does nothing to make the slow dependency healthy again. By itself it doesn't even stop calls to that dependency from continuing to queue up to its own pool's limit. Pair it with a circuit breaker (stop calling once it's clearly failing) and a timeout (bound how long each call can hold a bulkhead slot). The three are complementary, not substitutes for each other.
Circuit breakers and bulkheads protect a caller from a downstream dependency. Backpressure protects a service from its own callers. When incoming request volume exceeds what the service can actually process, something has to give. Either the service explicitly rejects the excess (backpressure), or an unbounded queue absorbs it silently until latency and memory both blow up.
An unbounded in-memory queue in front of a slow consumer feels safe, because nothing is ever rejected. But it converts a capacity problem into a latency and eventually a memory problem. Every request added to a growing queue waits behind everything already in it. So tail latency grows without bound even after the load spike ends, because the queue has to drain the entire backlog first. The system doesn't "fail" in an obvious way. It just gets slower and slower until it OOMs or every caller's own timeout fires anyway. The rejection happens, just later and more chaotically, after the system wastes resources on requests nobody will wait for.
| Technique | Idea | Signal to caller |
|---|---|---|
| Bounded queue + reject | Cap queue depth. Reject new work once full | HTTP 503 / 429 immediately, not after a long wait |
| Load shedding | Drop low-priority work first under pressure (e.g. by request class or client tier) | 429 for shed requests. Unaffected for priority traffic |
| Rate limiting | Cap requests per caller/key over a window (see Distributed Rate Limiter) | 429 with Retry-After |
| Adaptive concurrency limits | Measure actual latency, adjust the allowed in-flight request count (AIMD, like TCP congestion control) | 503 once the adaptive limit is hit |
| Reactive streams backpressure | Consumer signals producer how much it can accept (pull-based, not push) | Producer slows emission, no rejection needed |
A fixed "max 500 concurrent requests" limit has to be guessed and re-tuned every time the service's capacity changes (new hardware, a slower downstream, a code change). Netflix's concurrency-limits library instead measures actual observed latency continuously. It adjusts the allowed concurrency using an AIMD scheme: additive increase while latency is healthy, multiplicative decrease when it isn't. TCP congestion control uses the same idea to find available bandwidth without being told it in advance. The limit tracks the service's real, current capacity rather than a number someone guessed six months ago.
Explicitly rejecting the 5% of requests that exceed capacity, fast and with a clear signal, preserves good service for the other 95%. It also lets the caller retry with backoff or degrade gracefully. The alternative is silently accepting 100% of requests, serving all of them slowly, and eventually erroring out anyway. That is strictly worse for every caller, including the ones that would have been fine. Backpressure isn't a compromise on reliability. It's what makes the healthy majority of traffic reliable at all under real load.
None of these patterns is a substitute for the others. A production call path layers all of them, and the order matters. Each layer's job is to catch what the layer before it couldn't.
- 1Bulkhead: acquire a slot from this dependency's dedicated pool/semaphore. If the pool is full, fail immediately. A healthy dependency elsewhere is unaffected.
- 2Circuit breaker: if open, fail immediately without attempting the call at all. This check runs before the network call, so an outage costs nothing but a state check.
- 3Timeout: bound how long this specific attempt is allowed to take, so it can't hold its bulkhead slot forever.
- 4Retry with backoff + jitter, inside a budget: on a transient, idempotency-safe failure, retry a bounded number of times with increasing, jittered delay. The fleet-wide retry budget caps these retries.
- →Outcome recorded back into the circuit breaker's window either way. This is what lets the breaker trip on sustained failure and later probe for recovery.
Meanwhile, backpressure works on the receiving side of every service in this chain. It caps how much of this (retried, bulkheaded, breaker-wrapped) traffic the service will accept before it rejects the rest outright. This protects the service from every caller upstream. It is symmetric to how the caller protects itself from every dependency downstream.
Checking the circuit breaker before acquiring a bulkhead slot would waste a slot on a call that's about to fast-fail anyway. So the order of bulkhead and breaker is mostly a wash, but most implementations (resilience4j included) check bulkhead capacity first since it's cheaper. Two rules are non-negotiable. First, the timeout must wrap every individual attempt, not the whole retry loop. Otherwise one hung attempt blocks all remaining retries too. Second, retries must happen inside the per-attempt timeout/circuit-breaker check, not around it. Otherwise a breaker trip mid-retry-loop doesn't actually stop further attempts.
A brief trigger (a deploy blip, a GC pause, a network glitch) pushes the system into a self-sustaining bad state: full queues, tripped-open breakers everywhere, retry storms. That state persists after the original trigger is gone. The recovery attempts themselves (retries, reconnects, cache-refill stampedes) are now the load keeping it down. Fixing the root cause is necessary but not sufficient. Recovering from a metastable failure usually requires actively shedding load below the level that would let the system heal. That can mean deliberately rejecting traffic that could otherwise be served, until backlogs drain.
Every instance of a service may trip its breaker for the same dependency at roughly the same time (likely, since they're all seeing the same outage). Then they all enter half-open and send trial calls at roughly the same moment too. That is a mini thundering herd hitting a dependency that's mid-recovery. It can knock the dependency back down and re-trip every breaker in lockstep. Jittering the cooldown duration per instance (not a fixed 30s for everyone) spreads the probes out.
Bulkheads and breakers stop failure from spreading through direct call paths. But a shared resource underneath everything can still let one dependency's failure starve every service that touches it, bulkheads notwithstanding. Examples are a connection pool to a shared database, a shared cache cluster, or a shared message broker. This is why the resilience story has to include the data layer. That means connection pool limits per caller, cache stampede protection, and broker-level backpressure, not just the application-level call-wrapping patterns in this note.
A circuit breaker's fallback (return a cached value, a default, a degraded response) is good practice. But some fallbacks mask a real, ongoing outage: they return stale or wrong data with a 200 instead of surfacing that something is broken. That can turn a visible, alertable failure into an invisible correctness bug. Fallbacks need their own monitoring. Alert on fallback rate, not just on raw error rate. Otherwise a permanently-open breaker silently serving degraded data for weeks goes unnoticed.
A load balancer's shallow health check can report "healthy" for an instance whose actual request-serving path is completely broken. The shallow check is a TCP connect, or a trivial /health endpoint that doesn't touch the database. The broken case is a DB connection pool that is exhausted while the process is still up and answering pings. Health checks need to exercise enough of the real path to catch the failure modes that matter. But they must not be so deep that the health check itself becomes a source of load or false negatives.
| Symptom | Pattern | What it changes |
|---|---|---|
| One call occasionally hangs | Timeout | Bounds worst-case latency of a single attempt |
| Transient, isolated failures (one bad node behind an LB) | Retry with backoff + jitter | Recovers from failures that clear on their own, without herding |
| A dependency is systemically failing, not just occasionally | Circuit breaker | Stops sending load to something that can't currently serve it. Fails fast instead |
| One slow dependency degrades calls to unrelated dependencies | Bulkhead | Caps blast radius to the dependency that's actually failing |
| Your own service is overloaded by legitimate traffic volume | Backpressure / load shedding | Protects the majority of requests by explicitly rejecting the excess |
Timeouts and idempotency-safe, budgeted, jittered retries are close to mandatory on every outbound call. Circuit breakers and bulkheads earn their complexity on certain calls: to dependencies that can fail systemically and that share infrastructure with other, healthier traffic. Backpressure earns its complexity the moment a service is paged for an overload incident. It also earns it when a service serves traffic whose volume it doesn't fully control. Use all of them by default on a payments-adjacent or otherwise correctness-and-availability-critical path. Be more selective on low-stakes internal tooling. There, the operational overhead of four extra failure modes to reason about may not be worth it.
resilience4j (JVM), Polly (.NET), and the retry/circuit-breaker support built into service meshes (Envoy's outlier detection, Istio) implement all of this correctly. They come with sane defaults and already handle battle-tested edge cases. The value of understanding the mechanics is choosing and tuning thresholds correctly. It also helps you reason about failure modes in an interview or an incident. It is not a reason to reimplement a circuit breaker in production.
Follow-up questions that probe the design beyond the happy path. Click each to expand the answer.
A single caller retrying a single failed call is harmless. The problem is population scale. When a dependency degrades, every caller sees failures at roughly the same time and retries at roughly the same time. That retry traffic adds directly to the load on a service that's already failing to keep up. Without backoff and jitter, this compounds across retry layers roughly exponentially (my retries times my caller's retries times their caller's retries). The dependency then serves several multiples of its normal load. It can't recover even after the original cause of the slowdown clears, because the retry storm itself is now the load.
The fix isn't "don't retry". It has three parts. Exponential backoff with jitter spreads retries out instead of spiking them in lockstep. A retry budget caps total fleet-wide amplification. A circuit breaker stops retrying entirely once failure is the norm rather than the transient exception.
Closed is the normal state: calls pass through, and outcomes are recorded in a rolling window (typically count-based, e.g. last 20 calls, since time-based windows misbehave at low traffic volume). If the failure rate in that window crosses a threshold (commonly ~50%), the breaker trips to open.
In open, calls fail immediately with no network call at all. This fast-fail replaces waiting out a timeout. It protects the caller's own latency and stops sending load to a struggling dependency. After a cooldown timer, the breaker moves to half-open and allows one (or a small number of) trial calls through. If the trial succeeds, it goes back to closed with a fresh window. If it fails, it goes back to open for another cooldown. Half-open is limited to a small number of calls because a still-fragile dependency shouldn't get the full traffic spike the instant it shows the first sign of recovery.
Thread pool isolation gives each dependency its own dedicated pool of threads. Calls to that dependency can only ever consume threads from its own pool. So a slow dependency literally cannot touch the threads reserved for another. That's the strongest guarantee, but it costs memory and context-switch overhead per extra pool, which adds up with many dependencies.
Semaphore isolation uses the caller's existing thread but caps how many concurrent calls to a given dependency are allowed via a counting semaphore. It's cheaper, with no extra threads. But the calling thread itself still blocks for the duration of a slow call, up to the semaphore's limit. It bounds concurrency but not the calling thread's own exposure to latency. In practice, use thread pool isolation for a small number of critical or historically failure-prone dependencies. Use semaphore isolation (or fully async I/O, which sidesteps this trade-off entirely) for the rest.
It feels safe because nothing is explicitly rejected. But it converts a capacity problem into a latency and then a memory problem. Every new request queues behind the entire existing backlog. So tail latency grows without bound, and keeps growing even after the load spike that caused it ends, because the queue has to drain the whole backlog first. Eventually the process OOMs, or every caller's own client-side timeout fires anyway. So the effective rejection still happens, just later and more chaotically, after the system burns resources on requests nobody is still waiting for.
The fix is a bounded queue that rejects explicitly (503/429) once full. Ideally, pair it with load shedding that drops lower-priority work first. Also add an adaptive concurrency limit that adjusts the accepted in-flight count to the service's actual current capacity, rather than a static guessed number.
A metastable failure is a state the system enters after a brief trigger: a deploy blip, a GC pause, a short network glitch. The state then sustains itself even after the trigger is gone. The system's own recovery behavior (retries, reconnects, cache-refill stampedes, queued backlog) now generates enough load to keep it down. The system is stable at normal load and stable at zero load, but unstable in between, and the outage traffic is self-reinforcing.
That's why fixing the root cause (rolling back the bad deploy, letting the GC pause pass) is necessary but often not sufficient. The retry storm and backlog it triggered can keep the system down on their own. Recovery usually needs active load shedding, deliberately below what the system could otherwise serve. This lets backlogs drain and breakers close before traffic returns to normal levels.
A fallback that returns a cached or default value with a 200 status can mask a real, ongoing outage. From the outside, the system looks like it's working, but it's silently serving stale or degraded data. You need dedicated monitoring on fallback rate, not just raw error rate (which the fallback is specifically designed to suppress). Without it, a breaker can sit open for days or weeks serving degraded responses with nobody noticing. That turns a visible, alertable failure into an invisible correctness problem instead.
The fix is to treat "fallback engaged" as its own signal worth alerting on, separate from "request failed". Then someone investigates a permanently tripped breaker, even though it's technically "handling" every request without erroring.
It's a trade-off in both directions, same shape as picking a lock TTL. A threshold that is too sensitive (e.g. tripping at 20% failures), or a window that is too short, lets normal, brief blips trip the breaker. The breaker then blocks a dependency that would have recovered on its own, fast-failing traffic that could have succeeded. Too lax (80%+ threshold, huge window) means the breaker doesn't trip until the dependency is already deeply unhealthy and has absorbed a lot of pointless load first.
In practice I'd start from the dependency's normal baseline error rate (measured, not guessed) and set the threshold meaningfully above that. It should be high enough to not fire on routine noise, and low enough to fire well before a real outage compounds. Cooldown duration should roughly match how long the dependency typically takes to recover from the failure modes it actually has (a restart, a failover, a GC pause clearing). Jitter it per-instance, so every caller's breaker doesn't probe at the exact same moment and re-trip the dependency in a herd.
Per-call: a timeout bounding each individual attempt (not the whole retry loop). A bulkhead (thread pool or semaphore, dedicated to this dependency), so a slow fraud-check doesn't starve calls to, say, the payment-processor API running through the same service. A circuit breaker wrapping the call. It trips on sustained failure, so a real outage fast-fails instead of every payment attempt eating the full timeout. Retries only if the failure is clearly transient and the call is safe to repeat. For a fraud check specifically, that needs care. A duplicate fraud-check call is usually harmless (it's a read). But if it has any side effect (logging a flagged attempt), that needs an idempotency key too.
Critically, this needs an explicit fallback decision for what happens when the breaker is open. Does the payment proceed without a fraud check (risk), or does it fail closed and reject the payment (availability cost)? That's a product/risk decision, not a purely technical one. It's the kind of judgment call that's worth stating explicitly rather than assuming.
Three service layers each retry a failing dependency. How many calls reach the dependency, and how does a retry budget change it?
- 1Calls per request at the bottom: 3 × 3 × 3 = 27.
- 2Load on the dependency: 1,000 × 27 = 27,000 calls per second, which is 27 times normal.
- 3With a 10% budget at each layer, calls grow by at most 1.1 per layer: 1,000 → 1,100 → 1,210 → 1,331.
- 4The dependency now sees about 1.33 times its normal load. That is 20 times lower than 27.
Retries multiply across layers. A budget at each layer bounds the worst case at 1.1 per layer. A circuit breaker then cuts the remaining load to near zero. Set the budget as a share of the request rate, not as a fixed count per call.
One hung attempt can use the full 2 seconds. No time remains for the other attempts, so the retries never run. The timeout must wrap each attempt. A separate overall deadline can still cap the total time.
In-flight calls equal rate × latency. At 50 ms that is 200 × 0.05 = 10 threads. The pool fills when 200 × latency = 30, so latency = 0.15 s. That is 150 ms, only 3 times normal. After that, new calls queue or fail.
Count timeouts, connection errors, 5xx responses and 429s. Treat calls slower than the latency limit as failures too. Do not count other 4xx errors such as validation failures. Those are caller mistakes, and the dependency is healthy. Counting them trips the breaker for a dependency that works fine.
The load balancer keeps sending traffic to a pod that cannot serve it. Add a cheap check of the pool state to the readiness check. Keep the liveness check shallow, so a busy pod is not restarted. Only readiness reflects pool exhaustion. Do not call remote dependencies in the health check. If one dependency fails, every pod would report unhealthy and the whole service would go down.
Build a circuit breaker class wrapping an arbitrary function call. The closed state tracks the last N outcomes in a rolling window. It trips to open once the failure rate crosses a configurable threshold. The open state fails immediately without invoking the wrapped function. After a cooldown, transition to half-open and allow exactly one trial call to decide close vs. re-open.
Write a test harness that simulates a dependency going from healthy to 100% failing to recovered. Verify that the breaker trips at the right point and fast-fails while open (assert the wrapped function is never actually called). Verify that it correctly closes again once the dependency recovers.
Your service calls five dependencies. A database (critical, every request). A cache (optional: a miss just means slower response). A fraud-check API (critical for payment requests only). A recommendations service (optional, degrades gracefully to a static fallback). An email-notification service (fire-and-forget, failure is fine to ignore). For each, decide: does it need a bulkhead, a circuit breaker, retries, a fallback? What isolation level (thread pool vs. semaphore) for the bulkhead, if any?
Then: your service itself is starting to get overloaded during a traffic spike from an unrelated marketing campaign. Design the backpressure story. What gets shed first, what never gets shed, and what signal do rejected callers get back?
Postmortem summary: a 90-second network blip between two regions caused a spike in timeouts. Retries (fixed 1-second interval, no jitter, no budget) started across ~800 service instances simultaneously. The downstream service now received roughly 4x normal load from retry traffic alone. It started timing out on requests that would otherwise have succeeded, triggering more retries. The network blip resolved after 90 seconds, but the outage continued for another 40 minutes. Explain exactly why the outage outlived its trigger, and what should have prevented the retry storm from compounding in the first place. Then explain why the on-call engineer's first instinct ("just restart the downstream service") didn't work. What did they have to do instead to actually recover?
Covers retry storms and jitter, retry budgets, circuit breakers, and the metastable-failure recovery pattern of deliberately shedding load below normal capacity before ramping back up.