Failure, Timeouts & Retries

Services fail in more than one way, and the ways are not equally bad.

Crash is the good failure: fast and unambiguous. Slow is the one that takes whole systems down, because nothing detects it and every caller's threads pile up behind it.

A call without a timeout waits forever, which means a held thread, a held connection, and often a held database transaction. When a downstream service hangs, every thread in your service ends up parked on it. That is how one slow dependency takes down a healthy one.

Pick timeouts from measured latency, not round numbers. If p99 is 200 ms, a 1-second timeout gives real requests room while cutting off hung ones. A 30-second timeout is indistinguishable from no timeout.

Timeout budgets are the part most people miss. A user request has 3 seconds, calls A, which calls B, which calls C. The inner calls must fit inside what remains, or C is still working on a request A abandoned two seconds ago. Pass the remaining budget down and let each hop shrink it.

Retrying a failed call is obviously right, and naive retries are one of the most reliable ways to turn a small problem into an outage. A service slows, every client times out and retries, load doubles, it slows further, clients retry more. That is a retry storm, and the service cannot recover even after the original cause is gone, because the retries are the load now.

• Exponential backoff. 1s, 2s, 4s, 8s. Give the dependency room instead of hammering it.
• Jitter. Randomise the delay. Without it, everyone who failed together retries together and you have rebuilt the spike.
• A cap. Three attempts, not thirty. Then fail and let the caller decide.
• Retry only what is retryable. 5xx and timeouts, yes. 4xx, no: a malformed request stays malformed.

Retries help with a blip. When a dependency is truly down, continuing to call it wastes your resources and delays its recovery. A circuit breaker stops the calls.

Two benefits: your service stops burning threads on calls that will fail, and the struggling dependency gets the quiet it needs to recover.

Bulkheads are the companion. Give each dependency its own limited pool of connections or threads, so a hanging dependency can only consume its own share. Without them, a slow recommendations service exhausts the same pool checkout needs, and a nice-to-have takes down a critical path.

Service C slows. B's threads fill waiting on C. A's threads fill waiting on B. The front end stops responding, and the symptom is "the site is down" when one internal service was broken. That is cascading failure. Three things stop the spread:

• Bounded queues. An unbounded queue does not prevent overload, it hides it. Requests sit past their client timeout, so you do work nobody is waiting for. A bounded queue that rejects when full at least fails honestly.
• Load shedding. Above a threshold, reject immediately with 429 or 503. Serving 70% of traffic well beats serving 100% at 30 seconds each, which is really serving nobody.
• Back pressure. Push the "slow down" signal upstream instead of absorbing it silently, so producers stop generating work the system cannot handle.