Failure, Timeouts & Retries
Everything fails. The design question is what your system does in the seconds afterwards, and whether that response makes things better or much worse.
Prerequisites: One network call to a service you do not own.
Cover these firstNetworking & Protocols
After this: Set timeouts and retry policy deliberately, and recognise when a retry needs idempotency behind it.
Suggested first pass: Read sections 1–5, answer each section in your own words, then use the remaining failure modes and exercises as the advanced pass.
Services fail in more than one way, and the ways are not equally bad.
| Mode | What happens | Detection | Blast radius |
|---|---|---|---|
| Crash | Process dies, connections refuse | instant | Contained; traffic reroutes |
| Slow | Answers in 30s instead of 50ms | health checks pass | Spreads to every caller |
| Partial | One endpoint, or 5% of users | hidden in averages | Small but long-lived |
| Byzantine | Confident wrong answers | very hard | Corrupts downstream data |
Crash is the good failure: fast and unambiguous. Slow is the one that takes whole systems down, because nothing detects it and every caller's threads pile up behind it.
Turn the dangerous mode into a fast failure: tight timeouts, bounded queues, circuit breakers, and health checks that measure latency instead of merely returning 200. A component that gives up quickly is one the rest of the system can route around.