Stop a dead dependency from taking down its caller

applied · 70 min · Objective 5.4

Task

Show a cascading failure and then stop it. A caller with no timeout waiting on a dead dependency exhausts its own resources and fails -- which is how one service's outage becomes several.

Steps

  1. Run the caller with a bounded worker pool and NO timeout on its outbound call. Stop the dependency and drive steady client traffic. Record what happens to the caller's pool, its latency and its own availability in lab/integration/cascade.md.
  2. Add a timeout shorter than the client's patience. Repeat and record the difference -- the caller should now fail fast rather than hang.
  3. Add retries with exponential backoff and JITTER. Repeat, and record what the dependency sees when it comes back: measure the request rate in the first seconds after recovery, with and without jitter.
  4. Add a circuit breaker that opens after repeated failures and fails fast without calling. Repeat and record the caller's resource usage while the dependency is down.
  5. Write the summary: which of the three mechanisms protected the caller, which protected the dependency, and which protected the client's experience.

Verify

grep -Eci 'timeout' lab/integration/cascade.md
grep -Eci 'jitter' lab/integration/cascade.md
grep -Eci 'circuit breaker' lab/integration/cascade.md
grep -Ec '[0-9]+ ?(ms|s|req|request)' lab/integration/cascade.md

All four non-zero with real measurements. The jitter comparison is the interesting one: without it, every throttled client retries simultaneously and the retry itself becomes a load spike.

This is an independent study companion for CompTIA Cloud+ CV0-004 and is not produced by or endorsed by CompTIA.