Chase a latency regression to its process

applied · 50 min · Objective 5.5

Task

A service got slower and nothing in its own logs explains why. Reproduce that and track the cause to a specific process and resource using the USE method (Utilisation, Saturation, Errors) rather than guesswork. The deliverable is a measured before/after, not a feeling that it is faster.

Steps

  1. Measure the baseline properly: run a fixed load test and record p50 and p99 latency and throughput. One number is not a measurement of a distribution -- p99 is where the pain lives.
  2. Plant the regression: start a competing process that saturates ONE resource (a dd loop for disk, stress-ng --vm for memory pressure, a busy loop for CPU). Do not tell yourself which.
  3. Re-run the identical load test. Confirm the regression is real and quantify it: p99 went from X to Y.
  4. Apply USE across resources: mpstat -P ALL (per-CPU utilisation), iostat -x (disk util and await), free/vmstat (memory and swap), sar -n DEV (network). Find the one that is saturated.
  5. Attribute it to a process: pidstat -d for disk per process, pidstat for CPU, smem or /proc/<pid>/status for memory. Name the PID.
  6. Remove the competing workload and re-measure to confirm p99 returns to baseline -- closing the loop proves you found the cause, not a coincidence.
  7. Write the regression up as: symptom, resource, process, evidence, resolution.

Verify

# the load test is the measurement; capture the tail latency, not just the mean:
wrk -t2 -c50 -d20s --latency http://localhost:8080/ | grep -E '99%|Requests/sec'
mpstat -P ALL 1 3 2>/dev/null   # which CPUs, if any, are saturated
iostat -x 1 3 2>/dev/null       # disk util/await under load
pidstat -d 1 3 2>/dev/null      # per-process disk I/O -- names the culprit

The proof is the loop closing: p99 rises when the competing process runs and returns to baseline when it stops, and one resource's saturation lines up with both transitions. A fix that is not confirmed by re-measurement is a guess that happened to coincide with the symptom clearing.

Notes

The USE method is the discipline that keeps this from being a fishing trip: for each resource, check utilisation, saturation and errors in turn, and stop at the one that is saturated. p99 rather than mean latency matters because a service that is fast on average and terrible at the tail is the exact shape users complain about, and the mean hides it.