Chase a latency regression to its process

applied · 50 min · Objective 5.5

Before you start — what this lab needs

A disposable virtual machine running a service you can load-test (a web app with wrk or ab), on which you can plant a competing workload. Lab VM; snapshot first so the regression is repeatable.

dnf install -y httpd-tools sysstat stress-ng   # ab; iostat/pidstat/mpstat/sar; load
# wrk is not in RHEL's repositories -- use ab, as in the Verify, unless you have it

Task

A service got slower and nothing in its own logs explains why. Reproduce that and track the cause to a specific process and resource using the USE method (Utilisation, Saturation, Errors) rather than guesswork. The deliverable is a measured before/after, not a feeling that it is faster.

Steps

Measure the baseline properly: run a fixed load test and record p50 and p99 latency and throughput. One number is not a measurement of a distribution -- p99 is where the pain lives.
Plant the regression: start a competing process that saturates ONE resource (a dd loop for disk, stress-ng --vm for memory pressure, a busy loop for CPU). Do not tell yourself which.
Re-run the identical load test. Confirm the regression is real and quantify it: p99 went from X to Y.
Apply USE across resources: mpstat -P ALL (per-CPU utilisation), iostat -x (disk util and await), free/vmstat (memory and swap), sar -n DEV (network). Find the one that is saturated.
Attribute it to a process: pidstat -d for disk per process, pidstat for CPU, smem or /proc/<pid>/status for memory. Name the PID.
Remove the competing workload and re-measure to confirm p99 returns to baseline -- closing the loop proves you found the cause, not a coincidence.
Write the regression up as: symptom, resource, process, evidence, resolution.

Verify

# the load test is the measurement; capture the tail latency, not just the mean:
ab -n 5000 -c 50 http://localhost:8080/ | grep -E '^ *99%|Requests per second'
# (or, with wrk: wrk -t2 -c50 -d20s --latency http://localhost:8080/ | grep -E '99%|Requests/sec')
mpstat -P ALL 1 3 2>/dev/null   # which CPUs, if any, are saturated
iostat -x 1 3 2>/dev/null       # disk util/await under load
pidstat -d 1 3 2>/dev/null      # per-process disk I/O -- names the culprit

The proof is the loop closing: p99 rises when the competing process runs and returns to baseline when it stops, and one resource's saturation lines up with both transitions. A fix that is not confirmed by re-measurement is a guess that happened to coincide with the symptom clearing.

Notes

The USE method is the discipline that keeps this from being a fishing trip: for each resource, check utilisation, saturation and errors in turn, and stop at the one that is saturated. p99 rather than mean latency matters because a service that is fast on average and terrible at the tail is the exact shape users complain about, and the mean hides it.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.