Bring a broken service back under an incident clock

capstone · 75 min · Objective 5.5

Task

You inherit a web service that "was working last week" and now returns errors under load. It has more than one fault, planted across the four troubleshooting domains you have practised -- a network reachability fault, a storage fault, a security-context fault, and a resource-saturation fault. Working as if it were a real incident, restore the service to a measured baseline and produce a short writeup. This capstone is passed not when the service responds once, but when it meets a stated latency target under load with every control still enforcing.

Steps

  1. Triage in order, not by hunch. Start at reachability: from a client, classify the failure by timing and message (hang-then-timeout = DROP, instant refused = nothing listening or REJECT). Open the port and confirm you can now reach the service at all.
  2. The service is reachable but returns 403. Do not disable SELinux. Read ausearch -m avc -ts recent and ls -Z on the document root, add the correct semanage fcontext rule, restorecon, and confirm a 200.
  3. The service now serves but errors intermittently under write. Check df -h and df -i (inodes, the fault people miss): a full partition returns ENOSPC. Identify the largest consumers with du -x --max-depth=1 and clear or rotate what is safe to clear.
  4. Latency is still bad. Apply the USE method: mpstat, iostat -x, vmstat, sar -n DEV, find the saturated resource, and attribute it to a process with pidstat -d. Stop or throttle the competing workload.
  5. Establish the baseline you are being measured against: run a fixed load test and record p50 and p99. State the target explicitly before you claim success.
  6. Re-run the load test after each fix so you can attribute each improvement to a specific change -- and so a fix that did nothing is exposed rather than assumed.
  7. Write the incident up: timeline, each fault, the observation that identified it, the fix, and the before/after latency. Note which control you were tempted to disable and did not.

Verify

getenforce                                   # must still say Enforcing
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://server:8080/
wrk -t2 -c50 -d30s --latency http://server:8080/ | grep -E '99%|Requests/sec'
df -h /var /srv; df -i /var /srv             # space AND inodes both healthy
iostat -x 1 3 2>/dev/null                     # no single disk pinned at 100% util
ausearch -m avc -ts recent 2>/dev/null | tail -3   # no fresh SELinux denials

The capstone is passed when all of these hold at once: a 200, p99 under your stated target, both space and inodes healthy, no disk saturated, and getenforce still Enforcing. Any single check passing while another fails is a partial fix -- exactly the state a real incident is left in when someone stops at the first green light.

Notes

The discipline the whole domain has been building toward is here: a systematic order (reachability, then the application-layer denial, then storage, then resource saturation) beats jumping to the fault you find most familiar, because a multi-fault incident hides its later faults behind its earlier ones -- you cannot see the 403 until the port is open, or the I/O saturation until the disk has space. The habit that separates this from luck is re-measuring after every change, so each fix is attributable and "it works now" is a number, not a mood.