Bring a broken service back under an incident clock

capstone · 75 min · Objective 5.5

Task

You inherit a web service that "was working last week" and now returns errors under load. It has more than one fault, planted across the four troubleshooting domains you have practised -- a network reachability fault, a storage fault, a security-context fault, and a resource-saturation fault. Working as if it were a real incident, restore the service to a measured baseline and produce a short writeup. This capstone is passed not when the service responds once, but when it meets a stated latency target under load with every control still enforcing.

Steps

  1. Triage in order, not by hunch. Start at reachability: from a client, classify the failure by timing and message (hang-then-timeout = DROP, instant "Connection refused" = nothing listening or an iptables REJECT, instant "No route to host" over a working route = firewalld's default rejection). Open the port and confirm you can now reach the service at all.
  2. The service is reachable but returns 403. Do not disable SELinux. Read ausearch -m avc -ts recent and ls -Z on the document root, add the correct semanage fcontext rule, restorecon, and confirm a 200.
  3. The service now serves but errors intermittently under write. Check df -h and df -i (inodes, the fault people miss): a full partition returns ENOSPC. Identify the largest consumers with du -x --max-depth=1 and clear or rotate what is safe to clear.
  4. Latency is still bad. Apply the USE method: mpstat, iostat -x, vmstat, sar -n DEV, find the saturated resource, and attribute it to a process with pidstat -d. Stop or throttle the competing workload.
  5. Establish the baseline you are being measured against: run a fixed load test and record p50 and p99. State the target explicitly before you claim success.
  6. Re-run the load test after each fix so you can attribute each improvement to a specific change -- and so a fix that did nothing is exposed rather than assumed.
  7. Write the incident up: timeline, each fault, the observation that identified it, the fix, and the before/after latency. Note which control you were tempted to disable and did not.

Verify

getenforce                                   # must still say Enforcing
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://server:8080/
ab -n 5000 -c 50 http://server:8080/ | grep -E '^ *99%|Requests per second'   # wrk is not in RHEL's repos
df -h /var /srv; df -i /var /srv             # space AND inodes both healthy
iostat -x 1 3 2>/dev/null                     # no disk with climbing await and queue (%util alone misleads on SSDs)
ausearch -m avc -ts recent 2>/dev/null | tail -3   # no fresh SELinux denials

The capstone is passed when all of these hold at once: a 200, p99 under your stated target, both space and inodes healthy, no disk saturated, and getenforce still Enforcing. Any single check passing while another fails is a partial fix -- exactly the state a real incident is left in when someone stops at the first green light.

Notes

The discipline the whole domain has been building toward is here: a systematic order (reachability, then the application-layer denial, then storage, then resource saturation) beats jumping to the fault you find most familiar, because a multi-fault incident hides its later faults behind its earlier ones -- you cannot see the 403 until the port is open, or the I/O saturation until the disk has space. The habit that separates this from luck is re-measuring after every change, so each fix is attributable and "it works now" is a number, not a mood.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.