Name the resource before you tune anything

short · 30 min · Objective 5.5

Task

Every performance problem is one of four resources -- CPU, memory, disk I/O, or network -- and the whole skill is naming which before you change anything. Practise the triage against planted loads so that "the server is slow" becomes "the server is disk-bound, here is the process" in under a minute.

Steps

  1. Learn the top line first: run uptime and read the three load averages against the core count from nproc. Load above cores means a run queue is backing up -- but load does not say WHICH resource.
  2. CPU case: start CPU workers. Confirm with top that %us or %sy is near 100 and load tracks it. The tell is a process pinned at ~100% of a core.
  3. Memory case: start memory pressure. Read free -h and vmstat 1: the tell is not low free memory (Linux uses it for cache) but active swapping -- si and so columns non-zero. Distinguish cache from committed memory.
  4. I/O case: start heavy disk writes. In top, CPU is mostly idle but %wa (iowait) is high and load is still high. Confirm the disk with iostat -x 1 -- %util near 100 and high await. This is the case people misread as a CPU problem because load is high.
  5. For each, identify the guilty process with pidstat or top sorted by the right column.
  6. Write the one line that names the resource and the process.

Verify

uptime                       # load average vs nproc
vmstat 1 3                   # si/so reveal swapping; wa reveals I/O wait
iostat -x 1 3 2>/dev/null    # %util and await pinpoint a saturated disk

High load with idle CPU and high wa is the decisive I/O signature -- it is the one that looks like a CPU problem in uptime alone and is not. Non-zero si/so in vmstat is the memory signature; free memory being low without swapping is normal and not a problem to chase.

Notes

The mistake this lab prevents is tuning the wrong resource: adding CPU to an I/O-bound box, or "freeing" cache memory that was doing its job. Load average tells you something is queuing; it never tells you what. Always resolve load to a resource with vmstat/iostat/top before you touch a single knob.