Inspecting processes

Why this matters

"The server is slow" is the most common ticket you will ever receive, and it is answered by finding out what is running, what it is waiting for, and what it is consuming. That is this lesson.

The material also underpins the whole Troubleshooting domain — 22% of the exam. A process in state D and a process in state Z look similar in top and mean completely different things, and knowing which is which decides whether you have a storage problem or a badly written parent process.

The lesson

PID, PPID, and the process tree

Every process has a Process ID (PID), assigned at creation, and a Parent Process ID (PPID) identifying the process that started it. Processes form a tree rooted at PID 1 — systemd — which is started by the kernel at boot.

echo $$          # the PID of your current shell
pstree           # the whole tree, drawn
pstree -p         # with PIDs
pstree -u alice   # only alice's processes

The parent relationship is not decoration. When a parent dies, its children are re-parented to PID 1, which then cleans up after them. And when a parent fails to clean up after a dead child, you get a zombie — see below.

ps: a snapshot

ps prints processes at one instant. It has two incompatible option styles for historical reasons; you only need two invocations.

ps aux              # BSD style: every process, with owner and resource use
ps -ef              # UNIX style: every process, with PPID
ps -ef --forest     # as a tree
ps aux | grep nginx # the everyday filter
ps -u alice         # one user's processes
ps -p 1234 -o pid,ppid,stat,pcpu,pmem,cmd   # choose your own columns

ps aux columns worth reading: USER, PID, %CPU, %MEM, VSZ (virtual size), RSS (resident set size — actual physical memory), STAT (state), START, TIME (cumulative CPU time), COMMAND.

RSS is the number that matters for "is this using too much memory". VSZ counts address space the process has reserved but may never touch, so it is routinely enormous and routinely misleading.

ps -ef is the one to reach for when you need the PPID, because ps aux does not show it.

top and its friends: live

top          # live, refreshing
htop         # nicer: colour, mouse, scrolling, tree view
atop         # records history, and shows disk and network per process

Inside top: P sorts by CPU, M by memory, k kills a process, 1 shows each CPU core separately, q quits.

The header line is where the diagnosis usually starts. Load average gives one, five and fifteen-minute figures — roughly the number of processes running or waiting to run. Compare it with your core count: a load of 4.0 on four cores is fully busy but healthy; 40.0 on four cores is a queue.

The crucial subtlety: on Linux, load average includes processes blocked on disk I/O, not just CPU. So a very high load with low CPU usage means the system is waiting for storage, not computing. That single fact separates two completely different investigations, and top's %wa (I/O wait) figure confirms it.

atop is worth installing precisely because it keeps history — it can tell you what was happening at 3 a.m., which top cannot.

Per-resource detail

mpstat 2 5           # CPU stats per core, every 2s, 5 times
mpstat -P ALL 1      # all cores, every second
pidstat 2            # per-PROCESS CPU, every 2 seconds
pidstat -d 2         # per-process DISK I/O
pidstat -r 2         # per-process memory
pidstat -p 1234 1    # one process

mpstat answers "is one core saturated while the others idle?" — the signature of a single-threaded bottleneck. pidstat -d answers "which process is actually hitting the disk?", which top cannot tell you and which is exactly what you need when I/O wait is high.

Both come from the sysstat package.

Open files and sockets

lsof -p 1234         # everything this process has open
lsof /var/log/app.log  # which processes have this file open
lsof -i :443         # what is using port 443
lsof -u alice        # everything alice has open
lsof +D /mnt/data    # everything open under a directory

lsof is how you answer "why can't I unmount this filesystem" and "why is the disk still full after I deleted the file" — a process holding a deleted file open keeps its blocks allocated until it closes.

Tracing what a process is actually doing

strace -p 1234              # attach and print system calls
strace -c ls                # summary count by syscall
strace -e trace=openat ls   # only file-opening calls

strace shows every system call a process makes. When a program hangs with no useful logging, attaching strace usually shows it stuck on a single call — reading a socket that will never answer, or opening a file that does not exist. It is slow and intrusive, so it is a diagnostic tool rather than something to leave running.

/proc/<PID>

Every process has a directory under /proc holding its live state.

ls /proc/1234/
cat /proc/1234/cmdline    # the exact command line
cat /proc/1234/status     # state, memory, UIDs, thread count
ls -l /proc/1234/cwd      # working directory
ls -l /proc/1234/exe      # the binary, even if it has been deleted
ls /proc/1234/fd/         # open file descriptors
cat /proc/1234/environ | tr '\0' '\n'   # its environment

/proc/<PID>/exe is quietly useful during incident response: it still points at the binary even when the file has been unlinked from disk, which is a common malware behaviour.

Process states

The STAT column, and the reason to care about each:

Extra letters may follow: s session leader, < high priority, N low priority, + in the foreground.

Two of these carry real diagnostic weight.

D — blocked. The process cannot be killed, not even with kill -9, because it is inside a kernel call waiting for hardware. A pile of D-state processes means a storage problem: a failing disk, a hung NFS mount, a saturated SAN. The fix is upstream, at the storage, not at the process.

Z — zombie. The process is already dead; the entry survives only to hold its exit status until the parent reads it. A zombie consumes no CPU and no memory — just a slot in the process table. You cannot kill a zombie; it is already dead. The fix is to deal with the parent: signal it to reap, or restart it, after which init adopts and clears the orphans. A handful of zombies is harmless; thousands mean a buggy parent and eventually an exhausted process table.

Priority and nice

Every process has a priority, influenced by its nice value, which runs from -20 (greediest) to +19 (most generous). Higher nice means nicer to others, so lower priority.

nice -n 10 ./backup.sh     # start it with reduced priority
nice -n -5 ./urgent.sh     # raise priority — requires root
ps -eo pid,ni,comm         # show nice values
top                        # the NI column

Only root may set a negative nice value; any user may make their own processes nicer. That asymmetry is the point — you can always yield, never seize.

nice sets priority at launch. Changing it afterwards is renice, in the next lesson.

On the exam

• Load average on Linux includes processes blocked on I/O. High load with low CPU means storage, not compute.
• RSS is real memory; VSZ is address space and is misleading.
• State D cannot be killed, including with kill -9, because it is in uninterruptible sleep. Expect this as a scenario.
• A zombie is already dead — you cannot kill it. Fix or restart the parent.
• ps -ef shows PPID; ps aux does not.
• nice ranges -20 to +19; higher is lower priority; only root can go negative.
• lsof finds what is holding a mount point or a deleted file.
• When a parent dies, children are re-parented to PID 1.