Troubleshooting performance

Why this matters

Performance problems are reported as feelings — "it's slow" — and have to be turned into a number before you can do anything. Slow at what? Compared to when? By how much?

That is why exceeding baselines is on the objectives. Without a baseline you cannot tell a problem from a busy Tuesday, and you cannot tell whether your fix worked. A load average of 8 is alarming on two cores and idle on thirty-two; 200 ms of disk latency is broken for a database and irrelevant for a backup job.

The other reason this lesson matters is that there are only four resources — CPU, memory, disk, network — and the whole skill is identifying which one is the constraint before changing anything. Most wasted performance work is tuning the wrong one.

The lesson

The sixty-second triage

Before any theory, run these. They tell you which of the four resources is the problem:

uptime                     # load average: 1, 5, 15 minutes
vmstat 1 5                 # r, b, si/so, wa -- the densest single view
free -h                    # available, not free
df -h && df -i
top -o %CPU                # or htop
iostat -xz 1 5             # per-disk utilisation and latency
ss -s                      # socket summary

vmstat 1 5 is the highest-yield command on this list. Five columns answer almost everything:

• r — processes waiting for CPU. Consistently above your core count means CPU-bound.
• b — processes blocked on I/O. Non-zero means disk or network waiting.
• si/so — swap in/out. Non-zero and sustained means memory pressure.
• wa — CPU time spent waiting for I/O. High means the disk is the bottleneck.
• cs — context switches per second.

High load average, and what it actually means

uptime
cat /proc/loadavg
nproc                      # how many cores you have

Load average is not CPU utilisation. On Linux it counts processes that are runnable or in uninterruptible sleep — so it includes processes blocked on disk. This is why a machine with idle CPUs can show a load of 40: they are all waiting on a failing disk or a hung NFS mount.

The interpretation:

• Divide by core count. Load 8 on 8 cores is fully utilised; on 2 cores it is badly oversubscribed.
• Compare the three numbers. 1-minute much higher than 15-minute means it is getting worse right now; the reverse means it is recovering.
• High load with low CPU utilisation means I/O, and you should be looking at iostat, not at processes.

That last point is the one worth memorising, because a high load average with an idle CPU sends most people to the wrong resource.

High CPU usage and CPU bottleneck

top -o %CPU
pidstat -u 1 5                       # per process, over time
mpstat -P ALL 1 5                    # PER CORE -- reveals single-thread saturation
ps -eo pid,pcpu,pmem,comm --sort=-pcpu | head
perf top                             # which functions, if you need that depth

Read the top header rather than the process list first:

• us (user) high — application work. Normal, if the work is real.
• sy (system) high — kernel time. Excessive syscalls, network interrupt load, or context switching.
• wa (I/O wait) high — not a CPU problem; the disk is the constraint.
• st (steal) high — you are on a virtual machine and the hypervisor is not giving you the CPU you asked for. Nothing you do inside the guest will fix it; it is a noisy neighbour or an oversubscribed host.

st is the one people miss entirely, and it is a genuine exam item: steal time means the problem is outside the machine.

mpstat -P ALL matters because a single-threaded process pinning one core shows as 100% of one CPU and perhaps 12% overall. top averaged across cores hides it completely.

A real CPU bottleneck is r in vmstat consistently above core count, low wa, low st, and processes accumulating runtime. The answers are fewer processes, more cores, or more efficient code — in that order of practicality.

Swapping and out of memory

free -h
vmstat 1 5                            # watch si/so
swapon --show
cat /proc/meminfo | grep -i "available\|swap"
smem -rs uss | head                   # what each process really uses
sysctl vm.swappiness

Read available, not free. Linux uses spare memory for page cache, so free near zero is normal and healthy — the cache is reclaimable. available estimates what a new process could actually obtain, and it is the only memory number worth alerting on.

Swap being used is not a problem; swap being actively transferred is. A machine can sit with 2 GB of swap occupied by idle pages and perform perfectly. What matters is sustained si/so in vmstat — pages moving in and out continuously. That is thrashing, and it destroys performance because every access costs a disk read.

The symptom is memorable: the machine becomes unresponsive while the CPU is not busy, load climbs into double figures, and even typing at the console lags.

Out of memory is what follows when there is nothing left to reclaim. The kernel invokes the OOM killer and kills the highest-scoring process with SIGKILL:

dmesg | grep -i "out of memory\|killed process"
journalctl -k | grep -i oom

Exit code 137 is the signature, as in the hardware lesson.

Responses in order: find what is using the memory (smem, ps --sort=-rss); constrain it with MemoryMax= on the unit or --memory on a container; add RAM; and only then consider tuning vm.swappiness — lowering it makes the kernel prefer dropping cache over swapping, which suits a database but does not create memory that is not there.

Disabling swap entirely is usually wrong. Without it there is no room to evict genuinely idle pages, so the OOM killer arrives sooner and more abruptly.

High I/O wait and disk latency

iostat -xz 1 5
iotop -o                              # which PROCESS is doing the I/O
pidstat -d 1 5
vmstat 1 5                            # the wa column
biolatency                            # bcc/eBPF, if available

iostat -x columns that matter:

• %util — how much of the time the device was busy. Near 100% on a spinning disk means saturated; on SSD and NVMe it is misleading, because they handle parallel requests and can be at 100% with capacity to spare.
• r_await / w_await — average latency in milliseconds, and the number to trust. Single-digit ms is healthy for SSD; over 20 ms on a supposedly fast device means a problem. High disk latency is this measurement.
• aqu-sz — average queue depth. Deep queues mean requests are backing up.

High I/O wait means the CPU is idle waiting for storage. The fix is never more CPU. It is finding the process (iotop -o shows only those doing I/O), and then: reducing the I/O, adding cache, faster storage, or spreading load.

Low throughput with low latency is a different problem — a bandwidth limit, a throttled cloud volume that has exhausted its burst credits, or a single-threaded reader. Measure both before concluding.

dd if=/dev/zero of=/tmp/test bs=1M count=1024 oflag=direct   # sequential write
fio --name=r --rw=randread --bs=4k --size=1G --numjobs=4 --runtime=30
hdparm -tT /dev/sda

oflag=direct bypasses the page cache — without it you are measuring RAM.

Slow remote storage response

NFS, iSCSI, SMB and cloud volumes fail differently from local disks:

nfsstat -c                            # client stats: retransmits are the tell
mount | grep nfs
showmount -e nfsserver
ping -c20 nfsserver                   # latency AND loss to the server
mtr nfsserver
iostat -xz 1 5                        # the mount appears here too

The dominant cause is the network, not the storage. A few milliseconds of added latency per operation is invisible on a large sequential read and crippling on a workload of many small ones.

Two specifics:

A hung NFS mount puts processes into uninterruptible D state, unkillable, and the load average climbs with an idle CPU. hard mounts retry forever by design — which protects data and hangs the machine. soft,timeo= returns errors instead, at the risk of a truncated write. Neither is universally right; hard,intr where supported is the usual compromise.

Check rsize/wsize and the NFS version. Small transfer sizes multiply round trips, and NFSv3 over a high-latency link is far slower than v4.1 with its compound operations.

For cloud volumes, remember they are provisioned by IOPS and throughput, often with burst credits. A volume that was fast for an hour and is now slow has exhausted its burst balance — a metric on the provider's side, invisible from inside the guest, and a very common "the server got slower for no reason".

High context switching and blocked processes

vmstat 1 5                            # cs and in columns
pidstat -w 1 5                        # per-process context switches
cat /proc/interrupts

Context switching is normal; thousands per second per core is normal. What matters is a sharp rise without a matching rise in useful work, which points at:

• Too many runnable threads for the cores available — the scheduler spends its time switching rather than running. Over-sized thread pools do this, and the fix is fewer threads, which feels wrong and is right.
• Lock contention, with threads repeatedly sleeping and waking.
• Interrupt load from a busy network card — check /proc/interrupts for one CPU taking them all, which is fixed with IRQ affinity or receive-side scaling.

pidstat -w distinguishes voluntary switches (waiting for a resource) from involuntary ones (the scheduler preempted it). Mostly involuntary means CPU contention; mostly voluntary means waiting on locks or I/O.

Blocked processes are the b column in vmstat and D state in ps:

ps -eo pid,stat,wchan:25,comm | awk '$2 ~ /D/'
cat /proc/<pid>/stack

wchan names the kernel function the process is waiting in, which usually identifies the resource directly. As in the hardware lesson, a D-state process cannot be killed — you must fix what it is waiting for.

System unresponsiveness and sluggish terminal behaviour

The machine responds to ping, and everything is treacle.

uptime
vmstat 1 5
dmesg -T | tail -40
iostat -xz 1 3

The usual causes, in order of likelihood: memory thrashing (sustained si/so), disk saturation (wa high, %util near 100), an OOM killer loop killing and restarting a service repeatedly, or a failing disk doing retries that take seconds each.

Sluggish terminal behaviour deserves its own note because it splits cleanly in two, and the split is diagnostic:

• Slow to log in, fast once you are in — that is a name resolution problem, not a performance one. sshd doing a reverse DNS lookup on your address with a broken resolver waits for the timeout every time. UseDNS no in sshd_config fixes it, and the same shape causes slow sudo when the hostname does not resolve.
• Slow throughout the session — that is genuine resource starvation, or network latency/loss on the SSH connection itself.

Asking "slow to connect, or slow once connected?" resolves this in one question, and it is worth asking first because the first case is not a performance problem at all.

Slow startup

systemd-analyze                       # firmware, loader, kernel, userspace
systemd-analyze blame                 # slowest units, ordered
systemd-analyze critical-chain        # what actually DELAYED the boot
journalctl -b | grep -i "timed out\|failed"

blame and critical-chain answer different questions, and confusing them wastes effort. blame lists the slowest units, but a slow unit running in parallel with others delays nothing. critical-chain shows the dependency path that actually determined the total — that is the one to optimise.

Recurring causes: a unit waiting on the network (NetworkManager-wait-online is a frequent 30-second offender), a mount for a device that is not there timing out, DNS timeouts during startup, and fsck on a large filesystem.

For the pre-kernel portion, systemd-analyze reports firmware and loader time separately — if the firmware takes 40 seconds, nothing inside Linux will help.

Slow application response and slow response times

curl -w "dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} \
ttfb:%{time_starttransfer} total:%{time_total}\n" -o /dev/null -s https://app.example.com

That single command splits the latency into stages, and the stage that dominates tells you where to look:

• time_namelookup high — DNS, not the application.
• time_connect high — network latency or a saturated backlog.
• time_appconnect high — TLS handshake; certificate chain fetching or a slow key exchange.
• time_starttransfer high — the server is thinking. Now it is the application or its database.

Only in the last case is the application itself the problem, and people routinely start there.

Then, on the server: application logs with timings, slow query logs, and the four resources. A slow application is very often a slow database, and a slow database is very often a missing index or a lock.

Measure percentiles, not averages. A 40 ms mean with a 2-second 99th percentile means one request in a hundred is dreadful, and the average conceals it entirely. Users experience the tail.

High latency, jitter, packet drops, random disconnects and timeouts

ping -c100 target                     # loss %, and min/avg/max/mdev
mtr -rwc 100 target                   # per-hop loss and latency
ss -s                                 # socket summary, including retransmits
netstat -s | grep -i "retransmit\|drop\|overflow"
ip -s link show eth0                  # errors, dropped, overruns
ethtool -S eth0 | grep -i "drop\|error\|discard"

High latency is ping's avg. Jitter is the variation — mdev in ping's output, or mtr's standard deviation column. Jitter matters independently of latency for anything real-time: 100 ms of consistent latency is workable for voice, while 40 ms swinging by 60 ms is not.

Packet drops appear in three different places and mean three different things:

• ip -s link dropped/overruns on the host — the kernel could not keep up. Ring buffer too small (ethtool -G), or interrupts on one CPU.
• netstat -s retransmits — loss somewhere in the path.
• mtr per-hop loss — loss at a specific hop. Loss at an intermediate hop that does not continue to the destination is ICMP rate limiting, not a fault — this is the most misread output in network diagnostics.

Random disconnects and random timeouts share a small set of causes, and "random" is usually a pattern nobody has looked for yet:

• Idle timeouts in a firewall, NAT device or load balancer silently dropping connection state. The connection is not closed, it is forgotten, and the next packet vanishes. TCP keepalives (ClientAliveInterval for SSH) hold it open.
• Conntrack table full — nf_conntrack: table full, dropping packet in dmesg, on a busy firewall or NAT host.
• Duplex mismatch or a failing cable, from the networking lesson: fine when idle, dreadful under load.
• MTU problems, which present as a connection that establishes and then hangs on the first large transfer.
• DHCP lease expiry with a conflict, or an IP conflict — connectivity that works, then does not, then does.
• Wireless: interference, roaming between access points, power saving.

The approach is to make "random" specific. Log continuously and correlate: ping -D with timestamps, mtr --report-cycles on a loop, and the exact times from the application log. A pattern — every 60 minutes, only under load, only to one destination — is nearly always there, and it is the diagnosis.

High failed login attempts

lastb | head -40                      # failed logins, from /var/log/btmp
journalctl -u sshd | grep -i "failed password" | tail -40
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
faillock --user alice
ss -tn state established '( dport = :22 or sport = :22 )'

That awk pipeline is the counting idiom from the shell utilities lesson doing real work: it produces a ranked list of the source addresses attacking you.

This appears under performance because at volume it is a performance problem — thousands of authentication attempts per minute consume CPU on password hashing, fill logs, exhaust sshd's connection slots and can deny service to legitimate users.

Responses, in the order they are worth doing:

# 1. keys only -- removes the attack surface entirely
PasswordAuthentication no
PermitRootLogin no

# 2. rate limit
MaxStartups 10:30:60
MaxAuthTries 3

# 3. block repeat offenders
fail2ban-client status sshd
firewall-cmd --permanent --add-rich-rule='rule source address="203.0.113.0/24" drop'

Disabling password authentication is the real fix, because it makes the attack pointless rather than merely slower. Moving SSH to a non-standard port reduces log noise substantially and is not a security control — say so plainly rather than treating it as one.

A distinction the exam cares about: broad, distributed attempts against root and common usernames are internet background noise. Repeated failures against one real account, or successes after failures, or attempts from inside your network — those are an incident, and belong in the escalation path from the methodology lesson rather than in a tuning exercise.

Hardware errors

dmesg -T | grep -i "error\|fail\|corrected\|thermal"
journalctl -k -p err -b
edac-util -v                          # memory error counters
mcelog --client                       # machine check exceptions
smartctl -a /dev/sda
sensors                               # temperatures
ipmitool sel list                     # the hardware event log

Hardware degradation shows as performance loss long before it shows as failure, and each has a signature:

• Correctable ECC memory errors — corrected, so nothing fails, but each costs time and a rising count predicts an uncorrectable error. edac-util is the only place this appears.
• Disk sector reallocation — every retry is milliseconds of latency, so a dying disk looks exactly like a slow application.
• Thermal throttling — the CPU reduces its clock to stay within temperature. sensors and a lower-than-expected frequency in /proc/cpuinfo confirm it, and the cause is dust, a failed fan, or a datacentre cooling problem.
• PCIe errors or a link that has negotiated fewer lanes than it should, halving a storage controller's bandwidth.

A gradual, unexplained slowdown with no configuration change is hardware until proven otherwise, and dmesg -T plus the IPMI event log are where it is written down.

Exceeding baselines

The concept the whole lesson depends on. A baseline is what normal looks like, recorded when things are working, so "slow" can become "40% above the weekday mean".

sar -u 1 3                            # from sysstat: historical CPU
sar -r                                # memory
sar -d -p                             # disk
sar -n DEV                            # network
sar -q                                # load average
sar -f /var/log/sa/sa15               # a SPECIFIC PREVIOUS DAY

sar is the tool that answers "was it like this last week?" — the sysstat package collects and retains system activity, so you can compare today against the same hour on a normal day. Enabling it costs nothing and is the difference between diagnosing a regression and guessing at one. Enable it before you need it; systemctl enable --now sysstat.

What to record: CPU per core, memory available, disk latency and utilisation, network throughput and errors, load average, and the application's own latency percentiles and request rate.

Two points that make baselines useful rather than decorative. Baselines are seasonal — compare Monday 09:00 to Monday 09:00, not to Sunday 03:00. And re-baseline after deliberate change: a deployment that legitimately raises CPU by 20% makes the old baseline a permanent false alarm, and a permanent false alarm is how alerting dies.

On the exam

• Load average includes processes blocked on I/O, so high load with idle CPU means disk or network, not CPU. Divide by core count.
• vmstat 1: r runnable, b blocked, si/so swapping, wa I/O wait, cs context switches.
• In top: wa high is a disk problem; st (steal) high means the hypervisor is not giving you CPU — nothing inside the guest can fix it. mpstat -P ALL reveals one saturated core that the average hides.
• available, not free. Swap used is fine; sustained si/so is thrashing. OOM kill = exit 137. Do not disable swap.
• iostat -x: r_await/w_await are latency in ms and the number to trust; %util misleads on SSD/NVMe. iotop -o finds the process.
• High I/O wait is never fixed with more CPU.
• Remote storage: a hung NFS mount gives unkillable D state and high load with idle CPU; cloud volumes exhaust burst credits.
• curl -w splits latency into DNS, connect, TLS and time-to-first-byte — find the stage before blaming the application. Measure percentiles.
• systemd-analyze critical-chain, not blame, shows what delayed boot.
• Slow to log in but fine afterwards is reverse DNS, not performance.
• Jitter is variation, not latency. mtr loss at an intermediate hop only is ICMP rate limiting, not a fault.
• Random disconnects: firewall/NAT idle timeouts, full conntrack table, duplex mismatch, MTU. Make "random" specific by correlating timestamps.
• High failed logins are a performance problem at volume; the real fix is PasswordAuthentication no. Repeated failures against one real account is an incident, not noise.
• Gradual unexplained slowdown with no change is hardware — edac-util, smartctl, sensors, thermal throttling.
• sar gives you the historical baseline. Baselines are seasonal, and must be re-taken after deliberate change.