Troubleshooting hardware and storage

Why this matters

This is the objective with the longest list of named failures on the whole exam, and they are named for a reason: each has a characteristic symptom and a short path to the cause. Recognising "No space left on device on a disk that is 40% free" as inode exhaustion is worth more than any amount of general knowledge, because it turns a two-hour investigation into a two-minute one.

The common issues below are CompTIA's list. Learn them as symptom → first command → likely cause, which is how the exam presents them and how they arrive in real life.

The lesson

Server not turning on

Nothing on the screen, no response. Work outward from power:

• Power — is the PSU on, is the cable seated, is there a light? Redundant supplies fail silently, so the machine keeps running on one and then dies on the second failure.
• Out-of-band management — iDRAC, iLO or IPMI is the tool for this. It has its own network connection and works with the machine powered off, giving you power control, a console, and the hardware event log.

ipmitool -H mgmt01 -U admin -P "$PASS" chassis power status
ipmitool -H mgmt01 -U admin -P "$PASS" sel list       # hardware event log
ipmitool -H mgmt01 -U admin -P "$PASS" sensor list    # temperatures, voltages

The system event log is the thing to read: it records the PSU failure, the overheat shutdown, the memory error, with a timestamp, and it is available when the operating system is not.

• POST beeps or diagnostic LEDs — the vendor's manual decodes them, and they usually name the failed component directly.
• Memory — a machine that will not POST is very often one failed DIMM. Reseat, or remove all but one, and try again.

If it powers on but shows nothing, separate "not booting" from "no video" before going further — a serial console or IPMI console tells you which.

Kernel panic

The kernel has hit an unrecoverable error and stopped. The screen shows a trace and the machine is dead.

Common causes: a bad or missing kernel module, corrupted initramfs, failing memory, an unavailable root filesystem, or an incompatible driver after an update.

Boot the previous kernel from the GRUB menu. That single action both restores service and diagnoses it — if the old kernel boots, the new kernel or its initramfs is the problem.

# once you are in
journalctl -k -b -1              # kernel messages from the PREVIOUS boot
ls /var/crash/                   # kdump output, if configured
dmesg --level=err,crit
grubby --info=ALL                # which kernels are installed
dracut -f --kver 5.14.0-427.el9  # rebuild a broken initramfs

journalctl -k -b -1 is the key command, because a panicking machine cannot write its own logs to disk in time — but the previous boot's messages are there, and the last few before the panic name the module. Configure kdump on anything important; it writes a crash dump to /var/crash for exactly this.

For suspected memory, run memtest86+ from the boot menu. Intermittent panics with no pattern are memory until proven otherwise.

Server inaccessible

The machine is up but you cannot reach it. Work outward:

ping server01                    # is it on the network at all?
ssh -v user@server01             # verbose -- where does it stop?
nc -zv server01 22               # is the port open?

• Ping fails: network, routing, firewall, or the host is down. Ask the monitoring system whether it stopped reporting.
• Ping works, SSH refused: sshd is not running or not listening.
• Ping works, SSH times out: a firewall is dropping rather than rejecting.
• SSH connects then fails: authentication — wrong key, wrong permissions on ~/.ssh, an expired or locked account, or / full so PAM cannot write.

Use the out-of-band console. IPMI, virsh console, or a cloud provider's serial console reaches a machine whose networking is broken, which is precisely when you need it. That is the reason to enable a serial console in your template rather than when you need it.

A machine that responds to ping but does nothing else, with a very high load average, is usually out of memory and thrashing, or has exhausted a resource — see the memory leak section below.

Filesystem will not mount

mount -a                         # try everything in fstab, and read the error
dmesg | tail -30                 # the kernel's opinion, which is the useful one
blkid                            # UUIDs and filesystem types as they REALLY are
lsblk -f
cat /etc/fstab

The causes, in the order they occur:

Wrong UUID in /etc/fstab. A disk was replaced or a filesystem recreated, and mkfs generates a new UUID. blkid versus fstab shows it immediately.

The device is missing — a failed disk, an unassembled RAID array, an inactive LVM volume group:

cat /proc/mdstat                 # RAID status
mdadm --detail /dev/md0
vgchange -ay                     # activate volume groups
lvs && vgs && pvs

Filesystem corruption, which needs a check — on an unmounted filesystem:

umount /dev/sdb1
fsck -y /dev/sdb1
xfs_repair /dev/sdb1             # xfs uses its own tool, not fsck
xfs_repair -L /dev/sdb1          # zeroes the log -- DATA LOSS, last resort

Never run fsck on a mounted filesystem. It will corrupt what was previously merely damaged. And xfs_repair -L discards the journal, which means losing whatever it held — it is the option after everything else.

Wrong filesystem type or missing module — mount: unknown filesystem type 'xfs' means the module is not loaded or not installed.

The trap worth naming: a bad /etc/fstab entry stops the machine booting. systemd waits for the device, times out, and drops to emergency mode. Add nofail to any non-essential mount, and after editing fstab run mount -a before rebooting to confirm it parses.

OS filesystem full

df -h                            # space
df -i                            # INODES -- check both, always
du -xh /var --max-depth=2 | sort -h | tail -20
ncdu /var                        # interactive, if available

The -x matters: it keeps du on one filesystem instead of wandering into /proc and network mounts.

The usual culprits are logs in /var/log, the journal, package caches, old kernels, container images and layers, and core dumps.

journalctl --disk-usage
journalctl --vacuum-size=500M
dnf clean all
podman system prune -a
find /var/log -name "*.gz" -mtime +30 -delete

The deleted-but-open file is the case that defeats people: du says the space is used by nothing, df insists the disk is full. A file was deleted while a process still had it open, so the space is not released until that process closes it:

lsof +L1                         # files with a link count of 0
lsof -nP | grep '(deleted)'

Restart the process holding it — usually the application whose log you just deleted rather than truncated. > /var/log/app.log truncates in place and avoids the problem entirely.

A full root filesystem breaks things that look unrelated: logins fail, sudo fails, services will not start, because each needs to write something. Fix the space before diagnosing anything else on that machine.

Inode exhaustion

The signature failure, and the one worth memorising:

touch: cannot touch 'newfile': No space left on device

…on a filesystem df -h reports as 40% free.

df -i                            # Use% at 100% here
# find the directories with the most files
find /var -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -rn | head

An inode holds a file's metadata, and a filesystem is created with a fixed number of them. Millions of tiny files — a mail queue, a session directory, a cache, an application logging one file per request — exhaust the inodes while using almost no space.

The fix is deleting files, and the deletion is slow because there are millions of them. find ... -delete beats rm with a huge argument list.

The permanent fix is architectural: rotate and archive, or recreate the filesystem with more inodes (mkfs.ext4 -N), or move to xfs, which allocates inodes dynamically and does not have a fixed limit. Then monitor df -i, because nothing else will tell you.

Partition not writable

mount | grep " / "               # does it say (ro,...)?
dmesg | grep -i "read-only\|I/O error\|EXT4-fs error"

A filesystem mounted read-only when it should be writable has three causes, and they need very different responses:

The kernel remounted it read-only after an I/O error. This is a protective action — it saw corruption or a failing disk and stopped writing to limit the damage. Do not simply remount it read-write. Check the disk first; you are being told the hardware is failing.

fstab says ro, or the filesystem was mounted that way deliberately.

The disk is genuinely failing, which is the previous case's cause.

smartctl -a /dev/sda             # SMART health -- read this before anything
smartctl -t short /dev/sda
mount -o remount,rw /            # only once you know WHY it went read-only

Also check the obvious: a full filesystem, or one whose quota is exhausted, behaves like a read-only one for the affected user.

Device failure

smartctl -H /dev/sda             # overall health assessment
smartctl -a /dev/sda | grep -i "reallocated\|pending\|uncorrectable"
dmesg | grep -i "I/O error\|ata.*failed\|medium error"
cat /proc/mdstat                 # a RAID array with [U_] has lost a member
badblocks -sv /dev/sdb           # read-only scan -- slow

The SMART attributes that actually predict failure are Reallocated_Sector_Ct, Current_Pending_Sector and Offline_Uncorrectable. Any of them non-zero and rising means replace the disk. SMART's overall "PASSED" is not reassurance — plenty of disks fail while reporting PASSED, which is why the individual counters matter more than the verdict.

For a failed RAID member:

mdadm --detail /dev/md0
mdadm --manage /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
mdadm --manage /dev/md0 --add /dev/sdc1     # then watch /proc/mdstat rebuild

Replace a degraded array's disk urgently. A RAID 5 with one failed disk has no redundancy at all, and the rebuild itself is the highest-stress read the remaining disks will ever see — which is exactly when a second one fails.

Data corruption issues

sha256sum -c manifest.txt        # verify against known checksums
fsck -n /dev/sdb1                # check WITHOUT modifying
xfs_repair -n /dev/sdb1
dmesg | grep -i "checksum\|corrupt\|I/O error"
smartctl -a /dev/sda
edac-util -v                     # memory error counters

Causes: a failing disk, bad memory (which corrupts data on its way to disk, silently), an unclean shutdown, a failing controller or cable, or a filesystem bug.

Two things to hold on to. Corruption caused by bad RAM will keep happening after you repair the filesystem, and will corrupt your restore as well — which is why edac-util and memtest belong in this list. And a filesystem repair is not data recovery: fsck makes the filesystem consistent, which may mean discarding what it cannot reconcile. If the data matters more than the uptime, image the disk with ddrescue first and work on the copy.

Kernel corruption issues

A damaged kernel image or initramfs: the machine fails to boot, panics early, or reports "unable to mount root fs".

# from the GRUB menu, boot an older kernel first
rpm -V kernel-core               # verify the package against its manifest
dnf reinstall kernel-core
dracut -f --kver $(uname -r)     # rebuild the initramfs
grub2-mkconfig -o /boot/grub2/grub.cfg

A frequent and under-appreciated cause is /boot being full, so the initramfs was written truncated. df -h /boot first, remove old kernels with dnf remove --oldinstallonly, then rebuild.

GRUB misconfiguration

The Grand Unified Bootloader fails and you get a grub> or grub rescue> prompt, "no such partition", or a machine that hangs before the kernel loads.

# from the rescue prompt, find and boot manually
grub> ls
grub> set root=(hd0,gpt2)
grub> linux /vmlinuz-5.14.0 root=/dev/mapper/rhel-root
grub> initrd /initramfs-5.14.0.img
grub> boot

Then repair properly from a rescue environment:

mount /dev/mapper/rhel-root /mnt
mount /dev/sda1 /mnt/boot
mount /dev/sda2 /mnt/boot/efi
for d in dev proc sys run; do mount --bind /$d /mnt/$d; done
chroot /mnt
grub2-install /dev/sda                        # BIOS
dnf reinstall grub2-efi-x64 shim-x64          # UEFI
grub2-mkconfig -o /boot/grub2/grub.cfg

Editing /boot/grub2/grub.cfg by hand is the mistake — it is generated, and the next grub2-mkconfig discards your edit. Change /etc/default/grub and regenerate.

The single most common cause is a disk renamed or replaced so the UUID in the configuration no longer exists. GRUB's rescue prompt with ls is how you find out what is actually there.

Systemd unit failures

systemctl --failed                       # everything that failed, first command
systemctl status myapp                   # state, PID, and the last log lines
journalctl -u myapp -n 50 --no-pager
journalctl -xeu myapp                    # with explanatory hints
systemd-analyze verify /etc/systemd/system/myapp.service
systemctl cat myapp                      # the unit as systemd actually sees it

Read the status output carefully — the exit code and the last journal lines are usually the entire answer.

The recurring causes:

• ExecStart path wrong — status says status=203/EXEC. The binary does not exist or is not executable.
• status=200/CHDIR — WorkingDirectory does not exist.
• Permissions — the User= cannot read a file or bind a port below 1024 (which needs AmbientCapabilities=CAP_NET_BIND_SERVICE).
• A dependency has not started — check After= and Requires=.
• Edited the file and did not reload: systemctl daemon-reload after every unit change, or systemd keeps running the old definition. This is the most common of the lot.
• SELinux denying something. ausearch -m avc -ts recent will say so, and the failure otherwise looks like an inexplicable permission error.
• Start rate limiting — a service that fails repeatedly is refused further restarts until systemctl reset-failed.

Package dependency issues

dnf install nginx                     # read the error properly
dnf repoquery --requires nginx
dnf repolist                          # are the repos even enabled?
dnf clean all && dnf makecache
rpm -q --whatprovides /usr/lib64/libssl.so.3

apt install nginx
apt --fix-broken install
apt-cache policy nginx                # which versions, from which sources
dpkg --configure -a

Causes: a disabled or unreachable repository, a stale cache, mixing repositories that carry incompatible builds of the same library, or a package held back.

The dangerous "fixes" are the ones that resolve the message rather than the problem. rpm -i --nodeps and dpkg --force-depends install the package and leave it broken at runtime, in a way that surfaces later and looks unrelated. dnf install --skip-broken quietly skips the thing you were installing.

Third-party repositories are the usual root cause. dnf module, priorities, or --enablerepo for one transaction keep them from displacing distribution packages permanently.

Segmentation fault

A process tried to access memory it does not own, and the kernel killed it. Segmentation fault (core dumped).

dmesg | grep -i segfault
coredumpctl list
coredumpctl info 12345
coredumpctl gdb 12345                 # if you have the debug symbols
ulimit -c unlimited                   # allow core dumps in this shell

For an administrator the useful causes are rarely the program's own bug:

• A library mismatch — the binary was built against a different version of a shared library. ldd /usr/bin/app shows what it links to and what is missing.
• Corrupted binary or library, from a truncated download or a failing disk. rpm -V verifies against the package manifest.
• Bad memory, which produces segfaults in unrelated programs at random — the pattern that should send you to memtest.
• Stack exhaustion from deep recursion, or a resource limit.

One segfault in one program is a bug. Segfaults in several unrelated programs are hardware until you have proved otherwise.

Killed processes

A process disappears with no error of its own. The first question is who killed it:

dmesg | grep -i "killed process\|Out of memory"
journalctl -k | grep -i oom
grep -i oom /var/log/messages
systemctl status myapp                # "Killed" or signal in the status

The OOM killer is the usual answer. When the kernel cannot satisfy an allocation it selects a process — generally the one using the most memory — and kills it with SIGKILL, which cannot be caught or handled. The message names the process and its score.

Recognise it from exit code 137, which is 128 + 9. A container restarting with 137 ran out of memory; it was not your script.

# constrain the culprit rather than the victim
systemctl set-property myapp.service MemoryMax=2G
# or protect the important one
systemctl set-property db.service OOMScoreAdjust=-500

Other killers: a cgroup memory limit (the container case), systemd stopping a unit that exceeded RuntimeMaxSec, a supervisor's health check, or a human. SIGKILL leaves nothing behind, which is why the kernel log is the only record.

Memory leaks

Memory use rises steadily and never falls, until the OOM killer intervenes — often days after the deployment that caused it.

ps aux --sort=-%mem | head
top -o %MEM
free -h                               # look at "available", not "free"
smem -rs uss                          # unique set size: what is really the process's
watch -n 60 'ps -o pid,rss,cmd -p 12345'   # RSS climbing over hours = a leak
pmap -x 12345
valgrind --leak-check=full ./app      # for something you build yourself

Read available, not free. Linux uses all spare memory as cache, so free being near zero is normal and healthy. available is what a new process could actually get, and it is the only number worth alerting on.

The distinction that matters: a genuine leak is RSS rising monotonically over hours or days with no plateau. A process that grows to a steady state and stops is not leaking, it is using memory.

The practical mitigations while the bug is fixed: MemoryMax= on the unit so one process cannot take the machine down, Restart=on-failure so it recovers, and a scheduled restart if it must be endured. Monitor available and the process's RSS so you find out before the OOM killer does.

Unresponsive process

A process that will not exit, or one that has stopped doing anything:

ps -eo pid,stat,wchan:20,cmd | grep myapp
kill -TERM 12345                      # ask politely
kill -KILL 12345                      # if it will not go
strace -p 12345                       # what syscall is it in?
cat /proc/12345/stack                 # kernel stack, for D state
lsof -p 12345

The process state letter is the diagnosis:

• R running, S sleeping — normal.
• D — uninterruptible sleep, waiting on I/O. A process in D cannot be killed, not even with -9, because it is inside a kernel operation. The problem is the storage or the network filesystem it is waiting on, not the process. dmesg for I/O errors, or an unreachable NFS server — where mounting with soft,intr instead of hard would have prevented it.
• Z — zombie, already dead, kept in the table because its parent has not reaped it. Zombies cannot be killed — they are already dead. Kill or fix the parent; if the parent dies, init adopts and reaps them. A handful are harmless; thousands mean a buggy parent and will exhaust the process table.
• T — stopped, usually by SIGSTOP; kill -CONT resumes it.

strace -p on a hung process, showing it blocked in one syscall, usually names the resource it is waiting for in one line.

Quota issues

quota -u alice                        # one user's usage and limits
repquota -a                           # everything, per filesystem
edquota -u alice
quotaon -avug
xfs_quota -x -c 'report -h' /home     # xfs has its own tooling

The symptom is a user unable to write while df shows plenty of space — "Disk quota exceeded". Quotas limit blocks and inodes separately, so a user with space left can still be blocked by too many files, which is inode exhaustion in miniature and just as confusing.

The soft limit may be exceeded for a grace period; the hard limit never can. A user past the soft limit whose grace has expired is refused writes even though they are under the hard limit, and the message does not explain that. repquota shows the grace column, which is where the answer is.

Quotas must also be enabled in fstab (usrquota,grpquota) and turned on; half-configured quotas produce confusing partial enforcement.

PATH misconfiguration issues

bash: ansible: command not found

…for a command you know is installed.

echo $PATH
which ansible || command -v ansible
type -a ansible
ls -l /usr/local/bin/ansible
sudo env | grep PATH                  # root's PATH is DIFFERENT

The causes:

• The directory is not on PATH — common for /usr/local/bin, ~/.local/bin and anything installed by pip --user.
• PATH was overwritten rather than appended. PATH=/opt/tool/bin instead of PATH=$PATH:/opt/tool/bin removes everything, and the shell then cannot find ls. The fix in a live shell is export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.
• sudo resets PATH via secure_path in /etc/sudoers, which is why a command works for you and not under sudo. That is deliberate — it prevents a user's PATH from tricking root into running the wrong binary — so use the full path rather than weakening it.
• cron has a minimal PATH (often just /usr/bin:/bin), which is why a script that works interactively fails from cron. Set PATH at the top of the crontab or use absolute paths.
• Order matters: the first match wins, so a directory earlier in PATH can shadow the binary you expected. type -a lists every match, in order.

Missing or disabled drivers

Hardware that does not appear, or appears and does not work:

lspci -k                              # devices AND the kernel driver in use
lsusb
lsmod                                 # loaded modules
modprobe e1000e                       # load one
modinfo e1000e
dmesg | grep -i "firmware\|failed to load\|taint"
cat /etc/modprobe.d/blacklist.conf

lspci -k is the command. A device listed with no "Kernel driver in use" line has no driver bound, which is the whole diagnosis.

The causes:

• Firmware missing — the module loads but the device needs a binary blob from linux-firmware. dmesg says so explicitly, and it is very common on wireless and some network cards.
• Module blacklisted, often deliberately by a vendor driver's installer, in /etc/modprobe.d/.
• Out-of-tree driver not rebuilt after a kernel update — the module was built for the old kernel and does not load for the new one. DKMS exists to rebuild automatically; when it fails, the machine boots without its storage or network driver, which is memorable.
• Secure Boot refusing an unsigned module. The module is present, correct, and rejected. mokutil --sb-state reports whether Secure Boot is on, and the module must be signed and its key enrolled.
• Module missing from the initramfs, so a driver needed for the root filesystem is not there at boot. dracut -f --add-drivers "...".

On the exam

• df -h AND df -i. "No space left on device" with free space is inode exhaustion; xfs allocates inodes dynamically and avoids it.
• Space missing from df but not du is a deleted-but-open file — lsof +L1, restart the holder.
• Kernel panic: boot the previous kernel, then journalctl -k -b -1.
• Read-only filesystem after an I/O error is the kernel protecting you — check smartctl before remounting.
• SMART: Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable. A degraded RAID array must be rebuilt urgently.
• fsck only on an unmounted filesystem; xfs uses xfs_repair, and -L loses data.
• A bad fstab entry stops the boot — use nofail, and mount -a before rebooting.
• GRUB: never hand-edit grub.cfg; edit /etc/default/grub and regenerate. /boot full breaks initramfs generation.
• Systemd: systemctl --failed first; daemon-reload after every unit edit; 203/EXEC is a bad ExecStart path.
• Never --nodeps or --force-depends to clear a dependency error.
• Exit 137 is the OOM killer (128 + SIGKILL). Constrain with MemoryMax=.
• Segfaults in several unrelated programs mean hardware.
• Process states: D cannot be killed (waiting on I/O — fix the storage), Z is already dead (fix the parent).
• Quotas limit blocks and inodes separately, and a soft limit past its grace period behaves like a hard one.
• sudo and cron have their own PATH. First match on PATH wins.
• lspci -k: no "Kernel driver in use" means no driver. Check firmware, blacklists, DKMS after a kernel update, and Secure Boot signing.