Devices and kernel modules

Why this matters

The kernel talks to hardware through modules — drivers it can load and unload while running, rather than compiling into a single fixed image. That design is why one installation image boots on thousands of different machines.

Practically, this lesson answers two questions you will be asked constantly: what hardware is in this box? and why is this device not working? The second is nearly always a driver that is missing, not loaded, or loaded with the wrong options.

The lesson

Kernel modules

A module is a driver or feature that plugs into the running kernel. They live under /lib/modules/$(uname -r)/, which is why modules are tied to a specific kernel version — a fact that explains a great deal of upgrade pain.

lsmod                       # what is loaded now
lsmod | grep nvme           # is a particular driver in?
modinfo e1000e              # description, author, version, PARAMETERS, dependencies
modprobe e1000e             # load, resolving dependencies
modprobe -r e1000e          # unload, with its now-unused dependencies
insmod /path/to/module.ko   # load a file directly, NO dependency resolution
rmmod e1000e                # unload one module only
depmod -a                   # rebuild the module dependency map

modprobe versus insmod is the pairing to know. modprobe searches the module directory by name and pulls in whatever the module depends on. insmod takes a literal file path and does nothing else — if the module needs another one loaded first, it simply fails. modprobe is what you use; insmod exists for the case where you have built a module yourself and want to test it before installing it.

The same split applies coming back out: modprobe -r removes the module and any dependencies nothing else needs, while rmmod removes exactly one.

depmod builds modules.dep, the map modprobe consults. Installing a module by hand means running depmod -a afterwards, or modprobe will not find it.

lsmod output has three columns: the module, its size, and a use count with the list of modules using it. A non-zero use count is why rmmod refuses — something depends on it.

Module parameters are settings passed at load time, and modinfo -p lists what a module accepts:

modinfo -p e1000e
modprobe e1000e InterruptThrottleRate=3000    # load with a parameter

To make that persistent, and to control loading at boot:

# /etc/modprobe.d/e1000e.conf
options e1000e InterruptThrottleRate=3000

# blacklist a module so it is not auto-loaded
blacklist nouveau

# /etc/modules-load.d/mymodule.conf  — force a module to load at boot
mymodule

Blacklisting is the standard fix for a driver conflict. The usual case is nouveau, the open-source NVIDIA driver, which must be blacklisted before the proprietary one can take the card. Note that blacklisting only prevents automatic loading — a module explicitly required by something else may still come in.

After changing anything under /etc/modprobe.d/ that affects boot, rebuild the initrd, or the change will not apply during early boot:

dracut -f                    # RPM family
update-initramfs -u          # Debian family

What hardware is in this machine?

A tool per layer, and knowing which answers which question saves a lot of guessing.

lscpu          # CPU: model, cores, threads, architecture, virtualisation flags
lsmem          # memory ranges, online/offline blocks
lspci          # everything on the PCI bus: GPU, NIC, storage controllers
lspci -k       # ...and WHICH KERNEL MODULE is driving each
lsusb          # USB devices
lsblk          # block devices as a tree, with mount points
lshw           # a full hardware inventory, verbose
lshw -short    # a readable summary
dmidecode      # the firmware's own tables: serial numbers, DIMM slots, BIOS version
dmidecode -t memory
dmidecode -t system

lspci -k is the one to reach for when a device does not work. It shows the device and the driver bound to it. A device listed with no "Kernel driver in use" line has no driver — which is the actual problem, and points you at a missing firmware package or an unsupported chipset.

dmidecode reads the DMI/SMBIOS tables written by the firmware, so it reports things the kernel cannot infer: the machine's serial number, the exact memory modules and which slots are populated, the BIOS version and date. When you need to order matching RAM without opening the case, dmidecode -t memory is the answer.

lscpu is where you check virtualisation support (vmx for Intel, svm for AMD) before wondering why KVM will not start.

dmesg: the kernel's running commentary

dmesg                    # the kernel ring buffer
dmesg -T                 # human-readable timestamps
dmesg -w                 # follow, live
dmesg --level=err,warn   # only problems
dmesg | grep -i usb      # what happened when I plugged that in?
journalctl -k            # the same content, via the journal
journalctl -k -b -1      # the previous boot

dmesg -T and then plugging the device in is the fastest hardware diagnostic there is: the kernel narrates detection, driver binding, and any failure, as it happens. Disk errors, USB resets, out-of-memory kills and segfaults all land here first.

Note the buffer is finite and wraps, so on a long-running system early boot messages may be gone — journalctl -k retains them if the journal is persistent.

Sensors and out-of-band management

sensors                # temperatures, fan speeds, voltages
sensors-detect         # probe and configure, once, interactively

ipmitool sensor        # the same, via the BMC
ipmitool sel list      # the system event log
ipmitool chassis status
ipmitool power cycle   # power-cycle the host remotely

lm_sensors reads sensors from inside the running OS. Run sensors-detect once to identify the chips, then sensors reports on demand. Rising temperatures under load are the visible signature of a failing fan or blocked airflow, well before anything crashes.

ipmitool talks to the Baseboard Management Controller — a small computer on the motherboard with its own network connection and its own power. Because it runs independently of the operating system, it works when the machine is hung, or powered off. That is the point: you can read the hardware event log, check temperatures, and power-cycle a server from another continent without an OS. The vendor names — iDRAC, iLO, IMM — are all BMCs.

ipmitool sel list is where hardware faults are recorded: a failed DIMM, a predictive disk failure, a power supply that dropped out. Those events often never reach the OS at all.

initrd management

The initial RAM disk carries the drivers needed to reach the real root filesystem. It must be rebuilt whenever those change.

dracut -f                                 # rebuild for the running kernel
dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
lsinitrd | less                           # what is inside it

update-initramfs -u                       # Debian: rebuild for current kernel
update-initramfs -u -k all                # for every installed kernel
mkinitrd /boot/initrd-$(uname -r).img $(uname -r)   # the older tool

Rebuild after: adding a storage driver, changing an LVM or RAID layout, enabling disk encryption, or blacklisting a module involved in early boot.

Skip it and the symptom is stark — the kernel loads, then cannot find its root filesystem, and you land in an emergency shell. The system was fine a moment ago and the change looked unrelated, which is what makes it confusing. The recovery is to boot the previous kernel from the GRUB menu and rebuild.

Custom and embedded hardware

Embedded systems — routers, industrial controllers, single-board computers — run Linux on constrained hardware with tight power and storage budgets. Common differences: ARM rather than x86, a device tree instead of firmware enumeration, read-only root filesystems, and BusyBox providing a single binary that implements dozens of commands. Vendor kernels are frequently old and heavily patched, so drivers may not exist upstream at all.

GPUs appear in three roles now, and the driver situation differs for each. Graphics is the traditional one. Compute is the dominant one — CUDA or ROCm for machine learning and scientific work, where the GPU is a mathematics engine and never draws anything. Transcoding is the third, hardware video encode and decode for streaming.

lspci | grep -i vga        # is there a GPU, and what is it?
nvidia-smi                 # NVIDIA: utilisation, memory, temperature, processes
nvtop                      # an interactive per-process GPU monitor, like top

nvtop is the GPU equivalent of top and works across NVIDIA, AMD and Intel. On a machine doing GPU compute it answers the question top cannot: which process is using the GPU, and how much of its memory.

For NVIDIA specifically the recurring problem is driver conflict — the open-source nouveau driver claims the card, so the proprietary driver cannot. Blacklisting nouveau and rebuilding the initrd is the standard sequence, and it ties this whole lesson together: identify with lspci -k, blacklist in /etc/modprobe.d/, rebuild with dracut -f, reboot.

On the exam

• modprobe resolves dependencies and searches by name; insmod takes a file path and does not.
• lsmod's third column is the use count — a non-zero one is why rmmod refuses.
• Blacklisting in /etc/modprobe.d/ prevents automatic loading; nouveau is the standard example.
• Rebuild the initrd with dracut -f or update-initramfs -u after storage or early-boot module changes, or the system will not find root.
• lspci -k shows which driver is bound to a device. No driver line means no driver.
• dmidecode reads firmware tables — serial numbers, memory slots, BIOS version — that the kernel does not otherwise expose.
• dmesg -T gives readable timestamps and is the first place hardware problems appear.
• IPMI works when the OS is down, because the BMC is a separate processor.