RAID and mounted storage

Why this matters

Two things finish the storage stack: redundancy, so a dead disk is an inconvenience rather than an outage, and mounting, which is how any of that storage becomes a directory you can actually use.

Mounting is also where a specific and unforgiving mistake lives. A typo in /etc/fstab does not fail politely at the next reboot — it can stop the machine booting entirely and drop you into an emergency shell. One mount option, nofail, is the difference between a degraded server and an unreachable one.

The lesson

RAID levels

Redundant Array of Independent Disks combines several disks into one logical device. The level decides the trade between capacity, speed and survivability.

Three things worth understanding rather than memorising.

RAID 0 has no redundancy at all despite the name of the family. It is striping for speed, and it increases your risk — two disks means twice the chance of a failure that loses everything.

RAID 5's rebuild is its weak point. When a disk fails, rebuilding reads every block of every remaining disk. On large modern drives that takes many hours under full load, which is exactly when a second, already-aged disk is most likely to fail — and a second failure during a RAID 5 rebuild loses the array. This is why RAID 6 or RAID 10 are preferred for large disks.

RAID is not a backup. It protects against a disk failing. It does not protect against rm -rf, corruption, or ransomware, all of which are written faithfully and instantly to every disk in the array. A backup is a separate copy that a mistake cannot reach.

Software RAID with mdadm

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
mdadm --detail /dev/md0            # full status
mdadm --detail --scan >> /etc/mdadm.conf   # persist the array definition
cat /proc/mdstat                   # quick status, including rebuild progress

mdadm --manage /dev/md0 --fail /dev/sdb     # mark a disk failed
mdadm --manage /dev/md0 --remove /dev/sdb   # remove it
mdadm --manage /dev/md0 --add /dev/sdd      # add a replacement; rebuild starts
mdadm --stop /dev/md0

/proc/mdstat is the file to look at first. It shows every array, its members, and — during a rebuild — a progress bar and estimated finish time. The status string is the key part:

md0 : active raid1 sdc[1] sdb[0]
      976630464 blocks super 1.2 [2/2] [UU]

[2/2] [UU] means two of two devices present and both up. [2/1] [U_] means one is missing — the array is degraded and running with no redundancy left. That underscore is the thing to alert on.

Replacing a failed disk is fail → remove → add, and the rebuild begins automatically. Remember to persist the configuration with mdadm --detail --scan into /etc/mdadm.conf, or the array may not assemble at boot.

Mounting

mount /dev/sdb1 /mnt/data           # mount by device
mount UUID=1234-5678 /mnt/data      # by UUID — preferred
mount -t xfs /dev/sdb1 /mnt/data    # force the type
mount -o ro /dev/sdb1 /mnt/data     # with options
mount -a                            # mount everything in fstab not yet mounted
umount /mnt/data                    # by mount point
umount /dev/sdb1                    # or by device
mount                               # what is mounted right now
findmnt                             # the same, as a readable tree

Mounting onto a directory that already contains files does not delete them — they become invisible until you unmount, and the space they use still counts in df. That is one of the two reasons df and du disagree.

umount fails with "target is busy" when a process has a file open there, or someone's shell is sitting in the directory:

lsof +D /mnt/data          # what is using it
fuser -vm /mnt/data        # the same question, different tool
umount -l /mnt/data        # LAZY: detach now, clean up when free

umount -l is the pragmatic escape, but understand what it does: the mount disappears from the tree immediately while the filesystem stays alive until the last user releases it. It is not a forced unmount, and data can still be in flight.

/etc/fstab

/etc/fstab defines what mounts at boot. Six fields:

UUID=1234-5678  /srv/data  xfs  defaults,nofail  0  2
     1              2       3         4          5  6

1. Device — a UUID, a LABEL=, or a device path
2. Mount point
3. Filesystem type
4. Options, comma-separated
5. dump — legacy backup flag, effectively always 0
6. fsck order — 0 never, 1 the root filesystem, 2 everything else

Use UUIDs. /dev/sdb is assigned in detection order; add a disk and yesterday's sdb can become sdc, and your fstab now mounts the wrong device — or refuses to boot.

Always test before rebooting:

mount -a                # try every fstab entry
findmnt --verify        # check the file for errors without mounting

A bad fstab entry can stop the machine booting. systemd waits for the mount, fails, and drops to an emergency shell requiring the root password — over a console you may not have on a remote server. mount -a costs two seconds and catches it while you can still fix it.

Mount options

defaults    = rw,suid,dev,exec,auto,nouser,async

noexec,nosuid,nodev is the standard hardening trio for /tmp, /var/tmp, /dev/shm and any removable media. Together they mean a file dropped there cannot be run, cannot escalate through setuid, and cannot impersonate a device.

noatime is the easy performance win. By default reading a file writes its access time, so a read-heavy workload generates constant metadata writes. Turning it off removes them.

nofail is the one that saves you. An external or network device that is absent at boot will otherwise stall startup. With nofail the system notes it and carries on.

remount changes options in place — the standard move when the root filesystem has gone read-only after an error:

mount -o remount,rw /
mount -o remount,ro /srv/data

/etc/mtab and /proc/mounts

Three files describe mounting, and the distinction matters:

• /etc/fstab — what should be mounted at boot. Configuration you write.
• /proc/mounts — what is mounted, straight from the kernel. Always true.
• /etc/mtab — historically a userspace record of the same; now a symlink to /proc/self/mounts on current systems.

When they disagree, /proc/mounts is right — it is the kernel's own view rather than anything a tool wrote down. mount with no arguments reads it.

autofs

Mounting network shares permanently in fstab means every boot waits for them, and a server that is down delays or blocks startup. autofs mounts on demand instead: the filesystem is attached when someone accesses the path and unmounted again after a period of inactivity.

# /etc/auto.master
/mnt/nfs   /etc/auto.nfs   --timeout=60

# /etc/auto.nfs
data   -rw,soft   fileserver:/export/data

ls /mnt/nfs/data triggers the mount; sixty idle seconds later it goes away. For laptops and for any host with many rarely-used shares this is the right answer, and it removes the boot-time dependency entirely.

Network mounts

NFS — the Network File System, the Unix-native option. Preserves Unix ownership and permissions, which is why it suits Linux-to-Linux.

mount -t nfs fileserver:/export/data /mnt/data
# fstab
fileserver:/export/data  /mnt/data  nfs  defaults,_netdev,nofail  0 0

SMB — Server Message Block, Windows-native, spoken on Linux by Samba. The right choice for shares that Windows clients also use.

mount -t cifs //server/share /mnt/share -o credentials=/root/.smbcred,uid=1000

Put the username and password in a root-owned file with mode 600 rather than on the command line, where they land in shell history and in ps output for every user to read.

Two options matter for both: _netdev tells systemd this mount needs the network, so it waits rather than failing early; nofail stops an unreachable server making the machine unbootable. Network mounts want both.

For NFS specifically, soft returns an error when the server does not respond while hard retries forever. hard is the default and is safer for data integrity — but a hung hard mount produces exactly the uninterruptible D-state processes from the process lesson, unkillable until the server returns.

Inodes

Every file has an inode, a structure holding its metadata: permissions, owner, size, timestamps, link count, and pointers to the data blocks. It holds everything about a file except its name — names live in directories, which map names to inode numbers.

That design explains several things at once:

• Hard links are two names for one inode, which is why neither is "the original".
• Renaming is instant regardless of file size: only the directory entry changes.
• The inode count is fixed at format time. Run out and you cannot create files even with free space — df -i is the check, and millions of small files is the cause.

ls -i file          # the inode number
df -i               # inode usage per filesystem
stat file           # everything in the inode

On the exam

• RAID 0 has no redundancy. RAID 1 mirrors, 5 survives one disk, 6 survives two, 10 mirrors then stripes.
• RAID is not a backup — it does not protect against deletion or corruption.
• /proc/mdstat shows array status; [U_] means degraded.
• nofail in fstab prevents a missing device stopping the boot; _netdev makes systemd wait for the network.
• noexec,nosuid,nodev is the hardening set for /tmp and removable media.
• noatime removes a write on every read.
• Test fstab with mount -a before rebooting.
• /proc/mounts is the kernel's truth; /etc/mtab is a symlink to it.
• Use UUIDs, because device names are assigned in detection order.
• Out of inodes means "no space" with free blocks. df -i.