Files, directories and links

Why this matters

In Linux almost everything is a file: your documents, your configuration, your running processes, and your hard disk. Getting fluent with moving around the tree and manipulating what you find there is the foundation everything else sits on.

Two topics here are more subtle than they look and both are heavily examined: the difference between a hard link and a symbolic link, and the difference between a block and a character device. Neither is obvious, and both explain behaviour that otherwise seems arbitrary.

The lesson

Moving around and looking

pwd                 # print working directory — where am I?
cd /var/log         # absolute
cd ../..            # relative: up two
cd                  # with no argument: home
cd -                # back to the previous directory
ls                  # names
ls -l               # long: permissions, owner, size, date
ls -a               # include hidden files (those starting with .)
ls -lh              # human-readable sizes: 4.0K, 1.2G
ls -lt              # newest first
ls -lS              # largest first

ls -lah is the combination most administrators type without thinking.

cd - is genuinely useful and rarely taught: it toggles between the last two directories, which saves a lot of typing when you are comparing two places.

Creating, copying, moving, removing

touch newfile          # create empty, or update timestamp if it exists
mkdir project          # make a directory
mkdir -p a/b/c         # make parents as needed — no error if they exist
cp source dest         # copy a file
cp -r srcdir destdir   # copy a directory tree
cp -p file backup      # preserve mode, ownership and timestamps
cp -i file dest        # prompt before overwriting
mv old new             # move OR rename — the same operation
rm file                # delete
rm -r directory        # delete a tree
rm -f file             # force, no prompting
rmdir emptydir         # remove a directory only if it is empty

Three things worth fixing in your head now.

mv is both move and rename, because renaming is just moving within the same directory. There is no separate rename command.

rmdir fails unless the directory is empty, which is a feature — it is the safe way to remove something you believe should already be empty. When it refuses, that is information.

rm -rf has no undo and no recycle bin. There is no recovery short of backups. Habitually ls the path first, and be extremely careful with variables: rm -rf $DIR/ deletes the entire filesystem if $DIR is unset.

touch on an existing file updates its timestamps without changing content. That matters more than it sounds: build systems and find -mtime both key off timestamps.

Finding files

Two tools, and the difference is worth understanding because it explains why one is instant and the other is not.

find /var/log -name "*.log"            # by name, live, walks the tree
find /home -type d -name "cache"       # directories only
find / -size +100M                     # larger than 100 MB
find /tmp -mtime +7 -delete            # modified more than 7 days ago, delete
find . -type f -exec chmod 644 {} \;   # run a command on each result

locate passwd                          # by name, instantly, from an index
sudo updatedb                          # refresh that index

find searches the filesystem now; it is exact, current, and slow on a big tree. locate searches a database built periodically by updatedb; it is effectively instant but will miss files created since the last index run — and will happily list files you have already deleted. When locate gives a stale answer, that is why.

Inspecting a file

file report.pdf     # what IS this? Reads content, not the extension
stat report.pdf     # size, permissions, owner, and all three timestamps
lsof /var/log/syslog   # which processes have this file open?
lsof -u alice          # every file open by a user
lsof -i :80            # what is listening on port 80

file matters because Linux does not care about extensions. A file called script.txt can be an ELF binary; file tells you what it really is.

stat shows the three timestamps that confuse people: atime (last accessed), mtime (contents last modified) and ctime (inode last changed — permissions, ownership, or link count). ctime is not creation time, which trips up almost everyone the first time.

lsof answers "why can't I unmount this?" — something has a file open on it. It is also how you find what is holding a deleted file's disk space: a process with the file still open keeps the blocks allocated until it closes, which is why deleting a huge log sometimes frees nothing until you restart the service writing to it.

Comparing files

diff old.conf new.conf       # line differences
diff -u old.conf new.conf    # unified format, as used by patches
sdiff old.conf new.conf      # side by side, in two columns

diff output reads as instructions to turn the first file into the second. sdiff is easier on the eye when you want to compare rather than patch, since it puts them next to each other with |, < and > markers between.

Links

This is the section to slow down for.

ln target hardlink        # hard link
ln -s target symlink      # symbolic (soft) link

A hard link is a second name for the same data. Both names point at the same inode; neither is "the original". The data is freed only when the last name is removed. Because it references an inode, a hard link cannot cross filesystems and cannot point at a directory.

A symbolic link is a small file containing a path. It is a signpost. It can cross filesystems, can point at a directory, and breaks if the target is moved or deleted — leaving a dangling link that ls -l shows in red.

ls -l
lrwxrwxrwx 1 root root   11 Sep  9 10:02 latest -> app-2.1.log
-rw-r--r-- 2 root root 4096 Sep  9 10:01 data.db

The l at the start marks a symlink and -> shows its target. The 2 in the second line is the link count — this file has two names somewhere on the filesystem.

The practical rule: symlinks for almost everything — they are visible, obvious, and cross filesystems. Hard links for the specific case where you need the data to survive deletion of one name, and where everything is on one filesystem.

Device files in /dev

/dev holds files that are not files. They are the kernel's interface to hardware, and their type determines how you talk to them.

Block devices transfer data in fixed-size blocks and support random access — you can seek to any position. Disks and partitions: /dev/sda, /dev/nvme0n1, /dev/sda1. Marked b in ls -l.

Character devices transfer data a character at a time, as a stream, with no seeking. Terminals, serial ports, and the pseudo-devices: /dev/tty, /dev/null, /dev/zero, /dev/random. Marked c.

ls -l /dev/sda /dev/null
brw-rw---- 1 root disk 8, 0 Sep  9 09:00 /dev/sda
crw-rw-rw- 1 root root 1, 3 Sep  9 09:00 /dev/null

Note there is no size — instead there are two numbers, the major (which driver) and minor (which device that driver handles).

Special character devices are the pseudo-devices that have no hardware at all:

• /dev/null — discards everything written to it, returns EOF when read. The bit bucket, and the target of 2>/dev/null.
• /dev/zero — an endless stream of null bytes. Used to create files of a given size.
• /dev/random and /dev/urandom — cryptographic random data.

dd if=/dev/zero of=testfile bs=1M count=100   # a 100 MB file of zeroes

The block/character distinction is what decides whether a device can hold a filesystem: you can format a block device, and you cannot format a terminal.

On the exam

• Hard link versus symbolic link is close to guaranteed. Hard: same inode, one filesystem only, no directories, survives deletion of the other name. Soft: a path, crosses filesystems, works on directories, breaks if the target goes.
• Block versus character device: block is random-access and fixed-size (disks), character is a stream (terminals, /dev/null).
• ctime is inode-change time, not creation time.
• find is live and exact; locate is indexed and possibly stale. Know that updatedb refreshes it.
• mkdir -p creates parents and does not error on an existing directory.
• rmdir only removes empty directories.
• lsof identifies what holds a file or a mount point open.