Essential shell utilities

Why this matters

Linux has no single "do the thing" program. It has a few dozen small tools that each do one job, and a pipe operator that lets you bolt them together. Learn fifteen of them properly and you can answer almost any question about a system without installing anything.

This is also where the exam spends a lot of its command-recall budget, and where performance-based questions live: you will be given a file and asked to extract something from it.

The lesson

Reading files

cat /etc/hostname          # whole file to stdout
cat -n script.sh           # with line numbers
head -n 20 /var/log/syslog # first 20 lines (default 10)
tail -n 50 /var/log/syslog # last 50 lines
tail -f /var/log/syslog    # follow: print new lines as they arrive
less /var/log/syslog       # page through it
more /var/log/syslog       # the older pager

tail -f is the one you will use most. Watching a log while you reproduce a problem is the fastest diagnostic loop there is. Ctrl-C stops it.

less versus more: more only moves forward; less scrolls both ways, searches with /, and does not load the whole file into memory — which is why it opens a 4 GB log instantly. The old joke is that "less is more than more", and it is genuinely the right default.

Finding things inside files

grep prints lines matching a pattern, and is the workhorse.

grep root /etc/passwd            # lines containing "root"
grep -i error app.log            # case-insensitive
grep -v "^#" /etc/ssh/sshd_config  # INVERT: hide comment lines
grep -r "TODO" /home/dev/        # recurse through a directory
grep -c "404" access.log         # count matching lines
grep -n "fail" auth.log          # show line numbers
grep -A3 -B3 "panic" kern.log    # 3 lines of context either side

grep -v "^#" deserves special mention: stripping comments is how you find out what a configuration file actually sets, as opposed to what its 200 lines of commented examples suggest. Combine it with removing blank lines:

grep -v "^#" /etc/ssh/sshd_config | grep -v "^$"

Cutting and reshaping

cut extracts columns.

cut -d: -f1 /etc/passwd          # every username
cut -d: -f1,7 /etc/passwd        # username and shell
cut -c1-10 file.txt              # first 10 characters of each line

-d sets the delimiter, -f picks fields. It only handles a single-character delimiter and does not cope with runs of spaces, which is exactly when you reach for awk instead.

awk is a small programming language for column-oriented text. Two forms cover most real use:

awk '{print $1, $3}' access.log      # first and third whitespace-separated fields
awk -F: '$3 >= 1000 {print $1}' /etc/passwd   # usernames with UID >= 1000
awk '{sum += $5} END {print sum}' sizes.txt   # add up a column

$0 is the whole line, $1 the first field, NF the number of fields. Unlike cut, awk treats any run of whitespace as one separator, which is why it works on ps and df output where cut does not.

sed is a stream editor — it transforms text as it flows past.

sed 's/old/new/' file            # replace the FIRST match on each line
sed 's/old/new/g' file           # replace all matches
sed -i 's/old/new/g' file        # edit the file in place
sed -i.bak 's/old/new/g' file    # in place, keeping file.bak
sed -n '10,20p' file             # print only lines 10-20
sed '/^#/d' file                 # delete comment lines

sed -i without a backup suffix is irreversible. On a config file you care about, sed -i.bak costs nothing and has saved a great many afternoons.

Sorting, counting and de-duplicating

sort names.txt                   # alphabetical
sort -n sizes.txt                # numeric — 9 before 10, which sort alone gets wrong
sort -r names.txt                # reverse
sort -k3 -n data.txt             # numeric sort on the third field
uniq -c sorted.txt               # count consecutive duplicates
wc -l access.log                 # count lines
wc -w essay.txt                  # words

uniq only removes adjacent duplicates, which is why it is essentially always preceded by sort. The canonical idiom — worth knowing by heart — answers "what are the most common values in this column?":

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Extract the field, sort so duplicates group, count them, sort by count descending, take the top few. That one line answers "which IP is hammering us?"

Transforming and dispatching

tr 'a-z' 'A-Z' < file            # translate characters: lowercase to uppercase
tr -d '\r' < dos.txt > unix.txt  # delete characters — strips Windows line endings
tee output.txt                   # write to a file AND pass through
xargs                            # turn stdin into command arguments

tee solves the problem of wanting to both see and save output, and the sudo version is the standard way to write to a protected file from a pipeline:

echo "127.0.0.1 test" | sudo tee -a /etc/hosts

That works where sudo echo ... >> /etc/hosts fails, because the redirection is performed by your shell, not by sudo.

xargs builds command lines from input:

find . -name "*.log" | xargs rm          # delete every .log found
find . -name "*.log" -print0 | xargs -0 rm   # safe with spaces in names

Use -print0/-0 whenever filenames might contain spaces, which on a real system they eventually will.

Output, arithmetic, and the system's name

echo "Hello $USER"               # print, with variable expansion
echo -n "no trailing newline"
printf "%-10s %5.2f\n" "cpu" 93.5   # formatted output, like C
echo "5 * 3" | bc                # arbitrary-precision calculator
echo "scale=4; 10/3" | bc        # 3.3333
uname -a                         # kernel, hostname, architecture, all of it
uname -r                         # kernel release only

printf is preferred over echo in scripts because its behaviour is consistent across shells; echo -e and friends vary. bc matters because the shell only does integer arithmetic — $((10/3)) gives 3.

Aliases, history and sourcing

alias ll='ls -alF'               # define for this shell
alias                            # list all aliases
unalias ll
history                          # numbered list of past commands
history | grep ssh               # find that command you ran last week
source ~/.bashrc                 # run a file in the CURRENT shell

Aliases defined at the prompt vanish when the shell exits — put them in ~/.bashrc to keep them. And an alias is not available to scripts, which is a common surprise; scripts want functions or full paths.

Text editors

You will need one on a machine with nothing installed.

nano is the friendly option. Commands are listed along the bottom, ^ means Ctrl. Ctrl-O writes, Ctrl-X exits.

vi/vim is the one guaranteed to exist everywhere, and has modes, which is what makes it confusing at first:

• Normal mode (the default) — keys are commands, not text
• Insert mode — i to enter, typing inserts text
• Command mode — : to enter, for saving and quitting

The minimum viable set: i to start typing, Esc to stop, :w write, :q quit, :wq write and quit, :q! quit discarding changes. Learn those six and you can always repair a config file, which is the real reason this is on the exam.

On the exam

• sort | uniq -c | sort -rn is the counting idiom. Recognise it and know why the first sort is required.
• grep -v inverts the match. Expect it in a "show me the non-comment lines" scenario.
• cut -d handles one delimiter character; use awk for whitespace-separated columns.
• sed 's/x/y/' changes the first match per line; /g changes all.
• tee writes and passes through — the sudo tee idiom for protected files comes up.
• tail -f follows a growing file.
• Know how to exit vi. :wq and :q!. It is asked, and it is genuinely the most useful six keystrokes on this list.