Jobs, signals and scheduling

Why this matters

The previous lesson was about looking. This one is about acting: stopping a runaway process, backgrounding a long job so it survives your SSH session, and arranging for work to happen at three in the morning without you.

The signal material in particular repays precision. kill -9 is the thing everyone reaches for and usually the wrong choice, and knowing why it is wrong is worth more than the command itself.

The lesson

Foreground, background and jobs

A shell runs one foreground job at a time — the one receiving your keystrokes. Anything else runs in the background.

./long-task.sh &     # start in the background
jobs                 # list this shell's jobs
jobs -l              # with PIDs
fg                   # bring the most recent job to the foreground
fg %2                # bring job 2 forward
bg                   # resume a stopped job in the background
bg %1

The keyboard controls:

Ctrl-Z catches people out: the process is stopped, not finished. It sits in state T consuming its memory until you resume it with fg or bg, or kill it. Logging out with stopped jobs is why "I closed my terminal and the job died" happens.

Ctrl-D is not a signal at all — it sends end-of-file. At a shell prompt that means "no more input", so the shell exits. Inside cat it ends the input.

Surviving a disconnect

Background jobs still belong to your shell. When the connection drops, the shell receives SIGHUP and passes it on, killing them.

nohup ./long-task.sh &          # ignore SIGHUP; output to nohup.out
nohup ./task.sh > out.log 2>&1 &   # with your own redirection
disown -h %1                    # detach an already-running job

nohup is the exam answer. In practice tmux or screen are better, because they let you reattach and see the output rather than just surviving.

exec is the opposite idea: it replaces the current shell with the new program rather than starting a child. The new program inherits the shell's PID.

exec ./server        # the shell is gone; ./server IS this process now
exec 3< input.txt    # also used to open file descriptors in scripts

Running exec at an interactive prompt ends your session when the program exits, because there is no shell left to return to.

Signals

A signal is a small message to a process. There are dozens; four matter.

kill 1234              # sends SIGTERM (15) — the default
kill -15 1234          # explicitly
kill -9 1234           # SIGKILL
kill -1 1234           # SIGHUP
kill -HUP 1234         # by name
kill -TERM 1234        # names work for all of them: TERM, KILL, HUP, INT
kill -l                # list every signal
killall nginx          # by process NAME
pkill -f "python app"  # by pattern, matching the full command line
pgrep -f "python app"  # find first, before you kill

SIGTERM asks; SIGKILL compels. A process receiving SIGTERM runs its cleanup handler: flush buffers, finish the current transaction, close files, remove its PID file. SIGKILL is handled by the kernel and the process never sees it — so buffers are lost, files are left open, locks are left held, and a database can be left needing recovery.

So: always SIGTERM first, wait, and only then SIGKILL. Reaching straight for kill -9 is the habit the exam is testing you out of.

Two things kill -9 cannot do: kill a process in state D (uninterruptible sleep — it is not scheduled to receive anything), and kill a zombie (already dead). If kill -9 appears not to work, one of those two is why.

SIGHUP has a second life as the "reload" signal. Send it to nginx, sshd or rsyslog and they re-read their configuration without dropping connections. That is convention rather than a rule, but it is near-universal among daemons.

pkill -f is powerful and dangerous — the pattern matches the whole command line, so a loose pattern can match far more than you meant. Run pgrep -f with the same pattern first and read the list.

Process limits

Limits stop one process exhausting the machine.

ulimit -a              # every limit for this shell
ulimit -n              # max open file descriptors
ulimit -n 4096         # raise it for this shell and its children
ulimit -u              # max user processes
ulimit -c unlimited    # allow core dumps

Soft limits (-S) can be raised by the user up to the hard limit (-H); only root raises the hard limit. Permanent settings live in /etc/security/limits.conf or /etc/security/limits.d/, and for services in the unit file's LimitNOFILE=.

"Too many open files" is the classic symptom, and it is nearly always the file-descriptor limit rather than a disk problem. Busy web servers and databases routinely need this raised.

Scheduling: cron

cron runs commands on a repeating schedule.

crontab -e      # edit YOUR crontab
crontab -l      # list it
crontab -r      # remove it entirely — note there is no confirmation
sudo crontab -e -u alice   # edit another user's

Five time fields then the command:

* * * * *  command
│ │ │ │ └── day of week (0-7, 0 and 7 both Sunday)
│ │ │ └──── month (1-12)
│ │ └────── day of month (1-31)
│ └──────── hour (0-23)
└────────── minute (0-59)

0 3 * * *        /usr/local/bin/backup.sh    # 03:00 daily
*/15 * * * *     /usr/local/bin/check.sh     # every 15 minutes
0 9 * * 1-5      /usr/local/bin/report.sh    # 09:00, weekdays
0 0 1 * *        /usr/local/bin/monthly.sh   # midnight on the 1st
@reboot          /usr/local/bin/startup.sh   # once at boot

System-wide crontabs live in /etc/crontab and /etc/cron.d/, and take an extra field — the user to run as — between the schedule and the command. The drop-in directories /etc/cron.daily/, .hourly/, .weekly/ and .monthly/ take plain scripts with no schedule at all.

Cron's environment is minimal. PATH is typically just /usr/bin:/bin, HOME may be unset, and no profile is read. A script that works for you and silently fails from cron is nearly always calling something not on that short PATH. Use absolute paths, and capture output so failures are visible:

0 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Without redirection cron mails the output to the user, which on a server usually means it disappears.

anacron, at, and systemd timers

anacron exists for machines that are not on all the time. Cron simply skips a job whose moment passed while the machine was off; anacron notices on next boot that the job is overdue and runs it. It works in days rather than minutes, which is why it handles cron.daily and friends on laptops.

at schedules a one-off:

at 03:00                     # then type commands, Ctrl-D to finish
echo "/usr/local/bin/x.sh" | at 22:00
at now + 2 hours
atq                          # list pending jobs
atrm 3                       # remove job 3

Use cron for recurring, at for once, anacron for machines that sleep.

systemd timers are the modern alternative to cron: a .timer unit triggering a .service unit. They log to the journal, can depend on other units, support randomised delays, and — via Persistent=true — catch up on missed runs like anacron. Cron remains on the exam and everywhere in the wild, but new work increasingly uses timers.

systemctl list-timers        # what is scheduled, and when it next fires

Changing priority on the fly

renice -n 10 -p 1234      # make PID 1234 nicer
renice -n 5 -u alice      # every process owned by alice

renice adjusts a running process, where nice sets priority at launch. A normal user may only increase niceness, and may not undo it — lowering it again requires root. That is the same yield-but-never-seize asymmetry.

On the exam

• SIGTERM (15) is the default and is catchable; SIGKILL (9) cannot be caught, blocked or ignored, and skips all cleanup. Send 15 first.
• SIGHUP (1) conventionally means reload configuration.
• kill -9 cannot kill a D-state process or a zombie.
• Ctrl-Z suspends (state T); Ctrl-C interrupts; Ctrl-D is end-of-file, not a signal.
• nohup lets a job survive logout.
• Cron field order is minute, hour, day-of-month, month, day-of-week. */15 means every 15.
• /etc/crontab and /etc/cron.d entries carry an extra user field.
• Cron's minimal PATH is the standard cause of "works for me, fails in cron".
• anacron catches up missed jobs on machines that are powered off; at is one-off.
• A user can raise their nice value but not lower it again.