Watch a graceful shutdown and an abrupt one

short · 20 min · Objective 2.3

Task

Write a small program that handles SIGTERM and ignores nothing, then kill it both ways and observe the difference in what it leaves behind. The point is to see, rather than be told, why SIGTERM comes first.

Steps

  1. Write a script that creates a lock file, traps SIGTERM to remove it, and then waits:
   cat > worker.sh <<'EOS'
   #!/usr/bin/env bash
   touch /tmp/worker.lock
   cleanup() { rm -f /tmp/worker.lock; echo "cleaned up"; exit 0; }
   trap cleanup TERM
   while true; do sleep 1; done
   EOS
   chmod +x worker.sh
  1. Start it in the background and confirm the lock file exists.
  2. Send SIGTERM with plain kill <pid>. Confirm the lock file is gone and the handler ran.
  3. Start it again, and this time send SIGKILL with kill -9 <pid>. Confirm the lock file is still there -- no handler ran, because SIGKILL cannot be caught.
  4. Suspend a foreground job with Ctrl-Z, check its state letter with ps, and resume it in the background with bg.
  5. Start something with nohup and confirm it survives the shell exiting.

Verify

./worker.sh & sleep 1; PID=$!
test -f /tmp/worker.lock && echo "lock created"
kill "$PID"; sleep 1
test -f /tmp/worker.lock || echo "TERM cleaned up"
./worker.sh & sleep 1; PID=$!
kill -9 "$PID"; sleep 1
test -f /tmp/worker.lock && echo "KILL left the lock behind"
rm -f /tmp/worker.lock

All three echoes must print. The third is the lesson in one line: the same program, killed two ways, leaves the system in two different states.

Notes

A stale lock file is the mildest version of this. The serious versions are a half-written data file, an unflushed database buffer and a connection the other end still believes is open -- all of which SIGTERM would have let the program handle.