Write a unit, then break it four ways

short · 30 min · Objective 2.5

Task

Create a working service unit, then introduce each of the four failures that account for most systemd support tickets, and learn to read the exact status code that identifies each.

Steps

  1. Write a small script at /usr/local/bin/demo.sh that logs a line every few seconds, and make it executable.
  2. Write /etc/systemd/system/demo.service with ExecStart pointing at it, Restart=on-failure, and a [Install] section. Run daemon-reload, then enable --now, and confirm it is active.
  3. Break it the first way: edit the unit to change a setting and restart the service WITHOUT daemon-reload. Confirm the change has no effect, and explain why.
  4. Break it the second way: point ExecStart at a path that does not exist. Read the status output and find status=203/EXEC.
  5. Break it the third way: set WorkingDirectory to a directory that does not exist. Find status=200/CHDIR.
  6. Break it the fourth way: systemctl disable demo and reboot the container, confirming it does not start; then enable and confirm that it does. This is the "works until reboot" scenario.
  7. Finally, mask it and observe that even an explicit start is refused.

Verify

systemctl is-active demo && echo "running"
systemctl show demo -p ExecStart --value
systemctl status demo --no-pager | grep -o 'status=[0-9]*/[A-Z]*'
systemctl is-enabled demo
systemctl mask demo && systemctl start demo 2>&1 | grep -qi masked && echo "mask blocks start"
systemctl unmask demo

Step 3 is the one to dwell on: systemctl show reports the OLD ExecStart until daemon-reload runs, which is exactly why an edit appears to have done nothing.

Notes

Learn to read the status code rather than the prose. 203/EXEC and 200/CHDIR each name their cause precisely, and both are frequently misdiagnosed as permission problems because the message that reaches the operator says only that the service failed.