systemd and services

Why this matters

systemd is PID 1 on every current distribution. It starts everything, keeps it running, restarts it when it dies, and holds the logs. Practically every "the service isn't working" ticket is answered with systemctl status followed by journalctl -u.

It is also the layer where a specific and much-tested distinction lives: disabling a service and masking a service are not the same, and the difference is what stops something being switched back on behind your back.

The lesson

Units

systemd manages units — named objects it knows how to start, stop and depend on. The suffix gives the type.

Services are the everyday case. Targets replace the old runlevels: multi-user.target is a normal server (formerly runlevel 3), graphical.target adds a desktop (runlevel 5), rescue.target is single-user, and emergency.target is barely anything at all.

systemctl get-default              # which target this system boots to
systemctl set-default multi-user.target
systemctl isolate rescue.target    # switch now

Mounts are managed as units too. Mount units let systemd handle filesystems with dependencies — a service can require that a mount is present before it starts, which /etc/fstab alone cannot express. systemd generates a mount unit for every fstab entry automatically, which is why systemctl status var-log.mount works on a system where nobody wrote such a file.

Timers replace cron for scheduled work, with the advantages that they log to the journal, can depend on other units, and can catch up missed runs.

systemctl list-units --type=service      # loaded services
systemctl list-units --all               # including inactive
systemctl list-unit-files                # every unit and its enablement state
systemctl list-timers                    # what fires next
systemctl list-dependencies nginx        # what it needs

Unit files live in three places, and precedence matters:

• /usr/lib/systemd/system/ — shipped by packages. Do not edit.
• /etc/systemd/system/ — yours. Overrides the above.
• /run/systemd/system/ — runtime, volatile.

Managing a service

systemctl start nginx        # start now
systemctl stop nginx         # stop now
systemctl restart nginx      # stop then start
systemctl reload nginx       # re-read config WITHOUT dropping connections
systemctl reload-or-restart nginx
systemctl status nginx       # state, PID, recent log lines
systemctl is-active nginx
systemctl is-enabled nginx

reload versus restart is a real operational distinction. Reload asks the running daemon to re-read its configuration — nginx and sshd do this without dropping a single connection. Restart kills it and starts it again, breaking every active session. On a production web server that difference is visible to users. Not every service supports reload; reload-or-restart falls back.

start is not enable. Starting affects now; enabling affects boot. They are independent, and forgetting the second is the classic mistake:

systemctl enable nginx       # start at boot — does NOT start it now
systemctl disable nginx      # do not start at boot
systemctl enable --now nginx # both, in one command
systemctl disable --now nginx

A service that "stops working after a reboot" was almost always started but never enabled.

Masking

systemctl mask nginx         # make it impossible to start
systemctl unmask nginx

Disable removes the symlink that starts a unit at boot. Something else can still start it — another unit depending on it, a socket activation, a package update re-enabling it, or an administrator typing systemctl start.

Mask links the unit to /dev/null. It cannot be started at all, by anything, including as a dependency of something else. Attempts fail with "Unit is masked".

Mask when you need a guarantee: two conflicting services where one must never run, or a unit a package keeps re-enabling. Just remember to unmask before wondering why it will not start — a masked unit is a genuinely confusing symptom if you have forgotten you masked it.

Editing units, and daemon-reload

systemctl edit nginx           # a drop-in override in /etc/systemd/system/nginx.service.d/
systemctl edit --full nginx    # copy the whole unit to /etc and edit that
systemctl cat nginx            # show the effective unit, including drop-ins
systemctl revert nginx         # discard your changes

systemctl edit is the correct way to change a unit. It creates a small override file rather than modifying the package's copy, so a package update does not silently discard your work — and systemctl cat shows you the merged result.

daemon-reload is the step people forget. systemd caches unit files in memory. Change one on disk and systemd does not notice until told:

systemctl daemon-reload      # re-read unit files from disk

Edit a unit, systemctl restart, and see no change — that is why. systemctl edit runs it for you; editing files by hand does not.

Note the pair that sound alike and are not: systemctl reload nginx tells nginx to re-read its config; systemctl daemon-reload tells systemd to re-read unit files.

Reading the logs

journalctl -u nginx              # everything for this unit
journalctl -u nginx -f           # follow live
journalctl -u nginx --since "1 hour ago"
journalctl -p err -b             # errors only, this boot
journalctl -b -1                 # the PREVIOUS boot — for crash investigation
journalctl -xe                   # recent entries with explanations

journalctl -b -1 matters: after an unexpected reboot, the interesting logs are from the boot that ended, not the one you are in.

By default the journal may be volatile and lost on reboot. Persist it with Storage=persistent in /etc/systemd/journald.conf and a /var/log/journal directory — worth doing on any server you might have to investigate.

The system-configuration tools

systemd absorbed a set of *ctl utilities, each owning one area.

hostnamectl                       # hostname, OS, kernel, architecture
hostnamectl set-hostname web01
hostnamectl status

timedatectl                       # time, zone, NTP synchronisation state
timedatectl set-timezone Europe/London
timedatectl set-ntp true
timedatectl list-timezones

resolvectl status                 # DNS servers actually in use
resolvectl query example.com
resolvectl flush-caches

resolvectl matters because /etc/resolv.conf often lies. Where systemd-resolved is running, that file is a symlink to a stub pointing at 127.0.0.53, and the real upstream servers are held by the resolver. Reading /etc/resolv.conf tells you nothing useful; resolvectl status tells you which servers are genuinely being queried, per interface.

hostnamectl writes /etc/hostname and applies the change immediately, which the old approach of editing the file plus running hostname did not do reliably.

Tuning the kernel with sysctl

sysctl -a                                 # every kernel parameter
sysctl net.ipv4.ip_forward                # read one
sysctl -w net.ipv4.ip_forward=1           # set it, until reboot
sysctl -p                                 # reload from configuration

For a permanent change, write a file under /etc/sysctl.d/:

echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/99-forward.conf
sysctl --system                           # apply everything

These are the same values exposed as files under /proc/sys/, so sysctl net.ipv4.ip_forward and cat /proc/sys/net/ipv4/ip_forward are two views of one setting. sysctl -w is temporary; the file makes it survive a reboot. Enabling IP forwarding is the standard example — a router or a container host needs it, and it is lost at reboot if you only used -w.

Why is boot slow?

systemd-analyze                     # total firmware, loader, kernel, userspace time
systemd-analyze blame               # every unit, slowest first
                                    # (the objectives call this systemd-blame;
                                    #  there is no separate binary)
systemd-analyze critical-chain      # the chain that actually determined boot time
systemd-analyze plot > boot.svg     # a timeline

blame and critical-chain answer different questions, and confusing them wastes effort. blame ranks units by how long each took, but a slow unit that ran in parallel with others delayed nothing. critical-chain shows the dependency path that actually gated the boot. Optimise what critical-chain shows; blame is a list of suspects, not a verdict.

A frequent real answer is a network unit waiting for an interface that will never come up — NetworkManager-wait-online.service timing out after 30 or 90 seconds.

On the exam

• start affects now, enable affects boot. enable --now does both, and "works until reboot" means enable was missed.
• disable removes the boot symlink; mask links the unit to /dev/null so nothing can start it. Expect a scenario where a disabled service keeps coming back.
• daemon-reload after editing a unit file; systemctl reload is for the service's own configuration.
• Use systemctl edit for overrides rather than editing files in /usr/lib/systemd/system, which package updates overwrite.
• journalctl -u <unit> for one service; -b -1 for the previous boot.
• Targets replaced runlevels: multi-user.target ≈ 3, graphical.target ≈ 5.
• sysctl -w is temporary; /etc/sysctl.d/ makes it permanent.
• systemd-analyze critical-chain identifies what actually delayed boot; blame alone can mislead.