Instrument a service against an SLO

applied · 45 min · Objective 5.1

Before you start — what this lab needs

A disposable machine running a small web service and a way to generate load and faults:

podman run -d --name svc --rm -p 8080:80 nginx:1.25
dnf install -y httpd-tools   # for ab, to generate requests

Task

Define an SLI, an SLO and an error budget for a small service, measure the indicator, and build alerting that pages on the objective being at risk rather than on raw resource metrics. Then prove the alert routing and inhibition behave under a simulated incident.

Steps

Define the SLI precisely: the proportion of requests returning a non-5xx status over a window. Write down the SLO (say 99.5%) and compute the error budget it allows over that window.
Measure the SLI: drive requests with ab and compute the success rate from the access log. Confirm it meets the SLO under normal conditions.
Build an alert on the SLI, not on CPU: it fires when the measured success rate over the window falls below the objective. State why CPU utilisation would be the wrong thing to page on.
Inject faults so a fraction of requests return 500, enough to breach the SLO, and confirm the alert fires.
Measure latency percentiles, not the mean: show a case where the mean looks fine and the 99th percentile is bad, and page on the percentile.
Simulate a dependency failure that makes every instance fail at once, and confirm your alerting groups it into one notification rather than one per instance -- inhibition.
Confirm an alert that is not actionable is demoted to a dashboard, and articulate the test you used to decide.

Verify

# SLI under normal load meets the SLO:
ab -n 500 -c 10 http://localhost:8080/ >/dev/null 2>&1
awk '{print $NF}' access.log | awk '{t++; if($1>=500)e++} END{printf "success %.2f%%\n", 100*(t-e)/t}'
# after injecting 5xx, the SLI-based alert fires:
./slo-check.sh | grep -qi 'budget\|breach' && echo "SLO alert fired"

The alert firing on the SLI rather than on a resource metric is the discipline: you page on what the user experiences -- success rate and tail latency -- not on CPU, which can be high on a healthy system and low on a broken one.

Notes

The error budget reframes the SLO from a target to minimise to a quantity to spend. Unspent budget means you are shipping too cautiously; exhausted budget means stop deploying and stabilise. Alerting on the budget burn rate is more useful than alerting on any single request.