Make an alert flap, then stop it flapping

short · 40 min · Objective 3.3

Task

Configure an alert with no duration and no hysteresis against a metric that hovers near its threshold, count the notifications, then fix it twice and count again. Flapping is what gets an alert muted, and muted alerts are worse than absent ones.

Steps

  1. Emit a metric that oscillates around 80 -- say between 75 and 85 -- every few seconds. Configure an alert firing above 80 with no duration requirement.
  2. Run for ten minutes and count how many times the alert went to FIRING -- from the alerting component's own history, not from notifications. Alertmanager's grouping sends at most one update per group every five minutes by default, which quietly hides the flapping you are measuring; if you count notifications, set group_wait, group_interval and repeat_interval to seconds for the lab. Record in lab/alerts.csv.
  3. Add a duration requirement so the condition must hold for several minutes. Run again and count.
  4. Add hysteresis: fire above 90, clear below 70. Prometheus has no setting for this, so write it into the rule: fire when above 90, OR when above 70 while the alert is already firing -- x > 90 or (x > 70 and on() count(ALERTS{alertname="<yours>", alertstate="firing"}) > 0). Run again and count. With this metric it may never fire at all, which is an answer too.
  5. Write in lab/alerts.md the three counts, and then the harder question: for this metric, would you alert at all? State what user-visible symptom you would alert on instead.

Verify

awk -F, 'NR>1 && NF>=2 {n++} END {print n" configuration(s) measured"}' lab/alerts.csv
grep -Eci 'hysteresis' lab/alerts.md
grep -Eci 'symptom' lab/alerts.md
grep -Ec '[0-9]+ (notification|alert)' lab/alerts.md

Three counts, none higher than the one before, and a symptom-based alternative named. The final question is the real one: the best fix for a noisy cause-alert is usually to delete it and alert on what the user feels.

This is an independent study companion for CompTIA Cloud+ CV0-004 and is not produced by or endorsed by CompTIA.