Make an alert flap, then stop it flapping

short · 40 min · Objective 3.3

Task

Configure an alert with no duration and no hysteresis against a metric that hovers near its threshold, count the notifications, then fix it twice and count again. Flapping is what gets an alert muted, and muted alerts are worse than absent ones.

Steps

  1. Emit a metric that oscillates around 80 -- say between 75 and 85 -- every few seconds. Configure an alert firing above 80 with no duration requirement.
  2. Run for ten minutes and count the notifications produced. Record in lab/alerts.csv.
  3. Add a duration requirement so the condition must hold for several minutes. Run again and count.
  4. Add hysteresis: fire above 90, clear below 70. Run again and count.
  5. Write in lab/alerts.md the three counts, and then the harder question: for this metric, would you alert at all? State what user-visible symptom you would alert on instead.

Verify

awk -F, 'NR>1 && NF>=2 {n++} END {print n" configuration(s) measured"}' lab/alerts.csv
grep -Eci 'hysteresis' lab/alerts.md
grep -Eci 'symptom' lab/alerts.md
grep -Ec '[0-9]+ (notification|alert)' lab/alerts.md

Three counts, each lower than the last, and a symptom-based alternative named. The final question is the real one: the best fix for a noisy cause-alert is usually to delete it and alert on what the user feels.

This is an independent study companion for CompTIA Cloud+ CV0-004 and is not produced by or endorsed by CompTIA.