Build an event-driven pipeline end to end and make it observable

capstone · 140 min · Objective 5.6

Task

Pull the DevOps domain together: a change in source control flows through a pipeline to a deployed, event-driven application, and every failure mode from this domain is handled and demonstrated.

Steps

  1. Build the application: a producer publishing events, a topic, two subscribers with their own queues, idempotent consumers, dead-letter queues and correlation identifiers propagated throughout.
  2. Build the pipeline: on commit, build once, scan, test, publish by digest, deploy to staging, run smoke tests, then require an approval before deploying the SAME artefact to production. Record the digest at each stage in lab/cap5/promotion.md and confirm it never changes.
  3. Secure it: the pipeline exchanges a short-lived credential scoped to the target environment, the pipeline definition requires review, and third-party steps are pinned. Record each control in lab/cap5/controls.csv with the test that proves it.
  4. Make it observable: metrics for queue depth and consumer lag, centralised logs with the correlation identifier, and a trace showing one event's full path. Record in lab/cap5/observability.md the one question each signal answered that the others could not.
  5. Break it four ways and record the response each time in lab/cap5/failures.md: a poison event, a duplicate, a consumer outage, and a bad deployment. The first three must be handled automatically; the fourth must be rolled back by redeploying the previous digest, and you must record how long that took.

Verify

for f in promotion controls observability failures; do test -s lab/cap5/$f.* && echo "$f ok"; done
grep -Ec 'sha256:' lab/cap5/promotion.md
awk -F, 'NR>1 && NF>=2 {n++} END {print n" control(s) with a proof"}' lab/cap5/controls.csv
grep -Eci 'poison' lab/cap5/failures.md
grep -Ec '[0-9]+ ?(s|sec|second|min)' lab/cap5/failures.md

All four artefacts, an unchanged digest across every stage, at least four proven controls, and a measured rollback duration. If the digest changed between staging and production, the pipeline rebuilt instead of promoting, and the staging tests proved nothing about what production runs.

Notes

The digest check running through every stage is the assertion that ties this capstone together. It is the one property that makes a test result in staging mean something about production.

This is an independent study companion for CompTIA Cloud+ CV0-004 and is not produced by or endorsed by CompTIA.