Make retries duplicate the work, then make them safe

short · 45 min · Objective 5.4

Task

At-least-once delivery plus retries means duplicates, and duplicates corrupt any non-idempotent operation. Cause it with a counter, then fix it with an idempotency key.

Steps

  1. Run a queue and a consumer that increments a stored total by the amount in each message. Send ten messages totalling a known value and confirm the total is correct.
  2. Set the visibility timeout SHORTER than the consumer's processing time, and have the consumer receive in batches of ten (or run two consumers). Send the ten messages again and record the resulting total in lab/queue/dupes.md -- it will overshoot, and by how much tells you how many were redelivered. The batch matters: a single consumer taking one message at a time never overshoots, because nothing else is receiving while it works. Redelivery needs messages waiting out their timeout while something is free to receive them again.
  3. Fix it the wrong way first: lengthen the visibility timeout. Record that it works and state why it is not a fix -- what happens when a single message is unusually slow?
  4. Fix it properly: give each message an idempotency key, have the consumer record processed keys, and ignore repeats. Re-run with the short timeout and confirm the total is correct despite redelivery.
  5. Add a dead-letter path: introduce one message the consumer always fails on, confirm it is moved aside after a retry limit rather than blocking the queue, and record the depth alert you would configure.

Verify

grep -Eci 'visibility timeout' lab/queue/dupes.md
grep -Ec '[0-9]+' lab/queue/dupes.md
grep -Eci 'idempotency key' lab/queue/dupes.md
grep -Eci 'dead.letter' lab/queue/dupes.md

All four non-zero, with an overshoot recorded before the fix and a correct total after. Without the overshoot, either the visibility timeout was not actually shorter than processing or nothing was receiving while messages sat out their timeout -- a lone one-at-a-time consumer shows no duplicates at any timeout.

This is an independent study companion for CompTIA Cloud+ CV0-004 and is not produced by or endorsed by CompTIA.