Reproduce read-after-write failure against a replica

applied · 60 min · Objective 1.6

Task

Set up primary and replica databases locally, introduce replication lag, and produce the bug where an application writes a record and then cannot find it. It is a confusing failure in production and an obvious one once you have caused it deliberately.

Steps

  1. Start a primary and an asynchronous read replica in containers. Confirm replication is working by writing a row and reading it back from the replica.
  2. Write a short script that INSERTs a row against the primary and immediately SELECTs it from the replica, reporting found or not found. Run it 50 times and record the failure count in lab/lag.txt.
  3. Introduce lag: slow the replica, either by loading it or by pausing its container briefly. Re-run the 50 iterations and record the new failure count.
  4. Change the script so reads that must be current go to the primary. Re-run and record the failure count again.
  5. Write in lab/lag.txt the rule this gives you about which reads may go to a replica, and remove both containers.

Verify

grep -Ec '[0-9]+ (failure|not found)' lab/lag.txt
grep -Eci 'primary' lab/lag.txt
grep -Eci 'lag' lab/lag.txt

Three failure counts must appear and the middle one must be higher than the first. If lag produced no failures, the replica was not actually asynchronous or was not actually slowed.

This is an independent study companion for CompTIA Cloud+ CV0-004 and is not produced by or endorsed by CompTIA.