Cause a split-brain on purpose, then work out the quorum that prevents it

applied · 50 min · Objective 2.4

Task

Break the heartbeat path between the two keepalived nodes while both stay running, and watch both claim the virtual IP: a split-brain. Then work through the quorum arithmetic for clusters of different sizes, with and without a witness, to show which split each one survives and why two-node clusters need a witness most.

Steps

  1. With both nodes running and lin-srv holding the virtual IP, block VRRP on lin-b: nft add table inet lab, a chain on the input hook, and a rule dropping IP protocol 112 (VRRP). Save the rule list to lab/split/rule.txt.
  2. Wait ten seconds and save ip -br addr from both nodes to lab/split/both.txt.
  3. From win-srv, run arp -a (or ip neigh) and ping the virtual IP a few times, saving the output to lab/split/client.txt. Note which MAC address the client has for the virtual IP.
  4. Remove the rule with nft delete table inet lab and save ip -br addr from both nodes to lab/split/healed.txt.
  5. Write lab/split/quorum.csv with header nodes,witness,votes,majority,survives_even_split, with witness and survives_even_split as yes or no, for 2, 3, 4 and 5 nodes, each with and without a witness.

Verify

These checks run in a POSIX shell: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. A stock Windows PowerShell or Command Prompt has no awk or grep, so there the first line fails.

grep -Ec '112|vrrp' lab/split/rule.txt
grep -c '192.168.56.100' lab/split/both.txt
grep -c '192.168.56.100' lab/split/healed.txt
awk -F, 'NR>1 && $1==2 && $2=="no" {print "2 nodes, no witness: majority "$4", survives split: "$5}' lab/split/quorum.csv
awk -F, 'NR>1 && $1==4 && $2=="yes" {print "4 nodes + witness: votes "$3", majority "$4}' lab/split/quorum.csv

During the split the virtual IP appears twice -- once on each node -- and after healing, once. A two-node cluster without a witness has a majority of 2 and does not survive a split; four nodes plus a witness have five votes and a majority of 3, so a two-and-two split leaves one side with the witness and quorum.

Notes

keepalived has no quorum, which is why it split. That is the gap real cluster managers close: quorum stops the minority side, and fencing makes sure it has really stopped before the majority side takes over shared storage.

This is an independent study companion for CompTIA Server+ SK0-005 and is not produced by or endorsed by CompTIA.