Prove where the packet loss is

applied · 40 min · Objective 1.4

Task

Introduce measurable loss and latency deliberately, then measure them with the tools you would use in an incident and produce a statement you could defend: where the loss is, how much, and whether it is loss at all.

Steps

  1. Baseline first, because without it nothing that follows means anything. Run ping -c 100 to a nearby host and record loss, average and mdev. Run iperf3 -c <server> and record throughput.
  2. Introduce 100 ms of latency on the test interface: tc qdisc add dev eth1 root netem delay 100ms. Re-measure and confirm the ping average moved and the mdev did not.
  3. Add jitter: tc qdisc change dev eth1 root netem delay 100ms 40ms. Re-measure and confirm mdev is now large. State the difference between latency and jitter from your own numbers.
  4. Replace it with loss: tc qdisc change dev eth1 root netem loss 10%. Measure with ping, then with iperf3, and note how much more throughput suffers than the loss percentage alone suggests.
  5. Run mtr -rwc 100 through the impaired path and read the per-hop loss. Identify which hop the impairment is at.
  6. Look at the host's own counters: ip -s link show eth1 and ethtool -S eth1 | grep -i drop, and note whether the loss appears there -- it will not, because netem drops after the interface counters.
  7. Remove the impairment with tc qdisc del dev eth1 root and confirm the baseline returns.

Verify

tc qdisc show dev eth1
ping -c 20 -q <target> | tail -2          # loss % and rtt min/avg/max/mdev
mtr -rwc 50 <target> | tail -5
iperf3 -c <server> -t 5 | tail -3
ip -s link show eth1 | sed -n '4,6p'

You should be able to end with a defensible sentence such as: "10% loss appears from hop 2 onward and persists to the destination; throughput fell from 940 to 60 Mbit/s; host interface counters show no drops, so the loss is not local."

Notes

Step 6 matters more than it looks. Loss visible in ip -s link is the host failing to keep up -- a ring buffer too small, interrupts on one CPU. Loss visible only in end-to-end measurement is somewhere in the path. They are different problems with different fixes, and the counters tell you which.