Prove where the packet loss is

applied · 40 min · Objective 1.4

Task

Introduce measurable loss and latency deliberately, then measure them with the tools you would use in an incident and produce a statement you could defend: where the loss is, how much, and whether it is loss at all.

Steps

  1. Baseline first, because without it nothing that follows means anything. Run ping -c 100 to a nearby host and record loss, average and mdev. Run iperf3 -c <server> and record throughput.
  2. Introduce 100 ms of latency on the test interface: tc qdisc add dev eth1 root netem delay 100ms. Re-measure and confirm the ping average moved and the mdev did not.
  3. Add jitter: tc qdisc change dev eth1 root netem delay 100ms 40ms. Re-measure and confirm mdev is now large. State the difference between latency and jitter from your own numbers.
  4. Replace it with loss: tc qdisc change dev eth1 root netem loss 10%. Measure with ping, then with iperf3, and note how much more throughput suffers than the loss percentage alone suggests.
  5. Run mtr -rwc 100 through the impaired path and read the per-hop loss. Identify which hop the impairment is at. Here it shows from the FIRST hop onward, because netem is dropping this host's own outgoing probes -- loss that starts at hop 1 and persists to the destination points at the near end, not the path.
  6. Look at the host's own counters: ip -s link show eth1 and ethtool -S eth1 | grep -i drop, and note whether the loss appears there -- it will not. netem drops packets in the queueing layer, BEFORE they reach the driver whose counters those are. The queue keeps its own: tc -s qdisc show dev eth1 reports them as dropped.
  7. Remove the impairment with tc qdisc del dev eth1 root and confirm the baseline returns.

Verify

tc -s qdisc show dev eth1                 # the netem rule, and its dropped count
ping -c 20 -q <target> | tail -2          # loss % and rtt min/avg/max/mdev
mtr -rwc 50 <target> | tail -5
iperf3 -c <server> -t 5 | tail -3
ip -s link show eth1 | sed -n '4,6p'

You should be able to end with a defensible sentence such as: "10% loss appears from the first hop onward and persists to the destination; throughput fell from 940 to 60 Mbit/s; the interface counters show no drops, but the qdisc counter does -- the loss is being made on this host, before the driver." That last clause is the trap: interface counters that look clean do not prove the loss is elsewhere.

Notes

Step 6 matters more than it looks. Loss visible in ip -s link is the host failing to keep up -- a ring buffer too small, interrupts on one CPU. Loss visible only in end-to-end measurement is somewhere in the path. They are different problems with different fixes, and the counters tell you which.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.