Latency, throughput and routing problems

Why this matters

The previous lesson dealt with traffic that does not arrive. This one deals with traffic that arrives too slowly, which is harder because nothing is broken: every check passes, every component reports healthy, and the system is simply not good enough.

The core skill is measuring the right thing. "The network is slow" is three different problems — latency, throughput and loss — with different causes and different fixes, and treating them as one is why these investigations drift.

The lesson

Distinguishing latency from throughput from packet loss by measurement

Three distinct properties. Confusing them sends you in the wrong direction.

Latency is delay — how long a packet takes to arrive, usually measured as round-trip time. It is dominated by distance and by the number of hops, and it is close to a fixed property of the path. Within a region, single-digit milliseconds; across a continent, tens; intercontinental, well over a hundred. You cannot tune away the speed of light. If latency is the problem, the answer is to move things closer together or to make fewer round trips.

Throughput is volume per unit time. Limited by the narrowest link, by instance size (objective 1.2), by the protocol, and by latency in combination with window sizes.

Packet loss is packets not arriving. Small amounts have effects out of proportion to their size on TCP, because retransmission and congestion control respond sharply — a fraction of a percent of loss can halve effective throughput.

The relationship that explains many confusing cases: high latency limits throughput even on a fast link. TCP can only have so much data unacknowledged at once, so throughput is bounded by the window size divided by the round-trip time. A 10 Gbps link with 100 ms of latency and default window settings delivers a fraction of its capacity for a single stream. The symptom is a fast network that transfers files slowly, and the remedies are window scaling, parallel streams, or a protocol designed for long fat networks — not a bigger link.

Measure deliberately:

• Latency — round-trip time, and its variation (jitter), which matters more than the mean for real-time traffic.
• Throughput — a sustained transfer test, ideally with a single stream and with several, since the difference tells you whether you are window-limited or link-limited.
• Loss — over a sustained period, not a ten-packet sample.
• From the right places — between the actual endpoints, not from your laptop, and at the time the problem occurs.

Remember that many cloud networks deprioritise or suppress ICMP, so ping-based measurements can mislead. Prefer TCP-based tools.

Region and zone placement, and the chatty application that cannot tolerate distance

Placement is the most common cause of a cloud latency problem, and it is usually a design decision rather than a fault.

Cross-region traffic carries tens to hundreds of milliseconds and a per-gigabyte charge. Cross-zone traffic within a region adds a small amount of latency and, importantly, is billed in both directions (objective 1.8). Same-zone traffic is fastest and cheapest.

That produces a genuine tension the exam likes: spreading across zones is required for resilience (objective 3.2) and costs latency and money on every request between tiers. The resolution in practice is to keep tightly-coupled components in the same zone, replicated per zone, rather than distributing a single request path across zones.

The chatty application (objective 2.1) is where this becomes severe. An application making 200 sequential round trips per user action is invisible on a sub-millisecond local network and catastrophic at 80 ms:

200 × 0.5 ms = 0.1 seconds. 200 × 80 ms = 16 seconds.

Nothing is broken; the design multiplied the latency. The fixes are all about reducing round trips rather than reducing latency: batch requests, cache aggressively, denormalise, use bulk endpoints, or — most effectively — move the components together.

The related placement questions: put user-facing endpoints near users, with a CDN or edge presence; put compute near its data, because that is the chattiest relationship in most systems; and use placement groups (objective 2.4) where instances need the lowest possible latency between them.

Bandwidth limits tied to instance size, and the ceiling reached before the network's

A cloud-specific trap, and a favourite scenario.

Network bandwidth is a property of the instance size. A small instance may be limited to a fraction of a gigabit; a large one to tens. The underlying network is not the constraint — the instance's allocation is.

Consequences that look like network faults:

• Throughput plateaus at a number that is not a round network figure, because it is the instance's cap.
• The CPU is idle while the transfer is slow, which misleads people into investigating the network path.
• Storage traffic may share the same allocation on some instance types, so a large disk transfer and network traffic compete.
• Burstable network performance exists on smaller instances, with the same credit behaviour as CPU and storage (objective 1.2): good throughput initially, then a sustained ceiling at baseline. The signature is the same — fast at first, then consistently slower, with no configuration change.

Other caps in the same family: NAT gateways have per-flow and aggregate limits, and a single large transfer through one can hit a per-connection ceiling well below the gateway's total; load balancers scale but can throttle during a sudden spike unless pre-warmed or given time; and VPN gateways have their own throughput limits, which is a common reason a hybrid link underperforms a dedicated circuit (objective 1.3).

The diagnostic habit: before investigating the path, check the documented limits of every component in it. A throughput number that exactly matches a published cap is not a fault.

MTU, fragmentation and the VPN that breaks only large transfers

The classic "some things work and some do not" fault, and worth knowing because its signature is so distinctive.

MTU is the largest packet a link carries — typically 1500 bytes on Ethernet. Encapsulation adds overhead: an IPsec VPN, a tunnel, or an overlay network wraps each packet in additional headers, so the effective payload is smaller than 1500.

If a host sends a full-size packet with the don't fragment bit set and the path cannot carry it, the packet should be rejected with an ICMP "fragmentation needed" message telling the sender to use a smaller size. This is path MTU discovery, and it depends on that ICMP message getting back.

When ICMP is blocked, the message never arrives. The sender keeps sending packets that are silently discarded. The result is a PMTU black hole, and its signature is unmistakable once you know it:

• Small packets work perfectly — connections establish, pings succeed, short requests return.
• Large transfers hang or stall — file uploads, large API responses, database result sets, TLS handshakes with large certificate chains.
• It affects only traffic crossing the tunnel or overlay.

This is why "the VPN works but file copies hang" is a recognisable problem rather than a mystery.

The fixes, in order of preference:

1. Allow the necessary ICMP so path MTU discovery works. This is the correct fix, and blanket ICMP blocking is the underlying cause.
2. Lower the MTU on the interfaces using the tunnel.
3. Clamp TCP MSS on the tunnel device, so TCP negotiates a segment size that fits. This is the standard remedy on VPN endpoints and is very effective.

The opposite case is worth a mention: jumbo frames (typically 9000 bytes) improve throughput within a virtual network by reducing per-packet overhead — but they must be supported end to end, and traffic leaving via an internet gateway falls back to standard sizes. Enabling them on some hosts and not others causes exactly the same large-packet failures.

Path problems: peering, transit and the provider boundary you cannot see into

Sometimes the problem is genuinely not yours, and recognising that quickly is itself a skill.

Between the cloud and the internet, traffic crosses networks you neither own nor can inspect. Congestion at a peering point, a suboptimal route chosen by another network, or an outage several hops away can degrade performance for some users and not others. The signature is geographic — users in one region or on one ISP affected, others fine.

What you can do:

• Measure from multiple locations to establish the pattern, using synthetic monitoring from outside your own network.
• Check the provider's status page and any relevant network health dashboards, and remember that a provider-side issue can be real and unannounced for a while.
• Use the provider's backbone: a CDN or global accelerator brings traffic onto the provider's own network at an edge location near the user, replacing the unpredictable public path with a managed one. This is the strongest remedy for internet path problems and is the right answer for a scenario describing inconsistent performance by geography.
• Move closer to users, with a regional presence.

Between cloud networks, the picture is clearer but still layered: peering is non-transitive (objective 1.3), transit gateways and hubs add a hop and their own limits, and hybrid paths can involve a partner's network between you and the provider.

The escalation discipline: document the evidence before raising it. Timestamps, source and destination, the measurements, the paths, and what you have eliminated. A provider ticket with that content is answered quickly; one saying "the network is slow" is not. And know when to escalate — an investigation that has eliminated placement, instance limits, MTU and configuration, and shows a problem on a path you do not control, is finished on your side.

What to take into the exam

• Latency, throughput and loss are three different problems. Latency is mostly distance and cannot be tuned away; the fix is fewer round trips or closer placement.
• High latency limits throughput via the window/RTT relationship — a fast link that transfers slowly needs window scaling or parallel streams, not more bandwidth.
• Cross-zone traffic costs latency and is billed both ways. Keep tightly-coupled components in one zone, replicated per zone.
• Network bandwidth is capped by instance size, may share the storage allocation, and can be burstable — "fast then permanently slower". Check published limits before investigating the path.
• PMTU black hole: small packets fine, large transfers hang, only across a tunnel — caused by blocked ICMP. Fix by allowing ICMP, lowering MTU, or clamping TCP MSS.
• Geographic variation points at an internet path problem. A CDN or global accelerator moves traffic onto the provider's backbone. Escalate with measurements, not adjectives.