Troubleshooting network services

Why this matters

Objective 5.3 covers the faults that live above the cable: addressing, routing, switching and access control. These are the ones where everything looks healthy — link lights on, no errors, devices up — and traffic still does not arrive.

Nearly every fault here has a characteristic symptom that identifies it almost on its own. Learning the symptom-to-cause pairs is the efficient way through this objective, because exam questions are written as symptoms.

The lesson

Addressing faults

Incorrect IP address. A host configured outside its subnet's range. It can usually reach nothing, though it may reach hosts that happen to fall within the network its mask implies. Check against the IPAM record from domain 3.

Duplicate IP address. Two devices with the same address. The symptom is intermittent connectivity for both — sometimes one answers, sometimes the other — and it is the intermittency that identifies it. Operating systems usually log a duplicate address detection message, and arp -a from a third host shows the address flipping between two MAC addresses.

The usual cause is a statically configured device inside a DHCP scope with no matching exclusion, which is why the DHCP lesson treats scope, exclusion and reservation as one topic.

Incorrect subnet mask. The most instructive addressing fault, because the symptom is so distinctive: the host reaches some destinations and not others, with no obvious pattern.

That happens because the mask decides which destinations the host believes are local. Get it wrong and the host's bitwise AND produces the wrong network, so some addresses are wrongly treated as local (and ARPed for, which fails) while others are correctly sent to the gateway. The binary lesson's AND operation is what explains it.

If a scenario says "can ping some hosts on the same LAN but not others", check the mask before anything else.

Incorrect default gateway. The host reaches everything on its own subnet perfectly and nothing beyond it. That split is the signature. Local traffic never touches the gateway, so it works; anything remote is sent to an address that either does not exist or does not route.

Address pool exhaustion. The DHCP scope has no free addresses. New clients get an APIPA address in 169.254.0.0/16 and work only with other APIPA hosts on the same segment. Existing clients with valid leases are unaffected, which is why the symptom is "new devices cannot connect while everyone already on is fine".

Causes are a scope too small for demand, a lease time too long for a high-turnover network such as guest wireless, or a rogue device consuming addresses. The fix is usually a shorter lease before a larger scope.

Routing faults

Routing table problems are diagnosed by reading the table and asking what the router will actually do with a packet for the destination.

Apply the selection order from domain 2: longest prefix match first, then administrative distance, then metric. Most "the route is there but traffic is not taking it" puzzles resolve to a more specific route elsewhere in the table, or to a static route with AD 1 overriding a dynamic one.

Route selection faults are frequently asymmetric, and asymmetry is worth watching for. Traffic reaches the destination and the reply takes a different path — or no path. The classic presentation is a service that works in one direction only, and it is diagnosed by tracing from both ends rather than one.

Default routes are the catch-all, 0.0.0.0/0. Because it is the least specific route possible, it is used only when nothing else matches — which is exactly what makes a missing default route present as "internal works, internet does not". Two default routes pointing different ways, or a default route pointing at a dead next hop, produce intermittent or total loss of external connectivity while everything internal looks fine.

Switching faults

Incorrect VLAN assignment puts a device in the wrong broadcast domain. The symptoms depend on what else is there: no DHCP response (because the scope for that VLAN does not serve it, or no relay is configured), or connectivity to the wrong set of hosts.

The tell is a device that gets an address from an unexpected range, or none at all, while the port shows up and error-free. Check the port's access VLAN against what the device should be in, and remember that a port with no VLAN configured sits in VLAN 1.

Network loops are the most destructive layer 2 fault. Because an Ethernet frame has no TTL, a broadcast entering a loop circulates and multiplies until it saturates the segment.

The symptoms are unmistakable once seen: all link lights flashing in unison, switch CPU at 100%, the whole segment unusable rather than one device, MAC address flapping between ports in the logs, and management access to the switches themselves lost.

The cause is usually a cable plugged into two ports of the same switch, two switches connected twice, or a small unmanaged switch introduced by a user. Prevention is STP, plus BPDU Guard on access ports so an introduced switch shuts the port rather than joining the topology.

STP faults are the subtler relatives:

Root bridge selection. The root is the switch with the lowest bridge ID — priority first, then MAC address. Because every switch defaults to the same priority, the tie is broken by MAC address, and MAC addresses are effectively arbitrary — so the oldest switch usually wins, since older equipment tends to have lower addresses.

That is rarely what you want. The root should be a core switch, deliberately chosen by setting a low priority. A root election won by an access switch in a cupboard drags traffic through it, producing a network that works but performs inexplicably badly — traffic taking a long path for no visible reason.

Port roles are the outcome of the election: the root port is each switch's best path toward the root; a designated port forwards for a segment; a blocking or alternate port is the one STP has disabled to break the loop.

Port states in classic STP are blocking, listening, learning and forwarding, with a transition of 30 to 50 seconds. That delay is itself a fault source: an end device that waits 45 seconds for a port to forward often fails to get DHCP and falls back to APIPA — which is why PortFast exists for access ports, and RSTP for everything else.

A port unexpectedly in blocking state means STP believes there is a loop through it. That may be correct, or it may be the consequence of a badly placed root.

ACLs

ACLs deny traffic deliberately, which makes them easy to overlook — the network is doing exactly what it was told.

The characteristic symptoms:

• Traffic that works in one direction only, because the return path was not permitted on a stateless device.
• One application or port failing while everything else from the same host works.
• A fault that appeared immediately after a change, which is the change record from domain 3 earning its keep.

The three things to check, in order: rule order, because first match wins and a broad rule above a specific one makes the specific one unreachable; the implicit deny at the end, which drops anything not explicitly permitted; and direction, since a rule applied inbound on the wrong interface does nothing useful.

Most platforms count matches per rule, and reading those counters is the fastest way to see whether traffic is hitting the rule you think it is — or hitting one higher up that you had forgotten.

A useful discriminator between an ACL and a routing fault: an ACL drop usually produces a timeout, while a missing route often produces an ICMP unreachable. Not universal, but a good first guess.