Layer 2 and Layer 3 troubleshooting with show commands, ping, traceroute and captures

Why this matters

"It cannot reach the server" is the most common ticket in networking, and it says nothing about where the fault is. A help-desk analyst moving into networking, a network technician on call or a junior engineer has to turn that sentence into a location: a shut interface, a bad cable, a port in the wrong VLAN, a trunk that does not carry the VLAN, a missing route, or a filter. The tools for doing that are the same on every Cisco device: a handful of show commands, the log, ping, traceroute and, when those are not enough, a packet capture.

Without a method, people change things until the problem goes away, and often cause a second one. With a method, each command rules a layer in or out. This lesson walks up the stack, from the physical interface to the routing table, and shows what each tool's output means, including the outputs that look alike but point to different faults.

The lesson

Troubleshooting Layer 2 and Layer 3 with show commands, logs, ping, traceroute and captures

Work from the bottom up. Is the interface up? Is the port in the right VLAN, and does the trunk carry it? Does the device know the neighbour's MAC address? Is there a route? Is something filtering the traffic? Each question has a command that answers it.

Interface state and counters

show ip interface brief gives one line per interface: its IPv4 address, how the address was set, its status and its line protocol state. It does not show speed, duplex, errors or VLANs.

R1# show ip interface brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0/0   10.1.1.1        YES manual up                    up
GigabitEthernet0/0/1   10.1.12.1       YES manual administratively down down
GigabitEthernet0/0/2   10.1.13.1       YES manual down                  down

Read the two state columns together:

• administratively down / down: the interface has shutdown in its configuration. Enter no shutdown. A pulled cable never causes this state.
• down / down: a Layer 1 problem. The cable is unplugged or damaged, or the far end is shut down.
• up / down: the physical link is up but the line protocol is not. On an SVI, it usually means no active port carries the VLAN (lesson 8).
• up / up: working at Layers 1 and 2. Look higher.

For errors, use show interfaces. Two patterns matter most:

SW1# show interfaces GigabitEthernet1/0/7
GigabitEthernet1/0/7 is up, line protocol is up
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
     4120 input errors, 4087 CRC, 33 frame, 0 overrun, 0 ignored
     0 output errors, 0 collisions, 0 interface resets

Rising CRC errors on a full-duplex link with no collisions mean frames arrive corrupted: a damaged cable, a bad connector or electrical noise. A native VLAN or access VLAN error misplaces frames but does not corrupt them, so it never raises CRC counts.

SW1# show interfaces GigabitEthernet0/1
GigabitEthernet0/1 is up, line protocol is up
  Half-duplex, 100Mb/s, media type is 10/100/1000BaseTX
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 output errors, 2318 collisions, 0 interface resets
     0 babbles, 1844 late collision, 0 deferred

Late collisions on a half-duplex port are the classic sign of a duplex mismatch: this end runs half duplex and the far end full. Users report slow, unreliable transfers rather than a dead link. Lesson 1 covers these Layer 1 faults in depth.

A port can also be shut by the switch itself. show interfaces status err-disabled lists those ports with the reason:

SW1# show interfaces status err-disabled
Port      Name   Status       Reason
Gi1/0/5          err-disabled bpduguard

Here BPDU guard shut Gi1/0/5 because a small switch sent BPDUs into it. Remove the cause first, then bounce the port with shutdown and no shutdown. Disabling PortFast or clearing tables does not recover an err-disabled port. Lesson 12 covers BPDU guard.

VLANs, trunks and MAC addresses

show vlan brief lists each VLAN and its access ports. A port that is up and passing traffic but missing from every VLAN is almost certainly a trunk; trunks do not appear in this list. An access port with a voice VLAN still appears. Confirm a trunk with show interfaces trunk:

SW1# show interfaces trunk
Port        Mode         Encapsulation  Status        Native vlan
Gi1/0/24    on           802.1q         trunking      1

Port        Vlans allowed on trunk
Gi1/0/24    10,20

If users in VLAN 30 on SW2 cannot reach VLAN 30 on SW1, this output shows why: VLAN 30 is not allowed. The fix is switchport trunk allowed vlan add 30. Without add, the command replaces the list and cuts off VLANs 10 and 20.

For a single user, two commands confirm where they are at Layer 2. show interfaces switchport shows the port's mode and access VLAN. show mac address-table shows the VLAN and port on which the user's MAC address was learned:

SW1# show mac address-table address 0050.7966.6801
          Mac Address Table
-------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  20    0050.7966.6801    DYNAMIC     Gi1/0/24

The entry is DYNAMIC, learned from traffic, not configured. Gi1/0/24 is SW1's trunk to SW2, so the host is not on SW1 at all: it is reached through SW2, somewhere beyond the uplink.

Between Layer 2 and Layer 3 sits ARP. A router or host must resolve the next hop's MAC address before it can send a frame:

R1# show ip arp 10.1.1.50
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.1.1.50               0   Incomplete      ARPA

Incomplete means R1 sent ARP requests and got no reply. The fault is at Layer 2, or the server is down. A server firewall that drops ICMP would still answer ARP, so the entry would show a MAC address.

Routes and addressing

When Layer 2 is sound, check the route. show ip route with no address prints the whole table; show ip route followed by an address prints the entry that would be used. Lesson 13 explains the table in detail.

R1# show ip route
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.1.1.0/24 is directly connected, GigabitEthernet0/0/0
L        10.1.1.1/32 is directly connected, GigabitEthernet0/0/0

R1 cannot reach 198.51.100.50 because no entry matches it and the gateway of last resort is not set, so R1 drops the packet. The interface is clearly up: a shut interface would lose its connected and local routes.

When several entries match, the longest prefix wins, and that decides which router answers as the first hop of a traceroute:

R1# show ip route
Gateway of last resort is 10.0.0.9 to network 0.0.0.0
S*    0.0.0.0/0 [1/0] via 10.0.0.9
O     10.10.0.0/16 [110/20] via 10.0.0.1, 00:12:04, GigabitEthernet0/1
O     10.10.4.0/22 [110/30] via 10.0.0.2, 00:12:04, GigabitEthernet0/1
O     10.10.5.0/24 [110/40] via 10.0.0.3, 00:12:04, GigabitEthernet0/1

For 10.10.6.20, the /22 covers 10.10.4.0 to 10.10.7.255 and matches; the /24 is longer but covers only 10.10.5.x, so it does not. R1 forwards to 10.0.0.2, and that router is hop 1.

A Layer 2 switch with ip routing off is a host on its management VLAN. If its interface Vlan10, 10.1.10.5/24, is up/up and reaches hosts in 10.1.10.0/24 but nothing beyond, it lacks ip default-gateway. Turning on ip routing would make it a router that then needs routes of its own, which changes far more than needed.

Also check the host's own address. Worked example: a PC is set to 172.16.8.191 with mask 255.255.255.224 and gateway 172.16.8.161, and it cannot send anything.

1. The mask 255.255.255.224 is a /27. The block size is 256 − 224 = 32.
2. The blocks start at .128, .160, .192. The gateway, .161, sits in the block that starts at 172.16.8.160.
3. That block ends one below the next block, at 172.16.8.191. So 172.16.8.191 is the broadcast address.
4. The PC has been given the subnet's broadcast address, which no host may use. Valid hosts are .161 to .190.

Lesson 3 practises this method.

Reading the log

show logging shows recent messages, newest last. Two patterns come up often:

R1# show logging
%LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

The same four lines repeating every few seconds mean the link is flapping: a damaged cable, a loose connector or a failing port. An ACL or a routing protocol fault would produce different messages, not link state changes.

%CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (1), with SW2 GigabitEthernet1/0/24 (99).

This one means the two ends of a trunk use different native VLANs, 1 locally and 99 on SW2. Different allowed VLAN lists produce no CDP message. Lesson 10 explains CDP messages and lesson 29 explains syslog severity levels.

CDP also settles cabling questions. If show cdp neighbors on SW1 lists R1 on local port Gi1/0/1 with Port ID Gig 0/0/1, then R1's Gi0/0/1 is cabled there, whatever the documentation says. Port ID is always the neighbour's own port, and the entry exists only because R1 is sending CDP on it.

Ping

A standard ping sends five ICMP echoes and prints one character per echo:

• ! a reply arrived.
• . the echo timed out with no reply.
• U an ICMP destination unreachable came back, usually from a router with no route or an ACL deny.
• M the packet could not be fragmented (it was too big for a link and had DF set).

R1# ping 10.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.3.3.3, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5)

.!!!! is normal for a first ping: the first echo was dropped while the router resolved ARP for the next hop. A lossy link would scatter losses through repeated tests instead.

R1# ping 10.5.5.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.5.5.5, timeout is 2 seconds:
U.U.U
Success rate is 0 percent (0/5)

U.U.U means a router is sending unreachables back. The dots in between appear because IOS rate-limits the unreachables it sends. Five dots would mean silence.

Extended ping changes what is tested. Enter ping with no address to get the dialog:

R2# ping
Protocol [ip]:
Target IP address: 10.9.9.9
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 192.168.2.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0x0000ABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.9.9.9, timeout is 2 seconds:
Packet sent with a source address of 192.168.2.1
.....
Success rate is 0 percent (0/5)

Two uses matter:

• Source address. A standard ping uses the exit interface's address as its source, which the far side may know. Here R2 pings 10.9.9.9 fine normally, but users on 192.168.2.0/24 cannot. Sourcing the ping from R2's LAN address, 192.168.2.1, tests whether the far side has a return route to that LAN. The five dots say it does not. The one-line form is ping 10.9.9.9 source GigabitEthernet0/0/1.
• DF bit and sweep. If small pings succeed but large transfers stall, suspect a link with a smaller MTU. Set the DF bit so no router may fragment the echo, and sweep a range of sizes. The largest size that returns ! is the path MTU; larger sizes return M or .. Raising the repeat count alone never finds this, because the size stays small.

Traceroute

Traceroute sends probes with rising TTL values, so each router along the path reports itself in turn.

R1# traceroute 172.20.1.10
Type escape sequence to abort.
Tracing the route to 172.20.1.10
  1 10.0.12.2 1 msec 1 msec 1 msec
  2 10.0.23.3 2 msec 1 msec 2 msec
  3  *  *  *
  4  *  *  *

R1 has a route, since the first two hops answered. Replies stop after 10.0.23.3, so look at that router and the next device beyond it: one of them lacks a route or filters the traffic.

R1# traceroute 192.168.50.10
  1 10.0.12.2 1 msec 1 msec 1 msec
  2 10.0.12.1 1 msec 2 msec 1 msec
  3 10.0.12.2 2 msec 2 msec 1 msec
  4 10.0.12.1 2 msec 2 msec 2 msec

R1 owns 10.0.12.1 and R2 owns 10.0.12.2. Hops that alternate between the same two addresses are a routing loop: each router points to the other for 192.168.50.10, and the packet bounces until its TTL expires. A filter would show asterisks instead.

Packet captures

When show commands look healthy, a capture shows what is really on the wire.

No. Time      Source         Destination  Protocol Info
1   0.000000  Dell_3c:4d:5e  Broadcast    ARP      Who has 192.168.1.1? Tell 192.168.1.20
2   1.002113  Dell_3c:4d:5e  Broadcast    ARP      Who has 192.168.1.1? Tell 192.168.1.20
3   2.004087  Dell_3c:4d:5e  Broadcast    ARP      Who has 192.168.1.1? Tell 192.168.1.20

The PC asks for its gateway's MAC address and nothing answers. That is a Layer 2 fault, such as the access port sitting in the wrong VLAN. An ACL blocking ICMP on the gateway would not stop it answering ARP.

No. Time   Source     Destination  Protocol Info
1   0.000  10.1.1.20  10.2.2.5     TCP      54012 → 22 [SYN] Seq=0 Win=64240 Len=0
2   1.004  10.1.1.20  10.2.2.5     TCP      [TCP Retransmission] 54012 → 22 [SYN] Seq=0
3   3.010  10.1.1.20  10.2.2.5     TCP      [TCP Retransmission] 54012 → 22 [SYN] Seq=0

This host can ping 10.2.2.5 but cannot open SSH. The SYNs go unanswered and are retransmitted. That points to a filter dropping TCP port 22 somewhere on the path. A closed port would answer with a reset (RST), and a wrong default gateway on the server would break the ping replies too.

What the exam is really asking

• Administratively down vs down. The first is shutdown; the second is Layer 1.
• CRC vs late collisions. CRC errors on full duplex mean a bad cable or noise; late collisions on half duplex mean a duplex mismatch.
• Missing from show vlan brief usually means the port is a trunk (a routed port is missing too).
• A MAC address learned on a trunk means the host is beyond the neighbour switch.
• ARP Incomplete means no reply at Layer 2, not a firewall dropping ping.
• No match and no gateway of last resort means the router drops the packet.
• ip default-gateway vs ip routing on a Layer 2 switch.
• Ping characters. ! reply, . timeout, U unreachable; .!!!! is ARP.
• Extended ping uses. Source address tests the return path; DF bit plus sweep tests MTU.
• Traceroute patterns. Asterisks mark where replies stop; alternating hops mark a loop.
• Captures. Unanswered ARP is Layer 2; unanswered SYNs with working ping is a filter; RST is a closed port.
• err-disabled recovery. Remove the cause, then shutdown and no shutdown.