Troubleshooting networking

Why this matters

Network problems are the ones most often misdiagnosed, because the symptom is almost always the same — "it does not work" — and the cause can be at any of six layers, on either machine, or in between.

The cure is a fixed order. Work up from the physical link and stop at the first thing that is broken, rather than starting from the most interesting theory. Done properly it takes about ninety seconds to know which layer you are in, which is faster than any amount of guessing.

The lesson

The order to work in

Every one of the common issues below is found faster by running this sequence than by theorising:

ip link show                     # 1. is the interface UP? does it have carrier?
ip addr show                     # 2. does it have an address, and the right one?
ip route show                    # 3. is there a default gateway?
ping -c3 192.168.1.1             # 4. can you reach the gateway?
ping -c3 8.8.8.8                 # 5. can you reach the internet by IP?
dig example.com                  # 6. does DNS resolve?
curl -sv https://example.com     # 7. does the application layer work?

The first step that fails is where the problem is, and each has a different answer. Steps 4 and 5 succeeding while 6 fails is DNS, and no amount of inspecting the firewall will help. Step 5 failing while 4 succeeds is routing or upstream.

The single most valuable observation in this whole lesson: ping 8.8.8.8 works but ping google.com does not means DNS, always. It is the fastest diagnosis on the exam and in life.

Link down

ip link show eth0                # "state DOWN" or "NO-CARRIER"
ethtool eth0 | grep -i "link detected"
dmesg | grep -i eth0
ip link set eth0 up

NO-CARRIER means the physical link is not there: unplugged cable, dead switch port, failed transceiver, wrong SFP. state DOWN without NO-CARRIER means the interface is administratively down and just needs bringing up.

The distinction saves you from looking for a configuration problem when somebody has unplugged a cable, and from checking cables when the interface simply was not enabled.

ethtool -S eth0 gives per-interface error counters; rising rx_errors or rx_crc_errors point at a bad cable or a failing port rather than anything in software.

Link negotiation issues

Two devices must agree on speed and duplex. Autonegotiation works when both ends do it; hard-coding one end and leaving the other automatic is the classic failure, and the result is a duplex mismatch.

ethtool eth0                     # Speed, Duplex, Auto-negotiation
ethtool -s eth0 autoneg on
ethtool -s eth0 speed 1000 duplex full autoneg off   # only if BOTH ends are

A duplex mismatch is worth recognising by its signature: the link works, and is dreadful under load. Small pings succeed, throughput collapses, and ethtool -S shows rising collisions and late collisions. Because it works enough to look fine, it is regularly diagnosed as an application problem.

The fix is nearly always to set both ends to autonegotiate. Half duplex on a modern switched network is a symptom, never a plan.

A gigabit link running at 100 Mb/s is usually a damaged cable — a pair is broken, and negotiation falls back rather than failing.

Interface misconfiguration

ip addr show
nmcli connection show
nmcli device status
cat /etc/NetworkManager/system-connections/eth0.nmconnection
nmcli connection up eth0

The recurring mistakes:

• The wrong prefix length. /24 typed as /16, or vice versa. The host then believes remote addresses are local, sends ARP for them, and gets no answer — so nothing outside works while everything local does.
• No default route, covered below.
• Configured the interface but not persistently. ip addr add survives until reboot; the change belongs in NetworkManager or the distribution's configuration.
• The wrong interface. Predictable names (enp0s3, ens192) change when hardware changes, and a configuration bound to a name that no longer exists leaves the machine with no network at all.

Check the netmask against the gateway. If the gateway is not inside the interface's own subnet, routing cannot work, and this is one of the most common misconfigurations there is.

Subnet problems

A subnet mistake means the host has the wrong idea about what is local:

ipcalc -n 192.168.10.50/26        # network, broadcast, host range
sipcalc 192.168.10.50/26
ip route get 192.168.10.90        # is this local or via the gateway?

ip route get is the command that settles it: it tells you exactly how the kernel would send a packet to that address, which is the question you are actually asking.

Two hosts on the same wire with different prefix lengths — one /24, one /25 — can produce one-way communication, where A reaches B and B cannot reach A. It looks like a firewall and is not.

Cannot ping server

ping -c3 target
ping -c3 -I eth0 target           # from a specific interface
traceroute target
mtr target                        # continuous traceroute -- shows WHERE loss starts
arping -I eth0 192.168.1.50       # layer 2 -- does it answer ARP?
ip neigh show                     # the ARP cache

Before anything else: many hosts and firewalls block ICMP, so "cannot ping" is not proof a host is down. Test the port you actually care about:

nc -zv target 443
curl -sv --connect-timeout 5 https://target

If ping fails but the service answers, nothing is wrong. If both fail, work outward: arping proves layer 2 reachability on the local segment, and ip neigh showing FAILED for the target means ARP is getting no reply — the host is off, on a different VLAN, or your netmask is wrong.

mtr is the tool for anything beyond the local network, because it shows which hop loss begins at. Loss at an intermediate hop that does not continue to the destination is usually that router rate-limiting ICMP, not a fault.

Server unreachable

Reachable at the network layer but the service does not answer. Split it into three questions, and check on the server:

ss -tlnp | grep :443             # is it LISTENING, and on which address?
systemctl status nginx
firewall-cmd --list-all
nft list ruleset | head -40
ausearch -m avc -ts recent       # SELinux denying the port?

The listening address is the thing people miss. A service bound to 127.0.0.1:8080 works perfectly on the host and is unreachable from anywhere else, and ss shows it plainly. 0.0.0.0 or :: means all interfaces.

Then the distinction that tells you which side is at fault:

• Connection refused — the packet arrived and nothing was listening, or a firewall sent a reject. Fast failure.
• Connection timed out — the packet vanished. A firewall dropping silently, or a routing problem. Slow failure.

Refused is fast, dropped is slow, and that difference alone often tells you whether to look at the service or the network.

SELinux is worth checking explicitly: it confines which ports a service may bind, so nginx on port 8443 is denied until semanage port -a -t http_port_t -p tcp 8443 allows it. The error looks like a permission problem with no cause.

Misconfigured firewalls

firewall-cmd --state
firewall-cmd --list-all --zone=public
firewall-cmd --get-active-zones
firewall-cmd --add-service=https --permanent && firewall-cmd --reload

nft list ruleset
iptables -L -n -v --line-numbers          # legacy, still seen

The failures, in order of frequency:

Forgot --permanent. The rule works, and disappears at the next reload or reboot. Conversely, --permanent without --reload writes a rule that is not yet active — so the change appears to have done nothing.

Wrong zone. firewalld assigns each interface a zone, and a rule added to public does nothing if the interface is in internal. --get-active-zones first, always.

Rule order. In iptables and nftables, the first match wins. A permissive rule above a restrictive one makes the restrictive one dead code.

Both host and network firewalls. The host is clean and a security group, cloud firewall or physical appliance is dropping. Check from a host on the same segment to isolate which.

The safety rule: before changing firewall rules on a remote machine, arrange a way back in. firewall-cmd --timeout=300 applies a rule that reverts itself after five minutes, which is exactly the protection you want when the rule you are about to add might lock you out.

DHCP issues

journalctl -u NetworkManager | grep -i dhcp
nmcli connection show eth0 | grep ipv4
dhclient -v eth0                          # verbose, in the foreground
ip addr show eth0

A 169.254.x.x address is the diagnosis. That is APIPA/link-local, assigned when no DHCP server answered at all. The host is up, the interface is up, and nothing replied.

Causes: the DHCP server is down or out of leases, the relay is missing on a different subnet (DHCP is broadcast and does not cross routers without one), a switch port on the wrong VLAN, or the client is denied.

On the server:

journalctl -u dhcpd -n 50
cat /var/lib/dhcpd/dhcpd.leases | tail

A working DHCP lease with the wrong details — bad gateway, bad DNS — points at the scope options rather than the protocol. And a machine that gets an address and cannot reach anything is usually one where DHCP supplied no default gateway.

DNS issues

dig example.com                           # the full answer, with the server used
dig +short example.com
dig @8.8.8.8 example.com                  # ask a DIFFERENT resolver
dig -x 192.0.2.10                         # reverse lookup
host example.com
resolvectl status                         # what systemd-resolved is actually using
cat /etc/resolv.conf
grep hosts /etc/nsswitch.conf

Read the status in dig's output — it is the diagnosis:

• NOERROR with an answer — DNS is fine, look elsewhere.
• NXDOMAIN — the name genuinely does not exist. A typo, or a record that was never created.
• SERVFAIL — the resolver failed. Often DNSSEC validation failing, or an upstream problem.
• REFUSED — the server will not answer you. Usually an ACL.
• No response at all — the resolver is unreachable; check port 53 and the firewall.

dig @8.8.8.8 versus dig alone is the fastest split there is. If the public resolver answers and yours does not, the problem is your resolver. If neither answers, the record is wrong or missing.

Three specifics worth carrying:

/etc/resolv.conf is generated on most modern systems — by NetworkManager or systemd-resolved — so editing it works until the next DHCP renewal silently reverts it. Configure the resolver where it is actually managed.

/etc/hosts wins, because nsswitch.conf lists files before dns. A stale entry there overrides correct DNS and is invisible to every dig you run — getent hosts example.com is the command that shows what the system resolves, as opposed to what DNS says.

A short TTL is your friend during a migration, and a long one means a correct change takes hours to be visible. "I changed the record and it still goes to the old server" is usually caching, not a mistake.

Routing issues and the gateway

ip route show
ip route get 8.8.8.8              # exactly how this packet would be sent
traceroute -n 8.8.8.8
ip route add default via 192.168.1.1 dev eth0
sysctl net.ipv4.ip_forward        # only relevant if this host ROUTES

The failures:

No default route. Local traffic works, everything else fails with "Network is unreachable" — which is a distinct message from a timeout and tells you immediately.

The gateway is not on the interface's subnet, so it cannot be reached to be used. This follows directly from a wrong prefix length.

Two default routes with equal metrics, from two interfaces both taking DHCP. Traffic leaves by whichever wins, which may be the wrong one, and behaviour changes after a reboot. Set metrics deliberately.

Asymmetric routing — the reply comes back by a different path and is dropped by a stateful firewall that never saw the outbound packet. It presents as intermittent, which makes it one of the harder ones.

ip route get is the command to trust, because it reports the kernel's actual decision rather than your reading of the table.

IP conflicts

Two hosts with the same address. Intermittent connectivity, connections that work then fail, and ARP entries that change:

arping -D -I eth0 192.168.1.50    # duplicate address detection
ip neigh show | grep 192.168.1.50
journalctl -k | grep -i "duplicate address"
arp-scan --interface=eth0 --localnet

The kernel logs "duplicate address detected", which is conclusive when it appears. Otherwise the tell is a MAC address for the disputed IP that changes between checks.

Causes: a static address inside the DHCP pool, two machines cloned without changing configuration, or a device reconnected after its lease was reassigned. The fix is to keep static assignments outside the DHCP range, or to use DHCP reservations instead of static configuration.

MAC spoofing

Media access control spoofing is a host presenting a MAC address that is not its own. It has legitimate uses — failover pairs, licence-bound appliances, privacy randomisation on wireless — and illegitimate ones, such as bypassing MAC filtering or impersonating a gateway.

ip link show eth0                                    # "permaddr" reveals the real one
ip link set eth0 address 02:11:22:33:44:55
macchanger -r eth0
ethtool -P eth0                                      # permanent hardware address

Two ways it appears as a problem. Wireless MAC randomisation breaks MAC-based DHCP reservations and captive portals, because the device presents a different address each time — that is now default behaviour on phones and laptops, and it surprises people running MAC filtering.

And duplicate MACs on a switched network cause the switch's forwarding table to flap between two ports, producing intermittent loss for both hosts. That is the same symptom class as an IP conflict, one layer down.

The defences are worth naming: switch port security, dynamic ARP inspection and 802.1X. And the general point — MAC addresses are trivially changed, so MAC filtering is not a security control. It stops accidents, not attackers.

MTU mismatch

The maximum transmission unit is the largest frame an interface will send; 1500 bytes is standard Ethernet, 9000 is jumbo frames, and a tunnel or VPN reduces the usable size by its own header.

The signature is memorable: small packets work and large ones do not. SSH connects and then hangs when output scrolls. A web page loads its HTML and stalls on an image. ping succeeds and scp freezes.

ip link show eth0 | grep mtu
ping -M do -s 1472 8.8.8.8        # 1472 + 28 bytes of header = 1500
ping -M do -s 1400 8.8.8.8        # binary search downward until it succeeds
tracepath 8.8.8.8                 # reports the path MTU directly
ip link set eth0 mtu 1400

-M do sets "do not fragment", so an oversized packet is dropped rather than split — which is what makes the test conclusive. Work down until it passes and add 28 for the headers to get the real path MTU.

The underlying cause is usually PMTU discovery broken by a firewall blocking ICMP. Path MTU discovery works by receiving "fragmentation needed" ICMP messages; a firewall that blocks all ICMP as a "hardening" measure destroys it, and the connection hangs instead of adjusting. That is the reason not to block ICMP wholesale, and it is a recurring exam point.

Jumbo frames must be set consistently on every device in the path — hosts, switches, and any router. One device at 1500 in a path otherwise at 9000 produces exactly this failure, intermittently.

Bonding

Bonding aggregates several interfaces for redundancy or throughput.

cat /proc/net/bonding/bond0       # mode, active slave, per-slave link status
ip link show master bond0
nmcli connection add type bond ifname bond0 bond.options "mode=802.3ad,miimon=100"
nmcli connection add type ethernet ifname eth0 master bond0

The modes to know:

• active-backup (mode 1) — one link carries traffic, another takes over on failure. Needs no switch support at all, which makes it the safe default.
• 802.3ad / LACP (mode 4) — true link aggregation, more throughput, and requires the switch to be configured for it too.
• balance-rr (mode 0) — round robin; can reorder packets and upset TCP.

/proc/net/bonding/bond0 is the one file to read. It shows the mode, which slave is active, and each slave's link state.

The failures: LACP configured on the host and not on the switch (or on the wrong ports), which typically breaks the link entirely rather than degrading gracefully; a bond that has silently lost a member and is running with no redundancy, because nothing alerted; and miimon left at 0, so link failure is never detected and the bond keeps sending into a dead link.

Test failover deliberately, by unplugging one cable. An untested bond is a bond that fails over at the worst possible moment or not at all.

Dual stack issues

Dual stack means running IPv4 and IPv6 together, and the problems come from the two disagreeing.

ip -6 addr show
ip -6 route show
ping6 -c3 2001:4860:4860::8888
dig AAAA example.com
curl -4 https://example.com       # force IPv4
curl -6 https://example.com       # force IPv6
getent ahosts example.com         # what order will the system TRY?

The classic symptom: a host is slow to connect to some sites and fine with others. The name resolves to both an A and a AAAA record; the client prefers IPv6 by default; IPv6 is configured but not actually working; and every connection waits for an IPv6 timeout before falling back to IPv4.

curl -4 succeeding instantly while curl -6 hangs is the confirmation, and it takes ten seconds to run.

The right fix is to make IPv6 work or to disable it properly — not to leave it half-configured, which is the worst of both. "Half-configured IPv6" specifically means a link-local address and no working route, which is what a host has when someone disabled IPv6 in one place and not another.

Also worth knowing: firewall rules must be written for both stacks. iptables does not filter IPv6 — that is ip6tables — so a service correctly firewalled on IPv4 can be wide open on IPv6. nftables and firewalld handle both together, which is one of the better reasons to use them.

On the exam

• Work up the layers: link → address → route → gateway → internet by IP → DNS → application. The first failure is the problem.
• ping 8.8.8.8 works and ping google.com does not = DNS.
• NO-CARRIER is a physical problem; state DOWN is administrative.
• A duplex mismatch works and is dreadful under load — set both ends to autonegotiate.
• Refused is fast (nothing listening / reject); timed out is slow (dropped). Check the listening address with ss -tlnp — 127.0.0.1 is unreachable from elsewhere.
• firewalld: --permanent plus --reload, and check the zone. First matching rule wins in nftables/iptables.
• 169.254.x.x means no DHCP server answered.
• dig status: NXDOMAIN does not exist, SERVFAIL the resolver failed, REFUSED an ACL. dig @8.8.8.8 splits "your resolver" from "the record". /etc/hosts beats DNS; /etc/resolv.conf is generated.
• No default route gives "Network is unreachable"; the gateway must be inside the interface's own subnet.
• IP conflict: kernel logs a duplicate address; the MAC for that IP changes.
• MAC filtering is not security — addresses are trivially changed. Wireless randomisation breaks MAC-based reservations.
• MTU mismatch: small packets work, large ones hang. Test with ping -M do -s, and remember blocking ICMP breaks PMTU discovery.
• Bonding: /proc/net/bonding/bond0 is the file. active-backup needs no switch support; LACP requires switch configuration.
• Dual stack: connections that stall then succeed are IPv6 timing out and falling back. Firewall both stacks — iptables does not cover IPv6.