Hardware connectivity: cables, NICs and transceivers

Why this matters

A server that cannot reach the network is, for most purposes, a server that is down. Network faults are often blamed on configuration, such as addresses, DNS or VLANs, and later lessons in this domain cover those. But a large share of connectivity problems are physical: a damaged cable, a transceiver of the wrong type, a failing network adapter, or two ends that disagree about speed.

Physical faults are worth ruling out early, because they are quick to check and because no amount of configuration will fix a broken cable. This lesson covers reading link lights, cable and transceiver faults, adapter failures and teaming, speed and duplex mismatches, and the simple tests that isolate the faulty part.

The lesson

What link lights tell you

Network ports on servers and switches have LEDs that give the first and fastest information about a physical connection. Their exact meaning varies between manufacturers, so check the documentation, but the common pattern is:

• A link light shows that the two ends have established a physical connection. If it is off at both ends, there is no physical connection: an unplugged or broken cable, a failed port, a transceiver problem, or a port that has been disabled.
• An activity light flickers when traffic passes. A link with no activity at all on a busy server suggests the connection is up but traffic is not flowing, pointing towards configuration rather than cabling.
• Colour often indicates speed: for example, one colour for 10 Gbps and another for 1 Gbps. A port showing a lower speed than expected is a clue to a cable or negotiation problem.
• Amber or blinking patterns can indicate faults or errors.

Check both ends: the server's port and the switch port. A link light on at one end and off at the other points to a problem with one port or transceiver.

Link lights can also be read remotely. The operating system reports link state, the management controller may report adapter status, and the switch shows each port's state, speed and error counters.

Bad cables, wrong transceivers and mismatched fibre

Copper cables fail through damage: crushed or tightly bent cables, broken clips, damaged connectors, and cables pulled too hard during installation. The cable category must also suit the speed: 10 Gbps over copper needs at least Category 6 for short runs, or 6A for the full 100 metres. A cable that is marginal may give a link at a lower speed, or a link with errors.

Transceivers, the pluggable modules such as SFP+, SFP28 and QSFP used in server and switch ports, are a frequent source of problems:

• Wrong type: transceivers are made for particular speeds, media and distances. A short-range multimode module will not work over single-mode fibre, and a 1 Gbps module will not run at 10 Gbps.
• Compatibility: many switches and adapters accept only transceivers approved by their manufacturer, and disable or warn about others.
• Both ends must match: the transceivers at the two ends must use the same standard and wavelength.

Fibre has its own faults:

• Mismatched fibre: single-mode and multimode fibre are not interchangeable with each other's transceivers.
• Swapped strands: fibre links use a transmit and a receive strand, and each end's transmit must reach the other's receive. Crossed strands give no link.
• Dirty connectors: dust on a fibre end face blocks light, and is one of the most common causes of fibre faults. Connectors should be inspected and cleaned, and dust caps kept on unused ports and cables.
• Bends: fibre bent more tightly than its specified radius loses light.

Many transceivers report digital optical monitoring data, such as transmit and receive light levels, which the switch can show. Low receive power points to a dirty connector, a damaged cable, or the wrong fibre.

A failed NIC, and teaming failover

A network adapter can fail outright or partly, and can also be disabled or lose its driver.

Signs of a failed or failing adapter:

• the adapter disappears from the operating system, or shows as failed in Device Manager or in the ip link output;
• the link stays down even with a known-good cable and switch port;
• errors in the system log from the adapter's driver;
• intermittent drops, or rising error counts on the port.

Before replacing hardware, check the simple software causes: an adapter disabled in the operating system or firmware, a missing or outdated driver, or firmware that needs updating, a common fix for adapter problems, as the firmware lesson described.

On servers with NIC teaming, as the addressing lesson covered, a failed adapter should cause a failover to the remaining adapter, with the server staying connected. Two points matter. First, check that failover actually happened and that the team reports a failed member, and make sure this raises an alert, since a team that has quietly lost one member has lost its redundancy. Second, if both team members are connected to the same switch, a switch failure takes both down, which is why team members should connect to different switches.

Speed and duplex mismatches

Ethernet ports normally auto-negotiate speed and duplex with the device at the other end, agreeing on the fastest common speed and full duplex, sending and receiving at the same time.

Problems appear when negotiation fails or when one end is set manually and the other is not:

• Speed mismatch: usually, no link at all, or a link at a lower speed than expected, such as 1 Gbps on a 10 Gbps port. A cable or transceiver problem can also cause a link to negotiate a lower speed.
• Duplex mismatch: one end at full duplex and the other at half duplex. The link comes up and appears to work, but performance is poor, especially under load, and the ports log collisions, including late collisions, and errors such as CRC errors. This is a classic cause of a connection that works but is mysteriously slow.

The usual rule is to leave both ends on auto-negotiation. If one end must be set manually, set the other to the identical values. Setting one end manually while the other auto-negotiates is exactly how duplex mismatches are created, since the auto-negotiating end can detect speed but falls back to half duplex.

Check the negotiated speed and duplex at both ends: the operating system's adapter status, ethtool on Linux, and the switch port's status.

Testing with a loopback or a known-good cable

Physical troubleshooting is largely isolation: finding which component in the chain of adapter, transceiver, cable, patch panel and switch port has failed.

• Swap in a known-good part. Replace the cable with one known to work, try a different switch port, or swap the transceiver. This is often the fastest test of all, and each swap removes one suspect.
• Cable testers check copper cables for continuity, wiring faults and length; certifiers confirm a cable meets its category's performance. For fibre, a light meter measures light levels, and an inspection scope shows dirty or damaged end faces.
• Loopback tests check a port by itself. A loopback plug connects a port's transmit to its receive, so the port receives what it sends: if it passes, the port and adapter work, and the fault lies elsewhere. Fibre loopbacks do the same for fibre ports. The software loopback address, 127.0.0.1, tests the network stack in the operating system, not the hardware.
• Switch counters show errors on the port: CRC errors and other input errors usually point to the cable or transceiver.

Label cables and document which port connects where, as the cabling lesson recommended. It turns "which cable is this?" from an investigation into a lookup.