Cooling, airflow, and cabling the rack

Why this matters

Every watt a server draws ends up as heat, and a rack full of servers is a very effective heater. Servers are designed to be cooled in one specific way, and a rack that ignores it will run hot, throttle its processors, shorten the life of its drives and eventually shut equipment down to protect it.

Cabling is tied to the same problem. A back of rack tangled with cables blocks the hot air trying to leave, hides which cable goes where, and turns a five-minute drive replacement into an hour of tracing. This lesson covers how air should move through a rack, then the cables that share its space.

The lesson

Front-to-back airflow, and hot aisle against cold aisle

Almost all rack-mount servers use front-to-back airflow: fans draw cool air in through the front panel, pass it over the components, and exhaust hot air out of the back. Network switches increasingly offer the same arrangement, and where a switch blows the other way it needs to be ordered, or fitted with fans, to match the servers around it.

In a room with several racks, this is organised into aisles. Racks are placed in rows, and the rows are turned so that the fronts of two rows face each other across one aisle and their backs face each other across the next. The aisle the fronts face is the cold aisle, where cooled air is delivered. The aisle the backs face is the hot aisle, where exhaust collects and is drawn back to the cooling units.

The point is to stop hot exhaust from one row being pulled into the intakes of the next. Rows that all face the same direction do exactly that: the back of one row blows straight into the front of the row behind it. Larger sites go further with containment, putting doors and a roof over the cold or hot aisle so the two air streams cannot mix at all.

Temperature should be measured where it matters, at the server inlets in the cold aisle, rather than from a single thermostat on the wall.

Blanking panels, and why one empty unit can overheat a rack

Inside a rack, the same principle applies on a smaller scale. Hot air leaving the back of the servers will take any path back to the front, and an empty rack unit is an open path. Exhaust flows through the gap, mixes with the cool air at the front, and gets pulled straight back into the intakes of the servers beside it.

Blanking panels close that path. They are plain covers, often tool-less, fitted over every unused rack unit at the front of the rack. They are cheap, they look like decoration, and removing them is a common cause of servers that run hot for no obvious reason: the equipment is identical to its neighbours, the room is cool, and a gap two units above it is feeding it recycled exhaust.

The same thinking applies to anything that blocks the back of the rack. Dense cable bundles hanging over server exhaust vents trap the heat the fans are trying to push out.

Copper and fibre, connector types, and SFP, SFP+ and QSFP transceivers

Two families of cable connect servers to the network.

Copper twisted pair uses RJ45 connectors and is rated by category. Cat5e and Cat6 handle gigabit Ethernet to 100 metres; Cat6a carries 10 gigabit Ethernet (10GBASE-T) the full 100 metres. Copper is cheap and familiar, but it is limited in distance and sensitive to electrical interference.

Fibre optic cable carries light. It is immune to electrical interference and supports much longer distances and higher speeds.

• Multimode fibre is used for shorter runs within a building or data centre and uses cheaper optics.
• Single-mode fibre carries signals much further, across a campus or between buildings, at a higher optics cost.

The two are not interchangeable: the transceivers at each end must match the fibre type. The common fibre connectors are LC, small and the default in modern equipment, and SC, a larger square push-pull connector found on older equipment.

Higher-speed ports on servers and switches rarely take a cable directly. Instead they have a slot for a pluggable transceiver that converts the electrical signal to whatever the cable needs:

• SFP: 1 Gbps.
• SFP+: 10 Gbps, in the same physical size.
• QSFP+: 40 Gbps, and QSFP28 at 100 Gbps, in a larger form.

For short runs inside a rack, a direct attach copper (DAC) cable, which is a twinax cable with the transceivers moulded onto each end, is cheaper than optics and fibre. Vendors often restrict which transceivers their switches and network cards accept, so check compatibility before ordering; an unsupported transceiver may simply stay dark.

Cable management, labelling, and a service loop you can pull a server on

Good cabling is organised so that any single cable can be found, traced and replaced without disturbing the rest.

• Route cables through cable managers: horizontal managers between devices and vertical managers at the sides of the rack.
• Bundle with hook-and-loop straps, not tight cable ties. Ties pulled tight crush cables, damage fibre and have to be cut to change anything.
• Label both ends of every cable, so that the port it leaves and the port it reaches are recorded at each end.
• Use colour consistently, for example one colour per network or purpose, so the purpose of a cable is visible at a glance.

Servers on sliding rails need slack to move. A cable management arm at the back of the server folds the cables into a hinged arm that extends as the server slides forward, so the server can be pulled out for service while still connected and running. The slack it holds is the service loop. Too little slack and the cables are pulled out of their ports as the server moves; too much and the excess clutters the back of the rack and blocks airflow.

Measure cable lengths to suit the rack rather than buying long cables for convenience. The excess has to go somewhere, and it usually ends up draped across exhaust vents.

Redundant network paths, and keeping power and data runs apart

A server with two network ports should use them to remove single points of failure. Connect the two ports to two different switches, so that a switch failure or a switch firmware update leaves one path up. The two ports are then combined on the server, which the lesson on IP addressing and NIC teaming later in the course covers. Two ports cabled to the same switch protect against a failed cable or port, but not against losing the switch.

Keep power and data cables apart. Power cables can induce interference in nearby copper data cables, which shows up as errors and retransmissions that are hard to trace. A common layout runs power down one side of the rack and data down the other, and where the two must cross, they cross at right angles rather than running alongside each other. Fibre is immune to this interference, which is one more reason it is preferred for longer and busier runs.