Out-of-band management: iLO, iDRAC, IPMI and Redfish

Why this matters

Most server management happens in-band: you connect to the running operating system over the network and manage it with remote desktop, SSH or a management tool. That only works while the operating system is running and reachable. When a server has crashed, will not boot, is powered off, or has lost its network configuration, in-band management has nothing to talk to.

Out-of-band management is the way in that does not depend on the operating system at all. It is what lets an administrator in another building, or another country, see a crashed server's screen, power it back on, or install an operating system on it from scratch. For a server that lives in a remote data centre, it is the difference between a remote fix and a drive to site.

The lesson

What a baseboard management controller is, and why it works with the OS down

Out-of-band management is provided by a baseboard management controller (BMC): a small, separate computer built into the server's motherboard, with its own processor, memory, firmware and, usually, its own network port.

The BMC runs from the server's standby power, which is present whenever the server is plugged in, even when the server itself is switched off. So the BMC is up when the operating system has crashed, when the server is powered off, and before any operating system has been installed. It watches the hardware sensors continuously, and it can control the server's power and see its console.

Each major vendor has its own BMC product and name:

• HPE: iLO (Integrated Lights-Out).
• Dell: iDRAC (integrated Dell Remote Access Controller).
• Lenovo: XClarity Controller (XCC).
• Many other manufacturers, including Supermicro, simply call it the BMC or refer to it by the IPMI standard it implements.

Many BMCs can use either a dedicated management port or share one of the server's ordinary network ports. A dedicated port is the better choice, for reasons covered below.

Remote console, virtual media and remote power control

Three BMC features do most of the practical work.

The remote console, often called KVM over IP, shows the server's actual screen in a browser window and passes keyboard and mouse input to it. Because it captures the video output directly, it shows everything from the firmware setup screens and boot messages to an operating system's crash screen. This is what replaces standing at the rack's KVM drawer.

Virtual media lets an administrator attach a file on their own computer, such as an operating system installation ISO, to the server as if it were a DVD or USB drive plugged into it. Combined with the remote console, it makes it possible to install or reinstall an operating system on a server that nobody is standing next to.

Remote power control lets the BMC power the server on, off, reset it, or power-cycle it. A graceful shutdown asks the operating system to shut down; a forced power-off cuts power immediately, which is the equivalent of pulling the plug and should be the last resort.

BMCs also send alerts, typically by email or SNMP traps, when a sensor crosses a threshold or a component fails, and most can update the server's own firmware.

IPMI against Redfish, and querying a server's health from outside it

Two standards define how software talks to a BMC.

IPMI, the Intelligent Platform Management Interface, is the older standard, supported by almost every server BMC. It works over UDP port 623, and the usual command-line tool is ipmitool, which can read sensors, control power and read the event log. IPMI also provides serial over LAN, which carries a server's serial console over the network. IPMI's weakness is security: its design has well-known flaws, including authentication weaknesses that can leak password hashes, so it should never be reachable from an untrusted network.

Redfish is the modern standard, maintained by the DMTF. It is a RESTful API served over HTTPS that returns JSON, so it is protected by TLS and easy to use from scripts and management tools. Every Redfish service starts at the same root, /redfish/v1/, and organises the server into collections such as Systems, Chassis and Managers. A request such as a GET to a system's resource returns its power state, model and health; the chassis resources report temperatures, fans and power supplies.

The Redfish mockup server from the lab lesson at the start of this course serves exactly this structure, so the same requests you practise against it work against a real iLO, iDRAC or XClarity Controller. Where you have the choice, prefer Redfish over IPMI for new automation.

Default credentials, and the management port on its own network

A BMC can power off the server, change its firmware settings, mount media and see its console. Anyone who controls the BMC controls the server, whatever the operating system's security looks like. So the BMC must be secured deliberately:

• Change default credentials before the server goes into production. Some older equipment shipped with well-known default usernames and passwords; many vendors now print a unique password on a tag on each server. Either way, set strong credentials and use individual accounts rather than one shared administrator login.
• Put management ports on a separate management network or VLAN, reachable only from administrators' systems and never from the internet. This is the main reason to prefer a dedicated management port over a shared one.
• Keep BMC firmware up to date, because BMC vulnerabilities are real and are patched through firmware updates.
• Disable what you do not use, such as IPMI over LAN when Redfish or the web interface is enough, and use role-based accounts so that monitoring tools get read-only access.

Reading the hardware event log when the operating system cannot

The BMC keeps its own record of hardware events, called the System Event Log (SEL). It records events such as memory errors, power supply failures, fan failures, temperature thresholds being crossed and chassis intrusion, with timestamps, whether or not an operating system is running.

That independence is what makes it valuable. When a server rebooted overnight and the operating system logs simply stop, the SEL may show a power supply failure or a critical temperature at the moment it went down. When a server will not boot at all, the SEL is often the only record of why.

Vendors present it in their own interfaces, such as HPE's Integrated Management Log and Dell's Lifecycle Controller log, and it can be read with ipmitool or through Redfish's log services. Two habits make it useful: compare its timestamps with the operating system's logs to line up what the hardware saw with what the software saw, and save a copy before clearing a full log, so the history is not lost.