Hypervisors, hosts, guests and resource allocation

Why this matters

Most servers in service today are virtual machines. One physical server, the host, runs many virtual servers, the guests, each believing it has a machine to itself. Virtualisation made it cheap to give every role its own server, as the roles lesson recommended, and made servers easy to move, copy and restore.

It also creates problems that do not exist on physical hardware. Guests compete for the same processors, memory and disks, and a host that has promised more than it has will slow every guest on it at once. This lesson covers the kinds of hypervisor, how resources are shared out, how guests reach the network, how their disks are provisioned, and the division of responsibility between the host and the guests.

The lesson

Type 1 against type 2 hypervisors

A hypervisor is the software that creates and runs virtual machines, sharing the physical hardware among them.

A type 1 hypervisor, also called bare metal, runs directly on the hardware with no general-purpose operating system underneath. VMware ESXi, Microsoft Hyper-V, KVM on Linux, Xen and Proxmox are examples. Because nothing sits between the hypervisor and the hardware, type 1 hypervisors are efficient, stable and have a small attack surface. They are what servers use.

A type 2 hypervisor, also called hosted, runs as an application on top of an ordinary operating system. VMware Workstation, Oracle VirtualBox and Parallels are examples. They are convenient for desktops and for test labs such as the one built in the first lesson, but every guest depends on the host operating system, its updates and its restarts, and the extra layer costs performance. They are not used for production servers.

Whichever is used, the processor must support hardware virtualisation, Intel VT-x or AMD-V, and it must be enabled in the firmware settings. A hypervisor that refuses to start guests, or reports that virtualisation is unavailable, often needs nothing more than that setting switched on.

vCPUs, memory, and the risk of overcommitting either

Each guest is given virtual processors (vCPUs) and an amount of memory. The hypervisor schedules vCPUs onto the host's physical cores and maps guest memory onto physical memory.

Overcommitting means allocating more to guests in total than the host physically has. It is normal for processors: most guests are idle most of the time, so a host with 16 cores might run guests with 48 vCPUs between them without trouble. Too high a ratio, though, and guests queue for processor time; the guests see this as sluggishness even when their own processor use looks moderate. A related trap is giving a guest too many vCPUs: a guest with eight vCPUs may have to wait until eight cores are free together, so an over-provisioned guest can run slower than a smaller one.

Memory overcommitment is riskier. Hypervisors have techniques for reclaiming memory from guests, such as ballooning, where a driver inside the guest gives memory back, and dynamic memory, which adjusts allocation as demand changes. But if guests actually use more memory than the host has, the hypervisor has to swap guest memory to disk, and every guest on the host slows dramatically. For important servers, memory is usually allocated conservatively or reserved, guaranteeing the guest its full allocation.

Reservations, limits and shares (a relative priority used when resources are contested) let an administrator decide which guests win when the host runs short.

Virtual switches, virtual NICs and VLANs for guests

Each guest has one or more virtual NICs, connected to a virtual switch inside the hypervisor. Virtual switches come in three kinds:

• An external switch is bound to a physical network adapter, connecting guests to the physical network as if they were plugged into it.
• An internal switch connects guests to each other and to the host, but not to the outside network.
• A private switch connects guests only to each other, isolating them even from the host; useful for test networks.

The physical adapters behind an external switch are usually teamed, as the network lessons described, since a failure there would disconnect every guest on the host at once.

To place guests on different VLANs, the host's physical ports are connected to switch trunk ports, and each guest's virtual NIC is given a VLAN ID on the virtual switch. The guest itself needs no VLAN configuration; the virtual switch adds and removes the tags. A guest on the wrong network is usually a guest with the wrong VLAN ID on its virtual NIC, or a VLAN missing from the trunk.

Thin and thick provisioning, and snapshots -- which are not backups

A guest's disk is a file, or set of files, on the host's storage, and it can be provisioned two ways:

• Thick provisioning allocates the full size immediately. A 500 GB disk takes 500 GB of storage from the start. It is predictable and avoids the overhead of growing the file.
• Thin provisioning allocates space only as the guest writes data. A 500 GB disk holding 50 GB of data uses 50 GB. It saves space and allows storage to be overcommitted in the same way as memory, with the same danger: if thin disks grow until the underlying storage is full, every guest on it may stop at once. Thin-provisioned storage must be monitored, with alerts well before it runs out.

A snapshot records a guest's state at a moment so it can be returned to that point, which makes it ideal before a risky change such as a patch or upgrade. It works by freezing the original disk and writing all later changes to a separate delta file.

That is why a snapshot is not a backup. It depends on the original disk, on the same storage, so if that storage fails, the snapshot is lost with it. And snapshots cause problems when they are left in place: the delta file keeps growing, performance falls, and deleting an old snapshot means merging a large delta back in, which can take a long time. Take a snapshot for a change, and remove it once the change is confirmed.

What the host owns and what each guest owns: patching and licensing

Virtualisation splits responsibility between two layers, and both need managing.

The host needs its own firmware, hypervisor updates, drivers and hardware monitoring. Patching it means moving its guests elsewhere first, by live migration to another host in a cluster, or shutting them down in a maintenance window.

Each guest is a full server in its own right, with its own operating system to patch, its own antivirus, backups, monitoring and accounts. Guests also need the hypervisor's integration tools, such as VMware Tools or the Hyper-V integration services, which provide efficient drivers and let the host shut the guest down cleanly; these need updating when the host is.

Licensing follows the same split. The hypervisor is licensed on the host, and each guest's operating system and applications need licences too. Some licences change the arithmetic: Windows Server Datacenter edition, licensed on the host's cores, allows unlimited Windows guests on that host, while Standard edition covers only two. Some applications are licensed per physical core of the host a guest could run on, rather than the guest's own vCPUs, which can make a small guest expensive on a large cluster. Check the terms before building, because moving guests between hosts can take them out of compliance.