Hypervisors, virtual machines and containers, and what each one does

Why this matters

Very few servers in a modern data centre run one operating system on one box. A single physical server usually hosts dozens of virtual machines, and many applications now ship as containers. A junior network engineer who patches a cable into that server is not connecting one host; they are connecting every VM on it, in several VLANs, through a software switch they cannot see from the Catalyst console. Routers and firewalls themselves are increasingly software running on the same servers.

Without a clear picture of what a hypervisor does, the network side of virtualisation goes wrong in predictable ways. The server port is configured as an access port, and every VM outside one VLAN goes dark. A VM moves to another host during maintenance and someone opens a ticket because its MAC address "jumped" ports. A team tries to run a Windows application in a Linux container and wonders why it will not start. Each of these is a misunderstanding of the layers, not a hard technical problem.

The lesson

What hypervisors, virtual machines and containers are for

Virtualisation separates software from the hardware it runs on. There are two main ways to do it, and the CCNA expects you to tell them apart.

• A hypervisor creates virtual machines (VMs). Each VM is a complete computer in software: a guest operating system, with its own kernel and applications, running on virtual hardware.
• A container runtime creates containers. Each container is an application packaged with its libraries, running as isolated processes on the host's existing operating system kernel.

The difference is where the line is drawn. A VM virtualises the hardware. A container virtualises the operating system.

Hypervisors: Type 1 and Type 2

A hypervisor sits between physical hardware and the VMs. Its job has three parts:

1. Abstract the hardware. It presents each VM with virtual CPUs (vCPUs), memory, virtual disks and virtual NICs. The guest operating system thinks it has its own machine.
2. Share the hardware. When two VMs both need processor time at once, the hypervisor schedules their vCPUs onto the physical cores in turn. Both keep running. The guest operating system has no control over the physical CPUs; it only sees its virtual ones.
3. Isolate the VMs. One VM cannot read another's memory or disk, and a crash in one guest does not take down the others.

There are two kinds of hypervisor:

A Type 1 hypervisor is installed in place of an operating system. It boots on the hardware and runs the VMs itself. Hyper-V counts as Type 1 even when you enable it as a Windows feature, because it then loads beneath Windows, which becomes one more partition on top of it. A Type 2 hypervisor is a normal application. If you install VirtualBox on a Windows laptop to run a Linux lab VM, Windows is the host operating system and VirtualBox is a Type 2 hypervisor running on it.

Overcommitment is a normal practice. A host with 32 physical cores can run VMs that have been given 64 vCPUs in total, because most VMs are idle much of the time and the hypervisor schedules the busy ones onto the cores. This is CPU overcommitment. Memory can be overcommitted in a similar way, with more care. Do not confuse it with hyper-threading, which is a processor feature that exposes two hardware threads per core, or with CPU pinning, which ties a VM's vCPUs to specific cores so that they always run there. Passthrough gives a VM direct control of a physical device, such as a NIC, bypassing the virtual hardware for performance.

Virtual machines

A virtual machine is a guest operating system running on virtual hardware. Because each VM carries its own kernel, VMs on the same host can run different operating systems. One VM runs Windows Server, the next runs Linux, both on the same Type 1 hypervisor. That is the answer whenever a scenario needs different operating systems on one server.

VMs are also strongly isolated. A flaw in one guest kernel does not expose the others, because they do not share it; the hypervisor sits between them.

The cost is size and start-up time. Each VM stores a full operating system, typically gigabytes, and starting one means booting that operating system.

A VM is stored as files: a configuration file and one or more virtual disk files. That makes it easy to copy, snapshot and move. Live migration moves a running VM from one host to another with no shutdown. The VM keeps its MAC and IP addresses, so its sessions survive. That matters for the network, as you will see below.

Containers

A container packages an application with the libraries and dependencies it needs, but not a kernel. Containers on a host all share the host operating system kernel. The kernel keeps them apart with two features:

• Namespaces give each container its own view of processes, network interfaces, file systems and host names.
• Control groups (cgroups) limit how much CPU and memory each container can use.

Because nothing has to boot, starting a container means starting an application process on a kernel that is already running. Containers start in seconds or less and are often only megabytes in size. That is why teams that deploy many small services use them.

The trade-offs follow from the shared kernel:

• Isolation is weaker than a VM's. A flaw in the shared kernel can affect every container on the host, not just one.
• The operating system family is fixed. Linux containers need a Linux kernel. You cannot run an application that needs the Windows kernel in a Linux container. Containers need no hypervisor at all, although in practice they often run inside a VM.

Docker Engine is a common container runtime. A container image is the packaged, read-only template; a container is a running instance of an image. Images are stored in and pulled from a registry.

When an organisation runs hundreds of containers across many hosts, it uses an orchestration platform. Kubernetes is the best known. It schedules containers across a cluster of hosts, restarts failed ones and scales them up or down with demand. It is not a hypervisor and does not build VMs; the containers it manages still run on each host's operating system.

How virtual hosts reach the network

Inside the hypervisor is a virtual switch. It connects the virtual NICs of the VMs on that host to each other and to the host's physical network adapters. The guest operating system sees only its own virtual NIC and cannot switch traffic for other VMs. The virtual switch does that job, and it can place each VM in a VLAN and tag its frames on the way out of the physical uplink. Containers have the same idea on a smaller scale: each gets a virtual interface connected to a software bridge on the host.

So a host running VMs in VLANs 10, 20 and 30 through one physical uplink needs that physical switch port to carry all three VLANs, tagged. That means an 802.1Q trunk, not an access port (lesson 8 configures trunks in full, and lesson 9 covers edge ports for virtualised hosts):

SW1(config)#interface GigabitEthernet1/0/20
SW1(config-if)#description ESXi-host-01 vmnic0
SW1(config-if)#switchport mode trunk
SW1(config-if)#switchport trunk allowed vlan 10,20,30
SW1(config-if)#spanning-tree portfast trunk
SW1(config-if)#end
SW1#show interfaces GigabitEthernet1/0/20 trunk

Port        Mode             Encapsulation  Status        Native vlan
Gi1/0/20    on               802.1q         trunking      1

Port        Vlans allowed on trunk
Gi1/0/20    10,20,30

Port        Vlans allowed and active in management domain
Gi1/0/20    10,20,30

If the port were left as an access port in VLAN 10, it would carry only untagged VLAN 10 traffic. The VMs in VLANs 20 and 30 would be cut off, while the VLAN 10 VMs would work, which is a confusing half-failure.

Live migration shows up in the MAC address table. Before the move, the switch has learned the VM's MAC on host A's port:

SW1#show mac address-table address 0050.5689.1a2b
          Mac Address Table
-------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  20    0050.5689.1a2b    DYNAMIC     Gi1/0/20
Total Mac Addresses for this criterion: 1

After the VM moves to host B on Gi1/0/21, the hypervisor sends a frame on the VM's behalf from its new location. The switches see the same source MAC arrive on a new port and update their tables:

SW1#show mac address-table address 0050.5689.1a2b
          Mac Address Table
-------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  20    0050.5689.1a2b    DYNAMIC     Gi1/0/21
Total Mac Addresses for this criterion: 1

No new MAC address is assigned, spanning tree does not need to reconverge, and the default gateway does not change. The VM's address stays the same and the network simply learns where it now lives. Host B's port must allow VLAN 20, or the VM arrives on a host whose uplink cannot carry its traffic.

Network functions as virtual machines

The same idea applies to network devices. Network functions virtualization (NFV) replaces dedicated appliances, such as routers and firewalls, with software instances running on standard x86 servers. Each software instance is a virtual network function (VNF), and it can run as a VM or as a container.

A Cisco example is the Catalyst 8000V, a router delivered as a virtual machine. A company can run Catalyst 8000V instances on its VMware ESXi servers or in a public cloud. In that design ESXi is the Type 1 hypervisor and the Catalyst 8000V is a virtual router running as one of its VMs. It runs IOS XE, so you configure it with the same commands as a physical router.

R-VIRT1#show version | include Software|processor
Cisco IOS XE Software, Version 17.12.04
Cisco IOS Software [Dublin], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 17.12.4
cisco C8000V (VXE) processor (revision VXE) with 2028423K/3075K bytes of memory.

NFV is not the same thing as a software-defined campus fabric, which is a way of designing and controlling a whole campus network. NFV is about where a network function runs.

What the exam is really asking

• Type 1 versus Type 2. Type 1 runs on the hardware (ESXi, Hyper-V, KVM). Type 2 runs as an application on an operating system (VirtualBox, VMware Workstation). Docker Engine is neither; it runs containers.
• What is shared. Containers share the host kernel. VMs each have their own kernel. Any option saying "a VM shares the host kernel" or "a container includes its own kernel" has the two reversed.
• Isolation. VMs isolate more strongly, because of separate kernels. Containers rely on namespaces and cgroups.
• Speed. Containers start faster because they do not boot an operating system, not because the hardware is faster or the network stack is skipped.
• Mixed operating systems. Windows and Linux on one server means two VMs, not containers.
• The uplink to a virtual host. Several VLANs through one physical port means an 802.1Q trunk.
• Live migration. Same MAC, same IP; the switches relearn the MAC on the new port.
• Kubernetes orchestrates and scales containers. It is not a hypervisor.
• NFV and the Catalyst 8000V. A router or firewall as software on standard servers is NFV; the Catalyst 8000V is a virtual router running as a VM.
• More vCPUs than cores is overcommitment, not hyper-threading.