Behind the Millisecond Boot: Hardware Virtualization and KVM

Vantisso spins up a Firecracker MicroVM for every agent task and, when the task is done, reclaims everything down to the disk and the network. Boot times of 200ms for a cold snapshot restore, or 2ms with a warm pool, are not magic. They are possible because of the hardware virtualization built into modern CPUs, and KVM, which makes that hardware usable from Linux. This post looks at the infrastructure software that underpins how Vantisso runs.

The core problem of virtualization: how do you run a guest safely?

At its heart, a virtual machine is about running several operating systems at once, isolated from one another, on a single physical machine. And that turns out to be fundamentally hard.

Operating systems were designed to control the hardware exclusively. The kernel runs at the CPU's highest privilege level, performing sensitive work such as memory management, interrupt handling, and I/O device control directly. The CPU distinguishes these privilege levels using the notion of rings: traditional x86 has four levels, from Ring 0 (highest privilege) to Ring 3 (lowest), with the kernel running in Ring 0 and ordinary applications in Ring 3.

The problem is the guest OS inside a virtual machine. That guest OS also believes it controls the hardware directly, and it tries to execute Ring 0 privileged instructions. But if you let each guest OS drive the real hardware freely, multiple guests end up colliding over the same hardware resources, and the isolation you were after falls apart. To arbitrate, the hypervisor has to be the single point of control over the hardware. So how do you let a guest OS keep believing it controls the hardware while, in reality, it stays safely isolated under the hypervisor's control?

The limits of software virtualization

Before hardware virtualization arrived, this was solved in software.

• Binary translation — the hypervisor inspects the guest OS's instructions before they run and swaps dangerous privileged instructions for safe ones. It works correctly, but because every instruction stream from the guest has to be inspected and translated, performance inevitably suffers.
• Paravirtualization — the guest OS is modified to be virtualization-aware. Performance improves, but at the cost of having to change the guest OS's source code to support virtualization.

The millisecond boot and dozens of concurrent VMs that Vantisso aims for are hard to guarantee with these software approaches. That is why hardware virtualization is effectively the only answer.

Hardware virtualization: root mode and non-root mode

Modern CPUs ship with features dedicated to virtualization: Intel's VT-x and AMD's AMD-V. Because they add virtualization-specific instructions and execution modes on top of the existing x86 instruction set, they are commonly called virtualization extensions. On Linux you can confirm they exist through CPU flags, shown as vmx on Intel and svm on AMD.

On an architecture with these extensions, the CPU runs in two modes that are separate from the traditional Ring 0–3 structure:

• Root mode — the mode the hypervisor runs in. It actually controls the hardware.
• Non-root mode — the mode the guest runs in. The guest OS operates inside it.

The key thing to notice is that the mode distinction (root/non-root) and the privilege distinction (Ring 0–3) coexist without interfering with each other. As a result, Rings 0–3 still exist inside non-root mode too, so the guest OS's kernel can run in Ring 0 of non-root mode exactly as intended, with no modification at all. The guest OS believes it is controlling the hardware from Ring 0, when in fact it is running inside non-root mode, under the hypervisor's authority. As the CPU takes over the work that software virtualization once did laboriously through binary translation, operational complexity drops dramatically and performance stays close to bare metal.

As the diagram shows, Ring 0 and Ring 3 both exist inside root mode as well. The host kernel and the KVM module control the hardware directly from Ring 0 of root mode, while Firecracker, acting as the VMM, runs in Ring 3 of root mode and sends requests to KVM in Ring 0 through /dev/kvm.

Handing off control: VM Exit and VM Entry

So what happens when a guest OS, after doing nothing but internal computation in non-root mode, tries to perform a sensitive operation that actually touches the hardware, such as I/O, interrupt handling, or certain privileged instructions? At that point the CPU automatically halts the guest OS and hands control to the hypervisor. That transition from non-root mode to root mode is a VM Exit. Once the hypervisor has handled the event on the guest's behalf, a VM Entry returns control to the guest OS, which resumes from exactly where it stopped.

This VM Exit/Entry mechanism is the most important part of hardware virtualization. For ordinary computation the guest runs directly; the hypervisor steps in only at the moments that require intervention, like I/O, interrupt handling, or privileged instructions. Because there is no need to inspect every instruction the way software virtualization does, applications running on top of the guest OS reach near–bare-metal speed.

At these transitions, the guest OS's state (registers, control information, and so on) is saved in a dedicated control structure managed by the CPU — Intel calls it the VMCS (VM Control Structure) and AMD the VMCB (VM Control Block). On every VM Exit and VM Entry, the CPU uses this structure to save and restore the guest state. Compared with a snapshot, which writes the guest's entire state to disk, this is the CPU handling guest state at a far lower level, on every single transition.

KVM: the technology that turns Linux into a hypervisor

Most modern CPUs support hardware virtualization, but using it from an operating system requires system software that drives those features. On Linux, that role belongs to KVM (Kernel-based Virtual Machine). KVM is a module built into the Linux kernel that abstracts the CPU's hardware virtualization features (VT-x/AMD-V) and exposes an interface through which user-space programs can create and run virtual machines.

Once KVM is loaded, the Linux kernel itself becomes a hypervisor. Instead of building a separate hypervisor operating system, this design keeps the already battle-tested Linux scheduler, memory management, and device drivers, and simply layers virtualization on top through the KVM module. In this setup, each virtual machine is treated like an ordinary process from Linux's point of view, and the Linux scheduler schedules the VM just like any other process.

As the diagram shows, Firecracker is a user-space program (a VMM, or Virtual Machine Monitor) that runs on top of KVM. It never touches the hardware directly; instead it requests the creation and execution of virtual machines through the interface KVM provides. In other words, KVM handles the actual control of the hardware virtualization features, while Firecracker handles the VM's configuration and virtual-device emulation. The roles are split.

/dev/kvm: what actually flows back and forth is ioctl

The interface KVM provides is exposed as a special device file, /dev/kvm, and every user-space program communicates with KVM through it. When Firecracker asks KVM to "create a virtual machine," what actually happens is a call to the ioctl system call — Linux's standard mechanism for opening a device file and passing it specific commands — on /dev/kvm. Firecracker sends KVM a sequence of ioctl commands like these:

KVM_CREATE_VM                // Create a virtual machine
KVM_CREATE_VCPU              // Create a virtual CPU
KVM_SET_USER_MEMORY_REGION   // Allocate the guest OS memory region
KVM_RUN                      // Start running the guest OS (returns control on every VM Exit)

Calling KVM_RUN starts the guest OS running in non-root mode, and every time a VM Exit occurs, control returns to Firecracker from this call. Firecracker handles the event that triggered the VM Exit and then calls KVM_RUN again so the guest OS resumes from where it left off. In other words, the vCPU thread is simply spinning in this KVM_RUN loop. This is also why the Vantisso daemon runs with root privileges at runtime: accessing the hardware virtualization device through the /dev/kvm interface requires the corresponding permissions.

Three conditions for running Vantisso

To sum up, the conditions for Vantisso to make use of hardware virtualization come down to three:

• The CPU must support hardware virtualization — Intel VT-x or AMD-V
• The feature must be enabled in the BIOS/UEFI or equivalent
• Vantisso must have permission to access /dev/kvm

You can check all three on the host in two lines before installing:

# Whether the CPU supports virtualization extensions (Intel: vmx, AMD: svm)
grep -Eo 'vmx|svm' /proc/cpuinfo | sort -u

# Existence and permissions of the KVM device file
ls -l /dev/kvm

Once these three conditions are met, Vantisso uses hardware virtualization to boot MicroVMs in milliseconds.