Anatomy of MicroVM Storage — Sharing a Golden Image Without Copying It

One of the reasons Vantisso can start dozens of agents in milliseconds is a fundamental redesign of the storage layer. If the golden image had to be cloned in full every time an agent started, disk I/O would immediately become the bottleneck. Vantisso solves this structurally with the snapshot facility of Device Mapper (dm-snapshot), the Linux kernel's block device virtualization framework.

Device Mapper is the layer with which the Linux kernel creates logical virtual block devices on top of physical ones, and dm-snapshot is one of its features. It provides a copy-on-write mechanism in which several logical devices share a single read-only base image while each records only its own changes to a separate store. In other words, the essential role of dm-snapshot is to make dozens of MicroVMs behave as though each had its own independent disk, without the golden image ever being copied once.

Every MicroVM starts from a single golden image

Every MicroVM starts from one immutable master image: the golden image. It is a RAW ext4 disk image with Alpine Linux minirootfs, micro-init, the agent execution manager, and every dependency an agent needs, all preinstalled. Vantisso adopts COW as its default provisioning strategy, falling back automatically to ordinary file copying on hosts where the dm-snapshot tooling is unavailable.

The core idea of copy-on-write provisioning is simple: instead of actually copying the contents of a disk, write only the parts that change, when they change. With ordinary file copying, the entire golden image has to be copied to a new file and committed to disk with fsync before the VM can start. Starting several agents at once means writes amounting to a multiple of the image size land all at the same time; every VM competes to write to disk, saturating I/O bandwidth and producing wide variance in startup times. The more agents, the worse it gets. With COW, that whole-image copy disappears entirely. Behind that are two Linux facilities: loop devices and dm-snapshot.

The two pillars of COW: loop devices and dm-snapshot

A loop device is a Linux kernel feature that lets an ordinary file be treated as a block device. Attach a file such as rootfs.ext4 to a loop device with losetup, and the kernel will mount, read, and write it as if it were a real disk. Vantisso uses this mechanism to expose the golden image file and the changed-block store file as two separate loop devices. Note that the golden image side is attached read-only with losetup -r — the base must never change.

dm-snapshot then combines those two loop devices into a single virtual block device. The /dev/mapper/cow-<vm_id> device created with dmsetup lives in the host OS kernel space. To let the MicroVM use it as its own disk, Firecracker emulates this virtual block device as a virtio-blk virtual hardware device and exposes it to the guest. The guest OS kernel then recognizes the virtual disk as /dev/vda through its virtio-blk driver. Because Firecracker's direct kernel boot passes root=/dev/vda rw as a kernel parameter, /dev/vda is mounted as the ext4 root file system the moment the kernel boots. All micro-init has to do afterwards is mount virtual file systems such as /proc, /sys, and /dev.

What makes this design work is the transparency of the abstraction layer. The guest OS has no idea it is running on top of dm-snapshot. From the guest's point of view it is simply reading and writing an ordinary ext4 disk; whether that I/O is served from the golden image or recorded in the changed-block store is handled transparently by Device Mapper in the host kernel. Thanks to that transparency, the golden image can remain a standard ext4 image that knows nothing about COW, and the guest OS and agent execution manager run unmodified. The benefits of COW — fast provisioning and low disk usage — are achieved entirely at the host kernel level, leaving the guest's software stack and the storage optimization completely decoupled.

Reads and writes: the golden image is never touched

When a MicroVM issues a read, the changed-block store is checked first, and the block is fetched from the golden image only if it is not there. When it issues a write, the golden image is left untouched and only the changed block is recorded in the changed-block store. To the MicroVM it looks like a private disk of its own; in reality the golden image is shared while only the deltas are managed separately. The whole flow looks like this:

Reading the diagram from the bottom up makes the structure clear. On the left, the golden image file (rootfs.ext4) is the immutable read-only base, exposed as loop device A through losetup -r. On the right, the changed-block store file (vm-<id>.cow) is a pre-created sparse file whose initial size is effectively zero, exposed as loop device B through losetup. dmsetup combines the two into the virtual block device /dev/mapper/cow-<id>, which the MicroVM mounts. The two thick arrows in the diagram are the read and write paths described above: ① reads pull blocks absent from the changed-block store out of the golden image (loop A), and ② writes land only in the changed-block store (loop B). That no write arrow ever points at the golden image is, in essence, the whole design.

The operations that prepare the virtual block device during disk provisioning — two losetup calls and one dmsetup call — typically complete within tens of milliseconds. From there, the guest kernel boot and agent execution manager startup proceed exactly as they would with ordinary file copying. The virtual block device, loop devices, and changed-block store file created for provisioning are all cleaned up automatically when the VM is deleted (remove the virtual block device → detach the loop devices → delete the changed-block store file).

Why it is faster: provisioning cost decoupled from image size

The performance advantage of COW is not simply that it is "somewhat faster" — the cost structure itself is different. Breaking both approaches into stages shows exactly where the difference comes from.

Ordinary file copy:
  [full golden image copy + fsync]     ← proportional to image size
  [guest kernel boot + agent startup]  ← common to both

COW:
  [2× losetup + 1× dmsetup]            ← tens of milliseconds, independent of image size
  [guest kernel boot + agent startup]  ← common to both

The guest kernel boot phase is identical in both approaches. The difference in startup time therefore comes from exactly one stage: disk provisioning. And the decisive point is what each stage's cost is proportional to. The cost of a file copy grows with the size of the golden image, whereas COW provisioning is just a handful of kernel operations — losetup and dmsetup — and so stays at tens of milliseconds no matter how large the image grows, because no data is actually copied. As a result, COW startup drops to the point where the guest boot dominates, which is effectively the floor achievable in the current design short of a snapshot restore.

The same principle governs disk usage. The changed-block store starts at close to zero and grows only by the blocks that actually change, so adding one more VM increases disk usage by essentially nothing. Dozens of MicroVMs share one golden image physically.

The effect is most pronounced in multi-agent environments. When several agents start simultaneously, file copying puts writes amounting to a multiple of the image size into contention and saturates I/O bandwidth. COW, by contrast, only sets up loop devices and a virtual block device — there is no bulk write at all. Because disk I/O contention is eliminated at the root, startup time barely increases as the number of concurrently starting agents grows. Decoupling provisioning cost from both image size and agent count is the essence of this design.

Summary

To summarize, COW storage achieves three things at once. The golden image is never copied, so provisioning stays at a constant few tens of milliseconds; only deltas are recorded, so the disk growth per VM is effectively zero; and there are no bulk writes, so I/O contention does not appear as agent count grows. For reference, the startup performance of the current Vantisso implementation as of July 2026 summarizes as follows.