The Hidden Culprit Behind MicroVM Restore Latency: ARP Stall and How to Design It Away
MicroVM-based infrastructure makes active use of snapshot restore to bring up large numbers of AI agents or FaaS workloads instantly. Even with snapshots in place, however, restores intermittently incur delays of more than a second that are hard to account for. Engineers often mistake the cause for a storage disk I/O bottleneck or a page cache miss on the host. Profile the system at the network packet level, though, and the real culprit frequently turns out to be neither disk nor memory, but an "ARP resolution stall" in the L2/L3 network stack. This article covers what an ARP stall is and how to design a network architecture that sidesteps it intelligently.
Do not fall for the false bottleneck: the cache miss illusion
When a severe difference in speed appears between a cold restore (the first restore) and a warm restore (a restore from an already cached state), most systems engineers hypothesize a delay caused by the memory working set not yet being loaded into the host's page cache. Working from that hypothesis, one might attempt read-ahead optimizations such as madvise(MADV_WILLNEED) to pull the disk data into memory before the restore. Measured data, however, shows that such cache warming has almost no effect on actual restore speed. The real bottleneck, which accounts for more than 80% of the total delay, is not the process or disk I/O but the network stall hidden in the moment the host first attempts to communicate with the guest VM.
The technical mechanism behind an ARP stall
An ARP stall arises when the process by which a guest OS wakes from a snapshot and configures its new network interface falls out of step with the address resolution protocol (ARP) mechanism in the host kernel. A guest OS restored from a snapshot frequently does not broadcast the gratuitous ARP that announces to the network that it owns a given IP and MAC address, even once the address has been newly assigned (ip addr add). When the host daemon then goes to send its first probe packet to check the guest's status, the host's routing table holds no MAC address for the target IP. The host kernel therefore sends an ARP request and waits for a reply, and in the process burns through the full kernel ARP retry timeout of roughly a second for no good reason. What results is the absurd situation of a guest VM that has already been restored and stands ready to compute, while the host, unable to find the address at layer 2, is forced to suspend communication for about a second.
How serious an ARP stall is, in measured data
Profiling the wait that precedes the start of communication in a MicroVM environment makes the severity of an ARP stall clear. When one occurs immediately after a snapshot restore, the wait before the host establishes communication with the guest VM reaches about 870 ms. In a MicroVM environment where boot and memory restore themselves finish in milliseconds, 870 ms of network waiting is an overwhelming source of delay that eats up most of the overall responsiveness. In large-scale AI agent infrastructure, hundreds of VMs are spawned and destroyed every second. When a network delay of close to a second compounds across every one of them, it places a serious load on the orchestrator's synchronization logic.
The smart remedy: ARP priming
Tearing into the kernel inside the guest OS, or injecting a separate script to force a gratuitous ARP, is an inflexible way to approach the problem. Modern infrastructure architecture favors solving it on the host side, through preemptive network provisioning: ARP priming. The host daemon (or orchestrator) already knows the IP and MAC address that will be assigned to the guest as it restores the VM. Immediately before the host sends its status probe to the guest, then, it registers that IP-to-MAC mapping explicitly in the neighbor table of the host kernel. The host system now already knows the destination MAC address, so it can transmit packets immediately, without the time-wasting ARP request broadcast and the wait on the kernel timeout. Measuring performance after the ARP priming design was taken into the architecture confirmed that the network communication wait, which previously consumed about 870 ms, had fallen to a mere 1.8 ms or so. In summary, a smart network architecture that injects the guest's network state from the host side in advance lets us eliminate entirely the delay of about a second that undermines the immediacy of snapshot restore in MicroVM-based AI agent execution infrastructure.