Sandboxing at Scale: VMMs, Snapshots & the Local-Dev Problem
Chapter 9 settled the isolation question: microVMs are the sweet spot for running model-written code, giving each sandbox its own kernel yet booting in a fraction of a second. At platform scale the questions change shape. How many sandboxes fit on a host? How fast can one cold-start, since that latency lands directly on the agent? How do you orchestrate thousands? And the one every team hits within a week — how do you run any of this on a laptop, when the production isolator needs hardware virtualization your dev machine doesn't have? This chapter answers all four.
24.1 The Rust VMM landscape
The modern microVM monitors are written in Rust for a pointed reason: the VMM is part of the trusted computing base, and a memory-safe VMM is a smaller attack surface than a C one. They differ mainly in how much device model they expose.
| VMM | Lang | Character | Runs without KVM? |
|---|---|---|---|
| Firecracker | Rust | Minimal device model by design; fast boot; powers large serverless fleets | no — needs /dev/kvm |
| Cloud Hypervisor | Rust | Richer devices, hotplug, larger VMs, GPU passthrough; KVM and MSHV | no — needs KVM/MSHV |
| crosvm | Rust | Google's monitor (Firecracker forked from it); powers ChromeOS/Android | no — needs KVM |
| libkrun | Rust | A VMM as an embeddable library; basis for krunvm / podman machine | KVM on Linux · HVF on macOS |
| QEMU | C | The universal, heavy, flexible monitor; Kata's default | yes via TCG (emulated, slow) |
| gVisor | Go | Not a VMM — a user-space kernel intercepting syscalls (runsc) | yes (ptrace/systrap platform) |
24.2 The local-dev problem
Firecracker — and Cloud Hypervisor, and crosvm — require /dev/kvm, hardware virtualization exposed by the Linux kernel. Two facts collide with that. macOS has no KVM; it has Apple's Hypervisor.framework (HVF) instead. And many CI runners and nested-container environments don't expose nested virtualization at all. So the blunt truth your team discovers is correct: Firecracker can't run on a typical dev machine. The mistake is to let that dictate the production isolator. Instead, design to an isolation interface and make the implementation swappable.
run(code)→result contract, so the local isolator (a container, libkrun using macOS HVF, or gVisor's ptrace platform) and the production isolator (Firecracker microVMs) are swappable without changing the agent. In production a single golden snapshot is booted once and then resumed many times with copy-on-write memory — thousands of microVMs from one boot, each restored in milliseconds.The strategies, in order of fidelity: interface parity — ship the same OCI image, run it as a container locally and a microVM in production against an identical workload contract; local microVMs without KVM via libkrun/krunvm (HVF on Mac) or gVisor's ptrace platform, for closer behavior; remote sandboxes (E2B, Daytona) so no one needs local virtualization at all; and nested-virt CI on bare-metal or KVM-exposing instances for tests that must exercise the real isolator.
Define the sandbox by its contract — code in, result out, with declared resource and network bounds — and treat the monitor behind it as a swappable backend. This is what lets the local and production isolators differ (container vs. Firecracker) without the agent ever knowing, and lets you migrate VMMs later without rewriting the platform.
24.3 Scaling the fleet
Density and cold-start are the two levers. Each microVM is hardened by wrapping it with the jailer plus seccomp, cgroups, and namespaces. Cold-start is hidden with pre-warmed pools, and crushed outright by snapshot/restore: boot a VM once, capture its memory and device state, then resume new instances from that snapshot in milliseconds. With copy-on-write memory a single golden snapshot fans out to thousands of VMs at low marginal cost — the technique behind sub-second sandbox provisioning at fleet scale. Memory overcommit and ballooning push density further. Orchestration rides existing rails: firecracker-containerd or Kata Containers make microVMs OCI- and CRI-compatible, so Kubernetes schedules them through a RuntimeClass like any other workload. E2B and Fly.io build on Firecracker; Modal, Daytona, and managed Kata offerings on GKE/AKS take comparable approaches.
Resuming many VMs from one snapshot means they share whatever was in memory at capture time — RNG seeds, secrets, a frozen clock. Reused entropy breaks cryptography, and a baked-in secret leaks across tenants. Reseed randomness and resync the clock on restore, and inject per-tenant secrets after resume, never into the golden image.