How We Clone Firecracker VMs in Under 150ms: Bringing fork() to MicroVMs

Building on CodeSandbox.io’s playbook to bypass the disk and achieve sub-millisecond memory inheritance in AWS Firecracker.

Photo by Andy Wang on Unsplash

Processes have had fork() for fifty years: clones an entire address space in O(1), sharing every page copy-on-write, and let the parent and child diverge only when they actually write. Virtual machines, however, never had anything equivalent.

The standard approach to cloning a VM is snapshot and restore. You pause the guest, serialize its entire RAM to disk, and load that dump into a brand-new process. Memory goes to disk and comes back, resulting in gigabytes of I/O per clone.

Every single clone pays a pure RAM dump penalty that scales linearly at roughly 0.9 ms per MiB. For larger guests, this tax becomes a major bottleneck: cloning a 2 GiB guest costs 1,777ms, and a 1 GiB guest takes 953ms. As guests reach multi-gigabyte sizes, this delay stretches into seconds. We see this baseline tax even on much smaller instances, with a 300 MiB guest costing 366ms (varying from 309–531ms depending on how many pages had faulted in prior to the clone).

Then, we gave guest memory a file back. By utilizing a userspace daemon backed by userfaultfd to take over page-fault handling, the clone stopped being a disk operation. Today, a complete clone of a Firecracker microVM—including the process, RAM, vCPU state, and block device—lands at ~130ms.

RAM size doesn’t even register anymore: the clone total remains nearly the same at 300 MiB, 1 GiB, and 2 GiB.

Bypassing the Disk

The bottleneck existed because guest RAM was treated as anonymous memory — shareable with nobody, serializable only through disk. Fix that, and the dump disappears.

In our Firecracker fork, guest RAM became a MAP_SHARED mapping over a memfd. This is essentially a file whose content is the guest's live memory:

// Inside Firecracker: RAM is a shared file descriptor
let memfd = unsafe {
    libc::memfd_create(
        b"guest_ram\0".as_ptr() as *const libc::c_char,
        libc::MFD_CLOEXEC | libc::MFD_ALLOW_SEALING,
    )
};
unsafe { libc::ftruncate(memfd, ram_size as libc::off_t) };

let guest_ram = unsafe {
    libc::mmap(
        std::ptr::null_mut(),
        ram_size,
        libc::PROT_READ | libc::PROT_WRITE,
        libc::MAP_SHARED,
        memfd,
        0,
    )
};

Because the memfd is the guest's memory, it can be passed over a Unix socket via SCM_RIGHTS to a daemon that holds it alongside the parent. The daemon can read and write the guest's live pages directly. At clone time, it serves the child's page faults straight from them.

Firecracker’s fork creates the memfd-backed RAM, registers it with userfaultfd, and hands both file descriptors plus the region map to the daemon.

A Userspace Page Table

When memory is shared across generations of clones, the daemon needs to know where any given page lives. Is it in the VM’s own memfd? A parent’s live memfd? The read-only snapshot file? Or has it never been touched?

To track this, we built the Page State Table (PST) — one entry per guest page.

// Conceptual representation of the Page State Table
enum PageSource {
    Uninitialized,   // Never touched — install a zero page
    File,            // Read from the read-only base snapshot file
    VM(RawFd),       // Read from a live memfd (a parent's or the VM's own)
}

struct PageEntry {
    source: PageSource,
    offset: u64,
}
struct PageStateTable {
    entries: Vec<PageEntry>, // One per guest page
    live_memfd: RawFd,       // The VM's own private memory
    pages_to_parent: u64,    // Counter for inherited pages
}

At clone time, the child’s PST is a deep copy of the parent’s. This O(total_pages) operation costs 1–4ms at 300 MiB and still only 9–26ms at 2 GiB (524,288 entries). The parent’s own faulted-in pages are simply re-pointed back to it (roughly 1.3–2.7% of a 300 MiB guest).

Serving Faults

userfaultfd() delegates page-fault handling from the kernel to a userspace process. The daemon (written in Go) runs one serveFaults goroutine per VM:

// serveFaults: the per-VM fault loop (abridged)
pageBuf := make([]byte, pageSize) // pinned below: UFFDIO_COPY's src pointer
pinner := runtime.Pinner{}        // must not move while the kernel copies
pinner.Pin(&pageBuf[0])

for {
    n, err := syscall.Read(vs.uffdFD, msgBuf)
    if err != nil {
        if isEAGAIN(err) { time.Sleep(5 * time.Millisecond); continue }
        return
    }
    event, flags, address := parseUffdMsg(msgBuf)
    if event != uffdEventPagefault { continue }
    pageIdx := int((address - vs.regionBase) / pageSize)
    if pageIdx < 0 || pageIdx >= vs.pst.NumPages() { continue }
    if flags&2 != 0 { // uffd flags: bit 1 = write-protect fault (CoW)
        d.handleWriteFault(ctx, vs, address, pageIdx)
    } else {
        d.handleReadFault(ctx, vs, address, pageIdx, pageBuf)
    }
}

Notice a few critical details here:

• Raw Syscalls: The read uses the raw syscall instead of os.File to avoid the Go runtime poller turning our nonblocking uffd into a blocking read.
• Pinned Buffers: The page buffer is pinned with runtime.Pinner because UFFDIO_COPY's source pointer must stay put while the kernel copies, and Go's Garbage Collector moves heap objects.

The Clone

To clone, the worker pauses the parent and writes a state-only snapshot (vCPU registers and device state, taking ~10ms). Then, it freezes the parent, deep-copies its PST, and re-points its own pages at its memfd:

// CloneVM (abridged): freeze at T1, then inherit the page table
freezeStart := time.Now()
for _, r := range parent.regions {
    if err := WriteProtect(parent.uffdFD, r.BaseHostVirtAddr, r.Size, true); err != nil {
        wpFailures++ // log field "wp_failures" — must stay 0
    }
}
freezeDur := time.Since(freezeStart) // log field "freeze"

childPST := parent.pst.DeepCopy()
for i := 0; i < numPages; i++ {
    if childPST.Get(i).Source == SourceLocal { // parent's own faulted pages
        childPST.Set(i, PageEntry{Source: SourceVM, VMID: parentID})
        rePointed++ // log field "pages_to_parent"
    }
}
parent.cloned = true

The kernel applies Write-Protect to present pages only, meaning one whole-region ioctl covers exactly the parent’s resident pages without any touched-set bookkeeping.

Copy-on-Write

When a child writes a shared page, the kernel raises a WP fault. The handler reads the page’s pre-write content from the writer’s memfd, propagates it, and then un-protects it so the write can land.

// handleWriteFault: T1 -> descendants, then let the write land
oldPage := make([]byte, pageSize)
syscall.Pread(vs.memfdFD, oldPage, pageAlign(address-vs.regionBase))
d.propagatePage(ctx, vs.id, pageIdx, oldPage)
d.unprotectPage(ctx, vs, address, pageIdx)

The propagatePage function makes exactly one physical copy into a deterministic holder, and re-points everyone else at it. The writer gets its private page, and every other generation keeps the pre-write (T1) content. This is fork() semantics, fully implemented in userspace.

The Clone Flow, Measured

RAM size is effectively invisible to the clone process. When you grow the guest 6.8×, the interior clone time stays flat. The only things that scale with page count are the old-path RAM dump and the O(total_pages) PST deep copy (which remains strictly ms-class even at 2 GiB).

300 MiB (76,800 pages)~115–145ms

1 GiB (262,144 pages)~125–135ms

2 GiB (524,288 pages)~140ms

Why Fast Clones Matter in the Agentic era.

A clone at ~130ms isn’t just a latency improvement; it fundamentally changes what you can do with a VM. When forking a live VM costs less than a network round trip, VMs stop being long-lived infrastructure you have to carefully plan around. They become disposable iterations you can fork and discard freely.

• Closed iteration loops: Instead of booting and waiting minutes for provisioning, you fork the live, fully-provisioned parent in ~130ms, run an experiment, and discard the child. The parent continues exactly where it left off.
• Massive parallelism: Agents fan out over the same parent state. Copy-on-write means every agent starts from the parent’s live memory and diverges only where it actually writes.
• Free hypotheses: Destroying a clone costs nothing. Test a risky migration, a hostile payload, or a destructive config change — if it fails, the clone disappears with zero consequence.