← notes on building k7d

Notes on building k7d · Part 4

Anatomy of a 100 ms cluster fork

One memory file, the device writes KVM's dirty log misses, network identity across forks, and the timer state a restore has to carry.

August 2026 · ~14 min read

Part 1 gave the overview. This is the ground-level version: what actually happens, in order, when k7d forks a live 3-node Kubernetes cluster in ~105 ms — and the three places where "byte-identical" quietly breaks unless you know exactly where to look. Everything here runs in a single Rust daemon, ≤25k lines including the containerd shim, on plain Linux + KVM.

A note on numbers before we start: every latency in this post is an assertion in an integration test that runs in CI, with the budget constants in one source file the tests import. The methodology — named hardware, exact commands, what "under churn" means — is written up in the repo's benchmark post alongside LATENCY_BUDGETS.md. If a number here ever disagrees with those files, trust the tests.

Step zero: guest RAM is one file

Every k7d guest's memory is a single memfd — an anonymous file — that the daemon maps and hands to KVM as the guest's RAM. This one decision is most of the architecture. A file can be mapped twice. A file can be mapped MAP_PRIVATE, which is the kernel's native copy-on-write: reads go to the underlying pages, and the first write to any page transparently copies it. That's the same mechanism fork(2) has used on process memory forever. k7d's contribution is mostly the bookkeeping required to apply it to a machine without corrupting anything.

A warm fork of one VM goes like this. The source VM was previously "prepared" as a fork source: a snapshot of its memory was taken and KVM's dirty-page tracking switched on. At fork time, the daemon pauses the vCPUs, asks KVM which pages changed since the snapshot, copies only those pages into the child's view, restores the child's CPU and device state, and resumes both sides. Everything the guest didn't touch stays physically shared. With less than 25% of pages dirty, that's ~5 ms against a 50 ms enforced budget. A cluster fork is the same dance for N member VMs under one coordinated pause, plus a network step we'll get to.

The cost model this buys is the one RL actually wants: you pay for divergence. Fifty forks of a multi-GiB cluster fit on a 64 GB box because rollout #37 poking at one namespace dirties a small fraction of the guest. A fork that immediately writes everywhere degrades to a full copy — that's not the workload.

What the dirty log doesn't see

Here is the first place byte-identical silently breaks, and the most instructive bug in the whole project.

KVM's dirty-page log records writes made by guest CPUs. It knows nothing about anyone else writing guest memory — and in a VMM, other writers are everywhere. k7d's virtio-blk device completes a disk read by writing the data directly into guest RAM from a userspace thread: the payload, the used-ring entry, the status byte. None of those writes trap through KVM. None of them appear in the dirty log.

Follow the consequence through. Guest reads a file; the block device DMAs fresh bytes into pages the dirty log considers clean; you fork; the fork copies "dirty" pages and shares "clean" ones — from the pre-I/O snapshot. The child resurrects stale bytes on exactly the pages the device just wrote. No error, no crash, no log line. A database in the guest would see a page of its file quietly revert. Silent corruption, the worst kind.

The fix has two halves, and the second is sneakier than the first. First: because the block worker is in-process, it can keep its own page bitmap — every store into guest memory marks the page, data buffers and used-ring slots and status bytes alike — and the fork path merges that bitmap into KVM's before copying. Second: ordering. Pausing a VM must first make the device workers drain every in-flight request and only then park, and the fork must wait for that quiescence before reading the bitmaps — otherwise a completion lands after the merge and you're corrupted again, just more rarely. The same discipline applies to the network workers: an earlier bug had TAP workers happily mutating virtqueues while the vCPUs were frozen for dirty-page capture, which desynchronized the guests' TCP stacks from their rings and killed established connections on resume. Pause means pause, for every thread that can touch guest RAM. (Entries #43 and #29 in the repo's CHALLENGES.md have the full details.)

The general lesson, worth stating because it applies to any CoW-fork design: dirty tracking is per-writer. Enumerate everyone who can write guest memory — vCPUs, in-process device threads, kernel-side vhost devices — and account for each one. vsock in k7d is kernel-vhost, so its rings can't be bitmap-tracked the same way; they get re-seeded from guest memory on fork instead. Three writers, three different mechanisms, one union.

Network identity: same addresses, isolated bridges

Memory CoW forks one machine. A cluster is machines plus a network, and the network holds the cluster's identity: node IPs baked into TLS certs, etcd peer addresses, kubelet registrations, established TCP between members. Change the addresses and the cluster notices immediately — agent restarts at a second or two per node, TLS re-handshakes, multi-second convergence. Keep the addresses on a shared L2 and fork #2 ARP-fights fork #1.

k7d's answer: each cluster lives on its own private Linux bridge, and a fork gets a fresh bridge carrying the same guest IPs and MACs as the source. Inside the guest, nothing moved — same ARP cache, same certs, same established intra-cluster TCP, so kubelet, the CNI and the control plane just keep running. Outside, the bridges are disjoint L2 domains, so fifty forks with identical addresses never exchange a frame. From the guest's point of view nothing changed; from the host's, every fork is isolated. Kubernetes only needs the inside view to stay consistent.

The detail that makes this actually work is below the IP layer. A virtio-net queue is a ring in guest memory with positions tracked on both sides. The forked VM gets a fresh host-side TAP worker, and if that worker starts counting from zero against rings that are mid-conversation, it re-processes served descriptors and writes garbage used-ring entries; the guest's network softirq backs off and eth0 wedges. So on every restore, the new worker seeds its counters from the guest's own ring state — read used.idx out of guest memory, start from there. The kernel's vhost devices have an ioctl for exactly this; a userspace worker has to remember to do it by hand.

One caveat survives from Part 1: TCP to the outside world dies at the fork, because the far end never forked. Everything inside the forked set survives; the CI test for this holds a kubectl ?watch=true TLS stream open across the fork and asserts it keeps streaming.

Timer state: the frozen PIT

The third trap had the symptom furthest from its cause.

After a restore, an inner-k3s guest looked half-alive: ping fine, agent exec fine, TCP connect to the API server accepted — but the TLS handshake never completed, and k3s consumed exactly zero CPU. The tell, once I finally looked in the right place: date inside the guest was frozen while /proc/uptime advanced, and /proc/interrupts showed IRQ0 stuck at its pre-restore count.

The cause: k7d's minimal guests take their timer tick from the classic i8254 PIT — a chip the guest kernel programs exactly once at boot and reasonably assumes stays programmed, because real hardware doesn't spontaneously forget. But the restore path created a fresh, unprogrammed PIT in KVM. No PIT program, no timer interrupts, ever again. Every timer-driven wakeup in the guest — Go runtime schedulers, sleep, kernel timekeeping — parked forever. Interrupt-driven paths (vsock, ICMP, TCP handshake delivery) kept working, which is why the guest looked half-alive and why months of busybox-guest tests never caught it: a shell with no timer-driven workload doesn't miss its tick. Kubernetes, which is Go runtimes all the way down, absolutely does.

The fix is small — snapshot the PIT channel state with KVM_GET_PIT2, restore it with KVM_SET_PIT2, which re-arms KVM's in-kernel timer — but the shape of the bug generalizes: a machine's state includes every latch some driver set once and forgot. The same family produced a subtler sequel on multi-vCPU guests: the LAPIC's one-shot deadline timer lives in an MSR (IA32_TSC_DEADLINE), not in the LAPIC register page, so capturing the LAPIC without that MSR left secondary CPUs with a disarmed timer — intermittent hangs that moved between runs, masked on single-vCPU guests by the restored PIT. Restoring a machine faithfully means finding every piece of state a driver programmed once at boot, and the clocks hold several of them. Guest wall-clocks are also reset on fork so time never jumps backwards.

The waterfall

Two things in that figure deserve a sentence each. The headline ~104 ms is measured under load — the fixture scales a Deployment and churns ConfigMaps and pods at fork time — because quiet-host numbers overstate how the system behaves in practice; on a quiet host the same on-host operation runs ~25–40 ms. And the 50-fork batch works out to ~82 ms per cluster because the expensive part (the coordinated pause and dirty-page capture) is shared once across the whole batch, then children materialize in parallel.

What it took to get here: the ~2-second era

Honesty requires the before-picture. For a while the cluster fork took ~2 s, not ~105 ms, and the reason is instructive. The cluster pods originally mounted their shared files over virtiofs, whose host daemon (virtiofsd) is a separate process speaking vhost-user — and a separate process cannot participate in a MAP_PRIVATE CoW fork of the daemon's memory, nor can its FUSE session survive being cloned (the forked guest holds file handles a fresh daemon has never issued; the first page fault on an mmap'd binary answered EBADF and the guest's k3s died by SIGBUS). So every fork of a virtiofs-carrying VM was demoted to a full memory copy: 3 × ~3.2 GiB, ~2 seconds, every time.

The fix was to remove the reason for virtiofs: pack the read-only host mounts into erofs images served by the in-process virtio-blk device — the one with the dirty bitmap from earlier. Zero out-of-process device state, so every member VM takes the true CoW path, and ~2 s became ~105 ms in one architectural move. The general rule it left behind: every out-of-process device is a fork liability. Either its state can be carried across the fork, or it forces a copy, or it has to go.

The numbers, all in one place

All on the same ~€40/month Hetzner box from Part 1 (Ryzen 5 3600, 64 GiB, NVMe). Budgets are set at roughly 2× typical and enforced by tests in CI; some are deliberately generous because the full suite runs serially with shared KVM state, which inflates tails. The repo's benchmark write-up documents the exact commands to reproduce every row.

Next: once forking is cheap, the problem becomes lifecycle. Part 5 covers the snapshot tree that keeps N forks under a RAM and disk budget.