← notes on building k7d

Notes on building k7d · Part 1

I built a Rust VMM that forks a live Kubernetes cluster in ~100 ms

Fifty byte-identical copies of a live cluster, down to the TLS sessions, on one €40/month box.

August 2026 · ~12 min read

For the past while I've been building k7d: a small Rust VMM whose whole job is to fork running machines. Not "snapshot to disk, restore later". Fork, like a process forks: pause for a beat, split in two, both keep running. A warm single VM forks in ~5 ms. A live 3-node Kubernetes cluster (control plane, kubelets, pods, established TCP and all) forks in about 100 ms. Fifty copies of that cluster fit on one 64 GB box.

This post is the overview: why this needs to exist, a gentle pass on what a VMM even is, the three mechanisms that make a 100 ms cluster fork possible, and an honest list of what doesn't work yet. The rest of the series goes deep on each piece.

Why: RL needs identical environments, not reset ones

The thing that pushed me to build this is RL on agents whose environment is infrastructure. If you're training or evaluating an agent that operates Kubernetes (deploys charts, debugs workloads, handles incidents), every rollout needs its own cluster. Not one shared cluster with cleanup between episodes: state leaks, always. And not a cold kind cluster per trial either: booting a fresh Kubernetes cluster takes ~30 s and a full RAM bill per copy. At GRPO group sizes that's your whole budget going into waiting for clusters to come up.

And GRPO makes the requirement sharper than just "fast". Group-relative methods compare rewards within a group: N policies attempt the same task from the same start, and the learning signal is who did better. Here's the problem. If member A starts from a colder cache, a different etcd revision, or a half-ready Deployment than member B, the reward gap is noise, not signal. "We reset the environment between episodes" is exactly the failure mode. A reset cluster is similar to the original. Never identical.

A k7d fork is a copy of the live machine. Same memory, same disk, same in-cluster TLS sessions, same kube-apiserver state. Every member of the group starts from a byte-identical environment, then diverges only because of what its policy did. A reset environment is similar to the original. A fork is the original.

What a VMM actually is (the short version)

You don't need to be a VMM person to use k7d. But the 100 ms number only makes sense once you see how little a VM actually is, from the host's side.

On Linux, hardware virtualization is exposed through /dev/kvm. The kernel does the genuinely hard part: running guest code directly on the CPU. A virtual machine monitor is the ordinary userspace process that sets everything up around that. It allocates a slab of memory and tells KVM "this is the guest's RAM", creates virtual CPUs, and sits in a loop handling the moments when the guest touches something that isn't real (a disk, a NIC, a timer) by emulating the device and resuming the guest. QEMU is the maximalist version of this, every device since the floppy drive. Firecracker is a minimalist one. k7d is another minimalist one: ≤25k lines of Rust for the VMM plus its containerd shim, built around one idea.

The idea is a simple observation: the guest's entire RAM is just a memory mapping inside my process. Its disk is a file I own. Its NIC is a queue I service. Everything the machine can observe passes through the VMM, which means the VMM can copy all of it. Cheaply, if it's careful. The whole project is what happens when you take that observation seriously.

Memory: shared until someone writes

Copying 50 × several GiB of guest RAM would blow the latency budget and the RAM budget at the same time. So: don't copy it. Guest RAM lives in one file. A fork pauses the source for a moment, notes which pages changed since the last checkpoint, maps the child's memory as a copy-on-write view of the parent's, and copies only those dirty pages. Everything else stays physically shared until one side writes to it. It's the same mechanism fork(2) uses on process memory, applied to a whole machine.

This is why 50 forks of a cluster fit in 64 GB: you pay for divergence, not for the base. An RL rollout poking at one namespace dirties a small fraction of a multi-GiB guest. (A fork that immediately writes everywhere degrades to a full copy. That's not the workload.)

Getting "which pages changed" right is harder than it sounds — KVM's dirty-page log only sees CPU writes, so device I/O into guest memory needs its own tracking. Part 4 walks that whole path at ground level, bugs included.

Network: same identity, isolated bridges

Memory CoW gets you a fork of one VM. A Kubernetes cluster is several VMs plus a network, and the network is where naive forking fails. Every node has an IP. kubelet, etcd, the API server and every TLS cert in the cluster are bound to those IPs. Give the forked nodes new IPs and the cluster notices immediately: agents restart, TLS breaks, and you eat a multi-second recovery that makes "100 ms" meaningless. Keep the same IPs on a shared network instead, and fork #2 ARP-fights fork #1.

Ok, so: each cluster lives on its own private Linux bridge, and a fork gets a new bridge carrying the same guest IPs and MAC addresses as the source. From inside the guest nothing moved, so kubelet, the CNI and the control plane just keep running; from outside, the bridges are separate L2 domains, so fifty forks with identical addresses never see each other.

Without this, every fork forces kubelet restarts at a second or two per node, and the whole exercise collapses back to "reboot-ish". With it, the cluster genuinely does not notice it was forked. One honest caveat: TCP connections to the outside world do not survive, because the far end never forked. The bridge mechanics, and what happens to in-flight packets, are in Part 4.

Lifecycle: forks live in a budgeted tree

Once forking is this cheap, the failure mode inverts. It's no longer "forks are too slow", it's "the agent forked 300 times and the box fell over". An RL run or a tree search doesn't produce a list of environments. It produces a tree: fork a branch, try something, fork again from the promising state, abandon the dead end, go back to an earlier checkpoint and try the other door.

So the daemon manages exactly that tree, under an explicit budget:

base cluster ──► fork A ──► fork A1   (protected: winner)
             ├─► fork B                (pruned: low reward)
             └─► fork C ──► rollback ─► fork C'

The verbs mirror the search loop. tree_fork_batch opens N parallel rollouts from one checkpoint. tree_protect pins a winner so budget pressure can't kill it. tree_prune drops a losing subtree. tree_rollback brings back an earlier node without destroying it. tree_auto_evict enforces the RAM and disk caps you set. Your training loop owns rewards and policy; k7d owns environments and budgets. That split is deliberate. The agent is exactly the party you can't trust to clean up after itself. It all speaks JSON-lines over a Unix socket, so wiring it into a GRPO trainer is an afternoon, not an integration project.

Part 5 covers the tree machinery. Part 6 covers why I trust the eviction model enough to let it kill VMs on its own. Short answer: the bookkeeping is extracted to Lean and proven, because "the eviction logic had an off-by-one and deleted the winner" is not a bug you want to find empirically.

The numbers

All measured on one bare-metal Hetzner box at ~€40/month: Ryzen 5 3600, 6 cores, 64 GiB, NVMe. Nothing exotic.

The "enforced budget" column is the part I actually care about. Every row is an assertion in an integration test that runs in CI. If a latency claim drifts past its budget, a test fails, and the regression gets caught before anyone reads a stale number. Benchmark tables in READMEs rot; assertions don't. Also worth saying: the cluster-fork number is measured under load (a Deployment being scaled plus a ConfigMap/pod churn storm at fork time), because quiet-host numbers overstate how the system behaves when it matters.

For calibration: cold-booting that same cluster takes ~30 s. The fork is about 300× faster than the boot, and copy #50 costs its dirty pages, not another full guest of RAM.

Limitations, honest

• One daemon, one address space. Live CoW fork requires parent and child memory to be mappings in the same process. Guest-to-host isolation is still KVM, but isolation between sibling forks is weaker than Firecracker's one-jailed-process-per-VM. k7d is built for fleets of your own environments (RL rollouts, evals, CI), not for hostile multi-tenant isolation between forks.
• Outside TCP dies at the fork. Everything inside the forked set survives: in-cluster TLS, established TCP between member VMs, disk state. Connections to the outside world don't, because the far end never forked. Guest clocks are reset so time doesn't jump backwards.
• Single host, x86_64 Linux + KVM only. Cross-node fork is on the roadmap. Does not build on macOS or Windows.
• The CI cluster fixture is lean. The 3-node k3s fixture behind the headline numbers runs flannel, kube-proxy and a real in-cluster Deployment, but several stock add-ons (CoreDNS, Traefik, local-path storage) are disabled there today. Re-enabling them is queued fidelity work, not a redesign. The fork engine itself is N-node; the practical ceiling is host RAM, not the API.
• Young project. ≤25k lines for the VMM + shim, one primary test machine, no security audit yet. The small surface is by design. The audit gap is just honest.

Where the series goes from here

1. This post. Why, roughly how, and what it costs.
2. The design space. Firecracker snapshots, CRIU, Kata, E2B-style sandbox services. What each can and cannot fork, and why the gap existed.
3. The 45-second detour. Skippable prequel: my earlier disk-only fork built on qemu + Longhorn PVC snapshots. It worked, it took ~45 s, and it taught me exactly which problem I actually had.
4. Anatomy of the VMM. KVM, one memory file, virtio devices, a guest agent over vsock. A guided walk through the ≤25k lines.
5. The tree API. fork / protect / prune / rollback / auto-evict, and wiring it to a GRPO trainer.
6. Formal verification where it pays off. Kani on the unsafe memory arithmetic, Aeneas→Lean on the eviction model, and where I deliberately stopped.

Next: Why not Firecracker, Kata, QEMU, or an E2B-style sandbox?

If any of this is your kind of problem, the code is open. Apache-2.0, small enough to actually read. Issues and opinions welcome.