# Shipped: forkd v0.3 — BRANCH source-pause from 29 s to 205 ms (143×) on 4 GiB SSD

**2026-05-19.** Two releases today on [`deeplethe/forkd`][repo]: [v0.3.0][r1] in
the morning, [v0.3.1][r2] in the afternoon. Same line of work — diff snapshots
in BRANCH. Worth one short post.

[repo]: https://github.com/deeplethe/forkd
[r1]: https://github.com/deeplethe/forkd/releases/tag/v0.3.0
[r2]: https://github.com/deeplethe/forkd/releases/tag/v0.3.1

## What forkd is

A fork-on-write microVM primitive for AI agents. The killer move is `POST
/v1/sandboxes/:id/branch`: pause a running source sandbox, snapshot it, resume
the source, spawn N children that all `mmap MAP_PRIVATE` the same memory image.
Children inherit the source's exact state (every dirty page, every open
descriptor, every TCP socket) and diverge under copy-on-write.

It exists because Modal has this as their proprietary moat, CubeSandbox has it
on the roadmap, boxlite mentions snapshots but doesn't ship fork. The
open-source story for this primitive was missing.

## The honest before/after

forkd v0.2's BRANCH pauses the source while writing the full `memory.bin` to
disk. The
[v0.2 pause-window benchmark](https://github.com/deeplethe/forkd/blob/main/bench/pause-window/RESULTS-v0.2.md)
shows source pause is bandwidth-bound on the snapshot write — **163 ms on
tmpfs, 4.26 s on SATA SSD for a 513 MiB source**. Larger sources scale
linearly: **29.3 s on SSD for 4 GiB**.

v0.3 swaps the full memory.bin write for **Firecracker's Diff snapshot mode**,
which writes only pages dirtied since the previous snapshot. The daemon also
runs a background `cp` of the source's memory.bin in parallel with the source,
then resumes the source as soon as the (tiny) diff write finishes, and merges
the diff onto the pre-copied output. **Source pause becomes the diff write
only.**

The numbers, 75+ trials of real measurement on a 16 GiB / SATA SSD dev box:

| Source memory | v0.2 Full | v0.3 Diff | **Speedup** |
|---:|---:|---:|---:|
| 256 MiB SSD | 1.8 s | 241 ms | 7.5 × |
| 512 MiB SSD | 3.4 s | 226 ms | 15 × |
| 1024 MiB SSD | 6.9 s | 229 ms | 30 × |
| 2048 MiB SSD | 14.5 s | 222 ms | 65 × |
| **4096 MiB SSD** | **29.3 s** | **205 ms** | **143 ×** |

Source pause is now essentially **constant at ~200 ms regardless of source
memory size**. Each release ships full CSVs.

## The caveat that matters

143× is the **idle-source ceiling**, included for comparability with prior
art. Real fan-out workloads dirty more memory between restore and BRANCH; the
diff write itself becomes the cost. We swept this too (42 more trials):

| Dirty (MiB) | Full pause | Diff pause | Speedup |
|---:|---:|---:|---:|
| 0 (idle) | 13.7 s | 0.59 s | 23 × |
| 50 | 13.7 s | 0.92 s | 15 × |
| 100 | 13.8 s | 1.25 s | 11 × |
| 250 | 14.5 s | 2.40 s | 6 × |
| 500 | 14.1 s | 5.73 s | 2.5 × |
| 1000 | 14.4 s | 10.7 s | **1.3 ×** |

Linear regression: `diff_ms ≈ 500 + 10.2 × dirty_mib`. Crossover at ~65 %
source RAM dirty. **The realistic number for typical agent fan-out (30-300 MiB
dirty working set) is 6-15×, not 143×.**

## The deferred plan we explicitly walked back from

The original v0.3 plan was live-fork via [`memfd_create`][memfd] +
[`uffd_wp`][uffdwp], targeting ~30 ms pause regardless of source memory size
— the architecture CodeSandbox uses. Patch was sketched. Then we evaluated:

- Diff snapshots got 85 % of the original target headroom on vanilla
  Firecracker, no fork required.
- The remaining ~175 ms is mostly Firecracker control-plane (the PUT
  `/snapshot/create` round-trip plus vCPU state harvest). A memfd path
  wouldn't shrink that.
- The memfd value-add isn't sharing — today's `mmap MAP_PRIVATE` already
  gives kernel-CoW fan-out between siblings. memfd's real value is enabling
  uffd_wp tracking of source writes, only useful for the harder live-fork
  architecture (which has its own deferred open questions on source-divergence
  sync).
- Fork-maintenance cost is real and permanent: own musl-via-docker CI,
  rebase on every upstream Firecracker tag, track CVEs, weaken the "vanilla
  Firecracker" trust story users rely on.

We deleted the `firecracker-patch/` directory and filed
[issue #101](https://github.com/deeplethe/forkd/issues/101) with explicit
revival criteria. If two of (a) phase 2/3 ship and the floor is still the
bottleneck, (b) a specific user commits, (c) the uffd_wp design closes
end-to-end, (d) upstream accepts external memfd injection — we re-derive
the patch then.

[memfd]: https://man7.org/linux/man-pages/man2/memfd_create.2.html
[uffdwp]: https://man7.org/linux/man-pages/man2/userfaultfd.2.html

## What v0.3.1 added on top

v0.3.0 restricted diff mode to a sandbox's first BRANCH. Firecracker clears
the dirty bitmap on every snapshot/create, so the Nth diff would silently
lose pages dirtied before BRANCH N-1. v0.3.1 lifts this without a separate
shadow file: each BRANCH's output is, by construction, source's state at that
BRANCH's pause time — exactly the base the next diff needs. The daemon
tracks `last_branch_memory_path` per sandbox and chains.

5 consecutive `"diff": true` BRANCHes on the same sandbox: **4.7 s aggregate
source pause vs ~70 s if these had been Full = 14× over a multi-BRANCH
workflow**.

## What this means in one line

**Source pause for a typical agent fan-out drops 6-15× on commodity SSD,
up to 143× for idle sources, declining toward 1× as the source dirties
>50 % of its RAM.** Opt in with `"diff": true` on `POST
/v1/sandboxes/:id/branch`. Total BRANCH API latency is unchanged — only
source DOWNTIME shrinks (the right trade-off for live BRANCH from a
long-running agent where TCP / kvmclock / timers matter).

## Try it

```bash
pip install forkd                    # 0.3.1 on PyPI
sudo forkd-controller serve \
    --bind 127.0.0.1:8889 \
    --snapshot-root /var/lib/forkd/snapshots \
    --token-file /etc/forkd/token

curl -X POST http://127.0.0.1:8889/v1/sandboxes \
    -H "Authorization: Bearer $(cat /etc/forkd/token)" \
    -d '{"snapshot_tag":"langgraph-react","n":1}'
# spawn source...

curl -X POST http://127.0.0.1:8889/v1/sandboxes/<id>/branch \
    -H "Authorization: Bearer $(cat /etc/forkd/token)" \
    -d '{"diff": true}'
# pause_ms ~200 ms instead of ~14000 ms
```

Recipes and longer-form docs:
- [`docs/design/diff-snapshots.md`](https://github.com/deeplethe/forkd/blob/main/docs/design/diff-snapshots.md) — full design
- [`bench/pause-window/RESULTS-v0.3.md`](https://github.com/deeplethe/forkd/blob/main/bench/pause-window/RESULTS-v0.3.md) — every measurement we took
- [`docs/COMPARISON.md`](https://github.com/deeplethe/forkd/blob/main/docs/COMPARISON.md) — how forkd fits next to Modal / boxlite / CubeSandbox

## What's next

- **Phase 2 (1 week)** — NVMe + io_uring snapshot writer. Targets the
  underlying full-copy backbone (which still bottlenecks total BRANCH API
  latency on SSD even with Diff mode).
- **Phase 3 (1-2 weeks)** — pre-emptive background snapshot. Bounds
  pause-window regardless of source size including for sources with high
  memory churn.
- **Anomaly to profile** — pause_ms jumps ~5× between BRANCH 2 and BRANCH 3
  in the multi-BRANCH sweep even though diff size doesn't grow. Hypothesis:
  KVM control-plane accumulating state across consecutive snapshots. Doesn't
  block v0.3.1 (still 10× better than Full at every cell) but worth a profile.

PRs / issues / fork the repo. Always interested in real workloads to point
the bench at.
