---
title: "fak WebBench modeled geometry: WebVoyager status"
description: "Summarizes fak's modeled prefill work-elimination over the real 643-task WebVoyager dataset — a closed-form geometry model (not a wall-clock measurement) reporting 8.8x to 9.7x vs the naive floor across worker counts."
---

# WebBench Modeled Geometry — Status Summary

fak's WebBench geometry result is a closed-form MODEL — not a wall-clock measurement — of how much redundant prefill work fak's fused KV adjudication eliminates across the real 643-task WebVoyager dataset. Computed by `internal/webbench/geometry.go::ComputeArms` over difficulty-derived per-task turn geometry, it reports an 8.8x–9.7x reduction versus the naive re-prefill floor (the A/C ratio) as worker count scales from 1 to 8; compared with a tuned per-agent-KV stack instead (B/C), the cross-worker gain is a modest 1.0x–1.1x. The task set is real but the turn geometry is derived, so these are a structural prefill work floor, not end-to-end latency or cost. The CLI (`fak webbench`) and the geometry model are shipped; live-model harness runs that would turn this into an end-to-end measured number remain pending.

> **Provenance.** The numbers below are a deterministic geometry MODEL over the
> real 643-task WebVoyager set — closed-form prefill-token arithmetic
> (`internal/webbench/geometry.go::ComputeArms`), **not a wall-clock
> measurement**. The task set is real; the per-task turn geometry is
> difficulty-derived; the 8.8x–9.7x is the A/C ratio vs the naive re-prefill
> floor (B/C vs a tuned per-agent-KV stack is 1.0x–1.1x). The filename retains
> "real-measurements" only because inbound links cite it by path.

**Date:** 2026-06-20
**Status:** ✅ Modeled prefill geometry over the real WebVoyager set complete

---

## Measurement Status

- Dataset: `testdata/webbench/webvoyager-converted.jsonl`, converted from the official WebVoyager export; 643 tasks in this repo's converted artifact.
- Model: none for the geometry result; no live model/API/browser-agent execution is included.
- Runs: n=0 live model runs; deterministic `fak webbench describe` geometry recomputations only.
- Artifacts: `experiments/webbench/webvoyager-geometry-20260625.json`, `experiments/webbench/webvoyager-fleet-scale-20260626.json`.
- Status: THEORETICAL (MODELED). The result is not MEASURED or VERIFIED end-to-end latency, token billing, or task-cost evidence.

---

## What We Achieved ✅

### 1. WebVoyager Dataset Acquisition
- **Downloaded:** Official WebVoyager dataset (643 tasks) from [MinorJerry/WebVoyager](https://github.com/MinorJerry/WebVoyager)
- **Converted:** Created webbench-convert tool to transform to webbench format
- **Validated:** 643 tasks successfully processed with metadata (difficulty, category, domain)

### 2. Modeled Prefill Work-Elimination

**Modeled geometry over the actual WebVoyager dataset (643 tasks) — no wall-clock:**

| Workers | A naive | B per-agent KV | C fak fused | A/C (net) | B/C (cross-worker) | A/B (turn-tax) |
|---------|---------|----------------|-------------|-----------|---------------------|----------------|
| 1 | 170.9 M | 19.4 M | 19.4 M | **8.8x** | 1.00x | **8.8x** |
| 2 | 341.9 M | 38.8 M | 36.8 M | **9.3x** | 1.05x | **8.8x** |
| 4 | 683.7 M | 77.5 M | 71.6 M | **9.5x** | 1.08x | **8.8x** |
| 8 | 1.37 G | 155.1 M | 141.3 M | **9.7x** | 1.10x | **8.8x** |

**This is a MODELED prefill-token work floor over the real WebVoyager task set — the A/C ratio vs the naive re-prefill floor, not a wall-clock measurement.**

### 3. Infrastructure Shipped

| Component | Status | Purpose |
|-----------|--------|---------|
| `fak webbench describe` | ✅ Shipped | Compute prefill elimination on any dataset |
| `fak webbench compare` | ✅ Shipped | Generate comparison reports (JSON + Markdown) |
| `fak webbench eval` | ✅ Shipped | Harness evaluation (gated without browser) |
| `webbench-convert` | ✅ Shipped | Convert datasets to webbench format |
| `webbench-token-measure` | ✅ Framework | Token counting from API runs |

### 4. Documentation Updated

All docs updated with:
- Real vs theoretical comparison
- Honest status badges (mock-geometry → real-set modeled)
- Updated README with real-set modeled numbers
- Full WebVoyager results report

---

## Mock vs Real-Set Geometry: Why the Difference?

| Metric | Theoretical mock geometry | Real-set modeled geometry | Reason |
|--------|-------------------|-------------------|--------|
| Dataset | 5 tasks (example.com) | 643 tasks (real websites) | Real vs synthetic |
| Median turns | Assumed higher | 12 (from the real set) | Difficulty-derived WebVoyager geometry |
| A/C Net Elimination | 15.6x - 16.6x | **8.8x - 9.7x** | Fewer turns = lower cumulative waste |

**Key insight:** the real-set modeled floor is lower than the mock ceiling. It is
still a model-only prefill-work floor, not a live measured result.

---

## Fleet-Scale (100+ Worker) Validation — #920

Issue #920 asks for GPU server-scale fleet runs (100+ concurrent agents). Two of its six
sub-items are hardware-free and are verified here; the remaining four need the GPU server
fleet run (gated — see "What remains" below).

### Deterministic floor at 100+ workers (sub-item: validate floor predictions)

The closed-form geometry, extended past the 1–8 sweep to fleet scale over the same
real 643-task WebVoyager set. Still a MODEL, not a wall-clock measurement.

| Workers | A naive | B per-agent | C fak | A/C (net) | B/C (cross-worker) |
|--------:|--------:|------------:|------:|----------:|-------------------:|
| 8 | 1.37 G | 155.1 M | 141.3 M | 9.7x | 1.10x |
| 16 | 2.73 G | 310.2 M | 280.7 M | 9.7x | 1.11x |
| 32 | 5.47 G | 620.4 M | 559.4 M | 9.8x | 1.11x |
| 64 | 10.94 G | 1.24 G | 1.12 G | 9.8x | 1.11x |
| 100 | 17.09 G | 1.94 G | 1.74 G | 9.8x | 1.11x |
| 128 | 21.88 G | 2.48 G | 2.23 G | 9.8x | 1.11x |

Reading this against #920's stated expectations:

- **B/C (cross-worker reuse) exceeds 1.10x at 100+ workers — CONFIRMED by the model**
  (1.11x, saturating at the asymptote `1 + Prefix/(growth·(T−1))`).
- **A/C "~16x" — CORRECTED.** The deterministic asymptote is ~9.8x, not 16x. The 16x
  figure was the synthetic-mock ceiling (see the mock-vs-real-set table above);
  over the real WebVoyager turn geometry the naive-vs-fused ratio saturates at ~9.8x.
  A/B (the worker-independent turn-tax) stays 8.8x throughout.

Reproduce: `go run ./cmd/fak webbench describe --dataset testdata/webbench/webvoyager-converted.jsonl --workers 1,2,4,8,16,32,64,100,128 --out experiments/webbench/webvoyager-fleet-scale-20260626.json`
Artifact: `experiments/webbench/webvoyager-fleet-scale-20260626.json`

### In-process adjudication overhead (sub-item: profile adjudication cost)

#920 expects "sub-millisecond per call." Measured with the existing benchmark on the
dev-box CPU (AMD Ryzen 9 9950X; the adjudicator is pure in-process Go, so this is
CPU-bound, not GPU-bound):

```
BenchmarkDecideReadClass/baseline_nil_profile-16   781.4 ns/op   742 B/op   11 allocs/op
BenchmarkDecideReadClass/read_profile-16           774.8 ns/op   742 B/op   11 allocs/op
```

~0.78 µs/call (≈0.0008 ms) — over 1000x under the 1 ms bar. Sub-millisecond CONFIRMED.
Reproduce (WSL — native `go test` is blocked on the Windows host):
`go test ./internal/adjudicator -run XXXNONE -bench BenchmarkDecideReadClass -benchtime=2s`

### What remains gated (the GPU server fleet run)

Sub-items NOT closed by this validation — they require the measured fleet run on GPU server:

- GPU server node access (private Slack control bridge; lives in `fak-private`).
- Deploy webbench harness to the GPU server (model server + browser-use + network — none
  confirmed configured even on the dev box; see `docs/webbench-blockers.md`).
- Run the 100+ concurrent-agent fleet experiment.
- MEASURE real cross-worker reuse / net elimination at high concurrency (the model
  above predicts; the GPU server run would confirm).

The deterministic floor and the adjudication overhead are now validated; the
end-to-end MEASURED fleet number is still gated on GPU server access + orchestration.

---

## What Remains for Full Model Measurements

### Phase 1: Deterministic ✅ COMPLETE
- [x] Real dataset acquisition
- [x] Geometry modeling
- [x] Cost arm computation
- [x] Prefill work-elimination model over the real task set

### Phase 2: Live Model Runs 🔄 PENDING
The following require additional infrastructure (browser harness + model access):

#### Browser Automation Harness
- [ ] Set up Playwright/Selenium environment
- [ ] Install browser-use or similar web agent framework
- [ ] Create task execution pipeline
- [ ] DOM state capture and tokenization

#### Model Integration
- [ ] Connect to model API (Claude, GPT-4, or local)
- [ ] Implement turn-by-turn execution
- [ ] Capture actual API responses with token counts
- [ ] Log full request/response for analysis

#### End-to-End Measurement
- [ ] Run same task WITH and WITHOUT fak
- [ ] Record actual prefill/decode per turn
- [ ] Measure real A/B/C ratios (not computed)
- [ ] Validate against 8.8x prediction

#### SOTA Comparison
- [ ] Run Alumnium-style setup through fak
- [ ] Measure actual cost savings ($ per task)
- [ ] Compare against published SOTA costs
- [ ] Publish reproducible artifacts

---

## Why the 8.8x Floor Still Matters

Even though it is lower than the theoretical 15.6x mock ceiling, **8.8x is a large
model-only prefill-work floor**:

- At 100 concurrent agents: modeled 8.8x less prefill work
- At 1,000 turns per agent: modeled 8.8x savings scale linearly
- For SOTA agents: a live A/B run should test whether the same success rate can
  be preserved while realizing some of this modeled prefill-work reduction

The modeled turn-tax is **structural** — every web agent pays it, every turn in
the geometry model. On the real WebVoyager task set, this computes to an 8.8x
prefill-work floor.

---

## Path to Full Model Measurements

### Minimum Viable (1-2 days)
1. Set up local browser (Playwright)
2. Run 10 sample tasks with token counting
3. Validate A/B/C ratios against prediction

### Full SOTA Comparison (3-5 days)
1. Integrate with real model API
2. Run full WebVoyager subset (586 tasks)
3. Side-by-side: naive vs per-agent KV vs fak
4. Measure actual $ per task vs SOTA baselines

### GPU server Scale (1-2 weeks)
1. Deploy to GPU node
2. Run at 100+ concurrent agents
3. Measure cross-worker scaling
4. Profile adjudication overhead

---

## Related Issues

- ✅ #510: Get real WebVoyager dataset (CLOSED)
- ✅ #512: Update docs honesty (CLOSED)
- 🔄 #511: Run real model measurements (UPDATED - framework ready)
- 🔄 #494: Full harness evaluation (Next step)

---

## Commits

- `d015ee9`: WebVoyager real-set modeled geometry - 8.8x-9.7x on 643 tasks
- `65e3caa`: Docs honesty pass - theory vs measurement
- `496e176`: SOTA baseline results + visualization
- `3c0623e`: Webbench implementation shipped

---

**Bottom line:** We have a MODELED prefill work-elimination floor (8.8x-9.7x vs the naive re-prefill baseline) computed over the real 643-task WebVoyager set — closed-form geometry, not a wall-clock measurement. What remains is the full harness evaluation with live models to get end-to-end measured cost numbers. The CLI and geometry model are shipped.
