---
title: "SWE-bench Verified: fak vs bench, Comparable"
description: "A fak-native SWE-bench Verified benchmark directly comparable to the bench tool on cost, cache-reuse, turns, and adjudication, with resolve-rate GPU server-gated."
---

# SWE-bench Verified — fak ↔ bench, directly comparable

> **What this is.** A fak-native SWE-bench Verified benchmark whose results are
> **directly comparable** to the external **Benchmark tool** ("bench",
> `<benchmark-checkout>`) that runs the same 500-instance task set against
> an SGLang endpoint — on the four metrics fak is built to move. It runs **now, on
> this Mac, with no GPU / Docker / network**, and scales to a real coding-agent
> resolve run with a Qwen3.6-27B-class model on the GPU server via the **identical**
> upstream harness.
>
> **Status: the cost/cache comparison is SHIPPED & runnable; resolve-rate is wired
> and GPU server-gated.** `go build ./... && go vet ./... && go test ./internal/swebench/`
> green on this box (go1.26, darwin/arm64).

---

## Measurement Status

- Dataset: SWE-bench Verified 500-instance task set geometry, using the local difficulty map by default or `--dataset <princeton-nlp/SWE-bench_Verified export>` for real token geometry.
- Model: none for `describe`/computed value-stack rows; local adjudication microbench for in-process cost; Qwen3.6-27B-class GPU-server model required for resolve-rate.
- Runs: n=0 live coding-agent resolve runs in this document; deterministic geometry is runnable locally, and in-process adjudication is measured locally.
- Artifacts: generated by `fak swebench compare --out swebench-compare.json --md swebench-compare.md`; resolve-rate artifacts are pending a GPU-server run.
- Status: THEORETICAL (MODELED) for value-stack floors, MEASURED for local adjudication cost, and not MEASURED/VERIFIED for resolve-rate.

---

## Why this exists — the comparison contract

bench runs SWE-bench Verified as a **KV-cache / serving** benchmark: `bench
exec-sweep --swebench-preset verified --serve-preset <model> --target <dgx>
--sweep-workers 1,2,4,8` drives the upstream **`mini-swe-agent`** (`mini-extra
swebench`) against a self-hosted **SGLang OpenAI-compatible endpoint**, scrapes
SGLang's **Prometheus** for the server stream (TTFT/throughput/**cache-hit**/KV),
and grades predictions with the upstream `python -m swebench.harness.run_evaluation`.
It emits one `results_<run_id>.json` (`schema_version` 6; loader tolerates v4).

fak's whole thesis is *also* about the KV cache (the session value stack, prefix
reuse, the turn-tax). So the honest, direct comparison is: **fak's harness-cost /
cache-reuse metrics vs the same SWE-bench Verified workload bench measures** —
expressed in **bench's own metric vocabulary** (its `critical-metrics.md`: an
*agent stream* and a *server stream*) so the numbers slot side-by-side.

---

## The four metric families (keyed to bench's vocabulary)

Each is tagged **comparable** (bench measures an analog → a true head-to-head) or
**fak-native** (fak's differentiator, no bench analog), and **computed**
(deterministic arithmetic), **live** (measured here), or **gated** (GPU server-only).

| family | kind vs bench | provenance | headline (real 500-instance set) |
|---|---|---|---|
| **prefill / KV-reuse work-elimination** | **fak-native** (related to, not the same quantity as, server-side cache-hit) | computed | A/C **17.9×→23.4×** (workers 1→16); B/C **1.0×→1.31×** |
| **turns + tokens** | comparable — agent-stream `actual_agent_steps` | computed | Σ steps **9 846**, median **22**; turn-tax A/B **17.9×** (fak-native lever) |
| **in-process adjudication cost** | **fak-native** — bench has no analog | live (this Mac) | in-process p50 **~2.4 µs** vs spawn-per-hook **~5.8 ms** ⇒ **~2 400×** |
| **resolve-rate + safety** | comparable — agent-stream `resolved_count` / `pass_rate_pct` | gated (GPU server) | gated here (≈0 with the local 135M model); real number is a GPU server/Qwen3.6-27B run |

### The value-stack floor (deterministic, no model)

The prefill-token work-elimination is **exact arithmetic** over the real instance
geometry (sessionbench's proven A/B/C formula) — the *contention-free floor* under
any measured wall-clock, so it cannot drift with machine load:

| workers | A naive (re-prefill/turn) | B per-agent KV | C fak fused | **A/C** | **B/C** | A/B (turn-tax) |
|---:|---:|---:|---:|---:|---:|---:|
| 1 | 89.4 M | 4.99 M | 4.99 M | **17.9×** | 1.00× | 17.9× |
| 4 | 357 M | 19.95 M | 16.2 M | **22.1×** | 1.23× | 17.9× |
| 8 | 715 M | 39.9 M | 31.2 M | **22.9×** | 1.28× | 17.9× |
| 16 | 1.43 B | 79.8 M | 61.1 M | **23.4×** | 1.31× | 17.9× |

- **A/B = 17.9×** is the **turn-tax**: a re-prefill-every-turn harness reprocesses
  the whole growing context each turn; KV persistence reprocesses only the new
  result. Worker-independent.
- **B/C** is **cross-worker prefix reuse** — the system+tool preamble prefilled
  **once** for all workers vs **once per worker**. It grows with the worker sweep
  (the exact `--sweep-workers 1,2,4,8` axis bench uses). It is *thematically* the
  same "don't redo prefill" story bench's server-side `token_hit_ratio` also tracks,
  but it is a **different quantity** — an unbounded work-ratio (≥1), not a bounded
  0–100 % hit fraction — and a fak-vs-fak arm, **not** that server metric.

> **Honesty (METRICS-HONESTY discipline).** A/C, B/C and the turn-tax A/B are
> fak-vs-**fak-harness-arms** ratios (naive re-prefill vs per-agent-KV vs fak-fused),
> tagged **fak-native** and reported as such — **not** a head-to-head vs a tuned
> SGLang `kv_unified`+`seq_cp` server (which *also* reuses a shared prefix, so there
> is no fair-comparator win to claim here). They are a *deterministic floor* (exact
> token arithmetic), **not** a measured wall-clock; live TTFT/wall-clock is the gated
> GPU server number. The bucket-derived turn counts are **estimates** (labeled
> `geometry_source: difficulty`) — a recorded `mini-swe-agent` trajectory supplies
> real `actual_agent_steps` on the GPU server path. The only **comparable** (true
> head-to-head) families are turns (`actual_agent_steps`) and resolve-rate.

---

## Runnable now (this Mac) vs GPU server

| capability | here | GPU server |
|---|---|---|
| dataset ingest (500 ids + official difficulty buckets) | ✓ | ✓ + full problem-statement token geometry |
| value-stack floor (prefill work-elimination, all 4 families' computed parts) | ✓ | ✓ |
| in-process vs spawn-per-hook adjudication gate | ✓ (measured) | ✓ |
| resolve-rate (official harness) | gated (no Docker) — prints the exact GPU server command | ✓ with Docker + a capable model |
| live coding-agent solve (mini-swe-agent → `fak serve` → Qwen3.6-27B) | — | ✓ (roadmap: `fak serve` `/metrics` so bench scrapes fak like SGLang) |

---

## Reproduce

```bash
cd fak
# the value-stack floor on the real 500-instance set, across the worker sweep
go run ./cmd/fak swebench describe --workers 1,2,4,8

# the full four-family comparison + the adjudication gate, to JSON + markdown
go run ./cmd/fak swebench compare --workers 1,2,4,8 --with-adjudication \
  --md swebench-compare.md --out swebench-compare.json

# grade predictions into the resolve-rate (gated here; prints the GPU server command)
go run ./cmd/fak swebench eval --predictions preds.json

# side-by-side vs a real bench run
go run ./cmd/fak swebench compare --bench-result /path/to/results_<run_id>.json --md compare.md
```

`describe`/`compare` default to the local bench difficulty map when present; pass
`--dataset <princeton-nlp/SWE-bench_Verified export>` for real token geometry.

---

## The integration gap (named, not hidden)

To be **scraped by bench exactly like SGLang**, `fak serve` needs a Prometheus
`/metrics` route exposing `kernel.Counters()` (vDSO hits / quarantines / denies)
and the prefix-reuse stats. Today those counters live only in-process; the gateway
exposes no `/metrics`. Closing that makes fak a drop-in measured endpoint in bench's
`monitor` loop — the deepest form of "directly comparable." Until then, fak emits
the comparison artifact (`compare --out/--md`) and bench's `compare`/`chart`
pipeline can consume it.
