---
version: 0.16.0
date: 2026-06-18
headline: "SWE-bench Verified in fak: directly comparable to the bench harness on the metrics we care about."
themes: ["swebench", "bench", "metrics"]
highlights:
  - "New `fak swebench` verb: describe / eval / compare on the real 500-instance set."
  - "Value-stack floor on SWE-bench Verified: turn-tax 17.9x, cross-worker reuse up to 1.31x — deterministic, no model."
  - "In-process vs spawn-per-hook adjudication gate folds into the comparison (~2400x here)."
  - "Resolve-rate via the OFFICIAL swebench harness — runs on Docker boxes, honestly gated here."
  - "Metrics keyed to bench's own vocabulary; adversarially reviewed for honesty."
---

**TL;DR** — fak can now run **SWE-bench Verified** as a native benchmark whose
results are **directly comparable** to the external Benchmark tool ("bench") that
runs the same 500-instance set against an SGLang endpoint — on the four metric
families we care about (prefill/KV-reuse, turns+tokens, in-process adjudication
cost, resolve-rate+safety). It runs **now, on this Mac, with no GPU/Docker/network**
and scales to a real coding-agent resolve run with a Qwen3.6-27B-class model on the
GPU server via the identical upstream harness. Built understand→build→adversarially-review.

## SWE-bench Verified, comparable to bench (#swebench #bench #metrics)

- **What changed** — a new `internal/swebench` package + `fak swebench` verb
  (`describe` / `eval` / `compare`).
  - *Why:* bench runs SWE-bench Verified as a KV-cache/serving benchmark
    (`exec-sweep --sweep-workers 1,2,4,8` → `mini-swe-agent` → SGLang); fak's thesis
    is also about the KV cache, so the honest comparison is fak's harness-cost /
    cache-reuse metrics on the same workload, in bench's own vocabulary.
  - *How:* `instance.go` (official `princeton-nlp/SWE-bench_Verified` schema + loaders
    for the full dataset AND bench's offline 500-id difficulty map); `geometry.go`
    (per-instance P,T,D,R from real instance signal); `cost.go` (the deterministic
    prefill-token work-elimination floor — sessionbench's A/B/C formula per instance);
    `report.go` (the four-family comparison keyed to bench's `critical-metrics`
    vocabulary, JSON + markdown, ingests a bench `results_*.json`); `eval.go` (the
    official `swebench.harness.run_evaluation` seam).
  - *Measured here, real 500-instance set:* turn-tax A/B **17.9×**, cross-worker
    prefix-reuse B/C up to **1.31×** at 16 workers, adjudication fusion **~2400×**
    (in-process vs spawn-per-hook). Witness: `go run ./cmd/fak swebench compare
    --with-adjudication`; `fak/SWEBENCH-RESULTS.md`.

- **Honesty (METRICS-HONESTY discipline, adversarially reviewed).** The prefill
  A/C, B/C and turn-tax A/B are **fak-vs-fak ablation arms** (naive-re-prefill vs
  per-agent-KV vs fak-fused), tagged **fak-native** — *not* a head-to-head vs a tuned
  SGLang `seq_cp` server, and a deterministic floor, not a measured wall-clock. Only
  turns (`actual_agent_steps`) and resolve-rate are true head-to-heads. Resolve-rate
  is honestly **gated** on a no-Docker box (prints the exact GPU server command), never
  fabricated; the denominator mirrors bench's `_count_report`. A 3-lens adversarial
  review (correctness + honesty + bench-fidelity) verdict was *ship-after-must-fix*;
  every must/should fix landed before this release.

## Bigger models on the M3 (#model)

- **What changed** — measured rows for running 2×/4×-larger models on the M3's
  unified memory (the 36 GB unified bank busts the 8 GB/7B VRAM ceiling): a 4×-dense
  Qwen2.5-32B Q4 row at **6.7 tok/s @ 18.5 GiB**.
  - *How:* `fak/GPU*.md` / model-results docs (peer-authored, folded into this tag).
