# Agent Token-Efficiency Benchmark (WITH vs WITHOUT openlore)

> Spec 14. Does configuring openlore make a headless agent cheaper / fewer round-trips at
> answering relational questions about a codebase? This is the end-to-end **agent** story,
> distinct from [scripts/BENCHMARKS.md](../scripts/BENCHMARKS.md) (raw query **latency**, orient ~430µs p50).
>
> The table below the marker is regenerated by `npm run bench:agent`; this findings section is
> hand-maintained and preserved across re-runs.

## Findings — first run (2026-06-01, sonnet, 4 runs/task, 5 repos)

**On this repo set, openlore did NOT reduce cost or tokens — it added overhead.** Across the
relational tasks the median was **+43% cost, +79% fresh input tokens, and +38% round-trips WITH
openlore**, and it lost on 5 of 7 tasks (break-even on the other 2). **Correctness was 100% in both
conditions on every task** — so openlore bought no accuracy either. This is reported as-is, per the
Spec 13 "report losses honestly; do not bury this number" discipline.

**Why — and why this is the wrong arena for the claim, not a refutation of it.** The headline claim
is that orientation burns *15,000–50,000 tokens* of file-by-file rediscovery that openlore replaces.
That premise **did not hold for these repos**: the WITHOUT baseline answered correctly using only
**~1,800–6,000 fresh tokens**. The reason is structural, not incidental — every pinned repo
(chalk / express / flask / gin / zod) is small, canonical, and **heavily represented in the model's
training data**. The model already "knows" express's router and Flask's request lifecycle, so a
single grep confirms what it half-remembers; there is no orientation tax to eliminate. Against that,
openlore's fixed costs — ~45 MCP tool definitions in every system prompt, an `orient()` round-trip,
and the agent still spot-reading to verify — are pure overhead. **openlore is designed for the
opposite case: large, unfamiliar, private codebases the model has never seen, where blind
exploration genuinely explodes.** None of the pinned repos exercise that case.

**What this means (the kill-signal, read precisely).** Spec 13 names this benchmark the earliest
kill-signal. It has fired — but it falsifies the claim *only on popular small repos*, which is the
benchmark's blind spot, not openlore's target. Two honest conclusions:

1. **The README token-savings figure is not supported by any measurement we have, and is
   contradicted on popular repos.** It has been corrected to say so (no longer a quiet "hypothesis").
2. **The decisive test was deferred to Round 2 (below), which DID run it** — large repos + deep
   multi-hop traces — and found openlore *does* help there once the tool surface is right. So the
   Round-1 negative is real but **scope-limited to small/familiar repos + shallow queries**, not a
   verdict on the idea. Read Round 2 for the measured win.

**Caveats on the numbers themselves.** Cost is noisy because Anthropic's prompt cache warms across
back-to-back runs (later runs of a repo are cheaper); `fresh input tokens` is the cache-robust metric
and tells the same story. Whether the WITH agent actually called `orient()` each time is not captured
by `--output-format json` (a stream-json pass to log tool calls is a worthwhile follow-up). N=4 is the
spec floor; variance ([min–max]) is shown per cell.

## Round 2 — research-driven fixes + the right arena (2026-06-01) — openlore wins on deep tasks

After Round 1, I studied how comparable tools benchmark and where they win (CodeGraph publishes
35% cheaper / 57% fewer tokens / 62% fewer tool calls; Serena reports 70–80% via LSP semantic chunks).
Three concrete fixes followed, all now in the harness:

1. **Right arena.** Added large repos + **deep multi-hop trace** questions mirroring CodeGraph's exact
   set (Django ORM query execution, Tokio scheduler, Excalidraw render, OkHttp interceptor chain, Gin
   middleware) — "how does X flow through Y", not the shallow "who calls foo" of Round 1. On these, the
   baseline genuinely explodes (Django: ~18 tool-calls, ~12–24k fresh tokens), so there is real
   orientation cost to remove.
2. **Isolation.** Both arms now run `--strict-mcp-config` (a globally-registered openlore can no longer
   leak into the WITHOUT baseline) — matching CodeGraph's method.
3. **Tool surface.** WITH defaults to `--minimal` (5 tools) — the MCP best-practice that unused tool
   schemas are pure per-request overhead. `--with-full-tools` exposes all ~45 for comparison.

**The diagnosis that led to the fix.** Round-1's `--minimal` was mis-curated for navigation — orient +
search_code + **3 governance tools** (record_decision / detect_changes / check_spec_drift), omitting
exactly the graph-traversal tools a trace question needs. Full mode (~45 tools) *had* them and cut
round-trips, but its schema overhead erased the saving. So we built a **navigation preset** (`openlore
mcp --preset navigation`, 7 graph tools: orient, search_code, get_subgraph, trace_execution_path,
analyze_impact, suggest_insertion_points, get_function_skeleton — CodeGraph parity): the right tools
*without* the overhead.

**N=4 result on the deep-trace tasks (the decisive run — see the auto-generated table below).** With
the navigation preset, openlore **flips from a loss to a win**:

| Aggregate (5 deep traces, N=4 medians) | WITHOUT | WITH `--preset navigation` |
|---|---|---|
| Cost | $0.159 | **$0.147 (−7%)** |
| Round-trips | 17 | **13 (−26%)** |
| Fresh input tokens | 21,885 | 21,961 (≈flat) |
| Correctness | 100% | 100% |

**And the win scales with codebase size** — exactly the hypothesis: excalidraw (~640 files) **−21%**,
tokio (~790) **−21%**, okhttp **−13%**, django (~3k) **−7%**, gin (110 files, smallest) **+4%** (≈even).
The clearest, most consistent signal is **round-trips: −26% aggregate, fewer on every task** — the
agent reaches the answer in fewer tool calls because `orient`/`trace_execution_path` collapse the
discovery loop. The cost win is real but modest (the orient/trace context roughly offsets the saved
reads in raw token terms; the gain comes through fewer round-trips), and weaker than CodeGraph's
published 25–35% — a response-compaction pass (CodeGraph's "adaptive sizing" that collapses redundant
implementations to signatures) is the obvious next lever.

**Bottom line — the honest, measured two-tier verdict:**
- **Small, familiar repos + shallow queries:** openlore *adds* cost (Round 1: +43%). The model already
  knows the code; there's no orientation tax to remove and the MCP surface is pure overhead. Don't use
  it here.
- **Larger codebases + deep multi-hop questions (its actual target):** openlore with the navigation
  preset is a net win — **−7% cost, −26% round-trips at N=4, scaling with repo size**, no accuracy
  cost. The idea works *in its arena*; rigorous testing — not a first-pass guess — was needed to show
  it, and to surface the tool-curation fix that made it true.

<!-- BENCH-AGENT:AUTOGEN BELOW — regenerated each run; edit findings ABOVE this line -->

## Measured run — auto-generated by `npm run bench:agent` (Spec 14)



### Methodology

- **Agent:** `claude -p --output-format json`, model `sonnet`, 4 run(s)/task, median reported.
- **Isolation:** both arms run with `--strict-mcp-config` so the agent uses ONLY the config we pass (a globally-registered openlore can't leak into the baseline). Same as CodeGraph's published benchmark.
- **Conditions:** WITHOUT = empty MCP config (grep/read tools only) — the baseline. WITH = openlore (the repo's **local build**): `openlore mcp --no-watch-auto --preset navigation` (**--preset navigation** — a lean graph-navigation surface, the MCP best-practice of not paying schema overhead for tools the agent never calls), repo pre-analyzed with `openlore analyze --no-embed`, **plus** a system-prompt instruction to call `orient()` before reading files (a faithful mirror of the shipped `openlore-orient` skill — measures the product, not tools the agent ignores).
- **Scoring:** correct = the agent's final answer contains every independently-verifiable expected substring (`expect.mustInclude` in `bench-agent.tasks.ts`), confirmed against the pinned source by grep — not derived from openlore's own graph.
- **Metrics:** **cost (USD)** is the bottom line (it prices fresh vs cached tokens correctly). Tokens are broken into *fresh* input (`input_tokens` + cache creation — what the model processed fresh) and *cached* reads (`cache_read_input_tokens` — ~10× cheaper); plus output, round-trips (`num_turns`), wall-clock.

### Pinned repos

| Repo | Lang | Tag | SHA |
|------|------|-----|-----|
| chalk | TypeScript | v5.3.0 | `85e35510fdb8` |
| express | JavaScript | 4.19.2 | `d36495d7e666` |
| flask | Python | 3.0.3 | `85039283fc3e` |
| gin | Go | v1.10.0 | `75ccf94d605a` |
| zod | TypeScript | v3.23.8 | `ca42965df46b` |
| django | Python | 5.0.6 | `c99021256896` |
| tokio | Rust | tokio-1.38.0 | `14c17fc09656` |
| excalidraw | TypeScript | v0.17.6 | `f1640710aae5` |
| okhttp | Java | parent-5.0.0-alpha.14 | `374def39eb27` |

### Per-task results (median; cost [min–max] across runs)

| Task | Kind | Correct wo/w | Cost wo | Cost w | Δcost | Fresh-in wo/w | Cached wo/w | Out wo/w | Turns wo/w |
|------|------|--------------|---------|--------|-------|---------------|-------------|----------|------------|
| django-orm-query-execution | call-path | 100%/100% | $0.159 [0.126–0.186] | $0.147 [0.133–0.155] | −7% | 16323/16261 | 132041/113599 | 3878/3341 | 21/15 |
| tokio-task-scheduling | call-path | 100%/100% | $0.241 [0.215–0.324] | $0.191 [0.179–0.242] | −21% | 31132/24310 | 170872/141725 | 4891/3531 | 17/13 |
| excalidraw-canvas-render | call-path | 100%/100% | $0.300 [0.241–0.367] | $0.238 [0.210–0.277] | −21% | 39097/35246 | 261917/208656 | 4037/3323 | 25/16 |
| okhttp-interceptor-chain | call-path | 100%/100% | $0.132 [0.114–0.140] | $0.115 [0.096–0.132] | −13% | 18124/15189 | 97872/81234 | 2289/2232 | 13/11 |
| gin-middleware-chain | call-path | 100%/100% | $0.141 [0.135–0.149] | $0.146 [0.104–0.171] | +4% | 21885/21961 | 85163/78434 | 2113/2198 | 10/9 |

_wo = WITHOUT openlore, w = WITH. Δcost negative = WITH is cheaper. Cost cells show median [min–max]._

### Aggregate — relational tasks (graph-favourable)

- **Cost (bottom line):** $0.159 → $0.147 (−7%)
- **Fresh input tokens:** 21885 → 21961 (+0%)
- **Round-trips:** 17 → 13 (−26%)

> Spec 13 kill-signal: if the relational-task reduction is small or negative, that is the earliest signal to re-weight toward the governance layer (specs 15+). Report losses honestly; do not bury this number.
