# contextweaver — Benchmark Scorecard

> Auto-generated by `make scorecard`. Do not edit by hand.
> Source: `benchmarks/results/latest.json` (produced by `make benchmark`).

- Harness version: `1.2`
- Seed: `42`
- Rank cutoff `k`: `5`
- Token estimator: `CharDivFourEstimator` (deterministic, no model dependency)
- Answer-phase budget: `6000` tokens

All numbers below are reproducible deterministically by running
`make benchmark && make scorecard` from a clean checkout. Hardware and
Python version affect latency only; recall, drops, dedup, and token
counts are environment-independent.

---

## Routing accuracy & latency

Gold dataset: 200 hand-curated queries (`benchmarks/routing_gold.json`)
covering all 8 catalog namespaces. Each query is repeated three times
for latency percentile stability.

| catalog_size | queries | precision@5 | recall@5 | MRR | p50 (ms) | p95 (ms) | p99 (ms) |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 50 | 131 | 0.1191 | 0.5649 | 0.4978 | 0.423 | 0.548 | 0.759 |
| 83 | 200 | 0.0800 | 0.3825 | 0.3242 | 0.667 | 0.779 | 1.134 |
| 1000 | 200 | 0.0310 | 0.1475 | 0.1456 | 36.007 | 37.467 | 41.711 |

### Per-backend × per-size matrix

Generated by `make benchmark-matrix` (issue #208). The matrix lets
you compare the bundled `tfidf` baseline against the optional
`bm25` (core dep), `fuzzy` (`[retrieval]` extra), and embedding
backends (`embedding_hashing` stdlib baseline, `embedding_st`
`[embeddings]` extra — #266) across three catalog sizes.
Skipped backends are recorded explicitly.

| backend | catalog_size | queries | precision@5 | recall@5 | MRR | p50 (ms) | p95 (ms) | p99 (ms) | status |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---|
| bm25 | 100 | 200 | 0.0800 | 0.3825 | 0.3399 | 5.180 | 7.166 | 8.140 | ok |
| bm25 | 500 | 200 | 0.0470 | 0.2250 | 0.2165 | 28.506 | 36.817 | 38.989 | ok |
| bm25 | 1000 | 200 | 0.0330 | 0.1575 | 0.1525 | 70.705 | 86.805 | 111.716 | ok |
| embedding_hashing | 100 | 200 | 0.1050 | 0.5175 | 0.4360 | 0.898 | 6.847 | 7.225 | ok |
| embedding_hashing | 500 | 200 | 0.0560 | 0.2700 | 0.2674 | 9.733 | 38.126 | 44.182 | ok |
| embedding_hashing | 1000 | 200 | 0.0420 | 0.2000 | 0.1931 | 36.778 | 93.261 | 98.277 | ok |
| embedding_st | 100 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| embedding_st | 500 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| embedding_st | 1000 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| fuzzy | 100 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| fuzzy | 500 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| fuzzy | 1000 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| tfidf | 100 | 200 | 0.0800 | 0.3825 | 0.3220 | 0.930 | 1.039 | 1.102 | ok |
| tfidf | 500 | 200 | 0.0490 | 0.2325 | 0.2314 | 9.600 | 10.311 | 11.492 | ok |
| tfidf | 1000 | 200 | 0.0310 | 0.1475 | 0.1456 | 36.059 | 39.739 | 50.755 | ok |

### Mixed-namespace catalog (500 tools, head + long tail)

Generated by `python benchmarks/benchmark.py --matrix --mixed-shapes`
(issue #277). The mixed-shape catalog has one head namespace
(`analytics_xl`, 200 items), two mid-weight namespaces, four small
operational namespaces, and a 100-namespace long tail (one item
each) — a deliberately asymmetric distribution that contrasts
with the uniform 8-namespace pool used by the headline matrix.
Numbers here are intentionally lower: gold queries only cover
the natural 8 namespaces, so the synthetic head-and-tail tools
act as targeted noise items.

| backend | catalog_size | queries | precision@5 | recall@5 | MRR | p50 (ms) | p95 (ms) | p99 (ms) | status |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---|
| bm25 | 500 | 200 | 0.0260 | 0.1200 | 0.1065 | 19.612 | 23.858 | 25.717 | ok |
| embedding_hashing | 500 | 200 | 0.0340 | 0.1550 | 0.1376 | 10.514 | 39.809 | 42.553 | ok |
| embedding_st | 500 | — | — | — | — | — | — | — | skipped: missing sentence-transformers |
| fuzzy | 500 | — | — | — | — | — | — | — | skipped: missing rapidfuzz |
| tfidf | 500 | 200 | 0.0260 | 0.1200 | 0.1010 | 9.925 | 10.599 | 11.190 | ok |

### Per-namespace recall@k

Breakdown of recall@k by gold-set namespace (issue #209). Useful
for identifying which tool namespaces benefit most from a given
scoring backend.

| backend | catalog_size | namespace | queries | recall@5 |
|---|---:|---|---:|---:|
| bm25 | 100 | admin | 25 | 0.9200 |
| bm25 | 100 | analytics | 25 | 0.7200 |
| bm25 | 100 | billing | 25 | 0.1200 |
| bm25 | 100 | comms | 25 | 0.4000 |
| bm25 | 100 | crm | 25 | 0.1200 |
| bm25 | 100 | docs | 25 | 0.2800 |
| bm25 | 100 | infra | 25 | 0.0800 |
| bm25 | 100 | search | 25 | 0.4200 |
| bm25 | 500 | admin | 25 | 0.4600 |
| bm25 | 500 | analytics | 25 | 0.2400 |
| bm25 | 500 | billing | 25 | 0.5800 |
| bm25 | 500 | comms | 25 | 0.3600 |
| bm25 | 500 | crm | 25 | 0.0400 |
| bm25 | 500 | docs | 25 | 0.0800 |
| bm25 | 500 | infra | 25 | 0.0400 |
| bm25 | 500 | search | 25 | 0.0000 |
| bm25 | 1000 | admin | 25 | 0.5800 |
| bm25 | 1000 | analytics | 25 | 0.4000 |
| bm25 | 1000 | billing | 25 | 0.2400 |
| bm25 | 1000 | comms | 25 | 0.0000 |
| bm25 | 1000 | crm | 25 | 0.0400 |
| bm25 | 1000 | docs | 25 | 0.0000 |
| bm25 | 1000 | infra | 25 | 0.0000 |
| bm25 | 1000 | search | 25 | 0.0000 |
| embedding_hashing | 100 | admin | 25 | 0.4000 |
| embedding_hashing | 100 | analytics | 25 | 0.5200 |
| embedding_hashing | 100 | billing | 25 | 0.3200 |
| embedding_hashing | 100 | comms | 25 | 0.4400 |
| embedding_hashing | 100 | crm | 25 | 0.6400 |
| embedding_hashing | 100 | docs | 25 | 0.5600 |
| embedding_hashing | 100 | infra | 25 | 0.6800 |
| embedding_hashing | 100 | search | 25 | 0.5800 |
| embedding_hashing | 500 | admin | 25 | 0.5000 |
| embedding_hashing | 500 | analytics | 25 | 0.4400 |
| embedding_hashing | 500 | billing | 25 | 0.5800 |
| embedding_hashing | 500 | comms | 25 | 0.3200 |
| embedding_hashing | 500 | crm | 25 | 0.0400 |
| embedding_hashing | 500 | docs | 25 | 0.1600 |
| embedding_hashing | 500 | infra | 25 | 0.0400 |
| embedding_hashing | 500 | search | 25 | 0.0800 |
| embedding_hashing | 1000 | admin | 25 | 0.5800 |
| embedding_hashing | 1000 | analytics | 25 | 0.5200 |
| embedding_hashing | 1000 | billing | 25 | 0.3800 |
| embedding_hashing | 1000 | comms | 25 | 0.0000 |
| embedding_hashing | 1000 | crm | 25 | 0.0400 |
| embedding_hashing | 1000 | docs | 25 | 0.0000 |
| embedding_hashing | 1000 | infra | 25 | 0.0000 |
| embedding_hashing | 1000 | search | 25 | 0.0800 |
| tfidf | 100 | admin | 25 | 0.9200 |
| tfidf | 100 | analytics | 25 | 0.7200 |
| tfidf | 100 | billing | 25 | 0.1200 |
| tfidf | 100 | comms | 25 | 0.4000 |
| tfidf | 100 | crm | 25 | 0.1200 |
| tfidf | 100 | docs | 25 | 0.2800 |
| tfidf | 100 | infra | 25 | 0.0800 |
| tfidf | 100 | search | 25 | 0.4200 |
| tfidf | 500 | admin | 25 | 0.4600 |
| tfidf | 500 | analytics | 25 | 0.2000 |
| tfidf | 500 | billing | 25 | 0.5400 |
| tfidf | 500 | comms | 25 | 0.3600 |
| tfidf | 500 | crm | 25 | 0.0400 |
| tfidf | 500 | docs | 25 | 0.0800 |
| tfidf | 500 | infra | 25 | 0.0400 |
| tfidf | 500 | search | 25 | 0.1400 |
| tfidf | 1000 | admin | 25 | 0.5800 |
| tfidf | 1000 | analytics | 25 | 0.3600 |
| tfidf | 1000 | billing | 25 | 0.2000 |
| tfidf | 1000 | comms | 25 | 0.0000 |
| tfidf | 1000 | crm | 25 | 0.0400 |
| tfidf | 1000 | docs | 25 | 0.0000 |
| tfidf | 1000 | infra | 25 | 0.0000 |
| tfidf | 1000 | search | 25 | 0.0000 |

Reading the routing tables:

- `precision@5` is bounded by `1 / k` when each query has a single
  expected tool, so the headline accuracy signal is `recall@k` and `MRR`.
- Recall degrades predictably as the catalog grows — noise items
  compete with true matches; the routing-only experience for catalogs
  larger than ~200 items benefits from one of the optional retrieval
  backends (`bm25`, `fuzzy`) configured via `Router(scorer_backend=...)`.
- p50/p95 latencies stay under a millisecond at catalog_size ≤ 83.
  At catalog_size 1000 the p99 climbs into the tens of milliseconds
  because the beam search has to evaluate substantially more children
  per step; this is the regime where switching to a retriever-first
  shortlist (the `Retriever` protocol on the `EngineRegistry`) is the
  expected next step.

---

## Context pipeline scenarios

Reference event logs under `benchmarks/scenarios/` are pushed through
`ContextManager.build_sync(phase=Phase.answer)`. The firewall
intercepts every `tool_result`; large results become artifacts and the
prompt sees their summaries instead.

| scenario | events | included | dropped | dedup | tokens | budget | util % | artifacts | compaction |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| large_catalog | 60 | 60 | 0 | 0 | 1514 | 6000 | 25.2% | 15 | 1.00x |
| long_conversation | 82 | 82 | 0 | 0 | 2548 | 6000 | 42.5% | 21 | 1.41x |
| mixed_payload | 15 | 15 | 0 | 0 | 497 | 6000 | 8.3% | 4 | 1.64x |
| short_conversation | 18 | 18 | 0 | 0 | 496 | 6000 | 8.3% | 4 | 1.00x |
| stress_conversation | 147 | 136 | 7 | 4 | 6651 | 6000 | 110.9% | 32 | 3.29x |
| tiny_payload | 20 | 20 | 0 | 0 | 267 | 6000 | 4.5% | 5 | 1.00x |

Reading the table:

- `dropped > 0` means `select_and_pack` had to evict candidates to
  stay under the answer-phase budget — the `stress_conversation`
  scenario is sized to force this so the budget-driven selection
  stage shows up in benchmark output (#181).
- `dedup > 0` proves the Jaccard near-duplicate stage actively
  removed redundant context.
- `compaction > 1.0×` is the average ratio of raw artifact bytes to
  injected summary bytes — that's the firewall's load-bearing job.
- `compaction == 1.00×` rows (`tiny_payload`, `short_conversation`,
  `large_catalog`) are the firewall correctly no-op'ing on tiny
  inputs — every `tool_result` still passes through the envelope,
  but the summary equals the raw text so no token cost is saved
  *or* added (#271). The headline reduction figures further down
  reflect the firewall's value when payloads are actually large;
  small-payload rows guard against the impression that the
  firewall always wins.

### vs. naïve concat baseline

Token cost and coverage proxy against the no-op baseline of
concatenating *all* tool schemas + the full conversation history.
The naïve token count is measured with `tiktoken.cl100k_base`; the
coverage proxy is `items_included / event_count` (deterministic,
no LLM judge — see `scripts/baseline_naive.py`).

| scenario | naive_tokens | cw_tokens | token reduction | coverage proxy |
|---|---:|---:|---:|---:|
| large_catalog | 2767 | 1514 | 45.28% | 100.00% |
| long_conversation | 4365 | 2548 | 41.63% | 100.00% |
| mixed_payload | 2277 | 497 | 78.17% | 100.00% |
| short_conversation | 1946 | 496 | 74.51% | 100.00% |
| stress_conversation | 17482 | 6651 | 61.96% | 92.52% |
| tiny_payload | 1704 | 267 | 84.33% | 100.00% |

---

## Methodology

- **Deterministic seeds.** All catalog generation, scenario loading,
  and beam-search tie-breaking is seeded; identical inputs always
  produce identical outputs (`make benchmark` is a no-op on a fresh
  re-run for routing accuracy + context metrics; only latency
  varies with hardware).
- **No LLM calls by default.** The harness is pure-Python, stdlib +
  minimal core deps. The token estimator is `CharDivFourEstimator`
  so the headline numbers do not depend on `tiktoken`'s cached
  encoding state. The optional end-to-end real-model section (#269)
  is off by default and never runs in CI.
- **No network access by default.** The benchmark is safe to run
  in air-gapped CI environments. The end-to-end real-model capture
  (#269) and the `tiktoken` parity check (#268) require network or
  a cached encoding respectively, and degrade to a `skipped` row
  when unavailable.
- **Hardware variance.** Latency numbers are measured on the runner
  that produced `latest.json`. Treat them as ordering, not absolutes:
  the relative cost between catalog sizes is portable; the absolute
  microsecond count is not. See the **Hardware reference rig**
  section below for the canonical reference machine (#267).

### Hardware reference rig

**Reference rig** (numbers in this scorecard are calibrated for):

- GitHub Actions ubuntu-latest (2-core x86_64)
- System: `Linux`
- Machine: `x86_64`
- CPU logical cores: `2`
- Python: `3.10+`

> Absolute latency on other hardware will differ; the *relative* cost between catalog sizes and backends is portable.

**Measured on** (host that produced the current `latest.json`):

- System: `Linux`
- Machine: `x86_64`
- Processor: `x86_64`
- CPU logical cores: `4`
- Python: `CPython 3.11.15`
- Platform: `Linux-6.18.5-x86_64-with-glibc2.39`

### Token-estimator parity check

Quantifies the drift between `CharDivFourEstimator` (used everywhere
above) and `tiktoken.cl100k_base` on the gold-query corpus. This is
**measurement of the estimator**, not a routing metric — when the
two diverge, the headline token figures above are still self-
consistent, but readers comparing the numbers against an OpenAI
tokenizer expectation should apply the drift below (#268).

_Token-estimator parity check is unavailable on this run (`status: skipped: HTTPError`); the headline `CharDivFourEstimator` numbers elsewhere in this scorecard are still valid — they do not depend on `tiktoken` at runtime._

### Optional end-to-end real-model capture

Off by default. When enabled (`--with-real-model` flag plus
`CW_BENCH_LLM_PROVIDER` + `CW_BENCH_LLM_API_KEY` env vars) the
harness sends ≤5 gold queries through an OpenAI-compatible chat
endpoint and records prompt/completion tokens, USD cost, and
round-trip latency. Pure end-to-end, no fan-out, no retries; the
section exists so deployment-cost readers can sanity-check the
harness's offline token estimates against real provider usage (#269).

_Status: `skipped: offline by default`._

> Set CW_BENCH_LLM_PROVIDER + CW_BENCH_LLM_API_KEY and pass --with-real-model to enable.  See benchmarks/README.md.

See [`benchmarks/README.md`](README.md) for the full harness reference
and the per-scenario notes.

---

## Regenerating

```bash
make benchmark   # writes benchmarks/results/latest.json
make scorecard   # writes benchmarks/scorecard.md from latest.json
git diff --quiet benchmarks/scorecard.md   # passes on clean re-run
```

Per-backend matrices are generated via `make benchmark-matrix` (#208)
and now cover `tfidf`, `bm25`, `fuzzy`, `embedding_hashing`, and
`embedding_st` (#266). Weekly scheduled regeneration runs out of
`.github/workflows/scorecard-weekly.yml` (#207).

Optional capture flags:

- `--mixed-shapes` — emit the head-heavy + long-tail catalog matrix (#277).
- `--no-tiktoken-parity` — disable the estimator parity check (#268).
- `--with-real-model` — run the end-to-end real-model capture (#269).
