# Search Quality Benchmark

`docs-mcp-server` ships with a retrieval-quality benchmark that measures how well
`SearchTool` returns relevant chunks for natural-language queries. It is the
authoritative way to detect regressions when changing the embedding model,
chunking strategy, or ranking logic, and to compare this server against alternative
retrieval techniques.

This guide is the operator-facing entry point. For the detailed spec see
[openspec/changes/define-search-benchmark/specs/search-evaluation/spec.md](../../openspec/changes/define-search-benchmark/specs/search-evaluation/spec.md);
for developer internals (file layout, how to add a metric) see
[tests/search-eval/README.md](../../tests/search-eval/README.md).

## What it measures

**Headline (deterministic, gate regressions):**

| Metric | Question it answers |
|---|---|
| MRR | How high in the result list is the *first* relevant doc? |
| Recall@{3,5,10} | What fraction of the relevant docs did we find in the top-k? |
| nDCG@{5,10} | Did we rank the most relevant docs above the less-relevant ones? |
| Hit@{1,3,5} | Did we return *any* relevant doc in the top-k? |

**Observational (reported, do not gate):**

| Metric | Question it answers |
|---|---|
| Structural pass-rates | Are code blocks balanced? Is every chunk non-empty and URL'd? |
| Chunk coherence (LLM-judged) | Is each chunk a self-contained readable unit? |
| Content faithfulness (LLM-judged) | Do chunks faithfully reflect their source page? |
| Answerability (LLM-judged) | Could a downstream LLM answer the query from these chunks alone? |

Scope is **retrieval-only**. End-to-end RAG generation (running an LLM over the
retrieved chunks and judging the answer) is intentionally out of scope.

## Prerequisites

### 1. An LLM judge API key

The benchmark uses an LLM to score the three observational rubrics. The default
judge is `openai:gpt-5.4-mini`, so an `OPENAI_API_KEY` in `.env` is the
zero-configuration path. To use a different provider, set `DOCS_EVAL_JUDGE` to
one of the allowed identifiers in [tests/search-eval/judges.ts](../../tests/search-eval/judges.ts):

- OpenAI: `openai:gpt-5.4-mini` (default), `openai:gpt-5`
- Anthropic: `anthropic:claude-sonnet-4-6`, `anthropic:claude-opus-4-7`
- Google: `google:gemini-3-flash-preview`, `google:gemini-3.1-flash-lite`

Deprecated models (e.g. `gpt-4o-mini`) are rejected at startup.

### 2. The five benchmark libraries must be indexed locally

The shipped dataset ([tests/search-eval/dataset.yaml](../../tests/search-eval/dataset.yaml))
references 60 queries across five libraries. **Every one of those libraries must be
present in your local store before the benchmark runs**, or preflight will fail
with the exact missing list.

Run all five scrape commands once on the machine where you'll run the benchmark.
Every command excludes 2-letter language-prefix paths so translation mirrors
don't balloon the index — the dataset's qrels only reference English pages.

```bash
# React (react.dev has /es/, /zh-hans/, /ja/, etc.)
npx @arabold/docs-mcp-server@latest scrape react https://react.dev \
  --exclude-pattern '/^https:\/\/react\.dev\/[a-z]{2}(-[a-z]+)?\//' \
  --max-pages 300

# Python stdlib (docs.python.org has /zh-cn/, /ja/, /fr/, /es/, /ko/, etc.)
npx @arabold/docs-mcp-server@latest scrape python https://docs.python.org/3/ \
  --exclude-pattern '/^https:\/\/docs\.python\.org\/3\/[a-z]{2}(-[a-z]+)?\//' \
  --max-pages 500

# FastAPI (heavy i18n — /es/, /zh/, /zh-hant/, /em/, ~15 languages)
npx @arabold/docs-mcp-server@latest scrape fastapi https://fastapi.tiangolo.com/ \
  --exclude-pattern '/^https:\/\/fastapi\.tiangolo\.com\/[a-z]{2}(-[a-z]+)?\//' \
  --max-pages 200

# Vite (English-only, but cap pages to be safe)
npx @arabold/docs-mcp-server@latest scrape vite https://vitejs.dev/guide/ \
  --max-pages 200

# TailwindCSS
npx @arabold/docs-mcp-server@latest scrape tailwindcss https://tailwindcss.com/docs \
  --max-pages 300
```

> **Why exclude language prefixes?** FastAPI's docs site, for example, hosts the
> full doc tree in ~15 languages under `/es/`, `/zh/`, `/zh-hant/`, etc. A naive
> scrape pulls ~1300 pages where only ~100 are actually relevant to the
> benchmark. The regex excludes any URL whose first path segment is a 2-letter
> code (optionally with a suffix like `-hant`). English content lives at the
> root and is preserved.

`--max-pages` is a safety belt; the English doc tree for each library fits well
within these caps. You can drop or raise it if you want maximal coverage.

> **Subset run.** If you only want a quick sanity check, point `DOCS_EVAL_DATASET`
> at a custom dataset file that uses fewer libraries — the schema is identical.

### 3. Node 22

The store uses `better-sqlite3` with a pinned native binary. Always run the
benchmark on Node 22 even if the rest of your toolchain is newer.

## Running

```bash
# Measurement run: compute metrics, compare against baseline.json, exit non-zero on regression.
npm run evaluate:search

# Baseline-refresh: same run, but write the result to baseline.json and exit zero.
# Use this after intentional improvements, or for the very first run.
npm run evaluate:search:baseline

# Preflight only: confirm all required libraries are indexed.
npm run evaluate:search:preflight
```

Both modes produce, under `tests/search-eval/results/` (gitignored):

| File | Purpose |
|---|---|
| `summary.json` | Machine-readable summary for CI consumption |
| `promptfoo-raw.json` | Raw per-query promptfoo output |
| `cross-judge.json` | Cross-judge agreement, when `DOCS_EVAL_CROSS_JUDGE` is set |
| `dataset.flat.yaml` | Generated flat view of the dataset (don't edit) |

`baseline.json` lives in `tests/search-eval/baseline.json` and **is checked in**.
Its history is the audit trail for benchmark drift over time.

## First run

The repository ships with a populated `tests/search-eval/baseline.json` (recorded
against the default dataset / judge / embedding model). Your first run should be:

```bash
# 1. Confirm libraries are indexed.
npm run evaluate:search:preflight

# 2. Smoke-test the pipeline with 5 queries (~1 minute) — sanity check the
#    provider, assertions, judge, aggregator, and comparator. This writes to
#    `dataset.smoke.baseline.json`, NOT to the main baseline, so you can run it
#    safely without disturbing the checked-in reference numbers.
DOCS_EVAL_DATASET=tests/search-eval/dataset.smoke.yaml \
  npm run evaluate:search:baseline

# 3. Measure against the checked-in baseline. Exits non-zero on regression.
npm run evaluate:search
```

### When to refresh the baseline

`npm run evaluate:search:baseline` overwrites `tests/search-eval/baseline.json`.
Refresh it when:

- you've intentionally changed something that should shift retrieval (new
  embedding model, chunking config, scraper logic);
- you've reviewed the dataset and want the new numbers to be the new reference;
- a re-scrape or re-index moved scores in a direction you want to keep.

The comparator refuses to gate against a baseline recorded under materially
different config (different `datasetFile`, `embeddingModel`, `judge`, `topK`,
or `provider`). It prints the incompatibility list and exits zero — re-record
the baseline if this run is the new reference.

## Compare against another retrieval service

The benchmark supports running the same dataset, same judge, same metrics
against alternative retrieval providers — see how `docs-mcp-server` stacks
up against the alternatives. Out of the box, `local` (the in-repo
`SearchTool`) and `context7` ([Context7](https://context7.com/)) are
supported. Adding another provider is one entry in `tests/search-eval/run.ts`
plus a small exec-provider script.

```bash
# Record a baseline for the local provider (default — same as the canonical
# `npm run evaluate:search:baseline`).
npm run evaluate:search:baseline

# Record a baseline against Context7. Writes to a separate file so the
# canonical local baseline is untouched.
DOCS_EVAL_PROVIDER=context7 DOCS_EVAL_NO_CACHE=1 \
  npm run evaluate:search:baseline
# → tests/search-eval/baseline.context7.json

# Side-by-side report (IR / per-intent / LLM-judged / structural).
npx vite-node tests/search-eval/cli/compare-providers.ts \
  tests/search-eval/baseline.json \
  tests/search-eval/baseline.context7.json
```

The Context7 provider hits Context7's public `/v2/context` endpoint. It works
anonymously for short runs; set `CONTEXT7_API_KEY` in your environment for
sustained or CI use.

The comparator refuses to gate one provider's run against another's baseline
(it'd produce false regressions/improvements). For cross-provider comparison
always use the dedicated CLI above — it prints both sides and a coarse
"wins per metric" tally without pretending the deltas are regressions.

> **Asymmetry to know about.** Our IR metrics (MRR, Recall@k, nDCG@k, Hit@k)
> match by URL string against the dataset's qrels. For most libraries Context7
> returns per-page source URLs that match our qrels directly. TailwindCSS is
> the exception — Context7 attributes Tailwind chunks to GitHub source paths;
> the provider normalises these back to `tailwindcss.com/docs/...` URLs so the
> comparison is fair. Some Context7 info-snippets carry only a library-level
> `llms.txt` placeholder URL; the provider drops those rather than letting
> them poison IR metrics with always-wrong URLs.

### Performance note

Each query cold-starts a fresh `vite-node` process and re-initialises the
docService (which writes through `better-sqlite3` for schema migrations).
Running multiple provider processes in parallel deadlocks on the SQLite write
lock, so the default concurrency is **1**. Expect roughly 10–15 seconds per
query — the full 60-query baseline takes 10–15 minutes.

You can raise concurrency once we move to a long-running provider that
initialises the store once per run:

```bash
DOCS_EVAL_CONCURRENCY=4 npm run evaluate:search:baseline   # don't do this yet
```

## Reading the output

A run prints to stdout in roughly this shape:

```
=== docs-mcp-server search benchmark ===
judge=openai:gpt-5.4-mini  embedding=...  top_k=5  dataset=...  ts=...

─ Headline IR metrics (deterministic) ─
MRR         0.612  (baseline 0.605, +1.2% ▲)
Recall@5    0.844  (baseline 0.851, -0.8% ·)
nDCG@5      0.701  (baseline 0.704, -0.4% ·)
Hit@3       0.917  (baseline 0.900, +1.9% ▲)

─ Per-intent breakdown ─
api-lookup       n=19  MRR=0.78 R@5=0.91 nDCG@5=0.82 Hit@3=0.95
conceptual       n=15  MRR=0.55 R@5=0.78 nDCG@5=0.62 Hit@3=0.87
comparison       n=12  MRR=0.41 R@5=0.71 nDCG@5=0.55 Hit@3=0.83
troubleshooting  n=14  MRR=0.59 R@5=0.85 nDCG@5=0.70 Hit@3=0.93

─ Structural checks (deterministic, pass rate) ─
code_block_balance: 96.7%
non_empty_content:  100.0%
url_presence:       100.0%

─ LLM-judged (observational, not gating) ─
chunk_coherence        mean=3.85  n=60
content_faithfulness   mean=4.12  n=60
answerability          mean=3.40  n=60

✅ No regressions. 2 improved, 7 stable.
```

A regression run exits with code 1 and prints the offending metrics with relative
deltas. Tolerances default to 5% relative on headline metrics, 10% relative on
per-intent breakdowns; adjust via `tolerances` in the comparator if you need
something different.

## Customizing

| What | How |
|---|---|
| Use a different judge | `DOCS_EVAL_JUDGE=anthropic:claude-sonnet-4-6 npm run evaluate:search` |
| Enable cross-judge sampling | `DOCS_EVAL_CROSS_JUDGE=google:gemini-3.1-flash-lite DOCS_EVAL_CROSS_JUDGE_N=10 npm run evaluate:search` |
| Run a subset dataset | `DOCS_EVAL_DATASET=path/to/your.yaml npm run evaluate:search` |
| Bust promptfoo's cache after a re-index | `DOCS_EVAL_NO_CACHE=1 npm run evaluate:search:baseline` |
| Add a query | Append an entry to `tests/search-eval/dataset.yaml`, then refresh the baseline |
| Add a library | Append entries, scrape the library, also update `.github/workflows/eval.yml` |

> **About `DOCS_EVAL_NO_CACHE`.** Promptfoo caches by `(provider id, prompt, vars)`. If you re-scrape or re-index a library, the provider returns different chunks but promptfoo doesn't know — it serves stale cached output. Set `DOCS_EVAL_NO_CACHE=1` once after a re-index to force fresh provider calls + fresh judge calls. Default stays cached so subsequent measurement runs are fast and cheap.

The dataset currently ships with `status: draft` because the entries were
authored without per-entry community review. The `notes:` block at the top of
`dataset.yaml` contains a review checklist; once you've worked through it, change
`status: draft` → `status: reviewed`.

## CI

`.github/workflows/eval.yml` runs the benchmark on `workflow_dispatch` (manual)
and a weekly schedule (Mondays 07:00 UTC). The workflow scrapes the five
libraries into a cached store on the first run, then re-uses the cache on
subsequent runs. The benchmark is **deliberately not gated on PRs** — variance
and judge cost make per-PR gating premature. Regressions caught by the scheduled
run are visible in the workflow summary and uploaded as an artifact.

## Comparing HTML extractors (Cheerio vs. Defuddle)

The HTML pipeline supports two interchangeable content extractors:

- `cheerio` (default): the built-in `HtmlSanitizerMiddleware` with a hand-curated selector blocklist.
- `defuddle`: [Defuddle](https://github.com/kepano/defuddle), which scores DOM nodes and extracts the main article content.

Select the extractor with the `DOCS_MCP_SCRAPER_HTML_EXTRACTOR` env var (or
`scraper.htmlExtractor` in `config.yaml`). Both produce a Cheerio DOM that flows
through the same downstream middleware (normalisation, markdown conversion,
splitting), so the chunking and embedding behaviour stay identical — only the
boilerplate-stripping logic changes.

To run a head-to-head benchmark:

```bash
# 1. Lock in the baseline with the current Cheerio sanitiser.
DOCS_MCP_SCRAPER_HTML_EXTRACTOR=cheerio \
  npx @arabold/docs-mcp-server@latest remove react tailwindcss vite python fastapi  # if previously indexed
# …re-scrape all five libraries (see Prerequisites above)…
DOCS_MCP_SCRAPER_HTML_EXTRACTOR=cheerio \
  DOCS_EVAL_DATASET=tests/search-eval/dataset.yaml \
  npm run evaluate:search:baseline

# 2. Re-index with Defuddle.
DOCS_MCP_SCRAPER_HTML_EXTRACTOR=defuddle \
  npx @arabold/docs-mcp-server@latest remove react tailwindcss vite python fastapi
# …re-scrape all five libraries with the same commands…
DOCS_MCP_SCRAPER_HTML_EXTRACTOR=defuddle \
  DOCS_EVAL_NO_CACHE=1 \
  npm run evaluate:search
```

The measurement run diffs the Defuddle results against the Cheerio baseline and
fails on regression. Inspect `tests/search-eval/results/summary.json` for the
absolute numbers and `compare`'s console report for the per-metric deltas.

> **Why re-index between runs.** The extractor change happens during scraping,
> so the choice is baked into the stored chunks. Switching the env var without
> a fresh scrape compares the new code against old data.

## Reference benchmark runs

A historical record of the headline numbers for each configuration we've
benchmarked, against the 59-query dataset shipped in
[`tests/search-eval/dataset.yaml`](../../tests/search-eval/dataset.yaml). All
runs use `openai:gpt-5.4-mini` as judge at `temperature: 0` and
`text-embedding-3-small` for embeddings; comparisons are apples-to-apples
across the table.

| Metric                       | Cheerio (default) ¹ | Defuddle ² | Context7 (external) ³ |
|------------------------------|---------------------|------------|-----------------------|
| **Structural** (deterministic)                                                                  |
| code_block_balance           | **1.000**           | 0.983      | 1.000                 |
| non_empty_content            | 1.000               | 1.000      | 1.000                 |
| url_presence                 | 1.000               | 1.000      | 1.000                 |
| **Headline IR** (deterministic)                                                                 |
| MRR                          | **0.747**           | 0.718      | 0.733                 |
| Recall@3                     | 0.647               | 0.621      | **0.709**             |
| Recall@5                     | **0.732**           | 0.689      | 0.726                 |
| nDCG@5                       | 0.699               | 0.665      | **0.704**             |
| Hit@3                        | **0.831**           | 0.780      | **0.831**             |
| Hit@5                        | **0.881**           | 0.814      | 0.847                 |
| **Per-intent MRR**                                                                              |
| api-lookup (n=18)            | **0.861**           | 0.806      | 0.824                 |
| conceptual (n=15)            | 0.794               | **0.806**  | 0.767                 |
| comparison (n=12)            | 0.646               | **0.736**  | 0.563                 |
| troubleshooting (n=14)       | 0.637               | 0.494      | **0.726**             |
| **LLM-judged** (observational, not gating)                                                      |
| chunk_coherence              | 3.53                | 3.39       | **4.80**              |
| content_faithfulness         | 3.20                | 3.19       | **4.24**              |
| answerability                | 4.49                | 4.49       | 4.49                  |

¹ Default Cheerio extractor, recorded baseline at
  [`tests/search-eval/baseline.json`](../../tests/search-eval/baseline.json).
  This is what the comparator gates against. `code_block_balance` is 100% —
  fence-aware splitting in `TextContentSplitter` plus verbatim info-string
  preservation in `CodeContentSplitter` keep every chunk's fences balanced
  including VitePress/Shiki sequences like `` ```js{15-18} twoslash
  [server.js] ``.

² Defuddle extractor selected via `DOCS_MCP_SCRAPER_HTML_EXTRACTOR=defuddle`,
  all five libraries re-scraped through Defuddle on 2026-05-18 (1,155 indexed
  pages total). On this dataset Defuddle is a clear **regression** against
  the Cheerio default: headline IR metrics drop 4–6% relative, and
  `troubleshooting`-intent queries are hit hardest (MRR -22%, nDCG@5 -22%,
  Hit@3 -20%). `comparison`-intent queries improve (MRR +14%) — Defuddle's
  main-content pruning helps when the relevant content is the article body.
  Structural `code_block_balance` slips slightly to 98.3% as Defuddle
  occasionally collapses VitePress info-string sequences differently than the
  hand-curated Cheerio sanitiser does. **Recommendation:** keep `cheerio` as
  the default; Defuddle is worth revisiting per-site when the content is
  primarily article-shaped and the side-content is noise. See
  [Comparing HTML extractors](#comparing-html-extractors-cheerio-vs-defuddle).

³ External retrieval service [Context7](https://context7.com) measured via
  the alternate provider in
  [`tests/search-eval/baseline.context7.json`](../../tests/search-eval/baseline.context7.json).
  Included for comparison only; chunking, embedding, and ranking are not
  under our control. Context7 trades a small MRR deficit (-1.9%) for
  noticeably higher Recall@3 (+9.6%) and LLM-judged coherence
  (4.80 vs 3.53) — a hint that chunk semantic boundaries are the next
  improvement worth chasing locally.

### How to update this table

After any change that materially affects retrieval (chunker, embedding model,
ranking, scraper), refresh the local baseline and add or revise a column:

```bash
DOCS_EVAL_NO_CACHE=1 npm run evaluate:search:baseline
```

Then update the relevant column above with the new headline + structural + LLM
means and add a numbered footnote explaining what changed.

## Related

- [tests/search-eval/README.md](../../tests/search-eval/README.md) — file layout
  and how to add metrics
- [openspec/changes/define-search-benchmark/](../../openspec/changes/define-search-benchmark/)
  — proposal, design, full spec
