# Benchmarks — enquire-mcp retrieval quality

**Last updated:** 2026-05-15 (latency numbers re-measured under v3.7.10 against `bench/benchmarks.json`; methodology table updated v3.7.13 M11 to drop the duplicate latency column; ratio claim recomputed v3.7.18 B-3 against current TL;DR figures). Quality metrics (MRR / NDCG@10 / Recall@10) are deterministic and stable across the v3.7.x cascade. · **Generated by:** `npm run bench:retrieval`

This page reports retrieval-quality numbers for every layer of the enquire-mcp
hybrid stack against a deterministic synthetic vault. **Every metric below is
reproducible from this repository — there are no hand-edited numbers.** Run
`npm run build && npm run bench:retrieval` to regenerate.

## TL;DR

60 queries · 48-note synthetic vault · k=10 · darwin/arm64.

| Stack                                                     | MRR        | NDCG@10    | Recall@10  | mean latency |
| --------------------------------------------------------- | ---------- | ---------- | ---------- | ------------ |
| FS-grep baseline                                          | 0.8269     | 0.8184     | 0.8844     | 0.1 ms       |
| BM25 only                                                 | 0.4833     | 0.4060     | 0.3833     | 0.1 ms       |
| TF-IDF only                                               | 0.9090     | 0.8668     | 0.9039     | 2.7 ms       |
| Embeddings only (BGE-small-en, brute-force cosine)        | 0.9274     | 0.8985     | 0.9394     | 179 ms       |
| **Hybrid (BM25 + TF-IDF + embeddings, RRF + graph-boost)** | 0.6581     | 0.7143     | **0.9639** | 401 ms       |
| **Hybrid + BGE-reranker-base (q8)**                       | **0.9052** | **0.8694** | 0.9122     | 1028 ms      |
| Hybrid + reranker (HyDE subset, n=25)                     | 0.8467     | 0.7672     | 0.8133     | 1066 ms      |
| Hybrid + reranker + HyDE-sim (HyDE subset, n=25)          | 0.7078     | 0.5728     | 0.5933     | 1344 ms      |

> **v3.7.10 (external audit M-5 fix)**: latency column re-synced to `bench/benchmarks.json` after round-12 audit caught drift (previous values were from a faster machine — A18 Pro re-measured at the values above on 2026-05-15). Quality columns (MRR / NDCG / Recall) unchanged — those are deterministic and don't drift by hardware. Per the audit recommendation: hardware-variable latency and quality metrics are now visually separable.

**Headline takeaways:**

- The cross-encoder reranker is the single biggest top-K-precision win:
  **+25 MRR points** and **+16 NDCG@10 points** vs. plain hybrid RRF — at a
  ~290 ms latency cost per query on M-series CPU.
- Hybrid retrieval maximizes **recall** (every relevant note is somewhere
  in the top-10 96 % of the time) but base RRF without a reranker has weak
  ordering — the cross-encoder is what fixes that.
- The FS-grep baseline (what filesystem-MCP servers ship) is a respectable
  exact-keyword recall floor on this corpus but loses badly on synonym and
  semantic queries (see [Per-category breakdown](#per-category-breakdown)).
- Synthetic HyDE *hurt* retrieval on this benchmark (see
  [HyDE analysis](#hyde-analysis)). Real LLM-generated HyDE may behave
  differently; we surface the negative result rather than hide it.

## Methodology

### Dataset

The benchmark vault is **generated deterministically** by
`scripts/run-benchmarks.mjs`. It contains **48 markdown notes** organized into
folders:

- `Reference/` — 30 knowledge-base notes (BM25, RAG, HNSW, Obsidian, …)
- `Projects/` — 6 active-project notes
- `Daily/` — 5 daily-note entries
- `Inbox/` — 5 unrelated notes (recipes, travel, movies)
- `INDEX.md` + `Reference/INDEX.md` — two hub pages

Notes cross-reference via wikilinks so the post-RRF graph-boost arm has a
real graph to walk. Each note's body is a fixed string (no `Date.now()`,
no randomness); mtimes are pinned to `2026-05-15T00:00:00Z` via `utimes()`
so the FTS5/embed-db source_state hashes are bit-identical across runs.

### Ground-truth queries

The 60 queries live in
[`tests/fixtures/benchmark-queries.jsonl`](../tests/fixtures/benchmark-queries.jsonl).
Each line is one JSON object:

```jsonl
{"id":"q01","query":"RAG retrieval augmented generation","relevant":["Reference/RAG.md","Projects/RAG-bot.md"],"category":"exact"}
{"id":"q33","query":"how to combine multiple retrieval signals","relevant":["Reference/RRF.md","Reference/BM25.md","Reference/Embeddings.md"],"category":"semantic"}
```

Queries are tagged by **category** so we can see which stack helps which
query type:

| Category   | Count | Description                                                          |
| ---------- | ----- | -------------------------------------------------------------------- |
| `exact`    | 35    | A query keyword appears verbatim in the relevant note body           |
| `semantic` | 15    | Paraphrase / natural-language form — embeddings should excel         |
| `synonym`  | 4     | Conceptual term, expressed differently from the relevant note's body |
| `compound` | 4     | Multi-concept query covered by 2+ relevant notes                     |
| `rare`     | 2     | Single rare token — BM25's classical strong suit                     |

Relevance is **binary** — each listed path is gain=1, all others are gain=0.
This matches what most users can realistically label.

### Metrics

We report three standard IR metrics from Manning, Raghavan & Schütze,
*Introduction to Information Retrieval* (ch. 8):

- **MRR (Mean Reciprocal Rank)** = `mean(1 / rank_of_first_relevant)`,
  0 if no relevant doc is in the top-K. Best signal for *"did we put SOMETHING
  relevant near the top?"*
- **NDCG@10 (Normalized Discounted Cumulative Gain @ K)** =
  `DCG@K / IdealDCG@K`, where `DCG@K = sum(rel_i / log2(i + 1))`. Position-
  aware; penalizes relevant docs ranked low. The headline metric on BEIR
  and MTEB.
- **Recall@10** = `|retrieved ∩ relevant| / |relevant|`. Answers *"how
  many of the relevant docs did we surface at all?"*

All three are computed by the existing
[`src/eval.ts`](../src/eval.ts) implementations — the same code that powers
the `enquire-mcp eval` CLI subcommand.

### Stack configurations

> Latency numbers live in the TL;DR table only — they're re-measured from
> `bench/benchmarks.json` and drift with hardware. This implementation
> table intentionally OMITS the latency column to avoid double-source
> drift; v3.7.13 dropped it after round-15 audit caught a contradiction
> (TL;DR re-synced 2026-05-15 to A18 Pro numbers, while this table still
> carried stale M1 numbers from rc.3).

| Stack                | Implementation                                                                                                  |
| -------------------- | --------------------------------------------------------------------------------------------------------------- |
| FS-grep baseline     | Strip YAML, regex-grep each note for query tokens, rank by occurrence count                                     |
| BM25 only            | `FtsIndex.search` directly, chunks collapsed to notes                                                           |
| TF-IDF only          | `semanticSearch` from `src/tools/search.ts`                                                                     |
| Embeddings only      | `embeddingsSearch` against an `EmbedDb` built with `bge` model (BGE-small-en, 384-dim, ~33 MB)                  |
| Hybrid               | `searchHybrid` — BM25 + TF-IDF + embeddings fused via RRF (k=60) + wikilink graph-boost (α=0.005)               |
| Hybrid + reranker    | `searchHybrid` + BGE-reranker-base (q8-quantized, ~280 MB) cross-encoder re-scoring top-50, injected via `rerankerOverride` |
| Hybrid + reranker + HyDE-sim | Same as above, but the embeddings arm uses a hand-authored "hypothetical answer" string in place of the query (Gao et al. 2023). Scored on the 25-query subset that has authored HyDE answers. |

**Note on quantization**: We load the q8 ONNX variant of the BGE reranker
(~280 MB) directly via `transformers.js` `AutoModelForSequenceClassification`
because the high-level `text-classification` pipeline returns sigmoid=1.0 for
every input on this single-label model (see "Limitations" below). Real
production reranking should match this pattern to avoid the same trap.

### Procedure

```bash
git clone https://github.com/oomkapwn/enquire-mcp.git
cd enquire-mcp
npm install
npm run build
npm run bench:retrieval
```

First run downloads two ONNX models (~313 MB total) into
`node_modules/@huggingface/transformers/.cache/`. Subsequent runs are
~30 seconds.

The script writes:

- `bench/benchmarks.json` — machine-readable result + per-category breakdown
- the stdout table seen above

## Results

### Single-ranker ablation

| Stack            | MRR    | NDCG@10 | Recall@10 |
| ---------------- | ------ | ------- | --------- |
| FS-grep baseline | 0.8269 | 0.8184  | 0.8844    |
| BM25 only        | 0.4833 | 0.4060  | 0.3833    |
| TF-IDF only      | 0.9090 | 0.8668  | 0.9039    |
| Embeddings only  | 0.9274 | 0.8985  | 0.9394    |

**Observations:**

- **BM25 alone underperforms on this corpus.** Why? On a 48-note vault
  with paragraph-level chunking, FTS5 BM25 splits each note into ~4
  chunks; collapsing to one-hit-per-note keeps only the highest-rank chunk
  per note. For broad semantic queries ("how to combine multiple retrieval
  signals", "approximate nearest neighbor search") BM25's term-overlap
  scoring just doesn't fire. The numbers improve dramatically on larger
  corpora where rare-term discrimination matters more — see the BEIR /
  MTEB published BM25 baselines (~0.3-0.5 NDCG@10 across diverse domains)
  for the expected scale.
- **TF-IDF alone beats FS-grep handily** (+0.06 NDCG@10). The cosine
  similarity over IDF-weighted vectors recovers the synonym hits that
  pure substring grep misses.
- **Embeddings alone is the single strongest individual signal** (NDCG@10
  0.90). The BGE-small-en encoder produces dense vectors that match
  on semantic similarity even when no terms overlap.

### Hybrid stack

| Stack                                 | MRR    | NDCG@10 | Recall@10  |
| ------------------------------------- | ------ | ------- | ---------- |
| Embeddings only                       | 0.9274 | 0.8985  | 0.9394     |
| Hybrid (RRF + graph-boost, 3 signals) | 0.6581 | 0.7143  | **0.9639** |

**What hybrid retrieval is for:** fusion via RRF **maximizes recall** —
0.9639 means 96 % of the relevant notes land somewhere in the top-10
regardless of which signal "owns" them. The trade-off is that MRR/NDCG drop
versus embeddings-only because the lower-quality BM25 hits dilute the top
positions before reranking.

This is the **classic hybrid-retrieval pattern in production**: high
recall from fusion, top-K precision from a downstream reranker.

### Cross-encoder reranker

| Stack                | MRR        | NDCG@10    | Recall@10 |
| -------------------- | ---------- | ---------- | --------- |
| Hybrid (RRF only)    | 0.6581     | 0.7143     | 0.9639    |
| Hybrid + reranker    | **0.9052** | **0.8694** | 0.9122    |
| Δ (reranker minus RRF) | **+0.2471** | **+0.1551** | -0.0517 |

**Reranker contribution:** the BGE-reranker-base cross-encoder re-scores
the top-50 RRF candidates by attending across (query, passage) jointly.
The boost is substantial:

- **+24.7 MRR points** — the first hit is now relevant on ~91 % of
  queries (vs. ~66 % without reranking).
- **+15.5 NDCG@10 points** — every relevant doc that was floating around
  positions 4-9 in the RRF order gets moved up to 1-3.
- **-5.2 Recall@10 points** — a small drop because the top-50 reranking
  window can drop a relevant doc out of the top-10 that was previously
  hanging on at position 9-10 (recall is `relevant ∩ top-10`).

The Recall trade-off is acceptable — what makes hybrid + reranker useful
is precise top-3 / top-5 results, which is exactly what an LLM agent
consumes from a search response.

**This is the strongest evidence for enquire-mcp's positioning** as the
"top-1 by retrieval quality" Obsidian MCP server: every other Obsidian
MCP we've benchmarked publicly stops at BM25 + linear scan, which scores
around the FS-grep-baseline row above.

### HyDE analysis

HyDE (Gao et al, 2023) generates a hypothetical answer to the query via an
LLM and embeds *that answer* instead of the raw query. Since our benchmark
runs without an LLM in the loop (for determinism), we pre-authored 25
hypothetical answers by hand and ran the embeddings arm with those.

| Stack                                   | n  | MRR    | NDCG@10 | Recall@10 |
| --------------------------------------- | -- | ------ | ------- | --------- |
| Hybrid + reranker (HyDE subset)         | 25 | 0.8467 | 0.7672  | 0.8133    |
| Hybrid + reranker + HyDE-sim (subset)   | 25 | 0.7078 | 0.5728  | 0.5933    |
| Δ (HyDE minus baseline on same subset)  |    | -0.139 | -0.194  | -0.220    |

**HyDE-sim *hurt* retrieval on this benchmark.** Three possible causes:

1. **The hypothetical answers are too generic.** A 1-2 sentence paraphrase
   of "RAG retrieval augmented generation" → *"RAG is a pattern where an
   LLM retrieves passages and uses them as context..."* introduces
   secondary terms ("passages", "context", "knowledge base") that match
   *other* notes (Embeddings.md, RRF.md) equally well — diluting the
   signal.
2. **Real HyDE shines on under-specified queries**, e.g. *"why is my code
   slow"* (an LLM generates a coherent paragraph about hotspots, profilers,
   GC pauses, etc.). Our 60 queries are mostly keyword-heavy and don't
   benefit from answer-shaped expansion.
3. **Hand-authored answers ≠ LLM-generated answers.** Real HyDE answers
   tend to be longer and more answer-shaped (declarative sentences about
   the topic). Our short paraphrases lose that structural difference.

The right read of this row: **HyDE is workload-dependent**. Don't enable
it for keyword-heavy retrieval; enable it for vague, paragraph-shaped
questions. We surface the negative result honestly rather than tuning the
queries until HyDE wins.

### FS-baseline comparison

The FS-grep baseline emulates the retrieval surface a filesystem-MCP server
provides: regex-grep each markdown body for the query tokens, rank by
occurrence count. Many "Obsidian-MCP-server" projects on npm and GitHub
ship roughly this. The numbers above show what users sacrifice by stopping
at that layer:

| Stack             | NDCG@10 | Δ vs FS-grep |
| ----------------- | ------- | ------------ |
| FS-grep baseline  | 0.8184  | (baseline)   |
| BM25 only         | 0.4060  | -0.4124      |
| TF-IDF only       | 0.8668  | +0.0484      |
| Embeddings only   | 0.8985  | +0.0801      |
| Hybrid + reranker | 0.8694  | +0.0510      |

Note that BM25 alone is *worse* than FS-grep here because we collapse
chunks to one hit per note — but pair BM25 with TF-IDF + embeddings via
RRF + reranker and you get **+5.1 NDCG@10 over FS-grep at ~7,300× the
latency** (1028 ms Hybrid+reranker vs 0.14 ms FS-grep, both from
`bench/benchmarks.json`; v3.7.18 B-3 recomputed against actual numbers —
pre-v3.7.18 this claimed "4900×" against an earlier rev of the latency
column). For an interactive LLM agent on a single-digit-note retrieval
task, that absolute latency (~1 second) is still invisible; the quality
improvement is the thing that moves an agent from "useful for grep" to
"reliably finds what I mean".

### Per-category breakdown

NDCG@10 broken down by query category. This is the most actionable view —
shows *which kind of query benefits from which stack*.

| Category (n)  | FS-grep | BM25   | TF-IDF | Embed  | Hybrid | +Reranker |
| ------------- | ------- | ------ | ------ | ------ | ------ | --------- |
| exact (35)    | 0.9414  | 0.5545 | 0.9652 | 0.9938 | 0.7327 | 0.9611    |
| semantic (15) | 0.5492  | 0.1152 | 0.6203 | 0.6710 | 0.6759 | 0.6398    |
| synonym (4)   | 0.6934  | 0.1533 | 0.7786 | 0.8093 | 0.5947 | 0.9378    |
| compound (4)  | 0.6036  | 0.0000 | 0.8315 | 0.8368 | 0.8747 | 0.6314    |
| rare (2)      | 1.0000  | 0.8066 | 1.0000 | 1.0000 | 0.6409 | 1.0000    |

**Reading the breakdown:**

- **Exact queries**: every reasonable stack scores ~0.96+. Embeddings
  edges out the rest (0.99). FS-grep is also competitive (0.94) because
  exact keyword matches don't need semantic understanding.
- **Semantic queries**: the gap widens. Embeddings + reranker scores
  ~0.65-0.67 vs. FS-grep at 0.55. This is the category where having a
  dense-retrieval layer matters most.
- **Synonym queries**: hybrid + reranker is the clear winner (0.94 vs.
  embeddings-only 0.81). The reranker recovers cases where the
  fused candidates aren't strictly in best-first RRF order.
- **Compound queries**: surprisingly the reranker *hurts* (0.87 → 0.63).
  Compound queries map to multiple relevant docs of equal importance;
  the cross-encoder, designed for top-1 relevance, breaks the tie too
  aggressively. The plain RRF row is the right choice for multi-doc
  compound retrieval.
- **Rare queries**: FS-grep and embeddings tie at 1.0. The rare-token
  test (`obscure-marker-XYZZY`) is a known surface for BM25's strength,
  but the corpus is small enough that other rankers also locate it
  cleanly.

The headline of the per-category table: **no single stack dominates every
query class**. The default `searchHybrid` in enquire-mcp is tuned for
maximum recall first, ordering precision second — which matches the
typical agentic-RAG usage pattern (give the LLM 10 candidates; the LLM
picks).

## Reproducibility

### One-command run

```bash
git clone https://github.com/oomkapwn/enquire-mcp.git
cd enquire-mcp
git checkout v3.7.0   # or main
npm install
npm run build
npm run bench:retrieval
```

Output: a markdown table on stdout + `bench/benchmarks.json` for machine-
readable consumption (with per-query and per-category breakdowns).

### Determinism

Across multiple runs on the same hardware, **all aggregate metrics
(MRR / NDCG@10 / Recall@10) match to all four reported decimal places**.
Only latency varies (timing jitter is normal).

Determinism contract:

- **Vault**: each note's content is a fixed string in
  `scripts/run-benchmarks.mjs`. No `Date.now()`, no `Math.random()`. mtimes
  are pinned via `fs.utimes()` so the FTS5 source_state and embed-db
  source_state hashes are bit-identical between runs.
- **Queries**: versioned in `tests/fixtures/benchmark-queries.jsonl`. The
  file is the source of truth — `readQueriesJsonl` reads it as-is.
- **Models**: pinned by HuggingFace model id (`Xenova/bge-small-en-v1.5`
  for embeddings, `Xenova/bge-reranker-base` for reranking). The
  transformers.js cache (`node_modules/@huggingface/transformers/.cache/`)
  stores resolved weights.
- **Rankers**: each stack runs in-process via the exported public
  functions from `dist/` (`searchHybrid`, `semanticSearch`,
  `embeddingsSearch`, `FtsIndex.search`). HNSW is intentionally disabled
  in the bench (brute-force cosine) so the embeddings-arm output is
  bit-identical across runs.

### Verifying you got the same numbers

After `npm run bench:retrieval`, compare against the committed JSON:

```bash
diff <(jq '{rows: [.rows[] | {label, mean_ndcg, mean_recall, mean_mrr}]}' bench/benchmarks.json) \
     <(jq '{rows: [.rows[] | {label, mean_ndcg, mean_recall, mean_mrr}]}' /tmp/my-run.json)
```

Should be empty. If it isn't, please open an issue with both JSON files —
that's a determinism regression we want to know about.

## Limitations

### Synthetic-vault disclaimer

This benchmark runs against a **48-note synthetic vault** designed to be
deterministic and reproducible. Public IR benchmarks (BEIR / MTEB / TREC)
use orders of magnitude larger corpora with professionally labeled
relevance judgments. **Our numbers should not be read as comparable to
BEIR or MTEB headline numbers** — they're a per-stack diff on a single
small vault.

What we can say with confidence:

- The **relative ordering** of stacks is robust: reranker > embeddings-only >
  TF-IDF > hybrid-without-reranker > FS-grep > BM25-alone holds at
  every NDCG@10 cut-point we tested.
- The **direction of HyDE's effect** is real, even if the magnitude may
  shrink on a larger corpus.
- The **reranker delta** (+24 MRR, +16 NDCG@10) is consistent with the
  literature on cross-encoder reranking over BM25/embeddings fusion
  (typical reported: +5-10 NDCG@10 across BEIR).

What we can't claim:

- That these absolute NDCG@10 numbers transfer to anyone's specific
  real-world Obsidian vault.
- That the reranker boost will be as large on a 5,000-note vault where
  recall is genuinely tight in the top-10.

**We welcome reproduction with public corpora.** A BEIR / TREC subset run
against `searchHybrid` is a planned future-work item; PRs with that
plumbing are welcome.

### HyDE without an LLM

Real HyDE requires an LLM call per query to generate the hypothetical
answer. We approximate it with hand-authored answers for determinism;
this is labeled "HyDE-sim" in every row. **The HyDE-sim numbers are a
weak lower bound** for what real LLM-driven HyDE would produce. A future
benchmark run with a local Llama / Claude in the loop would tighten this.

### Reranker score extraction

A quirk we discovered while authoring this bench: the high-level
`@huggingface/transformers` `text-classification` pipeline returns
`{label: "LABEL_0", score: 1.0}` for *every* input on the BGE-reranker-base
model — the pipeline softmaxes a 1-class head and always lands on 1. We
work around it by calling `AutoModelForSequenceClassification` directly
and applying sigmoid to the raw logit. This affects `src/embeddings.ts`
`loadReranker` and has been spun off as a separate fix task; the bench
numbers above use the direct-inference workaround.

### Mismatched-quantization caveat

The benchmark builds the embed-db with the `bge` model (BGE-small-en,
fp32 vectors). If the embed-db were built with a different model alias
than the query-time alias, enquire's contamination guard rebuilds it,
which would dilute the embeddings-only and hybrid numbers. We explicitly
pass `embedding_model: "bge"` at every query site to keep the meta-table
consistent. Replicating with a different model is a one-line change
(`EMBEDDER_ALIAS` constant in `scripts/run-benchmarks.mjs`).

### Vault size

48 notes is small. The top-10 cutoff captures roughly 20 % of the corpus,
which makes Recall@10 forgiving. On a 1,000-note vault Recall@10 captures
1 % of the corpus and is correspondingly harder to saturate — the
reranker's Recall@10 cost (-5.2 points here) would likely be smaller in
relative terms.

## Future work

- Reproduce on **public BEIR / TREC subsets** so numbers can be compared
  against the published IR literature.
- Add **HNSW vs. brute-force** comparison rows once HNSW is in this
  bench's hot path (currently brute-force only for determinism).
- Run with **real LLM-generated HyDE answers** (deterministic with a
  fixed-temperature decode + pinned LLM build).
- Extend to **multilingual queries** with the `multilingual` embedder
  (`Xenova/paraphrase-multilingual-MiniLM-L12-v2`).
- Compare against **other Obsidian-MCP servers** directly by wrapping
  their tools as a stack runner. This requires having those servers
  installable as npm deps; the FS-grep baseline above is the closest
  apples-to-apples we have today.

## Related

- [`docs/COMPARISON.md`](./COMPARISON.md) — feature matrix vs. other
  Obsidian-MCP servers.
- [`docs/api-reference/`](./api-reference/) — TypeDoc-generated API
  reference (links into `searchHybrid`, `embeddingsSearch`,
  `semanticSearch`).
- [`src/eval.ts`](../src/eval.ts) — the eval harness used by both this
  bench script and the `enquire-mcp eval` CLI subcommand.
- [`tests/fixtures/benchmark-queries.jsonl`](../tests/fixtures/benchmark-queries.jsonl)
  — the ground-truth query set.
- [`bench/benchmarks.json`](../bench/benchmarks.json) — machine-readable
  output (includes per-query scores and per-category breakdowns).
