# Golden queries — design & policy

**Purpose:** Regression-test **Codemap internals** by comparing **`codemap query`** output to **checked-in expectations** (or subset matchers) on fixed corpora — **not** an LLM-in-the-loop eval. **Latency / tokens vs scanning:** [benchmark.md](./benchmark.md).

**Operational docs:** [CONTRIBUTING § Golden queries](../.github/CONTRIBUTING.md) · [benchmark § Fixtures](./benchmark.md#fixtures) · [benchmark § Agent eval harness](./benchmark.md#agent-eval-harness) (agent-eval harness — probe + live — reuses scenarios via `goldenId`) · Runner: [scripts/query-golden.ts](../scripts/query-golden.ts) · Setup: [scripts/query-golden/run-setup.ts](../scripts/query-golden/run-setup.ts) · Schema: [scripts/query-golden/schema.ts](../scripts/query-golden/schema.ts)

---

## Goals

| Goal                      | How scenarios help                               |
| ------------------------- | ------------------------------------------------ |
| **Catch regressions**     | Parser or schema drift → JSON diff vs golden     |
| **Encode good answers**   | Human-reviewed rows for representative queries   |
| **Stress realistic size** | Optional second corpus beyond `fixtures/minimal` |
| **Stay deterministic**    | Assertions on **query output**, not model prose  |

## Non-goals

- **Chat / SSE / auth** harnesses — out of scope here
- **Proving agents follow rules** — measure in the IDE or another project
- **Replacing** `src/benchmark.ts` — that stays **SQL vs glob/read time**; goldens add **correctness snapshots**

---

## How this fits other tooling

| Piece                     | Role                                                                        |
| ------------------------- | --------------------------------------------------------------------------- |
| `fixtures/minimal/`       | Tier **A** corpus; stable for CI                                            |
| `scripts/agent-eval/`     | Tier **A** agent-eval harness (`test:agent-eval`; reuses golden `goldenId`) |
| `src/benchmark.ts`        | Speed comparison (not golden row equality)                                  |
| `bun test`                | Unit tests for parsers, CLI, DB                                             |
| `CODEMAP_ROOT` / `--root` | Index **any** tree; Tier **B** uses env + optional gitignore                |

---

## No proprietary app code in this repo

We **do not** commit another product’s source tree, paths, business strings, or golden JSON **derived from** a private app (or any repo we do not own and license for redistribution).

| Safe to commit here                        | Not committed here                   |
| ------------------------------------------ | ------------------------------------ |
| **`fixtures/minimal/`** (trees we control) | Clones of private apps               |
| **Generic SQL** / **`--recipe`** ids       | App-specific path literals in assets |
| **Goldens** from **our** fixtures only     | Snapshots keyed to proprietary names |
| **Abstract `prompt` text** (intent labels) | Verbatim customer prompts            |

**Tier B:** Point `CODEMAP_ROOT` at a **local** clone; goldens for that tree stay **gitignored** (or private automation) — see [.gitignore](../.gitignore). For agent-eval on external fixtures, see [benchmark § Agent eval harness](./benchmark.md#agent-eval-harness) and [`.github/workflows/agent-eval-external.yml`](../.github/workflows/agent-eval-external.yml).

---

## Tier model

| Tier                       | Corpus                                   | When                       | Purpose                                                                                                                                                          |
| -------------------------- | ---------------------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **A (in-repo test bench)** | `fixtures/minimal` + `fixtures/golden/`  | Every PR / `bun run check` | **Canonical** for Codemap development — see [fixtures/README.md](../fixtures/README.md)                                                                          |
| **B (consumer-only)**      | Local path via `CODEMAP_*`               | Private app validation     | Goldens **gitignored** — not required to develop Codemap                                                                                                         |
| **Bench growth**           | `fixtures/minimal` + `CAPABILITIES.json` | Shipped (Phases 1–3)       | [testing-coverage.md](./testing-coverage.md), [fixtures/README.md](../fixtures/README.md); optional scale: [roadmap.md](./roadmap.md) (in-repo test bench scale) |

---

## Scenario shape (implemented)

Scenarios live in **`fixtures/golden/scenarios.json`** (Tier A) or optional **`scenarios.external.json`** / **example** (Tier B). The file may be a **bare array** of scenarios (legacy) or an object `{ "setup": [...], "scenarios": [...] }`. Optional top-level **`setup`** runs once after index, before scenarios — today **`ingest-coverage`**, **`clear-coverage`**, and **`seed-file-churn`** (see [run-setup.ts](../scripts/query-golden/run-setup.ts)); missing coverage files are skipped with a warning. Per-scenario **`preSetup`** runs after global setup (global setup restores after the scenario when `preSetup` mutates the index). Each scenario has **`id`**, **`sql` or `recipe`**, optional **`match`** (`exact`, `minRows`, `everyRowContains`, `everyRowFieldEquals`), optional **`budgetMs`**. Goldens: **`fixtures/golden/minimal/*.json`** etc. Refresh: **`bun scripts/query-golden.ts --update`**.

**Prompts** in JSON are **intent labels**, not pasted chat logs — pair with queries whose literals come from **fixture-owned** data (see [fixtures/qa/prompts.external.template.md](../fixtures/qa/prompts.external.template.md) for optional chat QA).

### Evidence columns (high-judgment recipes)

Some bundled recipes add optional **`reason`** (TEXT) and **`evidence_json`** (TEXT, JSON array) columns on each row — factual detection path for agents, not engine verdicts (Moat A — not pass/fail verdicts). Bounded subqueries cap evidence at three hops; list caps append `{"truncated":true}`. `unimported-exports` reasons: `no_direct_import`, `reexport_chain_possible`, `unresolved_import_blind_spot`. Goldens assert these columns when the recipe ships evidence (`boundary-violations`, `deprecated-symbols`, `unimported-exports`).

### Coverage columns (CRAP / enrichment recipes)

`high-crap-score` adds **`coverage_source`** (`measured` \| `estimated`) and **`effective_coverage_pct`** on each row — measured when `coverage` has a matching symbol row after `ingest-coverage`; otherwise graph-estimated tiers from test reachability. Goldens assert `coverage_source` when the recipe ships coverage semantics (`high-crap-score`); measured override is covered by `scripts/high-crap-score-measured.test.mjs`.

### Confidence columns (deletion-confidence recipes)

`coverage-confirmed-dead` adds **`confidence`** (`high` \| `medium`) on each row — **`high`** when static dead and ingested `coverage_pct = 0`; **`medium`** when static dead but the symbol has no ingested coverage row. Also **`reason`**, **`caller_count`**. Goldens: `coverage-confirmed-dead` (post-ingest mix) and `coverage-confirmed-dead-no-ingest` (`preSetup: clear-coverage`, `everyRowFieldEquals` on `confidence: medium`).

### Churn / hotspot columns (`churn-complexity-hotspots` recipe)

`churn-complexity-hotspots` ranks indexed files or symbols by git churn × cyclomatic complexity. File grain (default): **`file_path`**, null **`symbol_name`** / **`symbol_kind`** / **`line_start`**, **`max_complexity`**, **`avg_complexity`**, churn fields, scores. Symbol grain (`by_symbol=true`): **`symbol_name`**, **`symbol_kind`**, **`line_start`**, **`max_complexity`**. Optional **`path_prefix`** scopes to a subtree. Goldens: `churn-complexity-hotspots`, `churn-complexity-hotspots-by-symbol`, `churn-complexity-hotspots-path-prefix` (fixture churn seeded via **`seed-file-churn`**).

### Duplication columns (`duplicates` recipe)

`duplicates` returns one row per function-shaped symbol in a **`body_hash`** collision group: **`name`**, **`kind`**, **`file_path`**, **`line_start`**, **`line_end`**, **`body_hash`**, **`body_line_count`**, **`duplicate_count`** (in-scope group size after `path_prefix` / `min_body_lines`). Substrate column **`symbols.body_hash`** is populated at index for function-shaped symbols (`function`, `method`, `getter`, `setter`) when `body_line_count >= 2`. Goldens: `duplicates` (includes `src/bench/duplicate-body-{a,b}.ts` pair). False positives possible when unrelated functions share control-flow skeleton or sync vs async/generator bodies match — triage with `snippet`. Recipe caps at **50 rows** (no truncation marker).

---

## Status

| Area                          | State                                                                                                                                                                                                                                                                                                            |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Tier A runner + CI            | **`bun run test:golden`** + **`bun run test:agent-eval`** in `check` (CI Test job runs both; agent-eval reuses golden index when present)                                                                                                                                                                        |
| Tier A scenario coverage      | Scenarios cover core parser/schema surfaces, every bundled recipe (`templates/recipes/*.sql`), and SQL pin-down for substrate tables — see [testing-coverage.md](./testing-coverage.md). Guard: `scripts/query-golden-coverage-matrix.test.mjs`. Inventory: [scenarios.json](../fixtures/golden/scenarios.json). |
| Tier B external + schema      | **`test:golden:external`**, Zod in **`scripts/query-golden/schema.ts`**                                                                                                                                                                                                                                          |
| Subset matchers + budgets     | **`match`**, **`budgetMs`**, **`--strict-budget`**                                                                                                                                                                                                                                                               |
| Optional CI for public corpus | Deferred — [roadmap § Backlog](./roadmap.md#backlog)                                                                                                                                                                                                                                                             |

---

## References

- [benchmark.md](./benchmark.md) — speed methodology, Tier B, fixtures
- [architecture.md](./architecture.md) — schema, parsers
- [roadmap.md](./roadmap.md) — backlog
