# app/eval - Eval Discipline

Applies to agents working under `app/eval/`. Read alongside root `AGENTS.md`, which takes precedence on any topic not covered here.

---

## data/ - fixture management

**`app/eval/data/` is not committed.** It is shown as an untracked directory in `git status`. `.gitignore` only excludes `app/eval/data/longmemeval_*.json` by pattern; `locomo10.json` has no explicit gitignore rule but is also not tracked. Neither file ships in the repo. Every worktree must populate its own copy.

### What each file is

| File | Benchmark | Size | Source |
|---|---|---|---|
| `locomo10.json` | LoCoMo (10-conv subset) | ~5MB | https://github.com/snap-research/locomo |
| `longmemeval_oracle.json` | LongMemEval oracle split | ~15MB | https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned |

### How to populate a new worktree

Option A: copy from a sibling worktree that already has the files:

```bash
find . -name "locomo10.json" -not -path "*/target/*"
# then:
cp /path/to/found/locomo10.json app/eval/data/locomo10.json
```

Option B: download from source:

```bash
mkdir -p app/eval/data
# LoCoMo: obtain locomo10.json from the snap-research/locomo repo (see README there)
# LongMemEval oracle:
curl -sL 'https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json' \
  -o app/eval/data/longmemeval_oracle.json
```

No import script exists in this repo for the LoCoMo 10-conv subset. If one is added, reference it here.

### What silently SKIPs when a fixture is missing

Tests that call `eval_root().join("data/<fixture>.json")` print "SKIP" and return early if the file is absent. They do not fail: they silently produce no metrics. This is the bug that triggered this doc.

Tests that depend on `locomo10.json`:
- `test_locomo_benchmark`
- `test_locomo_gate_comparison`
- `save_locomo_baseline`
- `save_locomo_cross_rerank_baseline`
- `save_locomo_v2_with_pages_baseline`
- `benchmark_locomo_pipeline`
- `benchmark_context_path`
- `generate_e2e_context_tuples_locomo`
- `generate_e2e_context_tuples_locomo_api`
- `judge_e2e_context_locomo` (and variants)
- `generate_fullpipeline_locomo`
- `smoke_per_scenario_locomo` (and `_cli`, `_cli_batched`)

Tests that depend on `longmemeval_oracle.json`:
- `test_longmemeval_benchmark`
- `test_longmemeval_gate_comparison`
- `save_longmemeval_baseline`
- `save_longmemeval_cross_rerank_baseline`
- `save_longmemeval_v2_with_pages_baseline`
- `benchmark_longmemeval_pipeline`
- `benchmark_context_path_longmemeval`
- `generate_e2e_context_tuples_longmemeval`
- `generate_fullpipeline_lme`
- `smoke_per_scenario_lme`
- `judge_fullpipeline_lme` (and `_cli`)

### Sibling fixture trees

Three sibling subdirs live alongside `data/` and are tracked in git:

- `app/eval/kg_fixtures/*.toml` - hand-curated entity + relation ground-truth, consumed by `eval::kg_faithfulness`. See root `AGENTS.md` "KG-faithfulness bench".
- `app/eval/page_fixtures/*.toml` - hand-curated source memories + distilled page bodies, consumed by `eval::page_faithfulness`. See root `AGENTS.md` "Page-distillation faithfulness bench".
- `app/eval/fixtures/*.toml` - 41 LoCoMo/LME fixtures vendored from `7xuanlu/origin-app` (PR #148). Used by `eval_harness.rs` integration tests.

---

## baselines/ - artifact layout

`app/eval/baselines/` is gitignored. Two parallel artifact paths exist:

### Legacy path (single-file format)

```
app/eval/baselines/<benchmark>__<retrieval_method>__<hash>.json
```

Used by README citations and external references. Written by `report.save_baseline(&baseline_path)`. The `save_locomo_v2_with_pages_baseline` and `save_longmemeval_v2_with_pages_baseline` tests branch the filename suffix on `WENLAN_ENABLE_PAGE_CHANNEL`:
- page-channel ON (`WENLAN_ENABLE_PAGE_CHANNEL=1`): `...__with_pages.json`
- page-channel OFF (default, unset): `...__no_pages.json`

### Layered path (P0b schema, for compare-baselines)

```
~/.cache/origin-eval/baselines/l1_db/<task>/<variant_tag>__<comparable_hash>.json
```

Written by `save_full_report` via the `save_layered` helper in `eval_harness.rs`. Used by `compare-baselines`. The `comparable_hash` is a SHA-256[..8] over a fixed subset of `ReportEnv` fields (fixture revision, embedder revision, LLM provider class, LLM model, mcp_schema_hash, skill_prompt_hash, schema_version, schema_db_version, similarity_fn_name). The `variant_tag` string (not flags) is the load-bearing differentiator because `flags` is not yet included in the hash (Task #30).

### Dual-write convention

New PR-B tests call `save_baseline` for the legacy path AND `save_layered` for the layered path. Older tests (`save_locomo_baseline`, `save_longmemeval_baseline`, etc.) call `save_layered` too, via the same pattern.

Exception: `save_locomo_cross_rerank_baseline` and `save_longmemeval_cross_rerank_baseline` write legacy-only by design. This protects the pre-PR-B 0.684/0.883 disk artifacts as a stable reference (see Task 5 plan). New `__with_pages` variants dual-write.

---

## environment variables

| Variable | Purpose | Default | Consumed by |
|---|---|---|---|
| `SCENARIO_DB_ROOT` | Override root for cached scenario DBs | (auto-resolve) | `resolve_scenario_db_root_from_harness`, `cached_scenario_db_check.rs` |
| `EVAL_BASELINES_DIR` | Root of the `~/.cache/origin-eval` cache | `$HOME/.cache/origin-eval` | `baselines_root()`, `resolve_scenario_db_root_from_harness` |
| `EVAL_LOCOMO_LIMIT` | Truncate LoCoMo fixture to N samples | full fixture (10 conversations) | all `run_locomo_eval*` variants |
| `EVAL_LME_LIMIT` | Truncate LME fixture to N samples | full fixture | all `run_longmemeval_eval*` variants |
| `LOCOMO_LIMIT_CONVS` | (fullpipeline only) Limit to first N conversations | full fixture | `generate_fullpipeline_locomo` in `answer_quality.rs` |
| `EVAL_SCENARIO_CONCURRENCY` | Parallel scenario seeding (fullpipeline) | 1 | `generate_fullpipeline_locomo`, `generate_fullpipeline_lme` in `answer_quality.rs` |
| `EVAL_MAX_USD` | Cost cap for API-batch judge runs (pre-flight and per-batch) | none | `anthropic.rs` (`parse_eval_max_usd`, `submit_batch`, `estimate_batch_cost`) |
| `EVAL_MAX_WALL_SECS` | Wall-clock timeout for eval runs | 14400 (4h) | `wall_clock.rs` watchdog |
| `EVAL_ALLOW_WIPE` | Allow `clear_all_for_eval` to wipe DB | unset (refuses) | `open_or_seed_scenario_db` stale-cache recovery |
| `WENLAN_ENABLE_PAGE_CHANNEL` | Enable page-channel in `search_memory_with_reranker` | unset (page-channel OFF) | `db.rs:search_memory_with_reranker`, `locomo.rs:run_locomo_eval_cross_rerank_from_db`, `longmemeval.rs:run_longmemeval_eval_cross_rerank_from_db`, suffix branching in `eval_harness.rs` |
| `WENLAN_EVAL_ROOT` | Override `eval_root()` in test harness | `app/eval/` | `eval_root()` in `eval_harness.rs` |

---

## seed scripts and cached-DB workflow

### seed-scenario-dbs.sh

`scripts/seed-scenario-dbs.sh` copies `origin_memory.db` from the canonical fullpipeline DBs to the scenario_seeded layout:

```
~/.cache/origin-eval/scenario_seeded/locomo_v1/origin_memory.db
~/.cache/origin-eval/scenario_seeded/lme_v1/origin_memory.db
```

Run from the repo root:

```bash
bash scripts/seed-scenario-dbs.sh
```

Sources: `~/.cache/origin-eval/fullpipeline_locomo_tuples.db/origin_memory.db` and `~/.cache/origin-eval/fullpipeline_lme_tuples.db/origin_memory.db`. If those originals do not exist, run the fullpipeline harness first (`generate_fullpipeline_locomo` / `generate_fullpipeline_lme`).

### cached_scenario_db_check.rs

`crates/wenlan-core/tests/cached_scenario_db_check.rs` (L7 manual, `--ignored`) opens each scenario DB via `MemoryDB::new`, which replays migrations idempotently, then prints table counts and 3 sample pages per DB. Root resolution: `SCENARIO_DB_ROOT > EVAL_BASELINES_DIR/scenario_seeded > ~/.cache/origin-eval/scenario_seeded/`.

Run with:

```bash
cargo test -p wenlan-core --test cached_scenario_db_check -- --ignored --nocapture
```

---

## eval citation discipline

Numbers from `~/.cache/origin-eval/baselines/` carry guardrails that MUST be honored when citing them externally (Reddit, HN, Karpathy gist, vendor decks, README, blog).

- **Default run policy (lenient + N=1).** Routine/iteration evals default to **N=1 with the lenient judge** (`strict_batch_judge_prompt`'s default branch — field-standard, matches the official LongMemEval `evaluate_qa.py` judge and every benchmarked peer). These are **directional only**: fast feedback for "did this move the needle", never cited externally (the Single-run rule below still binds). The **strict, citable path is explicit and opt-in**: set `WENLAN_EVAL_JUDGE_STRICT=1` **and** run **N≥3 + stddev**. So lenient/N=1 = working number; strict/N≥3 = headline number. Rationale: on byte-identical answers the judge alone swings ±7–13pp at N=1 (noise run 2026-06-15: strict 61.3% vs lenient 68.7% on 60Q), so a single run is signal-for-iteration, not evidence-for-claims.
- **Single-run rule.** Any baseline with `env.is_single_run = true` MUST NOT be cited externally. Internal team references are fine but must be flagged "single-run, treat as scaffold." Full citation requires the P1.5 multi-run protocol (mean ± stddev over ≥3 runs, ideally 10).
- **Schema-version rule.** Cross-`env.schema_version` comparisons are refused by `compare-baselines` (exit code 2). Public claims that compare numbers across schema versions MUST regenerate both sides via current `save_*_baseline` tests.
- **Receipt-only rule (extends cost-receipt).** Regression thresholds, latency claims, accuracy improvements must have a measured-stddev or N≥3-run backing. No "improved X%" or "regressed Y%" without `compare-baselines` output AND multi-run inputs.
- **Per-case visibility.** Aggregate accuracy claims must include per-case breakdown when available. Headline-only numbers hide regressions (LoCoMo adversarial-cat-5 contamination is the canonical trap; see `feature/eval-semantic-gaps` discussion).
- **Layer attribution.** Public numbers must specify L1 / L2 / L3 / L4. No cross-layer averages without explicit weighting.
- **Commit policy — snapshot, not history.** Metric *values* MAY be committed to git as a **curated, env-stamped snapshot** (the current headline numbers) in a results doc or README section, overwritten per release. Each committed value carries its methodology inline (model, dataset, run count, repro command); single-run results are tagged "scaffold" and headline/external claims still require the Single-run + Receipt-only gates above (N≥3 + stddev). Do **not** commit a per-run *history series* — that is what the gitignored `append_history` file is for. Raw per-run baseline JSONs under `~/.cache/origin-eval/baselines/` stay **gitignored** (artifacts, reproduced by re-running, not source). The repo is Apache-2.0; the older blanket "never commit numbers to git" rule is retired in favor of this snapshot policy.

---

## pre-flight checklist

Before running any eval test:

```
- [ ] app/eval/data/locomo10.json present. If missing, populate via sibling worktree cp or source download
- [ ] app/eval/data/longmemeval_oracle.json present. If missing, download via curl (see above)
- [ ] Cached scenario DBs present at ~/.cache/origin-eval/scenario_seeded/{locomo_v1,lme_v1}/origin_memory.db. If missing, run: bash scripts/seed-scenario-dbs.sh
- [ ] Branch is clean: git status --short shows only intentional changes
- [ ] Limit chosen: EVAL_LOCOMO_LIMIT=2 for ~1min wiring smoke; EVAL_LOCOMO_LIMIT=20 for ~30min subset; unset for full fixture
- [ ] For A/B comparison runs: identical env except the ONE variable being tested
```

---

## subset eval methodology (PR-B page-channel)

Page-channel impact is measured by running the same test twice:

```bash
# page-channel ON (opt-in):
WENLAN_ENABLE_PAGE_CHANNEL=1 cargo test -p wenlan-core --test eval_harness save_locomo_v2_with_pages_baseline -- --ignored --nocapture

# page-channel OFF (default):
cargo test -p wenlan-core --test eval_harness save_locomo_v2_with_pages_baseline -- --ignored --nocapture
```

The cached consolidated scenario DB introduces cross-conversation noise compared to the per-conversation ephemeral DB used by `save_locomo_cross_rerank_baseline`. So numbers from `save_locomo_v2_with_pages_baseline` are NOT directly comparable to the published 0.684 LoCoMo bar. They are comparable to the OFF variant of the same runner (page-channel unset/default).

The `variant_tag` field (`cross_rerank_v2_pages` vs `cross_rerank_v2_no_pages`) is the load-bearing differentiator on the layered baseline path, because `comparable_env_hash` does not yet hash `flags` (pending Task #30).
