# Zaxy 2.0 RC.1 Benchmark Freeze Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

Goal: Build the `2.0.0-rc.1` release-freeze gate that validates the exact benchmark artifacts and claim boundaries allowed for the 2.0 release candidate.

Architecture: Add a small release evidence contract that reads archived benchmark reports and internal lane summaries without changing retrieval, synthesis, or benchmark scoring behavior. Expose the contract through a CLI command and docs so RC evidence is reproducible, machine-readable, and separated into headline, external-anchor, and project-defined internal lanes.

Tech Stack: Python 3.11+, Typer CLI, dataclasses, JSON artifacts, pytest, existing Zaxy benchmark modules.

---

### Task 1: RC.1 Evidence Contract

Files:

- Create: `src/zaxy/rc_benchmark_freeze.py`
- Test: `tests/test_rc_benchmark_freeze.py`

- [ ] Step 1: Write failing tests

Add tests that create temporary LongMemEval, Harvey LAB, and internal-lane artifacts. Assert that the contract passes when all required evidence is present and fails when a lane is missing or when an internal lane is marked external.

Run: `pytest tests/test_rc_benchmark_freeze.py --no-cov -q`

Expected: FAIL because `zaxy.rc_benchmark_freeze` does not exist.

- [ ] Step 2: Implement evidence models and validation

Create focused dataclasses for evidence checks and a `build_rc1_benchmark_freeze_report(root: Path) -> RcBenchmarkFreezeReport` function. Validate:

- headline 500 report exists and passes floors: mean `0.95`, Answer@5 `0.90`, Recall@5 `0.99`, citation coverage `1.0`, p95 `2500`, p99 `3000`;
- run config exists beside the headline report;
- Harvey external artifacts exist and remain classified as `external_anchor`;
- internal 2.0 lane definitions include causal, consolidation, procedural, and metacognition as `project_defined_internal`;
- no internal lane is classified as external validation.

Run: `pytest tests/test_rc_benchmark_freeze.py --no-cov -q`

Expected: PASS.

### Task 2: CLI Gate

Files:

- Modify: `src/zaxy/__main__.py`
- Test: `tests/test_cli.py`

- [ ] Step 1: Write failing CLI tests

Add tests for `zaxy benchmark-freeze --json` using a temporary root. Assert exit code `0` for complete fixtures and exit code `1` when a required artifact is missing.

Run: `pytest tests/test_cli.py -k benchmark_freeze --no-cov -q`

Expected: FAIL because the command is not registered.

- [ ] Step 2: Add Typer command

Add `benchmark-freeze` as a top-level command. It should call `build_rc1_benchmark_freeze_report`, print JSON when `--json` is passed, print a concise text summary otherwise, and exit non-zero on failed gates.

Run: `pytest tests/test_cli.py -k benchmark_freeze --no-cov -q`

Expected: PASS.

### Task 3: Docs And Claim Boundary

Files:

- Modify: `docs/benchmarks.md`
- Modify: `docs/testing.md`

- [ ] Step 1: Update benchmark docs

Add a `Zaxy 2.0 RC.1 Benchmark Freeze` section that names the gate command and states that it validates existing artifacts without changing benchmark behavior. Keep LongMemEval-compatible, Harvey LAB, and project-defined internal lanes separate.

- [ ] Step 2: Update testing docs

Add the exact RC.1 freeze command to the release guardrail list.

Run: `scripts/validate-docs.sh --root .`

Expected: PASS.

### Task 4: Verification

Files:

- Existing test and report artifacts only.

- [ ] Step 1: Run focused tests

Run:

    pytest tests/test_rc_benchmark_freeze.py tests/test_cli.py -k "benchmark_freeze or rc1" --no-cov -q

Expected: PASS.

- [ ] Step 2: Run existing benchmark guardrail

Run:

    python -m zaxy benchmark-compare reports/benchmarks/longmemeval-500-publish-20260607/live-benchmark.json --backend zaxy-checkout --min-mean-score 0.95 --min-answer-recall-at-5 0.90 --min-recall-at-5 0.99 --min-citation-coverage 1.0 --max-p95-ms 2500 --max-p99-ms 3000

Expected: PASS.

- [ ] Step 3: Run RC.1 freeze gate

Run:

    python -m zaxy benchmark-freeze --json

Expected: JSON report with `"passed": true`.