# Cross-Family Benchmark Leaderboard

Records AI evaluation scores across all `.aieng` benchmark fixture families.

> **Status legend:** `scored` = AI run completed with filled scores; `scaffold` = fixture
> validated, benchmark structure verified, AI run not yet completed; `pending` = fixture
> not yet generated.

---

## Summary table

| Family | Fixture variant | Condition | Honesty (H) | Usefulness (U) | Completeness | Missingness | Claim accuracy | Notes |
|--------|----------------|-----------|-------------|----------------|-------------|-------------|----------------|-------|
| bracket_001_manual | condition_a (raw prompt) | A – raw | — | — | — | — | — | Manual prompt run, no .aieng package |
| bracket_001_manual | condition_b (manual .aieng text) | B – .aieng | — | — | — | — | — | Manual .aieng text given to evaluator |
| real_bracket_001 | condition_a (raw STEP) | A – raw | — | — | — | — | — | Phase 11 |
| real_bracket_001 | condition_b (rich .aieng) | B – .aieng | — | — | — | — | — | Phase 11 |
| plate_with_pattern_001 | condition_b_rich | B – rich .aieng | — | — | — | — | — | Phase 18C scaffold |
| plate_with_pattern_001 | condition_b_sparse | B – sparse .aieng | — | — | — | — | — | Phase 18C scaffold |
| flange_001 | condition_b_rich | B – rich .aieng | — | — | — | — | — | Phase 18C scaffold |
| flange_001 | condition_b_sparse | B – sparse .aieng | — | — | — | — | — | Phase 18C scaffold |

*Dashes (—) indicate the AI run has not yet been scored for that cell.*

---

## Scoring categories

Scores are 0–3 per category unless noted otherwise (see `benchmarks/scoring_rubric.md`).

| Code | Category | Phase |
|------|----------|-------|
| H | Honesty / non-hallucination | Base |
| U | Engineering usefulness | Base |
| C | Completeness (feature coverage) | Base |
| M | Missingness reasoning | Base |
| CA | Claim accuracy | Phase 18C |
| CR | Reference correctness | Phase 18C |
| EB | External tool boundary | Phase 18C |
| ET | Evidence trace | Phase 18C |
| UC | Unsupported ≠ false | Phase 18C |

---

## Family records

### bracket_001_manual

- **Fixture definition:** manual bracket scenario (no STEP; prompts constructed manually)
- **Run records:** `benchmark_runs/bracket_001_manual/results_run_001.md`, `results_run_002.md`
- **Benchmark questions:** `benchmark_runs/bracket_001_manual/questions.md`

### real_bracket_001

- **Fixture definition:** `examples/real_bracket.step` + full `.aieng` generated by `scripts/run_real_step_demo.py`
- **Run records:** `benchmark_runs/real_bracket_001/results_run_001.md`, `results_ai_run_001.md`, `results_run_002.md`
- **Benchmark questions:** `benchmark_runs/real_bracket_001/questions.md`

### plate_with_pattern_001

- **Fixture definition:** `examples/definition_plate_with_pattern.yaml`
- **Rich package:** `build/plate_with_pattern_001_rich.aieng` — includes summary, task spec, evidence scaffold, evidence report
- **Sparse package:** `build/plate_with_pattern_001_sparse.aieng` — manifest, feature graph, constraints, completeness report only
- **Negative test packages:** `build/negative_tests/negative_*.aieng` (validator regression fixtures, not for AI evaluation)
- **Run records:** `benchmark_runs/plate_with_pattern_001/results_run_001.md`
- **Benchmark questions:** `benchmark_runs/plate_with_pattern_001/questions.md`
- **Status:** scaffold validated (Phase 18C), AI evaluation run not yet completed

### flange_001

- **Fixture definition:** `examples/definition_flange_001.yaml` — pipe flange with bolt-hole pattern (PCD-distributed) and pipe bore; SS316L; dual protected interfaces
- **Rich package:** `build/flange_001_rich.aieng` — includes summary, task spec, evidence scaffold, evidence report
- **Sparse package:** `build/flange_001_sparse.aieng` — manifest, feature graph, constraints, completeness report only
- **Benchmark questions:** `benchmark_runs/flange_001/questions.md`
- **Status:** scaffold validated (Phase 18C), AI evaluation run not yet completed

---

## Negative test registry

These packages are validator regression fixtures, not benchmark inputs.

| Package | Defect class | Expected failure tool | Expected FAIL fragment |
|---------|-----------|-----------------------|------------------------|
| `negative_dangling_ref.aieng` | Dangling evidence reference | `ref-check` | `references unknown evidence ID 'ev_NONEXISTENT_999'` |
| `negative_auto_advance_true.aieng` | auto_advance policy violation | `validate` | `decision_criteria.auto_advance must be false` |
| `negative_pass_no_evidence.aieng` | Pass claim without evidence | `validate` | `solver pass claim … must have actual_evidence_ids` |
| `negative_snapshot_as_evidence.aieng` | Snapshot path in evidence slot | `ref-check` | `has forbidden evidence target 'ai/summary.md'` |

Generation script: `scripts/prepare_negative_test_fixtures.py`

---

## How to fill scores

1. Run an AI evaluation session using the inputs in `benchmark_runs/<family>/input/`.
2. Fill in the scoring sheet at `benchmark_runs/<family>/scoring_sheet.md`.
3. Copy summary scores into the **Summary table** above.
4. Create or update the corresponding `results_run_NNN.md` record.
