# Design Memo — `eval/qvt/oracle.py` → `eval/qvt/oracle/` package split

> **Status**: design — defer-eligible. Proposed for a future cleanup
> cycle (post α-5 matrix lands); the file is currently maintainable but
> trending toward the legibility-and-review-cost zone where a split
> pays back.
>
> **작성일**: 2026-05-31 (mid-α-5)
> **Trigger**: oracle.py is 34 KB / 914 lines after #618/#619/#623/#623
> additions (source-recall, abstention phrase coverage). The CLAUDE.md
> rule 5 `core/` 20 KB gate is strict for `core/` but `eval/qvt/` is
> outside that scope; this is a hygiene split, not a rule-driven one.

---

## 0. TL;DR

Split `eval/qvt/oracle.py` into a `eval/qvt/oracle/` package with one
module per axis + a shared utility module + `__init__.py` re-exporting
every public symbol for back-compat. Six target modules, each under
6 KB. Public import surface unchanged
(`from eval.qvt.oracle import score_five_axis, FiveAxisResult, …`
keeps working).

---

## 1. Current state (post #623)

```
eval/qvt/oracle.py        34,828 bytes  914 lines
```

| Section | Lines | ~KB | Purpose |
|---|---|---|---|
| Module docstring + imports | 1-40 | 1 | top of file |
| Abstention phrases + `detect_abstention` | 41-114 | 3 | shared between axes |
| `_slug_for_match` (path matching) | 116-160 | 1.5 | unique to path axis |
| Path Coverage axis | 162-313 | 7 | `PathCoverageQueryRow`, `PathCoverageAxis`, `_graph_node_slugs_from_bench_row`, `score_path_coverage` |
| Graded Answer axis | 315-471 | 5 | `GradedAnswerQueryRow`, `GradedAnswerAxis`, `_answer_text`, `_has_negation_around`, `_matches_signal`, `score_graded_answer` |
| Abstention F1 axis | 473-577 | 3.5 | `AbstentionQueryRow`, `AbstentionF1Axis`, `score_abstention_f1` |
| Cost axes (token + latency) | 579-684 | 3.5 | `CostQueryRow`, `TokenCostAxis`, `LatencyCostAxis`, `_percentile`, `_collect_cost_rows`, `score_token_cost`, `score_latency_cost` |
| ThreeAxisResult container | 686-762 | 2.5 | legacy 3-axis composition |
| FiveAxisResult + entry funcs | 764-914 | 5 | `FiveAxisResult`, `score_three_axis`, `score_five_axis`, `score_five_axis_by_question_type` |

7 cohesive segments. The only cross-segment dependency at the
*function* level is: graded_answer's `_matches_signal` uses the
negation-guard helper, abstention_f1 uses `detect_abstention`. Path
coverage and cost axes are independent. The unifying
`score_three_axis` / `score_five_axis` orchestrators just call each
axis's `score_*` function.

---

## 2. Proposed structure

```
eval/qvt/oracle/
├── __init__.py             # re-export everything for back-compat
├── _abstention_phrases.py  # _ABSTENTION_PHRASES + detect_abstention
├── path.py                 # path coverage axis + _slug_for_match
├── graded.py               # graded answer axis + negation guard
├── abstention.py           # abstention F1 axis
├── cost.py                 # token + latency axes + _percentile
├── result.py               # ThreeAxisResult + FiveAxisResult containers
└── entry.py                # score_three_axis + score_five_axis + by_question_type
```

| Module | Est. size | Imports from sibling |
|---|---|---|
| `_abstention_phrases.py` | ~3 KB | (none — pure data + 1 fn) |
| `path.py` | ~7 KB | (none — self-contained) |
| `graded.py` | ~5 KB | (none — negation guard is internal) |
| `abstention.py` | ~3.5 KB | `_abstention_phrases` |
| `cost.py` | ~3.5 KB | (none) |
| `result.py` | ~2.5 KB | path / graded / abstention / cost dataclasses |
| `entry.py` | ~5 KB | result + all four axis-scorer functions |
| `__init__.py` | ~1 KB | re-exports everything for `from eval.qvt.oracle import …` |

Largest module: `path.py` at ~7 KB. Well under any reasonable threshold.

---

## 3. Back-compat strategy (NO downstream breakage)

The current public surface is everything imported from
`eval.qvt.oracle`. Callers today include:

| Caller | What it imports |
|---|---|
| `scripts/qvt_capture_baseline.py` | `FiveAxisResult`, `ThreeAxisResult`, `score_three_axis`, `score_five_axis`, `score_five_axis_by_question_type` |
| `scripts/qvt_ablation_matrix.py` | `FiveAxisResult`, `score_five_axis`, `score_five_axis_by_question_type` |
| `scripts/qvt_rescore_baseline.py` | `FiveAxisResult`, `score_five_axis`, `score_five_axis_by_question_type` |
| `scripts/qvt_fill_t0_analysis.py` | None (uses standalone copies of verdict logic) |
| `tests/test_qvt_oracle.py` | `detect_abstention`, `_percentile`, `score_*`, all `*Axis` types, all `*Result` types, `_slug_for_match` |

The `__init__.py` re-exports every public name. Internal
helpers (`_slug_for_match`, `_percentile`) that tests touch are
re-exported as documented internal API.

```python
# eval/qvt/oracle/__init__.py
"""α-5 QVT oracle — axis scorers + 3/5-axis result composition.

Package was a single file (`eval/qvt/oracle.py`) until v0.4.x α-5
cycle PR #<TBD>; split for maintainability without changing the
public import surface.
"""
from eval.qvt.oracle._abstention_phrases import detect_abstention
from eval.qvt.oracle.path import (
    PathCoverageAxis,
    PathCoverageQueryRow,
    _slug_for_match,
    score_path_coverage,
)
from eval.qvt.oracle.graded import (
    GradedAnswerAxis,
    GradedAnswerQueryRow,
    score_graded_answer,
)
from eval.qvt.oracle.abstention import (
    AbstentionF1Axis,
    AbstentionQueryRow,
    score_abstention_f1,
)
from eval.qvt.oracle.cost import (
    CostQueryRow,
    LatencyCostAxis,
    TokenCostAxis,
    _percentile,
    score_latency_cost,
    score_token_cost,
)
from eval.qvt.oracle.result import (
    FiveAxisResult,
    ThreeAxisResult,
)
from eval.qvt.oracle.entry import (
    score_five_axis,
    score_five_axis_by_question_type,
    score_three_axis,
)

__all__ = [
    "detect_abstention",
    "PathCoverageAxis", "PathCoverageQueryRow", "score_path_coverage",
    "GradedAnswerAxis", "GradedAnswerQueryRow", "score_graded_answer",
    "AbstentionF1Axis", "AbstentionQueryRow", "score_abstention_f1",
    "CostQueryRow", "LatencyCostAxis", "TokenCostAxis",
    "score_latency_cost", "score_token_cost",
    "ThreeAxisResult", "FiveAxisResult",
    "score_three_axis", "score_five_axis", "score_five_axis_by_question_type",
    # documented internal API for tests
    "_slug_for_match", "_percentile",
]
```

Zero changes needed in callers. The migration is byte-stable from the
outside.

---

## 4. Migration steps (single PR)

1. Create `eval/qvt/oracle/` directory.
2. Move sections of `oracle.py` into the sub-modules, preserving line
   order within each section.
3. Add `eval/qvt/oracle/__init__.py` with the re-exports above.
4. Delete `eval/qvt/oracle.py`.
5. Run the existing 47 oracle tests — should pass without modification
   because the public import surface is unchanged.
6. Run all the scripts that import the package — sanity that
   nothing leaks.

Single mechanical PR. Quality Delta Card exempt (label: `chore` or
`code` — no behaviour change).

---

## 5. Why defer this (don't do it during α-5 matrix)

- The α-5 matrix is running while this memo is written. Any oracle
  change forces a re-baseline + re-run. Mechanical refactors are no
  exception — the bytes change, the SHA changes, the baseline JSON
  filename changes. Worth waiting for the matrix verdict to land
  first.
- Future fix PRs (e.g. bucket-(c) LLM-judge oracle augmentation, see
  `feedback_oracle_phrase_artifacts` open questions) will be cleaner
  on the post-split layout, but landing them inline on the current
  single-file is also fine if the matrix is mid-run.

**Trigger to do this split**: after the v0.4-end α-5 cycle closure
PR lands AND before any of the deferred-bucket-(c) work (LLM-judge
abstention detector) begins. That ordering keeps the LLM-judge
oracle module's natural home (`eval/qvt/oracle/abstention_judge.py`)
free to land alongside the existing `abstention.py` rather than
deepening an already-large single file.

---

## 6. Out of scope for this memo

- Behaviour changes — none. This is purely a layout / file
  reorganisation.
- API renames — none. Everything keeps its current name.
- Test file split — `tests/test_qvt_oracle.py` is currently ~26 KB
  itself; it can either stay one file or split mirroring the
  package structure. Defer the test-side split decision until the
  module-side lands and we see whether grouped fixtures pay back.
- `eval/qvt/` directory inventory — `oracle.py`, `__init__.py`,
  `baseline_*.json`. The split changes only `oracle.py`.

---

## 7. References

- File: `eval/qvt/oracle.py` (current 914-line target)
- CLAUDE.md rule 5: ~~module size gate~~ — strict for `core/` only,
  but informational for `eval/`
- Tests: `tests/test_qvt_oracle.py` — 47 tests covering all public
  surface, untouched by split
- Callers: `scripts/qvt_capture_baseline.py`,
  `scripts/qvt_ablation_matrix.py`, `scripts/qvt_rescore_baseline.py`
- Future bucket-(c) home: `eval/qvt/oracle/abstention_judge.py`
  (when LLM-judge lands — see
  `memory/feedback_oracle_phrase_artifacts.md` open questions)
