# Design Memo — Bucket-(c) LLM-Judge Abstention Detector

> **Status**: design (deferred to post-α-5 closure). The cycle's
> phrase-based detector (`eval/qvt/oracle.py::detect_abstention`,
> #619 + #623) cleared 14 of the 19 original FN cases at zero FP cost
> by adding 10 narrow English refusal phrases. The remaining 6 FN sit
> in a region the phrase approach cannot safely enter without FP-
> flooding present-truth queries.
>
> **작성일**: 2026-05-31 (mid-α-5)
> **Trigger**: post-#629 oracle package split land; before any
> attempt to push abstention F1 past ~0.70 on the corrected baseline.
> **Companion docs**: `feedback_oracle_phrase_artifacts` (open
> questions), `v0.4-qvt-oracle-package-split.md` (target home).

---

## 0. TL;DR

When the phrase-list detector hits its precision ceiling, replace it
with a paired prompt-call to a local LLM (gemma3:12b judge model, same
shape as `v3prime_a3_v2_llm_judge.py` PR #611) that answers a single
binary question: *"Does this answer refuse to answer the user's
question?"* The judge sees the question + the answer in isolation,
no scoring rubric beyond the binary. Returned label drives
abstention F1 directly; phrase detector becomes a fast pre-filter.

Cost: ~1.5-2 s per query on local gemma3:12b. Acceptable for ~100
queries per matrix cell; deferred behind a flag for ~10,000 query
production loads.

---

## 1. Why the phrase detector hit its ceiling

The corrected baseline (post #623) on the canonical bench:

```
abstention_f1 = 0.704
  TP = 19   FP = 10   FN = 6   TN = 65
```

Inspecting the 6 remaining FN (the "still hallucinating per the
phrase detector" rows on null_query truth):

| qid | Answer (preview) | What phrase would catch it |
|---|---|---|
| q55 | "Based on the provided internal data, it is **not possible** to definitively name the country…" | `not possible to` |
| q67 | "The critical information needed to answer this question **is not available** in the current data set…" | `is not available` |
| q68 | "The core information required to complete this deduction…**is not available** in the current dataset…" | `is not available` |
| q71 | "Based on the provided source files, it **is not possible** to determine the first letter…" | `not possible to` |
| q63 | "The first letter of the state governed by Ron DeSantis is **F**. Verification: the following claims are **not directly supported** by the source data…" | (borderline — answered + caveat) |
| q64 | "The first letter of the Asian country is **I**." | (genuine confident wrong answer — no refusal phrase) |

Phrases that would catch the four unambiguous refusals (q55/q67/q68/q71)
are **`is not available`** and **`not possible to`**. Adding either to
the list:

| Added phrase | New FP | F1 impact |
|---|---|---|
| `is not available` | q9 ("The company is Amazon. Information about latest Q4 results is not available…"), q26 (Amazon ETF rating), q47 (FBI Director chart) — 3 partial-answer + disclaimer FPs | 0.704 → 0.71 (small TP gain offset by FP rise) |
| `not possible to` | q26 ("…it is not possible to confirm specific revenue increases…"), q43 (DeFi index movement caveat) — 2 disclaimer FPs | 0.704 → 0.70 (no net gain) |

The phrase-based approach is at its precision ceiling. Adding more
phrases doesn't make the detector better, just trades FN reduction
for FP injection at the same rate.

**Bucket**: (c) feature gap. The capability needed — "does this
answer constitute a refusal of the question, considering the
question's specificity" — is not within the phrase-matcher's reach.

---

## 2. Why an LLM judge is the right shape

The remaining FN cases all involve **partial-answer + disclaimer**
patterns:

- Correct refusal embedded in a longer hedge: q55, q67, q68, q71
- Partial answer + uncertainty caveat: q63
- Confident wrong answer with no refusal anywhere: q64

Distinguishing them programmatically requires understanding **what
question was asked** and **what the answer actually commits to** —
both context-dependent.

The same 2026-05-31 cycle's PR #611 (A3 v2 LLM-judge) already
established the gemma3:12b paired-comparison pattern for *answer
quality* judging. The same harness shape works here:

- One Ollama call per query (~1.5-2 s on gemma3:12b for a ~300-token
  prompt)
- Deterministic at temp=0.2
- Position-bias control unnecessary (binary single-input, not paired)
- Local — no external API cost

The judge answers one question only:

> *"Does this answer refuse to answer the user's question, in whole
> or in part such that the answer cannot be considered a substantive
> reply?"* — **yes** / **no** / (rarely) **partial — judgment call**.

The phrase detector becomes a fast pre-filter. Anything the phrase
list matches → fast-path to `abstain=True`. Anything the phrase list
misses → judge call. Anything the judge says is partial → tied to a
deterministic rule (e.g. "partial counts as answer for F1
calculation").

---

## 3. Module placement (post oracle package split)

```
eval/qvt/oracle/
├── __init__.py
├── _abstention_phrases.py    # current narrow phrase list (#619 + #623)
├── abstention.py             # AbstentionF1Axis + score_abstention_f1
├── abstention_judge.py       # ← NEW: LLM-judge backend
├── path.py
├── graded.py
├── cost.py
├── result.py
└── entry.py
```

`abstention_judge.py` exposes:

```python
def judge_abstention(question: str, answer: str,
                     model: str = "gemma3:12b",
                     temperature: float = 0.2,
                     timeout_s: float = 30.0) -> Optional[bool]:
    """Returns True if the judge classifies the answer as a refusal
    (or unsubstantive partial answer), False if it classifies it as
    a substantive answer, or None on judge failure (timeout / parse
    error). Caller falls back to phrase detector on None."""
```

`abstention.py::score_abstention_f1` gets a new optional
`use_judge: bool = False` argument:

```python
def score_abstention_f1(bench_results, fixture, *,
                        use_judge: bool = False,
                        judge_model: str = "gemma3:12b") -> AbstentionF1Axis:
    ...
    if detect_abstention(answer):
        is_abstain = True
    elif use_judge:
        judged = judge_abstention(question, answer, model=judge_model)
        is_abstain = bool(judged) if judged is not None else False
    else:
        is_abstain = False
    ...
```

Default `use_judge=False` keeps the phrase-only behaviour byte-stable
for existing callers (#619 + #623 phrase list + #618 source-recall +
the 5-axis composition). Operators opt in per run.

---

## 4. Prompt design

The prompt is built to keep the judge's task narrow. Drawing on the
A3 v2 paired-judge pattern (PR #611), but binary rather than paired:

```text
You are a strict judge of answer behaviour. You will see exactly
one question and the answer the system gave to it. Decide whether
the answer refuses to answer the question (in whole or in any
substantive part) or whether it gives a real, substantive answer
to the question.

A refusal includes:
- "I don't have / cannot find / no information / refuse / decline"
- "Source files / internal data do not contain / relate to / mention"
- "It is impossible / not possible to answer / determine / identify"
- "The information needed is not available / not present / lacking"
- "Cannot be determined / confirmed / identified / answered"

An answer that hedges with caveats but still names the requested
entity / fact / number is NOT a refusal — it is a substantive answer
with a disclaimer. Example: "The company is Amazon. However, the
revenue details are not contained in the source files." → not a
refusal (Amazon is the answer).

An answer that confidently states a wrong-looking fact without
refusing is also NOT a refusal. Example: "The first letter is I."
→ not a refusal (the system commits to "I" as the answer).

[QUESTION]
{question}

[ANSWER]
{answer}

Respond with one word: REFUSAL or ANSWER. No explanation.
```

Parsing: strip whitespace + lowercase + match `refusal` or `answer`.
Anything else → return `None` (caller falls back to phrase detector).

---

## 5. Validation plan

### 5.1 Re-score against the corrected baseline bench JSON

Run `abstention_judge.py` on the 100 queries of
`bench_3a961a3_multihop_rag_20260531_094341.json`. Expected outcome
based on the 6 FN inspection:

- q55, q67, q68, q71 → REFUSAL (correctly recovered as TPs)
- q63 → REFUSAL or ANSWER (judgment call; either is acceptable as
  long as it's consistent)
- q64 → ANSWER (correctly identified as confident-wrong, not refusal)

Predicted F1: **0.75 – 0.82** depending on q63 classification and
whether any new FPs surface on the truth=present side.

### 5.2 FP audit on truth=present queries

Run the judge on all 75 truth=present queries. Count how many it
classifies as REFUSAL (i.e., new FPs).

- If new FPs ≤ 3 → F1 climbs cleanly past 0.75
- If new FPs > 5 → judge is too aggressive; tighten prompt or fall
  back to phrase detector on borderline cases

### 5.3 Cost measurement

- Per-query: ~1.5-2 s on gemma3:12b local
- 100 queries × 2 s = ~3.3 min per scoring run
- Acceptable for matrix cells (each cell is already ~107 min). Not
  acceptable for ~10,000 query production loads — guard behind a
  flag.

### 5.4 Failure modes

- Judge timeout → return None, caller falls back to phrase detector
- Judge returns unparseable response → return None, caller falls
  back
- Judge model unavailable → caller silently uses phrase-only path
  (degrades to current behaviour, no exception)

---

## 6. Sequencing — when to implement

The implementation is **gated on three preconditions**:

1. **α-5 matrix closure PR** lands. The current numbers are reported
   against the phrase-only detector for cycle simplicity. Switching
   to judge-augmented mid-cycle would force a baseline re-capture
   and complicate the publishable narrative.
2. **Oracle package split** (PR #629 design) lands. The judge module
   has its natural home in `eval/qvt/oracle/abstention_judge.py`;
   slotting it in pre-split deepens the single-file problem.
3. **Audit of the cycle's `feedback_oracle_phrase_artifacts` rule**
   — confirm the 4-step verification is the dominant discipline. If
   the rule is unstable / disputed, judge augmentation muddies the
   teaching example. (Current expectation: stable.)

After all three, implement in a single feat PR with:
- New `eval/qvt/oracle/abstention_judge.py`
- Optional `use_judge=True` flag on `score_abstention_f1`
- Re-scored baseline comparison (phrase-only vs judge-augmented)
- 3 contract tests (REFUSAL accept, ANSWER accept, None fallback)

---

## 7. Cost & operational notes

- **Local judge model**: gemma3:12b. Already in operator's Ollama
  registry per α-5 cycle (used by `v3prime_a3_v2_llm_judge.py`).
- **No external API**: zero $-cost per query; pure local compute.
- **Latency**: 1.5-2 s per query on gemma3:12b. Multiply by suite
  size — balanced-100 = ~3.3 min, balanced-500 = ~17 min,
  full-2556 = ~85 min. Run frequency: per matrix cell when
  `--use-judge` flag is on; otherwise never.
- **Reproducibility**: temperature=0.2, fixed model tag. Same input
  → same output. Bench JSON stores the judge model tag so
  re-scoring later under a different judge model is detectable.

---

## 8. Open questions deferred to implementation

- **Default for which suites?** Probably opt-in via `--use-judge`
  flag at first; turn on by default after one cycle's stability data.
- **Caching?** Per-query judge results are deterministic (temp=0.2)
  — cache by `(question, answer)` hash to avoid re-judging across
  re-scores. Cache TTL: cycle-length (a day or two).
- **Q63 partial-answer classification**: judges differ on whether a
  correct-name-plus-disclaimer should count as REFUSAL or ANSWER.
  Spec a strict rule before implementation (default: name commits
  → ANSWER).
- **Per-question_type calibration**: do null_query refusals look
  different than (rare) refusals on truth=present queries? May
  warrant per-type prompt tweaks.

---

## 9. Relation to A3 v2 (#611)

PR #611 introduced the gemma3:12b paired-judge harness for *answer
quality comparison*. This memo's judge is the unpaired sibling for
*answer abstention classification*. Same model, same prompt-call
shape, same JSON-parse + None-on-failure pattern.

Implementation can lift the boilerplate (Ollama call wrapper, prompt
template structure, parse-or-None contract) from
`scripts/research/v3prime_a3_v2_llm_judge.py` rather than re-deriving.

Difference: this judge is harness-internal (lives in `eval/qvt/`);
A3 v2 is research-only (lives in `scripts/research/`). The harness
home is what makes this part of the canonical oracle, not a one-off
experiment.

---

## 10. References

- Bucket-(c) framing: `memory/feedback_oracle_phrase_artifacts.md`
  (4-bucket taxonomy + open questions section)
- Phrase-detector ceiling evidence: this memo §1 + PR #623 commit
  description
- Module placement post-split:
  `docs/design/v0.4-qvt-oracle-package-split.md` §2 target layout
- LLM-judge harness pattern: PR #611
  (`scripts/research/v3prime_a3_v2_llm_judge.py`)
- α-5 cycle closure (predecessor):
  `reports/research-runs/alpha-5-cycle-summary-DRAFT.md`
- Publishable framing: `reports/research-runs/alpha-5-publishable-narrative-DRAFT.md`
  (this memo's existence is itself a §9.1 open-question item in that doc)
