# Track C MuSiQue smoke handover — honest negative on JAMES reasoning (2026-06-12)

> **Status**: Track C 첫 reasoning measurement 완료. ⭐ honest negative
> tier landing per Track C plan §5 ladder.
> **NOT publication** — n=10/20 smoke; full sweep + cross-model
> (mxtral / claude) + cross-bench (TimeQA / TempReason) 후만.
>
> **사용자 catch (2026-06-11) 재확인**: "위 측정이 추론 성능을 측정한것이
> 맞아?" — LRB는 retrieval, RAB는 audit. 추론은 별도 측정 필요. Track
> C가 그 axis. 결과는 ⭐ honest negative — JAMES는 multi-hop reasoning
> 에서 literature와 동급.

---

## 1. 한 줄 요약

MuSiQue n=20 × 3 SUT (Vanilla / Naive-supersede / JAMES) × gemma3:12b
측정에서 **3 SUT cell-by-cell IDENTICAL** EM=0.150 / F1=0.246 /
support_recall=0.65. JAMES validity-window mechanism은 closed-book
multi-hop reasoning에 완전 무관 confirmed. Cycle γ Phase C.2 finding
(JAMES = literature family on multi-hop) 독립 pipeline에서 reproduced.

## 2. 측정 결과

### 2.1 gemma4:e4b smoke (n=10, capability 부족)

| SUT | EM | F1 | support_recall | n_empty |
|---|---|---|---|---|
| Vanilla | 0.000 | 0.000 | 0.55 | 1 |
| Naive | 0.000 | 0.019 | 0.55 | 1 |
| JAMES | 0.000 | 0.019 | 0.55 | 1 |

4B 모델 — multi-hop reasoning capability 부족. 모델이 거의 모든 query
에서 "Insufficient Information" 또는 empty 응답. Honest abstention.

### 2.2 gemma3:12b smoke (n=20, capability emergence + 3 SUT identical) ★★★

| SUT | EM | F1 | support_recall | n_empty |
|---|---|---|---|---|
| **Vanilla** | **0.150** | **0.246** | **0.65** | 0 |
| **Naive-supersede** | **0.150** | **0.246** | **0.65** | 0 |
| **JAMES** | **0.150** | **0.246** | **0.65** | 0 |

★ **Cell-by-cell IDENTICAL** to 4 decimal places. 12B 모델에서 reasoning
capability emerges (EM 0 → 0.15) — 4B → 12B cross-model identical
pattern as LRB v0.2.1.

## 3. 해석 — 왜 3 SUT 동등한가

MuSiQue 는 **closed-book multi-hop** 질문:
- 각 query 가 20 paragraphs 의 자체 corpus 제공
- 사용자가 ingest 하는 history (INGEST/UPDATE/SUPERSEDE/DELETE) 0
- 단일-shot 측정 — lifecycle events 자체 없음

→ 3 SUT의 차이 (Vanilla = keep all / Naive = remove old on supersede /
JAMES = validity window) 가 **활성화 안 됨**. 모든 SUT 가 동일하게
20 paragraphs 를 ingest 하고 동일한 token-overlap 으로 retrieve. 답변
생성도 동일 (같은 retrieved context).

이게 publishable finding: **JAMES 의 mechanism 은 task-specific** —
lifecycle / time-travel 시나리오에서 작동하지만, generic multi-hop QA
에서는 무력. cycle γ + Track C 두 측정으로 확인.

## 4. 외부 인정 요건 정합 (Track C plan §2의 7 요건)

| 요건 | 본 cycle 충족 |
|---|---|
| 외부 표준 fixture | ✓ MuSiQue-Ans (Trivedi 2022 TACL, Apache 2.0) |
| 외부 표준 scoring | ✓ SQuAD v1.1 norm EM/F1 (학계 표준, RAB H1 정합) |
| 사전 등록 | ✓ Track C C0 (#793) + plan (#792) |
| Deterministic | ✓ (no LLM judge, only SQuAD-norm) |
| Multi-SUT | ✓ 3 SUT (Vanilla / Naive / JAMES) |
| Multi-model | partial (gemma4:e4b + gemma3:12b ✓; mxtral / claude operator-attended) |
| 공개 artifacts | ✓ result.json + bench.jsonl committed |

→ **6/7 충족**. Multi-model 완성은 operator action 필요 (mxtral 측정
~1.5h, claude 측정 ~30min).

## 5. Track C plan §5 honest-tier ladder 매핑

```
LADDER:
  ⭐⭐⭐ specialty advantage = JAMES > V on temporal + = on multi-hop + = on faithfulness
  ⭐⭐ partial = JAMES > V on 1 axis only
  ⭐ honest negative = JAMES ≈ V on all measured axes
  ⭐ surprising negative = JAMES < V (root-cause cycle)
```

본 cycle landing: **⭐ honest negative** on multi-hop axis (3 SUT identical
on MuSiQue). 다른 axes:
- ✓ Temporal (LRB time-travel) — JAMES > V (Phase B ⭐⭐⭐)
- ✓ Audit chain (RAB AC/RF/PC) — JAMES > V (publication tier)
- ✗ Faithfulness (HR axis) — partial leg cleared (v0.2.4 smoke n=10
  shows V hallucinates on LRB-S1, but n too small)
- pending: TimeQA (temporal reasoning answer F1)

→ 통합 tier: **⭐⭐ specialty advantage candidate** (specialty axis 측정
완료, generic axis 측정 완료, TimeQA 완성 시 ⭐⭐⭐ candidate).

## 6. 회사 결재 자료 framing — 최종 정확 표현

```
자메스의 측정-evidenced 우위 axes:

  ✓ Audit chain reasoning  (RAB AC/RF/PC = 1.000 / Baseline = 0.275)
                            publication tier (DOI 10.5281/zenodo.20625533)
  ✓ Temporal retrieval     (LRB Phase B R@1 0.713-0.838 / Vanilla 0.225)
                            ⭐⭐⭐ cross-scenario validated, 3/4 cross-model
  ~ Hallucination resist.  (v0.2.4 HR smoke V=0.0 / J=0.6 on LRB-S1,
                            cross-NLI agreement RoBERTa+DeBERTa ✓)
                            partial leg (n=10 too small for tier)

자메스의 측정-evidenced "동등" axes (literature family):

  = Multi-hop reasoning    (MuSiQue n=20 × gemma3:12b: V = N = J cell-
                            by-cell identical EM=0.15 / F1=0.246)
                            Track C ⭐ honest negative — cycle γ Phase C.2
                            reproduced
```

→ "JAMES 추론 성능 우수" framing 측정-evidence 없음. "JAMES 시간 인지
검색 + 감사 추적 우수, 일반 추론 동급" 이 정확 표현.

## 7. 인프라

| 위치 | 역할 |
|---|---|
| `eval/external/lrb/answer_gen.py` | LLM 답변 생성 (Ollama + claude CLI dispatch) |
| `eval/external/lrb/answer_f1.py` | SQuAD v1.1 norm EM/F1 scorer |
| `scripts/research/track_c_musique_smoke.py` | MuSiQue smoke CLI |
| `eval/external/_fixtures/musique/musique_ans_v1.0_dev.jsonl` | 2417 dev queries (이미 repo) |

신규 코드 ~700 줄 (answer_gen 343 + answer_f1 165 + runner 213 = 721).

## 8. 산출 artifacts (deterministic, 재실행 가능)

```
reports/external/musique/
  track-c-musique-smoke-20260611T161450Z.{vanilla,naive-supersede,james}-gemma4-e4b.{result.json,bench.jsonl}
  track-c-musique-smoke-20260611T161841Z.{vanilla,naive-supersede,james}-gemma3-12b.{result.json,bench.jsonl}
```

재실행:
```bash
PYTHONPATH=. python scripts/research/track_c_musique_smoke.py \
  --n 20 --sut all --model gemma3:12b --k 5
```

## 9. 다음 cycle 후보

1. **claude S2 sweep** (현재 background 진행 중) — 4/4 ⭐⭐⭐ leg
2. **MuSiQue mxtral 측정** — capability ladder 47B point (~1.5h)
3. **TimeQA 측정** — Track C 의 **primary** temporal axis (operator action:
   data download + license 확인)
4. **v0.2.2 GraphRAG SUT** — Track C 4-SUT lineup 완성
5. **HR axis full sweep** — n=100 × 3 SUT × 3 model × 2 NLI = 18 cells
6. **TempReason 측정** — Track C secondary bench (operator action)

## 10. 관련

- 사용자 catch: "위 측정이 추론 성능을 측정한것이 맞아?" (2026-06-11)
- Track C plan: `docs/strategy/track-c-reasoning-measurement-plan-2026-06-11.md`
- Track C C0: `docs/research/track-c-c0-bench-selection-2026-06-11.md`
- v0.2.4 HR prereg: `docs/research/v024-hr-nli-axis-preregistration-2026-06-11.md`
- LRB Phase B handover (retrieval): `docs/handovers/v0.4-lrb-phase-b-cross-scenario-2026-06-11.md`
- v0.2.1 cross-model partial handover: `docs/handovers/v0.4-lrb-v021-cross-model-partial-2026-06-11.md`
- Cycle γ Phase C.2 memory: `project_cycle_gamma_phase_c2_retrieval_bottleneck`
- self-eval trap: memory `feedback_self_evaluation_trap`
- forced discovery hunt END: memory `feedback_alpha_cycle_discovery_loop_end`
- single-axis ablation 금지: memory `feedback_single_axis_ablation_misframing`
- measurement-driven discovery: memory `feedback_james_identity_measurement_driven`

---

*본 cycle 의 honest negative = 측정-evidenced. 결과를 부정 framing
으로 정정하지 말 것. measurement-driven discovery 정신 정합 — cycle γ
prior art positioning + Phase A naive≈JAMES finding과 동일 패턴.*
