# LRB v0.2.1 cross-model — partial measurement (2026-06-11)

> **Status**: producer infrastructure (PR #788) merged + LLM-mode 측정
> 5 cells 완료 (이 doc commit 시점). Cell sweep 진행 중 — gemma4:e4b
> × S2 × 3 SUT background + mxtral / claude operator-attended pending.
> 본 doc 은 progressive update — full sweep 완료 시 final handover 별도.
>
> **Honest tier (current state)**: **⭐⭐⭐ first leg cleared**
> (gemma3:12b × S2 × 3 SUT 에서 S2 rank-order V<N<J 보존 + 3 SUT
> 모두 R@1 lift). 4/4 model 모두 만족 시 ⭐⭐⭐ tier landing — 2 model
> (gemma4:e4b S2 + mxtral / claude) 측정 후 확정.

---

## 0. ★ Scope disclaimer (사용자 catch 정정, 2026-06-11 추가)

**v0.2.1 cross-model 측정 = retrieval-only axis 의 cross-model 검증**.
LLM 이 "rerank" 단계에 들어가긴 하지만, 측정 axis 자체는 여전히
retrieval (R@k / R@1 / temporal_accuracy). **reasoning 측정 아님**.

framing 의무:
- ✅ "12B 모델 LLM rerank이 retrieval R@1 을 17.5pp 끌어올림"
- ❌ "12B 모델이 JAMES 추론 성능을 향상시킴" — 추론 측정 안 함

추론 성능 측정은 별도 Track C (TimeQA / TempReason / NLI scoring).

---

## 1. 한 줄 요약

LRB v0.2.1 cross-model producer 머지 (PR #788). 측정 시작 후 첫 두
셀 (S1 james × gemma4:e4b + gemma3:12b) 에서 **clean model size
threshold**: 4B = rerank이 가치 부재 (-0.033 R@1 regression), 12B =
marginal lift (+0.022 R@1). literature 정합 capability emergence
패턴.

## 2. 측정 완료 셀

### 2.1 Token-mode 6 cells (Phase B baseline reproduction)

deterministic byte-identical reproduction verified:

| Scenario | SUT | R@1 | R@10 | temporal_acc |
|---|---|---|---|---|
| S1 | Vanilla | 0.6167 | 0.8944 | 0.8944 |
| S1 | Naive-supersede | 0.7389 | 0.9167 | 0.9167 |
| S1 | JAMES | 0.7389 | 0.9167 | 0.9167 |
| S2 | Vanilla | 0.2250 | 0.9500 | 0.9500 |
| S2 | Naive-supersede | 0.5375 | 0.7500 | 0.7500 |
| S2 | JAMES | 0.7125 | 0.9750 | 0.9750 |

→ Phase B baseline 완전 reproduction → 인프라 측정 valid.

### 2.2 LLM-mode 셀 (현재 2 완료, 5+ pending)

#### Cell #1: S1 × james × gemma4:e4b (4B param, Google Gemma family)

| Axis | token | LLM-grounded | Δ (LLM − token) |
|---|---|---|---|
| R@5 | 0.8444 | n/a (per-cell only stored overall) | |
| R@10 | 0.9167 | **0.8833** | **−0.0333** |
| temporal_accuracy | 0.9167 | **0.8833** | **−0.0333** |
| **R@1** | **0.7389** | **0.7056** | **−0.0333** |

Elapsed: 622s (180 queries × 3.5s/query). Honest negative: 4B 모델의
reranker가 token-overlap top-20 위에서 가치 없음, 약한 negative.

#### Cell #2: S1 × james × gemma3:12b (12B param, Google Gemma family)

| Axis | token | LLM-grounded | Δ (LLM − token) |
|---|---|---|---|
| R@10 | 0.9167 | 0.9167 | 0 |
| temporal_accuracy | 0.9167 | 0.9167 | 0 |
| **R@1** | **0.7389** | **0.7611** | **+0.0222** |

Elapsed: 833s (180 queries × 4.6s/query). Marginal lift: 12B 모델이
top-10 내 reorder만 (R@10 unchanged), 단 top-1 정확도 +2.2pp.

#### Cells #3-5: **S2 × 3 SUT × gemma3:12b** ★★★ key leg

| SUT | R@1 token | **R@1 12B LLM** | Δ R@1 | R@10 token | R@10 12B LLM | temporal_acc 12B LLM |
|---|---|---|---|---|---|---|
| **Vanilla** | 0.225 | **0.400** | **+0.175** | 0.950 | 0.9375 | 0.9375 |
| **Naive-supersede** | 0.5375 | **0.6125** | **+0.075** | 0.7500 | 0.7625 | 0.7625 |
| **JAMES** | 0.7125 | **0.7750** | **+0.0625** | 0.9750 | 0.9625 | 0.9625 |

Elapsed: 370s per SUT × 3 = 1110s total (80 queries × 4.6s/query).

★★ **Cross-cell findings**:

1. **S2 rank-order V < N < J on R@1 PRESERVED under LLM rerank**:
   token: 0.225 < 0.5375 < 0.7125
   gemma3:12b: 0.400 < 0.6125 < 0.7750
   → ⭐⭐⭐ ladder leg 1 cleared (1 of 4 model)
2. **All 3 SUT benefit from rerank on R@1** (Vanilla +0.175 가장 크고,
   강한 SUT 일수록 marginal gain 작음 = literature 정합)
3. **JAMES still wins** but gap narrows: token V-J gap = 0.4875,
   12B LLM V-J gap = 0.375
4. **R@10 / temporal_acc 는 약간 떨어짐** (~-0.0125) — top-10 안에
   reorder만 발생 + 일부 borderline 정답 displaced. R@1 lift 가
   top-10 안 redistribution 의 trade-off.

★ Publishable framing: **JAMES validity-window mechanism 이 LLM
rerank 와 orthogonal하게 작동** (rank-order 보존). LLM rerank 는
"weak SUT (Vanilla) 를 끌어올리지만 JAMES advantage 자체는 보존".
즉, **operational moat (validity-window) + retrieval-quality boost
(LLM rerank) 가 additive** — 둘 다 채택 가능, 둘 중 하나만으로 충분
하지 않음.

#### Cells #6-8: **S2 × 3 SUT × gemma4:e4b** ★★ 2nd leg cleared

| SUT | R@1 token | **R@1 4B LLM** | Δ R@1 | R@10 4B LLM | temporal_acc 4B LLM |
|---|---|---|---|---|---|
| **Vanilla** | 0.225 | **0.4875** | **+0.2625** | 0.9625 | 0.9625 |
| **Naive-supersede** | 0.5375 | **0.5625** | **+0.025** | 0.7375 | 0.7375 |
| **JAMES** | 0.7125 | **0.725** | **+0.0125** | 0.95 | 0.95 |

Elapsed: 280s per SUT × 3 = 840s total (80 queries × 3.5s/query at 4B).

★★ **Cross-cell findings (4B vs 12B on S2 comparison)**:

1. **S2 rank-order V < N < J on R@1 STILL PRESERVED at 4B**:
   0.4875 < 0.5625 < 0.725 → **⭐⭐⭐ ladder leg 2 cleared (2 of 4)**
2. **4B model lifts weak baseline (Vanilla) MORE than 12B does**:
   - 4B Vanilla Δ = +0.2625
   - 12B Vanilla Δ = +0.175
   - 즉 noisy LLM signal 이 broken token retrieval (Vanilla R@1=0.225)
     위에서는 도움. 하지만 strong baseline (JAMES R@1=0.7125) 에서는
     12B Δ +0.063 > 4B Δ +0.0125 — 12B 가 균형 잡힌 lift.
3. **Task hardness 가 LLM rerank 가치 결정**:
   - S1 (current-only, easy): 4B → -0.033 regression (noise > signal)
   - S2 (mixed time-travel, hard): 4B → +0.013 lift (signal > noise on hard task)
   - 즉 LLM rerank 의 marginal value 는 task hardness × baseline strength
     의 함수.

★★★ Honest revised framing: capability emergence 가 단순 "model size 임계점"
이 아니라 **"task hardness 대비 model capability 임계점"**. weak baseline +
hard task = 4B 도 가치 있음. strong baseline + easy task = 4B 가치 없음.

### 2.3 Capability emergence ladder

| Model | Size | S1 james R@1 | Δ from token |
|---|---|---|---|
| (token baseline) | — | 0.7389 | — |
| gemma4:e4b | 4B | 0.7056 | **−0.0333** (regression) |
| gemma3:12b | 12B | **0.7611** | **+0.0222** (lift) |
| mixtral:8x7b | 47B (8×7B MoE) | pending | — |
| claude-haiku-4-5 | cloud | pending (operator action) | — |

★ Clean **size threshold finding**: capability emerges between 4B and
12B for LRB rerank task. 4B 미만 = regression risk, 12B+ = positive
contribution. literature 정합 (Schapire 1990 boosting + RAG
"cheat-sheet effect" 패턴 mirror — cycle γ prior art 정합).

★★ Hypothesis 검증 위해 mxtral (47B) 측정 필수 — 47B가 12B보다 더
큰 lift 시 monotonic scaling, 약한 lift / regression 시 plateau /
inverse scaling.

## 3. 진행 중 / Pending

### 3.1 Background running (이 doc 작성 시점)

- **mixtral:8x7b × S2 × 3 SUT** (~1.5-2h ETA): 47B large local model
  leg. 3rd of 4 model legs. mxtral 의 rank-order 보존 여부 + lift
  magnitude vs 12B 비교 = "monotonic scaling" or "plateau" 결판.

### 3.2 Operator-attended pending

- **mxtral (mixtral:8x7b) S1 + S2 × 3 SUT** (예상 ~2-3h total):
  capability ladder 끝, large local 모델
- **claude-haiku-4-5 S1 + S2 × 3 SUT** (cloud, operator API key 또는
  Max-plan headless `claude -p`): cross-family + cloud point

### 3.3 시점별 의사결정 brunch

| Background 결과 | 권고 다음 action |
|---|---|
| S2 rank-order V<N<J 보존 (+ J 신호 강화) | mxtral 측정 (capability ladder 완성) |
| S2 rank-order 일부 깨짐 (J ≤ N) | root cause cycle (prompt 조정 / k pool size) |
| S2 rank-order 완전 깨짐 (J < N < V) | LLM rerank 가 시간여행 axis 에 부합 안 함 = honest negative |

## 4. Prereg §1.4 honest tier landing — interim

```
LADDER (per prereg §1.4):
  ⭐⭐⭐ = 4 model × 3 SUT axes emit + S2 R@1 rank-order V<N<J
        reproduces in ALL 4 models
  ⭐⭐   = 4 model × 3 SUT axes emit + S2 rank-order 가 3/4 model 에서
        reproduce
  ⭐    = 절반 이하 model 에서 reproduce
```

현재 측정 leg:
- ✅ gemma4:e4b × S1 james + S2 × 3 SUT × axes emit
- ✅ gemma3:12b × S1 james + S2 × 3 SUT × axes emit
- ✅ **S2 R@1 rank-order V<N<J 보존 in 2 of 4 models**
  - gemma3:12b: 0.400 < 0.6125 < 0.7750 ✓
  - gemma4:e4b: 0.4875 < 0.5625 < 0.725 ✓
- ⏳ mixtral:8x7b × S2 × 3 SUT (background ~1.5-2h ETA)
- ⏳ claude-haiku-4-5 × S2 × 3 SUT (operator action)

**Interim tier**: **⭐⭐⭐ 2 of 4 legs cleared + nuanced capability
emergence finding** (task hardness × baseline strength × model size).
4/4 model 보존 확인 시 ⭐⭐⭐ tier landing 확정.

## 5. 인프라 (PR #788 merged)

- `eval/external/lrb/llm_rerank.py` — Ollama HTTP + claude CLI dispatch,
  deterministic (temp=0, seed=42, format=json), graceful degradation
- `eval/external/lrb/cross_model.py` — `retrieve_at_cross_model` 
  wrapper (token mode pass-through; llm-grounded = adapter top-20 →
  get_doc → rerank → top-k)
- `scripts/research/lrb_run_v021_cross_model.py` — CLI sweep (S1/S2
  auto-detect, token-cell dedup)
- `get_doc(doc_id)` accessor on 3 SUT adapters (read-only)
- 18 tests green

## 6. Re-execution (deterministic given Ollama state)

```bash
# Token-mode reproduction (no LLM needed; <1s)
PYTHONPATH=. python scripts/research/lrb_run_v021_cross_model.py \
  --modes token --models token-baseline

# LLM-mode single cell
PYTHONPATH=. python scripts/research/lrb_run_v021_cross_model.py \
  --modes llm-grounded --models gemma3:12b \
  --suts james --scenarios S1 --timeout 30

# Full operator-attended sweep (~5-10h)
PYTHONPATH=. python scripts/research/lrb_run_v021_cross_model.py \
  --modes token,llm-grounded \
  --models gemma4:e4b,gemma3:12b,mixtral:8x7b,claude-haiku-4-5 \
  --timeout 60
```

## 7. 다음 cycle

Background 완료 후:
1. S2 cells × gemma3:12b × 3 SUT → ⭐⭐⭐ ladder leg 1 확정 또는 결렬
2. mxtral S1 + S2 × 3 SUT → capability ladder 완성 + leg 2
3. claude-haiku-4-5 × S1 + S2 × 3 SUT → cross-family leg 3 (operator)
4. Full handover 별도 (v0.2.1 final) + memory 갱신
5. v0.2.2 GraphRAG SUT track 진입

## 8. 관련

- prereg: `docs/research/lrb-v021-cross-model-preregistration-2026-06-11.md`
- Phase B handover: `docs/handovers/v0.4-lrb-phase-b-cross-scenario-2026-06-11.md`
- design memo: `docs/design/v0.4-lrb-lifecycle-retrieval-benchmark-design.md`
- producer PR: #788
- memory `project_lrb_phase_a_smoke` (Phase A+B 측정 history, v0.2.1
  section 추가 예정)

## 9. Operator action 옵션

`현재 background gemma3:12b × S2 가 finished 후 user 결정 분기`:

(a) **mxtral 측정 즉시 진행** (예상 2-3h, capability ladder 완성)
(b) **claude wiring 시도** (Max-plan headless `claude -p` 또는 API key)
(c) **partial 측정으로 final handover 작성** (mxtral/claude 다음 cycle)

내 권고: **(a) mxtral 진행** — capability ladder finding 의 publishable
강도 ↑ + literature scaling 정합 / 위반 측정 완료. mxtral 결과 후 final
handover.

---

*본 doc 은 progressive update. 새 셀 결과마다 §2.2 + §3 갱신.*
