# v0.3.x — Measurement Framework Track (V3' series 후속)

> Status: open — 2026-05-23 V3'.e (PR #440) + Robin 26b cross-stack (issue #448) 으로 3-axis framework 확정. 6 방향 후속 plan locked.
> Created: 2026-05-23
> Companion documents:
>   - `reports/promo-assets/v3prime-e-substitution-synthesis-result.md` — V3'.e result doc + §External validation + §Cross-stack replication
>   - `docs/handovers/v0.3.x-ali-collaboration-track.md` §Track 5 — joint paper narrative
>   - `docs/handovers/v0.3.x-gemma4-feedback-track.md` — internal evidence pile
>   - `scripts/research/v3prime_e_mode_split.py` — V3'.e driver
>   - https://github.com/triavalabs/gemma4-26b-mode-split — Robin's companion repo + JSON
>   - `reports/research-runs/v3prime-e-mode-split-20260523T051851.json` — JAMES e4b raw data

## Purpose

V3'.a/.b/.c/.d (PR #407) + V3'.e (PR #440) + Robin 26b cross-stack (2026-05-23 issue #448) sweeps가 합쳐서 **재현 가능한 3-axis LLM cost behavior measurement framework**를 만들어냈다. Robin이 우리 JSON schema를 cross-stack 분석 template으로 채택하면서 framework가 "data we publish" → "schema another lab analyses against"로 격상.

이 handover는 그 framework을 강화·발전시키는 6 방향 follow-up plan을 stage별로 정리. 다음 세션이 이 페이지만 읽으면 어느 방향부터든 시작 가능하게.

## Three confirmed axes (recap)

| # | Axis | Owner | Evidence | Status |
|---|---|---|---|---|
| 1 | Mode split (substitution=결정적, synthesis=가변) | Robin (LinkedIn 2026-05-22 + 26b 결정성 강화 issue #448) | T-sweep + bit-for-bit determinism on substitution arm (1 unique response across 40 calls on 26b) | Confirmed on both stacks |
| 2 | Workload gradient (heavy/light/no synthesis) | JAMES (V3'.e + V3'.a~d) | 4-stage 0/10 + e-commerce 14/20 + verbatim 20/20 @ cap=400 | Confirmed on e4b |
| 3 | Model-scale efficiency (synthesis cost ∝ 1/param-count; substitution invariant) | Robin (26b sweep, 2026-05-23) | 26b synthesis ~9× more token-efficient than e4b on same fixture | New — single point so far |

3-author lock된 headline: **"Substitution is free. Synthesis costs in proportion to what it has to invent."** (Ali 독립 재도출 + Robin endorse + JAMES origin)

## Six follow-up directions

| # | 방향 | Stage | 작업량 | 효용 종류 | Collaborator pattern |
|---|---|---|---|---|---|
| 4 | Substitution bypass verification on e4b | 1 (즉시) | 1일 | Publishable mechanism (axis 1 강화) | land 후 issue #448 응답 + Ali 한 줄 notice |
| 6(J) | Methodology spec standalone | 1 (즉시) | 3-5일 | Framework owner 위치 영구화 | land 후 informational DM 2개 |
| 1 | Adaptive Budgeting in Cognitive Middleware | 2 (단기) | 1-2주 | JAMES product 직접 격상 | land 후 launch-tracker row + DM 없음 (내부 변경) |
| 2 | Task-weight metric formalization | 2 (단기) | 1-2주 | Cost prediction predictor | land 후 publish 시 schema 공유 |
| 5 | Auto-routing on Provider Contract | 3 (중기) | 1-2주 | JAMES = auto-routing local Graph-RAG | design preview to Ali (Provider Contract 위 layer니까), land 후 informational DM |
| 3 | Cross-family generalization (Llama/Qwen/DeepSeek) | 3 (중기) | 2-3주 | Framework universal vs Gemma-only 판정 | Robin에게 사전 한 줄 notice + 결과 first-share |
| 6(I) | Joint paper consolidation | 4 (publication) | 2-3주 | Collaboration capital 회수 | **active consent + co-byline (Track 5 본질)** |

## Stage-by-stage 실행 계획

### Stage 1 (Week 1, 즉시)

**작업 1.A — Direction 4: Substitution bypass verification (1일)**

목표: Robin Finding 1 ("40/40 unique=1, mode bypasses sampling")이 e4b에서도 holds 하는지 검증.

- [ ] `scripts/research/v3prime_e_mode_split.py` patch: `call_ollama()` return에 `raw_response_text` 필드 추가 (full text, 디버깅 + unique 비교용)
- [ ] `summarize()` 함수에 unique-output count column 추가 (`unique = len(set(r['raw_response_sha256'] for r in runs))`)
- [ ] V3'.e re-run on gemma4:e4b (`python scripts/research/v3prime_e_mode_split.py --n 20`), 3분
- [ ] 결과 분석:
  - substitution arm 40 calls 중 unique=1 → **publishable mechanism**, axis 1 강화
  - unique > 1 → 26b에서만 bypass라는 model-scale-mechanism 발견 (그것도 가치)
- [ ] `reports/promo-assets/v3prime-e-substitution-synthesis-result.md` §Cross-stack replication subsection에 e4b unique-count 결과 append
- [ ] PR 생성 — 단일 commit, driver patch + result doc update

**작업 1.B — Direction 6(J): Methodology spec standalone (3-5일)**

목표: 다른 lab이 우리 protocol을 인용·확장할 수 있는 standalone spec.

- [ ] `docs/research/v3prime-protocol-v1.md` 신규 — JSON schema spec (`meta`, `runs[arm][cap][run_idx]`, 각 필드 의미), fixture design guideline, statistical floor (N≥20, T=0.2 권장 등), reproducibility checklist
- [ ] Robin 26b sweep + JAMES V3'.a~d/.e를 사용 사례로 인용
- [ ] PR 생성 — docs only, no code change

**Stage 1 Collaborator pattern**: 작업 1.A land 후 — issue #448에 substantive 응답 ("We re-ran on e4b and unique=N, mirrors your Finding 1" — Robin이 ball을 명시 우리에게 던졌음). 1.B land 후 — DM 2개 ("methodology spec landed at <URL>, no action needed").

### Stage 2 (Week 2-3, 단기)

**작업 2.A — Direction 1: Adaptive Budgeting (1-2주)**

목표: PR #399의 `DEFAULT_MAX_TOKENS = 4096` hard floor를 weight-aware budgeting으로 대체.

- [ ] `core/reasoning/budget.py` 신규 — `TaskBudget.assess(stage, prompt, context) → num_predict`
  - V3' 데이터에서 측정된 weight를 hardcoded rule로 시작 (heuristic: substitution-pattern detect → 200, light synthesis → 800, heavy synthesis → 4096)
  - fallback: `done_reason=length` 응답 시 자동 cap 2× retry
- [ ] `core/reasoning/{query_rewriter,planner,reflect,verify}.py` — 4 stage가 hardcoded 4096 대신 `TaskBudget().assess()` 호출
- [ ] `tests/test_adaptive_budget.py` 신규 — 각 stage가 적절한 cap을 요청하는지 contract pin
- [ ] STEP 7 bench in PR body (CLAUDE.md rule #2)
- [ ] 기대 effect: 평균 토큰 사용량 -60~80% (substitution 작업은 200으로 충분)

**작업 2.B — Direction 2: Task-weight metric formalization (1-2주)**

목표: "task weight"을 prompt에서 자동 계산 가능한 metric으로 정의.

- [ ] 5-7 level fixture 설계 — substitution / light retrieve / single-step synthesis / multi-step synthesis / structured-JSON synthesis / 4-stage cognitive / open-ended generation
- [ ] 각 level에서 V3' protocol으로 measure
- [ ] 후보 metrics 회귀 fitting: output entropy estimated / logical step marker count / verb modality / output template constraint / domain vocab depth
- [ ] `docs/research/v3prime-task-weight-metric.md` 결과 + best-fit metric
- [ ] 작업 2.A의 `budget.py`에 metric-based assess() 옵션 layer 추가 (옵션 — hardcoded heuristic은 fallback으로 유지)

**Stage 2 Collaborator pattern**: 2.A는 JAMES 내부 변경 → launch-tracker row만. 2.B는 publish 시 schema 공유.

### Stage 3 (Month 2, 중기)

**작업 3.A — Direction 5: Auto-routing on Provider Contract (1-2주)**

목표: Track 1 Provider Contract 위에 backend selection layer.

- [ ] `core/reasoning/router.py` 신규 — `select_backend(prompt, context, budget) → Backend`
  - input: prompt + 2.A의 `TaskBudget` 결과
  - 의사결정: substitution / light → 작은 모델, heavy → 큰 모델 또는 cloud
- [ ] env: `JAMES_ROUTER_RULES` — 라우팅 정책 외부화
- [ ] `tests/test_router.py` — 라우팅 결정 contract pin + 모든 backend가 Provider Contract conformance suite 통과한다는 architectural test
- [ ] **Ali에게 design preview** (작업 시작 1주 전): `docs/design/v0.3-router-on-provider-contract.md` link + "Provider Contract 자체 변경 zero, 위 layer" 명시

**작업 3.B — Direction 3: Cross-family generalization (2-3주)**

목표: Substitution/synthesis split이 Gemma 4 family architectural property인지 LLM general property인지 판정.

- [ ] Ollama pull: `llama3.1:8b`, `llama3.1:70b`, `qwen2.5:7b`, `qwen2.5:32b`, `deepseek-v2:16b`
- [ ] V3'.e 그대로 (driver model arg만 변경)
- [ ] 각 family 결과 표 + 통합 분석
- [ ] `docs/research/v3prime-cross-family-2026-MM.md`
- [ ] **Robin에게 사전 한 줄 notice** (작업 시작 시): "Running cross-family this week. Whatever shape comes back, you see first." → 결과 도착 시 그녀에게 first-share

**Stage 3 Collaborator pattern**: 3.A는 Ali design preview + land 후 informational DM. 3.B는 Robin first-share.

### Stage 4 (Month 3-4, publication)

**작업 4 — Direction 6(I): Joint paper consolidation**

목표: Stage 1-3에서 모인 데이터 + 3 axes를 joint paper로.

- [ ] Track 4 scope-lock template 적용 — `reports/promo-assets/cross-experiment-scope-lock-2026-MM.md` 작성, Ali + Robin 사인
- [ ] outline 합의 (3-author, role-split):
  - Ali — deployment context / managed-vs-sovereign comparison (Track 3 swap 결과)
  - Robin — mode split + model-scale efficiency (axis 1 + 3)
  - JAMES — workload gradient + adaptive budgeting + auto-routing (axis 2 + product impact)
- [ ] draft → 양쪽 review → submission (arXiv 또는 dev.to long-form 시리즈)

**Stage 4 Collaborator pattern**: **active consent + co-byline (Track 5 본질)**. Track 4 scope-lock note가 governance.

## Done-when per direction

| 방향 | Done when |
|---|---|
| 4 (substitution bypass) | e4b unique-count 측정 결과가 result doc에 박힘 + issue #448 응답 댓글 |
| 6(J) (methodology spec) | `docs/research/v3prime-protocol-v1.md` published + DM 2개 sent |
| 1 (adaptive budget) | `core/reasoning/budget.py` ship + 4 stage 전환 + STEP 7 bench PR body + `test_adaptive_budget.py` 통과 |
| 2 (task-weight metric) | `docs/research/v3prime-task-weight-metric.md` published + best-fit metric 명시 |
| 5 (auto-routing) | `core/reasoning/router.py` ship + Ali design preview sent + Provider Contract 변경 zero |
| 3 (cross-family) | `docs/research/v3prime-cross-family-*.md` published + Robin first-share confirmed |
| 6(I) (joint paper) | Track 4 scope-lock 사인 + submission complete (arXiv or dev.to) |

## CLAUDE.md rule alignment

| Rule | This track |
|---|---|
| #1 — no domain features before v1.0 | ✅ — 모두 platform-skeleton or research, no domain pack |
| #2 — bench numbers on `core/retrieval`/`graph`/`reasoning` PRs | Direction 1, 5는 core/reasoning → STEP 7 bench 필수 |
| #3 — self-evolution opt-in only | Unrelated |
| #4 — architecture changes via `architecture` label PR | Direction 5의 router는 borderline — Provider Contract 위 layer이지만 architecture document touch 가능. 검토 필요 |
| #5 — no file in `core/` exceeds 20 KB | 신규 모듈 (`budget.py`, `router.py`) — file 크기 모니터링 필요 |

## v0.4 / v0.5 정합성

- **v0.4 Layer 4 (Lifecycle Semantics — EVENT/TEMPORAL + GOVERNANCE)**:
  - Direction 1 (adaptive budget) → GOVERNANCE 측면 결합 (budget 결정 audit trail = T1 audit invariant 안)
  - Direction 5 (auto-routing) → EVENT 측면 결합 (routing 결정 = event, T1 audit + T7 invariant 안)
  - design 시 v0.4 hook 사전 잡기 — 더블 작업 방지

- **v0.5 First Domain Pilot**:
  - Direction 1 + 5의 결합 = domain pack 별 model 자동 routing → customer onboarding playbook의 핵심 가치 propose
  - Direction 2의 task-weight metric은 domain-agnostic 측정 (어느 도메인이든 동일 metric 적용)
  - Direction 3 cross-family는 domain pack 사용자가 family 선택 가능하게 함

## How a future session uses this page

1. Stage 1.A (Direction 4) 먼저 — 가장 작은 작업, 가장 큰 비대칭 upside
2. Stage 1.B (Direction 6(J)) — 1.A와 평행
3. Stage 2.A (Direction 1) — JAMES product 직접 격상
4. Stage 2.B (Direction 2) — 2.A의 input
5. Stage 3은 Stage 2.A/B land 후
6. Stage 4 (joint paper)는 Stage 1-3 데이터 풍부할수록 강함

각 Stage 진입 시:
- 작업 시작 — TaskCreate로 stage 단위 task 등록
- 진행 중 — Collaborator pattern (informational notice / first-share / pre-design notice) 적용
- land — launch-tracker row 추가 + 해당 doc update + 필요시 DM
- next stage — 이 페이지의 stage table 다시 읽고 시작

## 한국어 요약

V3'.e + Robin 26b cross-stack 데이터로 LLM cost behavior의 3-axis measurement framework이 자메스 손에. 6 방향 후속 plan (4 substitution bypass / 1 adaptive budget / 2 task-weight metric / 6J methodology spec / 5 auto-routing / 3 cross-family + 6I joint paper) 을 4 stage로 정리. 모든 방향이 platform-skeleton 강화 — 도메인 분화 없음. Track 1 success pattern (land first + "no action needed" DM) 유지, joint paper만 active consent. v0.4 Layer 4 + v0.5 Domain Pilot에 직접 plus.
