# 2026-05-07 · Opportunity D pre-survey — submit_prefetch_plan exists but is dead code

> **2026-05-07 EOD update**: original ROI projection corrected
> in [`544d00d`](../../plans/M_pf-radix-prefetch-wiring.md). Plus
> M_ibp Phase 0 license-or-kill (`9432289`) showed ARLE already
> 1.80× past vLLM at the multi-tenant shared-prefix workload —
> the niche M_pf would help is one ARLE is already winning. M_pf
> formal ROI now 5-15% TTFT improvement at HIT cases, NOT the
> 4× faster originally projected. M_pf demoted to **P3** in
> M-final roadmap.

## Priority & ROI (corrected 2026-05-07 EOD)

**Priority**: **P3** (deprioritized post-M_ibp ABANDONED finding).
ARLE's existing chunk-boundary cascade pattern already covers most
of the multi-tenant shared-prefix scenario without M_pf wiring.

**ROI basis (corrected)**:
- Substrate cost: zero (`submit_prefetch_plan` already built with
  backpressure + queue-depth + PlanTicket). Implementation is
  wiring + 1 helper, ~80 LOC.
- Expected gain at multi-tenant shared-prefix:
  - Cache HIT bytes (6k prefix × 36 layers × FP8) ≈ 135 MB
  - PCIe 4.0 ~32 GB/s → ~14 ms (CONTIGUOUS) H2D, NOT 144 ms as
    originally claimed (per-layer-serial overestimate corrected
    in `544d00d`)
  - 5–15% TTFT improvement at HIT cases (small, not 4×)
- M_ibp ABANDONED finding (`9432289`) shows ARLE already 1.80×
  past vLLM at the very workload M_pf would target.

**Negative case**:
- Single-tenant / no-prefix-share workloads: zero benefit.
- Speculative prefetch evicts useful T0 blocks under pressure.
- Wasted bandwidth on cancelled requests.

**Kill criteria**:
- < 5% TTFT improvement at multi-tenant shared-prefix bench → revert.
- Eviction storm metric increase → revert + add backpressure.

**Why P3 vs original P1**:
- M_ibp Phase 0 showed ARLE already wins at the M_pf target shape.
- Original "4× faster than vLLM" projection was math error;
  corrected to 5-15%.
- Higher-priority items (M_world1 Phase 0 baseline + long-ctx
  4k TTFT gap) crowd out M_pf.

## Discovery

While codex implements M3.9 Phase 0 instrumentation, parallel
source-survey of `infer/src/kv_tier/coordinator/builder.rs`
revealed:

- L93: `submit_fetch(blocks: Vec<FetchRequest>) -> Option<FetchTicket>`
  — blocking-style fetch (request needs this KV to proceed)
- L145: `submit_prefetch_plan(blocks: Vec<PrefetchPlanRequest>) -> Option<PlanTicket>`
  — **speculative prefetch**, intended for "KV might be needed soon"

`grep -rn "submit_prefetch_plan" infer/src/`:
- builder.rs:145 (definition)
- builder.rs:110 (rustdoc reference)
- ZERO call sites elsewhere

**The prefetch infrastructure is dead code.** Wired into the
KVTierCoordinator (with backpressure, fetch_queue_depth, etc.)
but no scheduler path fires it.

## Why this matters for "world-first long-sequence engine"

For long-context workloads with shared system prompts (RAG,
code completion, agentic tools), many requests share long prefix
KV. When a fresh request arrives:

**Today (admission-blocking fetch)**:
1. Request arrives → admission queue
2. Admission runs `lookup_or_stage` → classifies blocks as
   StagingFromHost
3. `submit_fetch` → request waits for H2D copy to complete
4. Then prefill runs
5. **TTFT includes the H2D wait**

**With prefetch wired (Opportunity D)**:
1. Request arrives → admission queue
2. **As soon as prefix MATCH detected**, fire `submit_prefetch_plan`
   on the matched blocks (non-blocking)
3. Admission doesn't wait — request enters prefill immediately
4. By the time prefill reaches the layer that needs the cached
   KV, the H2D has likely completed (overlapped with prefill of
   the unmatched suffix)
5. **TTFT excludes the H2D wait**

For a 8k prompt with 6k cached prefix from system prompt:
- 6k tokens × hidden_dim × 36 layers × 2 bytes (fp8) = ~135 MB
- At PCIe 4.0 ~32 GB/s = ~4 ms per layer × 36 = ~144 ms total
  H2D
- Overlapping with prefill (which takes seconds) → 0 ms wait

For long-context multi-tenant workloads, this could deliver a
substantial TTFT improvement that the current admission doesn't.

## Architecture survey

Current code path for prefix HIT at admission
(`scheduler/cuda/runtime/admission.rs:147-227`):

```
1. lookup_or_stage(prompt_tokens) → LookupOutcome with HitKind
   per matched block (ReadyOnGpu | StagingFromHost | StagingFromDisk | Miss)
2. If all ReadyOnGpu → direct GPU page attachment
3. If some below-T0 → "staged readmission plan" via submit_fetch
   (line 552) — admission-blocking fetch
4. Fall through paths for stale/cold prefill
```

For the prefetch wiring, the simplest insertion point is
**before step 1** in admission.rs's `build_prefix_admission_plan`:
a "speculative classify" pass that runs `lookup_or_stage` BEFORE
the request even enters the waiting queue, fires
`submit_prefetch_plan` for any host/disk blocks, then admission
proceeds normally a few ticks later.

## Required design

| # | Task | LOC | Risk |
|---|---|---|---|
| 1 | Add `peek_prefix_classify(tokens)` helper to `RadixCache` (read-only lookup, doesn't bump ref counts) | ~30 | Low |
| 2 | At HTTP request entry (`request_handle_engine.rs`), call peek + submit_prefetch_plan when host/disk blocks match | ~50 | Medium (extra GPU memory pressure if speculative prefetch evicts useful blocks) |
| 3 | Cache-aware admission: when admission's lookup_or_stage runs later, if prefetch has completed, blocks are now ReadyOnGpu and direct-attach path fires | 0 (existing logic) | Low |
| 4 | Bench: long-ctx multi-tenant workload (e.g. shared 4k system prompt + 4 different short user queries) | 0 | n/a |

**Total: ~80 LOC, codex-sized**.

## Risks

- **R1 — Prefetch evicts useful blocks**: at high pool utilization,
  speculatively prefetching blocks could evict T0 blocks needed
  by other in-flight requests. Mitigation: prefetch only triggers
  when pool utilization < threshold (e.g. 70%). Coordinator
  already has `fetch_backpressured()` (types.rs:132) which
  prefetch can respect.
- **R2 — Speculative prefetch wastes bandwidth**: if request is
  cancelled before prefill, the prefetch was wasted H2D. Cost
  is small for typical workloads.
- **R3 — Race with cache eviction**: between prefetch completion
  and request reaching prefill, the cache might evict the
  prefetched blocks (if other requests pressure the pool).
  Mitigation: temporarily soft-pin prefetched blocks via the
  existing `soft_pin_keepalive_ticks` mechanism.

## Cross-references vs M-final roadmap

This is **Opportunity D** in `M-final-world-first-integration-roadmap.md`
(`d16effe`). Promotion from "future opportunity" to "ready-to-implement"
because:
- Infrastructure (`submit_prefetch_plan`) already exists
- Wiring location (admission.rs `build_prefix_admission_plan`) is
  small and well-understood
- M_d.1 namespace just landed — prefix correctness is now solid
- Expected gain is concrete (long-ctx multi-tenant TTFT cliff)

## Recommendation

**Move Opportunity D into the active roadmap as M_pf** (prefetch).
Ordering: after M3.9 Phase 1A v3 (Mixed-path restoration, in
codex's current pipeline), before M_b.2 Phase 2.

Estimated cumulative impact:
- M3.9 Phase 1A v3: long-ctx 8k TTFT 4961 → ~2500 ms (Mixed restored)
- M_pf: shared-prefix long-ctx TTFT 2500 → ~600 ms (eliminate H2D)
- Combined: ARLE long-ctx with shared prefix BEATS vLLM 2367ms by ~4×

## Bench Status

No new bench. Source-only finding. Validation bench requires
Opportunity D implementation + a multi-tenant shared-prefix
workload (not yet in scripts).

## Rule

- **Dead code in the codebase often signals "infrastructure that
  was built but the wiring is missing"**. Periodic
  `grep -rn submit_prefetch | grep -v test | grep -v doc` style
  audits surface these. Each one is an Opportunity D candidate.
- The pattern: someone in the past built the substrate (M_d.1
  level work), planning was correct, but the integration tick
  never landed. Surveying surfaces them efficiently.
