# #36 PrefixAwareAdmission Layer 2 direct-HTTP bench — KILL on warm-mix workload

## Context

Following the GuideLLM JSONL invalidation (codex's 8150bfe), Claude
built `scripts/bench_36_warmmix_direct.py` (3b4c89b) per codex's
"Next Step" recommendation: aiohttp streaming bench with warm/cold
labeling and /v1/stats capture. This errors entry documents the
Claude-self-driven A/B run using that script.

Prior layers:
- Layer 1 substrate: PASS (gate fires under cold-only pressure,
  079639c counter)
- Layer 2 mechanism via GuideLLM: BLOCKED (8150bfe — JSONL parse
  drains 256 rows in 1.3s)
- Layer 2 mechanism via direct HTTP: PASS (this entry — workload
  exercises cache, prefix_hit_rate=57.2%, deferrals=73 in arm B)
- **Layer 2 perf license: KILL** (this entry — every metric regressed
  in PrefixAware vs QueueBound baseline)

## Goal

Type: license-or-kill PrefixAwareAdmission as multi-tenant default
on the deterministic warm/cold prefix-mix workload from
`scripts/gen_36_warm_prefix_mix.py`.

## Hypothesis

Per `docs/research/2026-05-10-36-followup-warm-mix-bench-spec.md`:
PrefixAware should improve warm-session p50 TTFT ≥20% with cold p95
no more than 3× warm p95 (no starvation).

## Commands

Identical server-restart-between-arms protocol per codex's 8150bfe
errors entry, only difference = bench tool (direct-HTTP vs GuideLLM):

```bash
# Workload (deterministic, regenerated by codex's prior tick): 256 reqs
# = 153 warm (4 sessions × ~38 reqs sharing 1024-tok prefix) +
#   103 cold (unique random 1280-tok prompts)

# Server (one config per arm)
CUDA_HOME=/opt/cuda \
TORCH_CUDA_ARCH_LIST=8.9 \
INFER_TILELANG_PYTHON=/home/ckl/projects/arle/.venv/bin/python \
INFER_HYBRID_W4A8_PREFILL=1 \
INFER_PREFILL_GRAPH=1 \
./target/release/infer \
  --model-path infer/models/Qwen3-4B-W4-hybrid-zpfix \
  --port 8765 \
  --num-slots 8 \
  --max-seq-len 5120 \
  --admission-policy <queue-bound|prefix-aware> \
  --cold-headroom 253

# Bench (NEW direct-HTTP harness)
python3 scripts/bench_36_warmmix_direct.py \
    http://127.0.0.1:8765 Qwen3-4B-W4-hybrid-zpfix \
    bench-output/36-warm-mix.jsonl \
    --concurrency 8 --warm-count 153
```

## Environment

- Commit: `b96a1e7` (skill v1.10.0 + /v1/stats parse fix landed mid-bench)
- GPU: RTX 4070 Ti SUPER 16 GiB
- CUDA: `/opt/cuda`, `TORCH_CUDA_ARCH_LIST=8.9`
- Model: `infer/models/Qwen3-4B-W4-hybrid-zpfix`
- Workload: 256-row JSONL, 153/103 warm/cold split
- Concurrency: 8

## Results

### Per-arm bench output (Claude self-driven, n=1)

| Metric | Arm A (queue-bound) | Arm B (prefix-aware) | Δ% | License gate |
|--------|---:|---:|---:|---|
| Wall time | 59.5s | 61.2s | +2.9% | regression |
| Throughput | 505 tok/s | 488 tok/s | **−3.4%** | KILL gate (≤+10% required) |
| Warm p50 TTFT | 66 ms | 66 ms | 0% | NO IMPROVEMENT |
| Warm p95 TTFT | 230 ms | 269 ms | **+17%** | regression |
| Warm p99 TTFT | 699 ms | 695 ms | -0.6% | flat |
| Warm Total p50 | 1546 ms | 1545 ms | 0% | flat |
| Cold p50 TTFT | 136 ms | 137 ms | +0.7% | flat |
| Cold p95 TTFT | 1048 ms | **2241 ms** | **+114%** | KILL gate (worsens cold tail) |
| Cold p99 TTFT | 4747 ms | 4348 ms | -8.4% | mild improvement |
| Cold Total p50 | 2061 ms | 2060 ms | 0% | flat |
| Cold/Warm p95 | 4.56× | **8.33×** | +83% | **STARVATION KILL** (≤3× required) |

### Server-side /v1/stats arm B (text/plain key=value, raw curl)

```
requests=257
prefix_hit_rate=57.2%
prefix_skip_rate=45.5%
session_affinity_hit=0          ← workload has no session_id, expected
prefix_aware_admit_deferrals=73 ← gate FIRED 73 times during 256-req workload
matched_prefix_tokens=0         ← no session_id path, expected
resume_prefill_tokens=1351
kv_util=50%
peak_mem=15855.6 MB
engine_ttft_us=150000           = 150 ms server-side
engine_itl_p50_us=15000         = 15 ms server-side
engine_batch_occupancy=0.5
```

## Root Cause

PrefixAware policy at `cold_soft_cap=3` (= max_waiting=256 −
cold_headroom=253) defers cold requests aggressively. Three issues:

1. **Cold queue back-pressure**: 73 cold requests deferred → they
   queue behind warm requests at saturated 8-slot pool. When they
   eventually admit, they hit the slot at high contention. Cold p95
   jumps from 1048ms (queue-bound, FIFO) to 2241ms (prefix-aware,
   deferred until last).

2. **No warm benefit at this op-point**: warm p50 unchanged (66ms in
   both arms). The 8 slots are already saturated and 4 sessions ×
   ~38 reqs serialize naturally — PrefixAware doesn't add warm
   throughput because the limiting factor is slot count, not
   admission queueing.

3. **Starvation ratio worsens**: cold/warm p95 ratio 4.56× → 8.33×.
   Predicted in spec doc as "warm < 3× cold" KILL gate; this is
   8× = severe.

## Mechanism evidence (PASSES, but doesn't help perf at this op-point)

- gate fires: `prefix_aware_admit_deferrals=73` (vs 0 in QB)
- prefix cache reuses: `prefix_hit_rate=57.2%` (above 50% target)
- workload exercises cache: warm/cold split working as designed

So the substrate is correct; the **policy choice is wrong for this
workload at this concurrency/slot ratio**.

## Decision

**KILL `--admission-policy prefix-aware` as production default.**
Substrate stays in main (079639c counter, 7c8fd61 wiring); CLI flag
remains opt-in for further investigation. DO NOT change the
SchedulerAdmissionPolicy::QueueBound default at types.rs:214.

## What this rules out (and what it doesn't)

Rules OUT:
- PrefixAware as a global default for the studied workload shape
  (4 sessions × 38 reqs × 1024-tok shared prefix, 8 slots, c=8)
- The current `cold_headroom = max_waiting/4` heuristic at low
  max_waiting values (256 − 253 = 3 is too aggressive)

Does NOT rule out:
- PrefixAware on **higher-concurrency** workloads (c=32, c=64) where
  slot count is no longer the binding constraint
- PrefixAware on **explicit session_id** workloads where the policy
  can use session-affinity preference (current workload had
  `session_affinity_hit=0`)
- PrefixAware on **mixed-tenant SLA** workloads where warm sessions
  have priority and cold tail latency is not the metric of interest
- A different `cold_headroom` value (e.g. 64 = 25% of 256, the
  documented default; we forced 253 to make cold_soft_cap=3 fire
  hard for evidence)

## Three follow-up paths if PrefixAware is to be revisited

1. **Sweep cold_headroom**: 64, 128, 200, 250 → find the knee where
   cold tail doesn't worsen but warm wins
2. **Add session_id workload variant**: extend
   `gen_36_warm_prefix_mix.py` to emit explicit session_id labels,
   exercise session-affinity path that this run had at zero
3. **Higher-concurrency workload**: c=32 with same warm-mix spec —
   test whether slot saturation is the real binding constraint vs
   admission policy

None of these are P0; #40 Path B.2 (c44788f) already delivered the
single-stream gap closure. PrefixAware is parked as opt-in flag with
KILL evidence at one op-point.

## Rule

**A policy substrate that works at the unit-test level can KILL on
production-shape benches when its gate triggers compete against the
binding constraint.** The PrefixAware substrate worked exactly as
designed (gate fires, deferrals counted, cache reuse measured) but
the chosen op-point made cold-deferral compete with slot scarcity,
worsening every perf metric.

For future admission-policy work: license requires both substrate
mechanism evidence AND op-point sensitivity sweep, not just
single-shape A/B at one cold_headroom setting.

Companion to skill v1.10.0 anti-pattern #28 lesson on this session:
the bench command pair (paste-buffer briefs, server-restart protocol)
worked cleanly **after** the hallucinated CLI-flag was caught and
corrected — emphasizes that the cooperative codex/Claude pipeline
recovers from individual mis-claims when each layer cross-validates.

## Cross-references

- Substrate complete survey: `docs/research/2026-05-10-36-prefix-aware-admission-substrate-complete-bench-pending.md` (5e902da)
- Counter audit: `docs/research/2026-05-10-36-codex-counter-audit-clean.md` (60ffa41)
- Brief gap (SUPERSEDED): `docs/research/2026-05-10-36-brief-gap-bench-server-restart-protocol.md` (0f4d0ae, then ee2c5b0 supersession)
- Hallucination errors: `docs/experience/errors/2026-05-10-claude-hallucinated-grep-output-cli-flag.md` (ee2c5b0)
- Layer 2 GuideLLM invalidation: `docs/experience/errors/2026-05-10-36-warmmix-guidellm-jsonl-invalid.md` (8150bfe codex)
- Direct HTTP harness: `scripts/bench_36_warmmix_direct.py` (3b4c89b + b96a1e7 fix)
- Spec doc: `docs/research/2026-05-10-36-followup-warm-mix-bench-spec.md` (5453ee4)
- Skill v1.10.0 anti-pattern #28: `.claude/skills/kernel-optimization/SKILL.md`
- M_b3 plan (SUPERSEDED): `docs/plans/M_b3-prefix-aware-admission-step1-directive.md`

## Status

#36 PrefixAwareAdmission KILLED as production default for the
studied workload op-point. Substrate retained in main as opt-in
`--admission-policy prefix-aware`. Three follow-up paths documented
for if/when the policy is revisited at a different op-point. n=1
in this entry — re-bench with n=3 σ-tight only if a follow-up path
is pursued.
