# M3.5 — Collapse production scheduler loops into one shared CPU policy

> Follow-up to [`M3-unified-schedule-ir.md`](M3-unified-schedule-ir.md).
> Closes the gap M3 S7 honestly flagged: M3 unified the IR (`+388` net) but
> did NOT collapse the two ~1700-line production scheduler decision loops
> (`scheduler/cuda/core.rs::step` + `backend/metal/scheduler.rs::MetalScheduler`).
> M3.5 is the structural -800-line follow-up.

## P0 finding (2026-05-07)

`wc -l` survey:

| File | Lines | Role |
|---|---|---|
| `infer/src/scheduler/cuda/core.rs` | 1742 | CUDA `Scheduler<M>::step()` decision body |
| `infer/src/backend/metal/scheduler.rs` | 1017 | Metal `MetalScheduler` + `MetalSchedulerConfig` |
| `infer/src/backend/metal/runtime.rs` | 3129 | Metal `run_metal_scheduler_runtime` (executor on top of MetalScheduler) |

Total **5888 lines** in the two loop tracks. The IR work in M3 left both
loops untouched on the *decision* side — they each derive their own
`LogicalServePlan`-equivalent inline. That's the chunk M3.5 collapses.

`runtime.rs` is the MLX hot path executor and is **not** in M3.5 scope —
it consumes the plan and runs MLX kernels. Keep it intact.

## 1. Design

### 1.1 Shared CPU policy

Extract a new module `infer/src/scheduler/policy.rs` that exports:

```rust
pub fn step_decision(
    pending: &[PendingRequest],
    running: &[RunningRequestSnapshot],
    config: &SchedulerPolicyConfig,
    signals: &SchedulerSignals,
) -> LogicalServePlan;
```

This is **pure CPU**; it does not touch GPU state. Both backends call it
with their own snapshot of the queue / slots.

The `SchedulerPolicyConfig` is a backend-neutral subset of
`SchedulerConfig` containing only the *decision*-relevant fields
(`max_slots`, `chunked_prefill_size`, `max_num_batched_tokens`,
`spec_enabled`, `spec_draft_k`, `prefix_cache_enabled`, etc.). Backends
keep their full config separately for execution-side knobs.

`SchedulerSignals` is the existing CUDA-side signals struct
(`scheduler/policy.rs` has prior art) lifted to be backend-neutral —
admission decisions, prefix-cache hit signals, spec-acceptance feedback,
etc.

### 1.2 What disappears

After M3.5 lands:

| Currently inline in CUDA loop | Source after M3.5 |
|---|---|
| chunked-prefill admission heuristic | `policy::admit_for_prefill` |
| decode-prefill mixed-step assembly | `policy::build_logical_plan` |
| spec-decode candidate row selection | `policy::select_spec_rows` |
| sparse-self-spec view building | `policy::build_sparse_views` |

CUDA `core.rs::step` shrinks to:

```rust
fn step(&mut self) {
    let snapshot = self.snapshot_for_policy();
    let plan = policy::step_decision(&snapshot.pending, &snapshot.running, &self.policy_config, &self.signals);
    self.lowering.execute(&plan)?;  // unchanged from M3 S4
    self.process_outputs();
}
```

Metal `scheduler.rs` similarly shrinks. **Combined target: drop ~800 lines
between core.rs + metal/scheduler.rs**, matching the original M3
acceptance target.

### 1.3 What does NOT disappear

- `runtime.rs::run_metal_scheduler_runtime` — Metal's MLX executor stays.
  M3.5 only touches its **caller** in `MetalScheduler`, not the body.
- `scheduler/cuda/{prefill,decode,spec_path}.rs` — execution helpers,
  not decision logic. Unchanged.
- `RequestState` / Slot lifecycle — backend-private, unchanged.

## 2. Migration steps

| Step | File | LOC delta (target) | Days | Owner |
|---|---|---|---|---|
| T1. Add `infer/src/scheduler/policy.rs` with `step_decision` + helpers; pull CPU-only logic out of `scheduler/cuda/core.rs` into the new module. | +500 / -500 | 1 | Codex |
| T2. CUDA `Scheduler<M>::step` calls `policy::step_decision`. Run `e2e + greedy_consistency + spec_decode_correctness` to confirm zero regression. | -300 in core.rs | 1 | Codex |
| T3. Lift `SchedulerSignals` and `SchedulerPolicyConfig` to backend-neutral. | ~50 | 0.5 | Codex |
| T4. Metal `MetalScheduler::next_step` calls `policy::step_decision`. Run Metal Rust typecheck on Linux + Apple Silicon smoke. | -400 in metal/scheduler.rs | 1.5 | Codex |
| T5. wins entry recording the actual line delta. Bench: e2e Phase 1-4, greedy_consistency, scheduler:: CPU tests. Spec-decode test trio. | 0 (docs) | 0.5 | Claude |

**Total**: ~4.5 days, target net delta around -800 lines (the original
acceptance target for M3 that M3 itself didn't hit).

## 3. Acceptance

- `wc -l infer/src/scheduler/cuda/core.rs infer/src/backend/metal/scheduler.rs` reports a combined delta ≤ -700.
- All M3 verification gates still pass:
  - `cargo test --release -p infer --features cuda --test e2e`
  - `INFER_DETERMINISTIC=1 cargo test --release -p infer --features cuda --test greedy_consistency`
  - `cargo test --release -p infer --features cuda --test spec_decode_correctness`
  - `cargo test --release -p infer --no-default-features --features no-cuda scheduler::`
  - `cargo clippy -p infer --features cuda -- -D warnings`
  - `cargo check -p infer --no-default-features --features metal,no-cuda`
- The new `infer/src/scheduler/policy.rs` builds clean under all three
  feature combos.
- Performance neutral: bench guidellm-quick on local 4070 Ti SUPER
  shows ≤ 1% throughput delta on prefill / decode / mixed workloads.

## 4. Risks + retreat

- **R1 — Spec-decode admission has scheduler-side state mutations** (e.g.
  `spec_acceptance_tracker` write-back on each step) that don't fit a
  pure CPU policy function. Mitigation: pass the relevant trackers as
  `&mut` references; or have `policy::step_decision` return both the
  plan AND a `PolicySideEffects` struct that the caller applies to
  request state. Pure-fn interpretation is preferred.
- **R2 — Metal's packed-varlen logical offsets need backend-specific
  interpretation that CUDA's paged-KV doesn't share**. Already addressed
  by M3's `logical_kv_offset` design (see M3 plan §1.2). Should not
  resurface in M3.5.
- **R3 — `wc -l` delta < target` after T2/T4** because some "decision"
  logic is genuinely intertwined with execution helpers and resists
  extraction. Retreat: T5 wins entry honestly records what landed; if
  delta is e.g. -400 instead of -800, declare M3.5 partially complete
  and open M3.6 for the residual 400 lines.

## 5. Out of scope

- Changes to `runtime.rs` (Metal MLX executor) — that's M5 territory
  (cross-backend `ModelForward`).
- Changes to `infer/src/scheduler/cuda/{prefill,decode,spec_path}.rs`
  internals — execution helpers, kept as-is.
- Schedule policy ALGORITHM changes — M3.5 is purely a structural
  collapse, not a policy redesign. Algorithm changes go through
  `scheduler/AGENTS.md` review.

## 6. References

- Parent: [`M3-unified-schedule-ir.md`](M3-unified-schedule-ir.md)
- M3 wins entry that flagged the gap:
  [`2026-05-07-m3-unified-scheduler-ir.md`](../experience/wins/2026-05-07-m3-unified-scheduler-ir.md)
  — see "Line Delta" section.
- Existing CUDA inline body: `infer/src/scheduler/cuda/core.rs`
- Existing Metal inline body: `infer/src/backend/metal/scheduler.rs`
- `LogicalServePlan` IR (M3 output): `infer/src/scheduler/plan.rs`
