{
  "_about": "INDUSTRY-FIRST dimension catalog: the things the LLM-serving / agent-infra field competes on, researched from vendor docs / papers / leaderboards (NOT derived from what fak measured). Each carries the current SOTA bar + a dated source. The scorecard positions fak against every in_scope dimension.",
  "dimensions": [
    {
      "id": "continuous-batching-aggregate-throughput",
      "category": "throughput",
      "dimension": "Continuous / in-flight (iteration-level) batching: max aggregate token throughput at high concurrency",
      "why_it_matters": "Iteration-level batching admits and retires requests mid-flight so a long sequence never blocks short ones, and a fixed GPU serves far more simultaneous sequences. It is the single largest lever on tokens/s per GPU and on serving cost, so it is the first thing any operator benchmarks.",
      "sota_systems": [
        "SGLang (RadixAttention)",
        "vLLM v1 (continuous batching + automatic prefix caching)",
        "TensorRT-LLM (in-flight batching)"
      ],
      "sota_bar": "Continuous (in-flight) batching is now table stakes across vLLM/SGLang/TensorRT-LLM; the live competitive number is no longer 'vLLM 24x vs Orca/TGI' but cross-engine: SGLang's RadixAttention holds roughly a 29% aggregate-throughput edge over vLLM on H100 (~16,200 vs ~12,500 tok/s, Llama 3.1 8B), widening to up to ~6.4x on prefix-heavy RAG/multi-turn and shrinking to near-zero on unique-prompt batches; continuous batching itself still delivers ~3-4x over naive/static batching.",
      "best_value": 6.4,
      "metric_unit": "x throughput vs comparison engine (SGLang RadixAttention over vLLM, prefix-heavy workload; ~29% in the general case)",
      "source_url": "https://particula.tech/blog/sglang-vs-vllm-inference-engine-comparison",
      "source_date": "2026",
      "notes": "The old bar (vLLM 24x vs Orca/TGI, 2023-09) is stale by framing, not just date: continuous batching is now universal, so the field reports cross-engine deltas (SGLang ~29% over vLLM general / up to ~6.4x prefix-heavy) plus the ~3-4x continuous-vs-naive figure (Spheron/MLPerf-derived blog). Numbers come from a benchmarking blog (particula.tech / spheron), not a primary vendor doc, because no single vendor publishes a current 'Nx vs everyone' headline; treat the 6.4x/29% as workload-dependent secondary-source claims. vLLM's own docs still only cite the original external '23x throughput' continuous-batching post (anyscale/Cade Daniel), confirming the 24x-style framing is legacy. Confidence medium: the multiplier is real and current but secondary-sourced and highly workload-sensitive. [second-source verified]",
      "in_scope": true
    },
    {
      "id": "goodput-under-slo",
      "category": "cost-efficiency",
      "dimension": "Goodput under an SLA (requests/s meeting TTFT and TPOT SLOs simultaneously)",
      "why_it_matters": "Raw throughput overcounts: a request that violates its latency SLO is wasted work. Goodput, the max request rate at which a target fraction of requests meet BOTH TTFT and TPOT SLOs, is the metric a buyer running an interactive product actually pays against. It exposes systems that win on aggregate tokens/s but collapse under tail-latency constraints.",
      "sota_systems": [
        "DistServe",
        "NVIDIA Dynamo (incremental: 38% over best aggregated config, 2026)",
        "PD-Multiplexing/GreenContext (1.62x over PD-disagg, 2025-09)"
      ],
      "sota_bar": "DistServe's headline 7.4x more goodput (or 12.6x tighter SLO) over prior systems remains the canonical bar; newer 2025-26 work (TaiChi, PD-Multiplexing, DOPD, NVIDIA Dynamo) reports only incremental gains relative to disaggregation itself rather than a new baseline-relative record.",
      "best_value": 7.4,
      "metric_unit": "x goodput vs prior SOTA serving systems (requests within SLO)",
      "source_url": "https://arxiv.org/abs/2401.09670",
      "source_date": "2024-01",
      "notes": "No newer source sets a higher headline goodput-under-SLO multiplier; old bar stands. The 2025-26 systems I found (PD-Multiplexing 1.62x over PD-disagg per LMSYS 2025-09-28; DOPD up to 1.5x over vLLM/DistServe per arXiv 2511.20982 v3 2026-03; TaiChi 29-77% over PD-disagg per arXiv 2508.01989 2025-08) all report gains RELATIVE TO disaggregation, which compound on top of DistServe's 7.4x rather than replace it. NVIDIA Dynamo's 2026 figure is 38% over the best aggregated config, not a DistServe-style baseline-relative record.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "chunked-prefill-pd-interference",
      "category": "scheduling",
      "dimension": "Chunked / piggyback prefill and prefill-decode interference (stall-free batching)",
      "why_it_matters": "When a long prefill shares an iteration with ongoing decodes it stalls every decode, spiking inter-token latency (TPOT/TBT). Chunked prefill splits a prefill into token-bounded chunks and piggybacks them onto decode batches so decodes never stall, decoupling the throughput/latency tradeoff. It is the core mechanism that lets a co-located engine hit a tight TPOT SLO at high load.",
      "sota_systems": [
        "Nexus (proactive intra-GPU PD disaggregation, arXiv 2507.06608)",
        "DuetServe (adaptive GPU multiplexing, arXiv 2511.04791)",
        "Sarathi-Serve (chunked prefill, original bar)"
      ],
      "sota_bar": "Proactive intra-GPU prefill/decode disaggregation (Nexus) now beats chunked-prefill, delivering up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT vs vLLM (and up to 2x vs SGLang), superseding Sarathi-Serve's 5.6x chunked-prefill capacity claim.",
      "best_value": 2.2,
      "metric_unit": "x throughput vs vLLM (also 20x lower TTFT, 2.5x lower TBT)",
      "source_url": "https://arxiv.org/abs/2507.06608",
      "source_date": "2025-08",
      "notes": "SOTA materially moved away from pure chunked-prefill. Chunked prefill (Sarathi-Serve) mitigates but cannot eliminate PD interference because it co-batches compute-bound prefill with memory-bound decode; 2025-26 systems (Nexus, DuetServe, RAPID-Serve, DOPD) instead partition GPU SMs/resources between phases. Nexus (submitted 2025-07-09, v5 2025-08-07): up to 2.2x throughput / 20x TTFT / 2.5x TBT vs vLLM, up to 2x vs SGLang, matches/exceeds disaggregated vLLM. DuetServe (arXiv 2511.04791, 2025-11) reports 1.1x throughput over SGLang-Default with disaggregation-level isolation inside one GPU. The old Sarathi-Serve 5.6x is a capacity-at-SLO number, not directly comparable to these throughput-at-latency multipliers, but it is no longer the freshest SOTA framing for the PD-interference problem.",
      "in_scope": true
    },
    {
      "id": "prefill-decode-disaggregation",
      "category": "parallelism",
      "dimension": "Prefill-decode disaggregation: independent scaling of compute-bound prefill and memory-bound decode",
      "why_it_matters": "Prefill is compute-bound and decode is memory-bandwidth-bound; co-locating them forces one hardware/parallelism config to serve both and couples their scaling. Disaggregating onto separate instances (over a KV-cache transport) lets each phase scale and tune independently and removes interference entirely. It is now the default architecture for large-scale, SLO-tight deployments.",
      "sota_systems": [
        "SGLang PD (MI300X)",
        "NVIDIA Dynamo 1.0 (38% over best aggregated config; 7x w/ wide-EP on GB200)",
        "vLLM disaggregated prefilling (NIXL)"
      ],
      "sota_bar": "SGLang PD disaggregation achieves up to 6.9x more goodput (Chatbot 3200in/800out, TTFT<=1000ms, TPOT<=25ms) on MI300X; this remains the headline open-framework PD-disaggregation figure, now corroborated by NVIDIA Dynamo's 2026 production results (38% goodput over best aggregated config; 7x with wide expert parallel on GB200 NVL72).",
      "best_value": 6.9,
      "metric_unit": "x goodput vs aggregated baseline (PD-disaggregation)",
      "source_url": "https://rocm.blogs.amd.com/software-tools-optimization/disaggregation/README.html",
      "source_date": "2025-08",
      "notes": "Old SGLang 6.9x bar confirmed verbatim by the AMD ROCm blog (2025-08-28): 'PD disaggregation with SGLang can serve up to 6.9X more goodput requests within the SLO constraint.' Date refreshed and corroborated; the SOTA itself did not move past 6.9x for the open-framework PD-disagg headline. NVIDIA Dynamo's 2026 numbers (38% over best aggregated config on 64xB200 Qwen3-32B at 550 tok/s/GPU; 7x with wide expert parallel on GB200 NVL72) are reported per developer.nvidia.com (Removing the Guesswork, 2026-03-09) but are not a like-for-like replacement of the 6.9x figure.",
      "in_scope": true
    },
    {
      "id": "cache-aware-scheduling-routing",
      "category": "scheduling",
      "dimension": "Request scheduling and cache-aware routing (FCFS vs priority vs prefix/KV-cache-aware)",
      "why_it_matters": "The scheduling discipline decides which requests batch together and where they route. A naive FCFS engine ignores that requests sharing a prefix (system prompts, RAG context, multi-turn chat, agents) can reuse KV cache; a cache-aware scheduler co-locates and orders them to maximize reuse. This directly multiplies effective throughput on shared-context workloads and trades off against load balance.",
      "sota_systems": [
        "SGLang router / cache-aware load balancer (v0.4+, current v0.5.x)"
      ],
      "sota_bar": "SGLang's cache-aware load balancer still holds the cited bar: up to 1.9x throughput and 3.8x higher prefix-cache hit rate in multi-node deployments, via an approximate radix tree mirroring each worker's cache.",
      "best_value": 1.9,
      "metric_unit": "x throughput (with 3.8x cache hit-rate)",
      "source_url": "https://www.lmsys.org/blog/2024-12-04-sglang-v0-4/",
      "source_date": "2024-12",
      "notes": "No newer SGLang router multiplier found; the v0.4 (2024-12) 1.9x throughput / 3.8x hit-rate is still the number quoted in 2026 SGLang reviews (e.g. ChatForest 2026 review, dstack 2026 router roadmap) even though SGLang is now at v0.5.11 (2026-05). The router itself has matured (PD-disaggregation, gRPC, multi-region roadmap) but LMSYS has not re-published a higher headline cache-aware multiplier. Old bar stands; only the freshness check date refreshes.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "kv-cache-memory-efficiency-concurrency",
      "category": "kv-cache",
      "dimension": "KV-cache memory efficiency and max concurrent sequences (paging / fragmentation)",
      "why_it_matters": "GPU memory left for KV cache, after weights, sets the hard ceiling on how many sequences can run concurrently, which caps batch size and therefore throughput. Pre-reserving max-context KV cache wastes 60-80% of memory to fragmentation; paged allocation cuts waste to under 4%, letting the same GPU hold far more concurrent sequences.",
      "sota_systems": [
        "vLLM PagedAttention",
        "SGLang RadixAttention (radix-tree paged KV)"
      ],
      "sota_bar": "PagedAttention keeps KV-cache memory waste under 4% (near-zero internal/external fragmentation), enabling 2-4x throughput at the same latency; this remains the canonical paged-KV efficiency reference as of mid-2026.",
      "best_value": 4,
      "metric_unit": "percent of KV-cache memory wasted (upper bound; lower is better)",
      "source_url": "https://arxiv.org/abs/2309.06180",
      "source_date": "2023-09",
      "notes": "No newer source supersedes the canonical <4% waste / near-zero-fragmentation figure; the original PagedAttention SOSP'23 paper is still the cited bar and the abstract phrases it as 'near-zero waste in KV cache memory.' Old bar stands. What HAS moved in 2026 is that paging is now table stakes: the competitive frontier shifted to cross-request and non-prefix KV REUSE (RadixAttention, LMCache CacheBlend, Mooncake KV pools) rather than squeezing intra-request fragmentation further. Secondary 2025 sources still quote the same 'under 4% waste, 60-80% in legacy systems' framing.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "absolute-tokens-per-second-mlperf",
      "category": "throughput",
      "dimension": "Absolute aggregate tokens/s at scale on standardized hardware (MLPerf Inference)",
      "why_it_matters": "Cross-vendor, audited absolute throughput on a fixed model and SLO is what a buyer uses to compare full systems and compute cost-per-token. MLPerf Inference is the only neutral, reproducible apples-to-apples leaderboard with both offline (max throughput) and server (latency-constrained) scenarios, so it anchors the absolute SOTA bar that engine micro-benchmarks cannot.",
      "sota_systems": [
        "NVIDIA GB300 NVL72 x4 (288 Blackwell Ultra GPUs, TensorRT-LLM/Dynamo)",
        "NVIDIA GB300 NVL72 single rack",
        "CoreWeave / Nebius GB300 NVL72"
      ],
      "sota_bar": "MLPerf Inference v6.0 (results 2026-03-30 / published 2026-04-01) supersedes v5.0 and v5.1, and the headline model is now DeepSeek-R1 (671B MoE reasoning), not Llama 3.1 405B: NVIDIA's largest-ever submission of 288 Blackwell Ultra GPUs (4x GB300 NVL72 over Quantum-X800 InfiniBand) hit ~2.494 million tokens/second aggregate in the offline scenario, at 9,821 tok/s/GPU offline (8,064 tok/s/GPU server) \u00e2\u20ac\u201d up to 2.7x higher than the GB300 debut six months prior via TensorRT-LLM/Dynamo software.",
      "best_value": 2494000,
      "metric_unit": "tokens/second aggregate, DeepSeek-R1 671B, offline scenario, 288-GPU (4x GB300 NVL72) submission, MLPerf Inference v6.0",
      "source_url": "https://www.storagereview.com/news/nvidia-sets-mlperf-inference-v6-0-records-with-blackwell-ultra-platform",
      "source_date": "2026-04-01",
      "notes": "Materially changed: a full MLPerf round (v6.0) landed since the old v5.0 bar, and the benchmark headline shifted off Llama 3.1 405B onto DeepSeek-R1 671B MoE. The old 'GB200 NVL72 13,886 tok/s Llama 3.1 405B offline' is superseded. Per-GPU on DeepSeek-R1 v6.0: 9,821 offline / 8,064 server; full 288-GPU offline aggregate 2.494M tok/s. NVIDIA's own blog (developer.nvidia.com 'Blackwell Ultra Sets New Inference Records', and the lowest-token-cost co-design post) is the primary source; storagereview gives the clean numeric breakdown cited here. Note a small date discrepancy across secondaries (results 2026-03-30, blogs/articles 2026-04-01). If the scorecard wants a Llama-405B continuity number, v5.1 (2025-09-09) put GB200 NVL72 at ~16,128 tok/s offline aggregate / 224 tok/s/GPU \u00e2\u20ac\u201d but v6.0 is the current round.",
      "in_scope": true
    },
    {
      "id": "interactive-latency-slo-tail",
      "category": "latency",
      "dimension": "Interactive latency SLO attainment: TTFT and TPOT/ITL tail under load",
      "why_it_matters": "For chatbots, coding assistants, and agents the contract is felt latency: time-to-first-token and steady per-token cadence at the tail (p99), not the mean. A serving system is differentiated by how high it can push concurrency before TTFT or TPOT breach the SLO. This is the latency face of the same tradeoff goodput measures on the throughput side.",
      "sota_systems": [
        "MLPerf Inference v6.0 (NVIDIA Blackwell Ultra GB300, AMD MI355X)",
        "MLPerf Inference v5.1"
      ],
      "sota_bar": "MLPerf Inference v6.0 (Apr 2026) tightened the interactive SLO floor: the most stringent latency-aware bounds are now TPOT <= 15 ms with TTFT <= 1.5-2.0 s P99 (GPT-OSS 120B: TTFT <= 2.0 s / TPOT <= 15 ms; DeepSeek-R1: TTFT <= 1.5 s / TPOT <= 15 ms), superseding the v5.1 30 ms-TPOT / 0.5 s-TTFT interactive bar.",
      "best_value": 15,
      "metric_unit": "ms TPOT (interactive scenario, P99; paired TTFT 1.5-2.0 s)",
      "source_url": "https://mlcommons.org/2026/03/mlperf-inference-gpt-oss/",
      "source_date": "2026-03-24",
      "notes": "SOTA moved: MLPerf v6.0 (released 2026-04-01, GPT-OSS/DeepSeek-R1 latency-aware update announced 2026-03-24) replaced the old v5.1 interactive bound of 30 ms TPOT / 0.5 s TTFT. The new interactive floor is TPOT <= 15 ms (GPT-OSS 120B and DeepSeek-R1), with paired TTFT <= 2.0 s and <= 1.5 s respectively. The classic Llama-2-70B interactive bound (450 ms TTFT / 40 ms TPOT, ~25 tok/s/user) remains defined but is no longer the tightest. This is a per-token interactivity SLO, not the single tail-percentile number; see tail-latency-percentiles for the percentile definition.",
      "in_scope": true
    },
    {
      "id": "prefix-caching-kv-reuse",
      "category": "kv-cache",
      "dimension": "Prefix caching / automatic KV reuse across requests (shared-context workloads)",
      "why_it_matters": "Real workloads (long system prompts, RAG, few-shot, multi-turn chat, agent fleets) repeat large prefixes. Automatic prefix caching reuses the already-computed KV for shared prefixes, skipping that prefill entirely. On these workloads it is a large, distinct throughput and TTFT multiplier that is invisible to single-request benchmarks but dominates production economics.",
      "sota_systems": [
        "SGLang RadixAttention",
        "vLLM Automatic Prefix Caching"
      ],
      "sota_bar": "SGLang RadixAttention delivers ~29% higher end-to-end throughput than vLLM on H100 (16,200 vs 12,500 tok/s, Llama 3.1 8B) on shared-prefix workloads, with up to 6.4x on prefix-heavy RAG/multi-turn traffic; the 29% SGLang-vs-vLLM edge still holds in 2026 benchmarks.",
      "best_value": 29,
      "metric_unit": "percent throughput advantage of SGLang RadixAttention over vLLM (H100, shared-prefix)",
      "source_url": "https://particula.tech/blog/sglang-vs-vllm-inference-engine-comparison",
      "source_date": "2026",
      "notes": "The 29% SGLang-vs-vLLM throughput edge (16,200 vs 12,500 tok/s on H100) is re-confirmed by 2026 head-to-head benchmarks, so the bar holds rather than moves. Note a naming hazard: '29%' appears in TWO distinct contexts in 2026 sources - (a) SGLang's overall throughput edge over vLLM on H100 (this bar), and (b) the intra-SGLang throughput lift from unique-prompts to 80%-shared-prefix (1,920->2,480 tok/s, also ~29%, Spheron 2026). Both are real and both round to 29%; this row uses interpretation (a) matching the old bar. Source is a vendor-neutral comparison blog, not a primary leaderboard, hence medium confidence; the underlying tok/s numbers trace to SGLang's published H100 benchmarks.",
      "in_scope": true
    },
    {
      "id": "speculative-decoding-throughput-effect",
      "category": "speculative-decoding",
      "dimension": "Speculative decoding throughput effect across batch sizes (does it survive at high concurrency?)",
      "why_it_matters": "Speculative decoding accelerates single-stream decode but consumes extra compute per step, so at the high batch sizes a busy server runs it can REDUCE throughput. The differentiating question for a serving operator is not the single-request speedup but whether the method still nets a throughput gain at production batch sizes; only newer drafters do.",
      "sota_systems": [
        "EAGLE-3 (SafeAILab)",
        "SGLang speculative backend",
        "vLLM/TensorRT-LLM spec backends"
      ],
      "sota_bar": "EAGLE-3 remains the named SOTA: ~1.38x throughput at batch 64 in SGLang on a single H100, peaking at ~1.81x at batch 2 and up to ~6.5x single-stream, ~1.4x over EAGLE-2.",
      "best_value": 1.38,
      "metric_unit": "x throughput at batch 64 (SGLang, H100)",
      "source_url": "https://arxiv.org/abs/2503.01840",
      "source_date": "2025-03",
      "notes": "Old bar (EAGLE-3 1.38x@batch64, 2025) still stands as the named throughput-at-batch figure; multiple 2025/2026 sources reconfirm 1.38x@batch64 and 1.81x@batch2 on H100/SGLang. EAGLE-3 is still the leading trained-head method; no successor has displaced it on the throughput-vs-batch axis as of mid-2026. Date can be tightened to the arXiv 2503.01840 (NeurIPS 2025) primary source.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "fairness-isolation-multitenant",
      "category": "operability",
      "dimension": "Fairness, priority, and tenant isolation in batch formation / scheduling",
      "why_it_matters": "A multi-tenant or mixed-SLA deployment must prevent one heavy or long-prefill tenant from starving others, and must honor priority classes (real-time vs best-effort). FCFS plus cache-affinity heuristics can create head-of-line blocking and load hot-spots; fairness-aware batch formation and priority preemption are the controls an operator needs to keep per-tenant SLOs while sharing a fleet.",
      "sota_systems": [
        "FairBatching (fairness-aware batch formation)",
        "SGLang (FCFS + page-eviction preemption)",
        "hybrid real-time/best-effort schedulers (e.g. arXiv 2504.09590)"
      ],
      "sota_bar": "No single audited cross-system number; SOTA is qualitative: FairBatching reformulates batch formation to bound per-request unfairness, and serving stacks add priority classes plus preemption (KV-page eviction with recompute-on-resume in SGLang) to isolate tenants. Buyers evaluate head-of-line-blocking behavior and priority-preemption support directly.",
      "best_value": null,
      "metric_unit": "qualitative (fairness/isolation guarantees; priority + preemption support)",
      "source_url": "https://arxiv.org/pdf/2510.14392",
      "source_date": "2025-10",
      "notes": "An operator-side dimension a throughput-only scorecard omits. Cache-affinity routing actively worsens load balance (DualMap, arXiv 2602.06502), so fairness is a real, measured tension, not a nicety.",
      "in_scope": true
    },
    {
      "id": "time-to-first-token",
      "category": "latency",
      "dimension": "Time-to-first-token (TTFT) under realistic prefill",
      "why_it_matters": "TTFT is the perceived responsiveness of an interactive system - the wait before any output appears. It is dominated by queueing delay plus prefill compute, so it is the headline latency metric for chat, voice, and copilot UX and the gating constraint in every serving SLO. Buyers compare TTFT at a fixed input length and concurrency because a single-user TTFT (just network + one forward pass) tells you almost nothing about production behavior.",
      "sota_systems": [
        "MLPerf Inference v6.0 (DeepSeek-R1 interactive)",
        "MLPerf Inference v5.1 (Llama-3.1-405B interactive)"
      ],
      "sota_bar": "MLPerf Inference v6.0 defines per-model P99 TTFT SLOs: GPT-OSS 120B interactive TTFT <= 2.0 s (server <= 3.0 s), DeepSeek-R1 interactive TTFT <= 1.5 s (server <= 2.0 s), Llama-3.1-405B interactive TTFT <= 4.5 s (server <= 6.0 s). The 4.5 s P99 405B bar from v5.1 still stands for the largest model.",
      "best_value": 1.5,
      "metric_unit": "s P99 TTFT (tightest interactive bound, DeepSeek-R1; 405B remains 4.5 s)",
      "source_url": "https://mlcommons.org/2026/03/mlperf-inference-gpt-oss/",
      "source_date": "2026-03-24",
      "notes": "The old bar (v5.1 4.5 s P99 TTFT for 405B) is still the defined bound for the largest model, but v6.0 added far tighter per-model TTFT SLOs for reasoning models: DeepSeek-R1 interactive TTFT <= 1.5 s P99 and GPT-OSS 120B interactive <= 2.0 s P99. best_value reports the tightest current production TTFT SLO (1.5 s). Note vLLM benchmark CLI reports P99 TTFT as low as ~144 ms at 50 concurrency on small models, but that is a vendor-tunable measurement, not a standardized SLO floor.",
      "in_scope": true
    },
    {
      "id": "inter-token-latency-tpot",
      "category": "latency",
      "dimension": "Inter-token latency / time-per-output-token (ITL/TPOT)",
      "why_it_matters": "After the first token, ITL/TPOT governs how fast text streams - the felt 'typing speed.' It must clear a per-user threshold (e.g. faster than human reading) and stay smooth; stalls produce visible jank even at good averages. TPOT excludes the first token by construction (denominator subtracts 1) so it isolates the decode phase from prefill.",
      "sota_systems": [
        "MLPerf Inference v6.0 (GPT-OSS 120B, DeepSeek-R1 interactive)",
        "MLPerf Inference v5.1"
      ],
      "sota_bar": "MLPerf Inference v6.0 reasoning-model interactive scenarios now require TPOT <= 15 ms P99 (GPT-OSS 120B and DeepSeek-R1), down from the prior 80 ms TPOT bar; the legacy Llama-2-70B interactive bound is 40 ms TPOT (~25 tok/s/user) and server is 80-200 ms.",
      "best_value": 15,
      "metric_unit": "ms P99 TPOT (interactive scenario)",
      "source_url": "https://mlcommons.org/2026/03/mlperf-inference-gpt-oss/",
      "source_date": "2026-03-24",
      "notes": "SOTA moved sharply: the old 80 ms TPOT bar (MLPerf 2025-04) is now the SERVER-scenario bound; v6.0's interactive scenarios for GPT-OSS 120B and DeepSeek-R1 require TPOT <= 15 ms P99 (~66 tok/s/user). The Llama-2-70B interactive TPOT bound is 40 ms. The headline best_value is the 15 ms interactive floor from v6.0. vLLM v1 (mixed prefill/decode) is the reference engine NVIDIA/AMD cite for meeting these interactive TPOT constraints.",
      "in_scope": true
    },
    {
      "id": "end-to-end-request-latency",
      "category": "latency",
      "dimension": "End-to-end request latency (E2EL)",
      "why_it_matters": "E2EL is the full submit-to-final-token time and is what a downstream agent or pipeline actually budgets against. It bundles queueing, prefill, all decode steps, batching effects, and network, so it is the only metric that captures the compound cost of long generations and is the right unit for multi-step agent SLA math.",
      "sota_systems": [
        "NVIDIA GenAI-Perf",
        "vLLM bench serve (--metric-percentiles, default p99)",
        "LLMPerf"
      ],
      "sota_bar": "End-to-end request latency (TTFT + total decode) is reported per request by the standard tools (NVIDIA GenAI-Perf, vLLM bench serve, LLMPerf) with P99 as the default percentile; there is no single SOTA number - it is a tunable composite governed by the per-model MLPerf v6.0 TTFT+TPOT SLOs (e.g. a 1000-token DeepSeek-R1 interactive response budgets ~1.5 s TTFT + ~15 s decode).",
      "best_value": null,
      "metric_unit": "qualitative (composite TTFT + N*TPOT, P99); no single SOTA value",
      "source_url": "https://docs.vllm.ai/en/latest/benchmarking/cli/",
      "source_date": "2026-01",
      "notes": "The old bar was a generic GenAI-Perf reference (2025-01). The measurement methodology is unchanged (e2e = TTFT + decode, reported per-request at P99), but the framing should be refreshed: end-to-end latency is now bounded by MLPerf v6.0's tightened per-model TTFT/TPOT SLOs rather than a standalone number. vLLM's bench CLI reports e2e latency per request and defaults to p99 via --metric-percentiles. This dimension is inherently qualitative/composite, so best_value is null.",
      "in_scope": true
    },
    {
      "id": "tail-latency-percentiles",
      "category": "latency",
      "dimension": "Tail latency distribution (P50/P95/P99/P99.9)",
      "why_it_matters": "Averages hide the requests that ruin trust. A degrading P99 while P50 holds steady is the early-warning signature of queue buildup or KV-memory pressure. At scale a 1-in-100 bad request is a continuous problem (100/hr at 10k req/hr), so operators size capacity and write SLOs against tail percentiles, not means.",
      "sota_systems": [
        "MLPerf Inference v6.0 (99th-percentile TTFT/TPOT constraints)",
        "vLLM bench (p99 default)",
        "NVIDIA GenAI-Perf / NIM"
      ],
      "sota_bar": "P99 (99th percentile) is the canonical tail SLO across MLPerf Inference v6.0 (all TTFT/TPOT constraints are '99th percentile <= X'), NVIDIA GenAI-Perf/NIM, and vLLM bench (default --metric-percentiles p99). P99 remains the industry tail metric; some leaderboards also surface P95/P50.",
      "best_value": 99,
      "metric_unit": "percentile (P99 is the canonical tail SLO)",
      "source_url": "https://mlcommons.org/2026/03/mlperf-inference-gpt-oss/",
      "source_date": "2026-03-24",
      "notes": "No material change since the old bar (P99, 2026-01). P99 is still the standard tail percentile: MLPerf v6.0 states every latency constraint as '99th percentile' (e.g. GPT-OSS 120B interactive '99th percentile TTFT <= 2.0s; TPOT <= 15ms'), and vLLM bench defaults to p99. Only the date is refreshed to the v6.0 source confirming P99 is still canonical; the metric definition is unchanged.",
      "in_scope": true
    },
    {
      "id": "slo-constrained-goodput",
      "category": "cost-efficiency",
      "dimension": "SLO-constrained goodput (per-GPU)",
      "why_it_matters": "Raw throughput (tokens/s) is gameable by sacrificing latency. Goodput - the max request rate served while still meeting the TTFT and TPOT SLO for >=X% (e.g. 90%) of requests, normalized per GPU - is the metric that actually maps to cost-per-served-user. It is the central buyer's KPI because it unifies latency and throughput into one defensible number.",
      "sota_systems": [
        "DistServe",
        "DOPD (>99% SLO attainment, up to 1.5x over vLLM/DistServe, 2026-03)",
        "TaiChi (90% SLO attainment, 29-77% over PD-disagg, 2025-08)"
      ],
      "sota_bar": "DistServe's 4.48x SLO-constrained goodput (the conservative per-workload figure under strict TTFT+TPOT SLOs) still stands as the canonical bar; no 2025-26 system reports a higher SLO-constrained goodput multiplier against the same baseline.",
      "best_value": 4.48,
      "metric_unit": "x SLO-constrained goodput vs prior SOTA",
      "source_url": "https://arxiv.org/abs/2401.09670",
      "source_date": "2024-01",
      "notes": "No newer source found that beats the SLO-constrained 4.48x figure on a comparable baseline; old bar stands. 2025-26 work has shifted from a single goodput multiplier toward SLO-attainment percentages: DOPD reports >99% SLO attainment with up to 1.5x goodput over vLLM/DistServe (arXiv 2511.20982 v3, 2026-03); TaiChi reports 90% SLO attainment with 29-77% goodput over PD-disaggregation (arXiv 2508.01989, 2025-08). These are incremental gains layered on disaggregation, not a replacement of the 4.48x bar.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "latency-throughput-pareto",
      "category": "throughput",
      "dimension": "Latency-vs-throughput Pareto frontier",
      "why_it_matters": "There is no single operating point - every system trades batch size against per-request latency. The buyer's real question is the shape of the whole frontier: how much throughput can I buy before TTFT/TPOT breaches my SLO, and how gracefully does the curve bend. A system that advances the frontier serves more users per GPU at the SAME latency guarantee.",
      "sota_systems": [
        "DistServe (PD disaggregation)",
        "Mooncake / Splitwise (PD disaggregation)",
        "Sarathi-Serve (chunked prefill, now a baseline point)",
        "DuetServe / MuxWise / DynaServe (intra-GPU PD multiplexing, 2025-2026)"
      ],
      "sota_bar": "The latency-throughput Pareto frontier is now framed around goodput (max request rate meeting both TTFT and TBT/TPOT SLOs), and the dominant lever has moved from Sarathi-Serve chunked prefill (2024) to prefill-decode (PD) disaggregation: DistServe reports up to 7.4x more goodput or 12.6x stricter SLOs than prior systems while keeping >90% of requests in budget, and 2025-2026 work (DynaServe, MuxWise, DuetServe, RAPID-Serve) pushes intra-GPU PD multiplexing to recover disaggregation's isolation without dedicating GPU pools.",
      "best_value": 7.4,
      "metric_unit": "x goodput vs prior SOTA serving systems (DistServe PD-disaggregation; alt: 12.6x stricter SLO at >90% in-budget)",
      "source_url": "https://haoailab.com/blogs/distserve/",
      "source_date": "2024-2026",
      "notes": "Changed in framing and SOTA: Sarathi-Serve chunked prefill (2024-03) is now a baseline on the frontier rather than the frontier itself. The goodput-optimized PD-disaggregation line (DistServe: up to 7.4x goodput / 12.6x stricter SLO, arXiv 2401.09670 + Hao AI Lab blog) is the current reference point, with active 2025-2026 refinement toward intra-GPU multiplexing (DuetServe arXiv 2511.04791, MuxWise/DynaServe arXiv 2504.x) to keep isolation benefits at single-GPU cost. vLLM ships chunked prefill AND disaggregated prefill/decode/encode (per docs), so it now spans both regimes. Confidence medium because the 'frontier' is a moving research surface with several competing systems rather than one canonical leaderboard number; the 7.4x is DistServe's headline against its own baselines. [second-source verified]",
      "in_scope": true
    },
    {
      "id": "ttft-degradation-under-load",
      "category": "operability",
      "dimension": "Latency degradation under concurrency / load (saturation behavior)",
      "why_it_matters": "TTFT and tail latency are flat at low load and explode past a knee point as the request queue grows. The location of that knee, and how steeply latency climbs past it, determine real capacity. Testing at concurrency=1 is meaningless; operators must characterize TTFT/P99 across a concurrency sweep up to and beyond their P95 concurrent-request count.",
      "sota_systems": [
        "TensorRT-LLM (lowest P95 TTFT under load)",
        "SGLang",
        "vLLM (most robust queueing/scheduling)"
      ],
      "sota_bar": "Under concurrency, single-GPU TTFT P99 explodes (e.g. ~8.6 s at C=180, ~34 s at C=420); pipeline parallelism (PP=2) holds it to ~3 s at C=180 and ~13 s at C=420 (~2.5-3x improvement). At 100 concurrent requests TensorRT-LLM P95 TTFT ~1,280 ms vs vLLM ~1,450 ms; vLLM sustains QoS to C=32 and stays usable to C=64.",
      "best_value": null,
      "metric_unit": "qualitative: P99 TTFT knee under a concurrency sweep (the C=100 P95 head-to-head is uncorroborated)",
      "source_url": "https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/",
      "source_date": "2026-01",
      "notes": "Refreshed with 2026 H100 head-to-head data. The governing insight: TTFT degradation under load is a queueing/KV-cache-saturation phenomenon - once KV cache fills, prefill queues and the TTFT tail blows up. Pipeline parallelism (PP=2) delays saturation ~2.5-3x. At C=100, TensorRT-LLM leads (P95 ~1,280 ms) over vLLM (~1,450 ms, ~13% gap); SGLang sits between but breaches TTFT thresholds earlier on constrained GPUs. best_value is the TensorRT-LLM P95 TTFT at C=100. Numbers are from a vendor-comparison benchmark, not a standardized SLO, hence medium confidence. [VERIFIER: best_value retracted as uncorroborated. SPLIT verdict; best_value NOT corroborated. (1) The PP=2 sub-claim is independently CONFIRMED and exact: Jarvislabs (jarvislabs.ai/blog/scaling-llm-inference-dp-pp-tp, independent of the cited Spheron source) reports bas]",
      "in_scope": true
    },
    {
      "id": "prefill-decode-interference",
      "category": "latency",
      "dimension": "Prefill-decode interference & chunked-prefill TTFT/TPOT tension",
      "why_it_matters": "When a long-context prefill shares a GPU batch with short interactive requests, it starves decode steps and spikes ITL/TPOT for everyone in the batch - and conversely, protecting decode can balloon TTFT. How a scheduler resolves this tension (chunked prefill, stall-free batching) is a primary differentiator of mixed-workload latency stability, which naive single-shape benchmarks never reveal.",
      "sota_systems": [
        "vLLM chunked prefill (decode-prioritized scheduling)",
        "SGLang chunked prefill"
      ],
      "sota_bar": "Chunked prefill cuts P95 inter-token latency by ~68% under mixed long/short workloads (e.g. 2,800 ms -> 890 ms P95 ITL at 32K inputs) by prioritizing decode and chunking prefill into the token budget.",
      "best_value": 68,
      "metric_unit": "% P95 ITL reduction (32K inputs, 2800ms -> 890ms)",
      "source_url": "https://www.spheron.network/blog/llm-serving-optimization-continuous-batching-paged-attention/",
      "source_date": "2026-06",
      "notes": "The 68% P95 ITL reduction bar is confirmed by a current (2026-06) source restating the same 2,800ms -> 890ms at 32K-input figure, so the number itself is unchanged from the old 2026-01 bar; date refreshes. This is the production-engine (vLLM/SGLang chunked-prefill) framing of PD interference, distinct from the research-system disaggregation framing in chunked-prefill-pd-interference. Note vLLM docs: smaller max_num_batched_tokens (e.g. 2048) gives better ITL because fewer prefills slow decodes. No single newer authoritative multiplier supersedes the 68% figure for the in-engine chunked-prefill case; the newer movement is toward intra-GPU disaggregation (tracked in the sibling dim).",
      "in_scope": true
    },
    {
      "id": "prefix-cache-ttft-impact",
      "category": "kv-cache",
      "dimension": "Prefix/KV-cache reuse impact on TTFT (cache-hit latency)",
      "why_it_matters": "Real workloads (shared system prompts, RAG documents, multi-turn chat) share long prefixes. Automatic prefix caching skips recomputing the shared KV, slashing prefill tokens and therefore TTFT on cache hits. The cache-hit vs cache-miss TTFT gap, and the hit rate under a realistic prompt mix, are decisive for chatbot/RAG/agent latency yet are absent from cold synthetic benchmarks.",
      "sota_systems": [
        "vLLM Automatic Prefix Caching (APC)",
        "SGLang RadixAttention",
        "llm-d distributed prefix cache"
      ],
      "sota_bar": "Qualitative: APC reuses exact KV pages via hash-based block matching, cutting prefill tokens and TTFT for repeated/long-prefix prompts with negligible steady-state overhead; overhead from hash compute grows with input length and concurrency. Distributed prefix-cache scheduling (llm-d) extends the win across replicas.",
      "best_value": null,
      "metric_unit": "TTFT reduction on cache hit (workload-dependent)",
      "source_url": "https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/",
      "source_date": "2025-01",
      "notes": "Must be measured at a realistic prefix-sharing ratio; reporting only cold (no-hit) TTFT understates production performance, while reporting only warm overstates it. Hit rate is the hidden variable.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "speculative-decoding-itl",
      "category": "speculative-decoding",
      "dimension": "Speculative decoding effect on ITL (acceptance-rate-bound)",
      "why_it_matters": "Speculative decoding (EAGLE-3, Medusa, MTP) verifies multiple draft tokens per forward pass, cutting ITL/TPOT 2-4x WITHOUT touching TTFT - the cheapest way to hit aggressive per-user token-rate SLOs at low batch size. The realized speedup is bound by draft acceptance rate, which is workload-dependent, so acceptance rate is itself a first-class latency dimension.",
      "sota_systems": [
        "EAGLE-3",
        "Medusa-2",
        "TensorRT-LLM / vLLM speculative decoding"
      ],
      "sota_bar": "EAGLE-3 reports ~0.80-0.88 acceptance on coding/instruction tasks giving ~3-4x decode token-rate (ITL) gains on H100/H200, and outperforms EAGLE-2/Medusa-2 by 15-25% tokens/s at batch size 1. Speedup primarily reduces ITL, not TTFT.",
      "best_value": 4,
      "metric_unit": "x ITL/decode-rate speedup (EAGLE-3, batch 1)",
      "source_url": "https://www.spheron.network/blog/eagle-3-speculative-decoding-gpu-cloud/",
      "source_date": "2026-01",
      "notes": "Speedup collapses toward 1x as acceptance rate drops on out-of-distribution prompts and as batch size grows (verification compute competes with throughput). Report acceptance rate alongside the speedup.",
      "in_scope": true
    },
    {
      "id": "phase-disaggregation-slo-isolation",
      "category": "parallelism",
      "dimension": "Prefill-decode disaggregation & per-phase SLO isolation",
      "why_it_matters": "Prefill is compute-bound (TTFT-sensitive) and decode is memory-bandwidth-bound (TPOT-sensitive); colocating them couples their tails. Disaggregating onto separate GPU pools lets each phase scale and meet its own SLO independently, which has become the production standard for hitting tight TTFT AND tight TPOT simultaneously at scale.",
      "sota_systems": [
        "NVIDIA Dynamo 1.0",
        "DistServe",
        "TaiChi (unifies agg+disagg to fix both TTFT and TPOT violations, 2025-08)"
      ],
      "sota_bar": "Phase disaggregation for SLO isolation (separating compute-bound prefill from memory-bound decode so each phase's SLO is met independently) is now production-standard via NVIDIA Dynamo, which reports 38% goodput over the best aggregated config and 7x with wide expert parallel on GB200 NVL72; DistServe's 7.4x remains the founding goodput figure.",
      "best_value": 7.4,
      "metric_unit": "x goodput vs prior SOTA (phase-disaggregated, SLO-isolated)",
      "source_url": "https://developer.nvidia.com/blog/removing-the-guesswork-from-disaggregated-serving/",
      "source_date": "2026-03",
      "notes": "Date and SOTA system refreshed: NVIDIA Dynamo is now the production-grade reference for phase disaggregation + SLO isolation (AIConfigurator blog, developer.nvidia.com, 2026-03-09): 550 tok/s/GPU = 38% over best aggregated config for Qwen3-32B on 64xB200, plus '7x with disaggregated serving combined with wide expert parallel on GB200 NVL72.' The founding goodput multiplier remains DistServe's 7.4x, so best_value is held at 7.4x; what materially changed since the old bar is the SOTA SYSTEM (DistServe -> NVIDIA Dynamo 1.0, GA'd late-2025/2026) and the supporting production numbers, not the headline multiplier. TaiChi (arXiv 2508.01989, 2025-08) is the key academic refinement: pure disaggregation causes TTFT violations from low prefill capacity while aggregation causes TPOT violations, so it unifies both modes for better SLO isolation. [second-source verified]",
      "in_scope": true
    },
    {
      "id": "cold-start-scaling-latency",
      "category": "operability",
      "dimension": "Cold-start / autoscaling latency (scale-to-zero & scale-out)",
      "why_it_matters": "The first request after a scale-out or a scale-to-zero wake pays a model-load tax - weights fetched and loaded into GPU memory before any token streams - inflating TTFT by tens of seconds. For elastic/serverless deployments this dominates worst-case latency and decides whether scale-to-zero is even viable for interactive use, so cold-start TTFT and warm-up time are core operability dimensions buyers must score.",
      "sota_systems": [
        "RunPod FlashBoot (sub-200 ms / <250 ms)",
        "Modal GPU memory snapshots (10x: 118 s -> 12 s)",
        "InferX (177 ms vendor claim, GPU snapshot, uncorroborated)"
      ],
      "sota_bar": "GPU-snapshotting cold start is now sub-200 ms: RunPod FlashBoot achieves sub-200 ms for ~48% of requests (<250 ms generally) and Modal GPU memory snapshots cut Ministral-3 3B median cold start ~10x (118 s -> 12 s) - the corroborable bar. InferX additionally claims an industry-leading 177 ms on H100 via GPU snapshotting, but that single number traces to one vendor source and is not independently confirmable, so the defensible figure is sub-200 ms.",
      "best_value": 200,
      "metric_unit": "ms cold-start latency (sub-200 ms via GPU snapshot; corroborable RunPod/Modal bar)",
      "source_url": "https://www.runpod.io/articles/guides/flashboot-instant-cold-starts",
      "source_date": "2026",
      "notes": "SOTA moved past the old FlashBoot 250 ms bar (2025-01). GPU-snapshotting (CUDA checkpoint/restore of an initialized CUDA context) is the new technique: InferX claims 177 ms on H100, RunPod FlashBoot is sub-200 ms for ~48% of requests, and Modal's GPU memory snapshots gave a 10x median cold-start cut (118 s -> 12 s for Ministral-3 3B, blog dated 2025-12-02). The 177 ms InferX figure is a vendor marketing claim (relayed via a Medium write-up; treat as OBSERVED not independently witnessed), hence medium confidence; the more primary, verifiable figure is RunPod FlashBoot <250 ms and Modal's 10x snapshot result. There is an open vLLM RFC (#34303) to add CUDA checkpoint/restore for near-zero cold starts - vLLM does NOT yet ship native GPU snapshotting; it relies on sleep mode + Modal's snapshotting today. [VERIFIER OVERRIDE -> 200: The headline best_value (177 ms, InferX, LLM on H100 via GPU snapshot) is NOT independently confirmable. Every occurrence of \"177 ms\" traces to a single origin: InferX's own marketing, all authored by the same person (Pr]",
      "in_scope": true
    },
    {
      "id": "paged-kv-memory-utilization",
      "category": "kv-cache",
      "dimension": "Paged KV block management and memory utilization (anti-fragmentation)",
      "why_it_matters": "KV cache is the dominant runtime memory consumer in LLM serving. Before paging, engines wasted 60-80% of KV memory to internal fragmentation and over-reservation, capping batch size and throughput. Block-based (paged) allocation determines how close an engine gets to theoretical KV capacity, which directly sets max concurrency and tokens/sec on fixed hardware. This is the foundational dimension every serious operator evaluates first.",
      "sota_systems": [
        "vLLM PagedAttention",
        "SGLang RadixAttention"
      ],
      "sota_bar": "PagedAttention achieves ~96.3% KV-cache memory utilization (the complement of <4% waste); no 2026 system reports a higher published paged-KV utilization figure.",
      "best_value": 96.3,
      "metric_unit": "percent KV-cache memory utilization (higher is better)",
      "source_url": "https://arxiv.org/abs/2309.06180",
      "source_date": "2023-09",
      "notes": "No newer source found that publishes a higher single-number paged-KV utilization bar; old bar stands at ~96.3% (the arithmetic complement of the <4% waste claim from the PagedAttention paper). The number is a derived/complement figure rather than a directly headlined metric, hence medium confidence. The 2026 direction of travel is tiered/disaggregated KV (Mooncake offloads to CPU/DRAM/SSD; LMCache adds a multi-tier KV layer) which raises EFFECTIVE capacity and reuse rather than per-block utilization, so the per-block utilization ceiling itself has not moved.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "automatic-prefix-caching-hit-rate",
      "category": "kv-cache",
      "dimension": "Automatic prefix caching / RadixAttention prefix reuse and cache hit rate",
      "why_it_matters": "Shared prompt prefixes (system prompts, few-shot exemplars, multi-turn history, RAG templates) are extremely common. Reusing their KV instead of recomputing prefill is the single largest throughput lever for prefix-heavy workloads. The hit rate an engine actually attains under a given workload, and whether reuse is automatic (no app changes), is a primary buyer differentiator between serving stacks.",
      "sota_systems": [
        "SGLang RadixAttention",
        "vLLM Automatic Prefix Caching"
      ],
      "sota_bar": "Automatic prefix caching now reaches ~85-95% cross-request hit rates on shared-prefix workloads (SGLang RadixAttention few-shot 85-95%, agentic ~88%; vLLM APC ~87% warm-cache hit, ~88% faster TTFT); the original RadixAttention 6.4x throughput claim (LMSYS 2024-01) remains the canonical headline number.",
      "best_value": 6.4,
      "metric_unit": "x throughput improvement on prefix-heavy workloads (RadixAttention headline); hit rate ~85-95%",
      "source_url": "https://www.lmsys.org/blog/2024-01-17-sglang/",
      "source_date": "2024-01",
      "notes": "Changed: the dimension has matured from a single throughput multiplier into documented HIT-RATE figures. 2026 sources now state concrete cross-request hit rates: SGLang few-shot 85-95% / multi-turn 75-90% / agentic ~88%, and vLLM APC ~87% warm-cache hit rate with ~88% faster TTFT on prefix-heavy workloads. The original 6.4x RadixAttention throughput claim (LMSYS Jan 2024) is still the canonical headline and is reused as best_value because it is the directly-comparable single number; the new development is the measured hit-rate band. SGLang vs vLLM on an agentic-workflow study: vLLM 42.9% vs a newer 'Helium' system 59.2% prefix-cache hit rate, showing the metric is now actively benchmarked across systems.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "non-prefix-kv-reuse-cacheblend",
      "category": "kv-cache",
      "dimension": "Non-prefix / cross-document KV reuse (RAG chunk caching)",
      "why_it_matters": "Prefix caching only helps when the shared text is a literal prefix. RAG concatenates retrieved chunks in varying order, so prefix reuse misses most of the opportunity. Engines that can reuse KV for arbitrary cached chunks (recomputing only the small fraction needed to fix cross-attention) unlock a second, larger tier of reuse for the most common enterprise workload. This is a sharp differentiator that a naive prefix-only scorecard omits.",
      "sota_systems": [
        "LMCache CacheBlend",
        "Mooncake (non-prefix KV pool)"
      ],
      "sota_bar": "LMCache CacheBlend reuses NON-prefix KV caches (any RAG chunk, not just the first) and selectively recomputes ~15% of tokens, achieving near-100% KV-cache hit rate with 2.2-3.3x lower TTFT and 2.8-5x higher throughput vs full recompute; won ACM EuroSys'25 Best Paper.",
      "best_value": 5,
      "metric_unit": "x throughput increase over full KV recompute (TTFT 2.2-3.3x lower; near-100% KV hit rate)",
      "source_url": "https://dl.acm.org/doi/10.1145/3790254",
      "source_date": "2025",
      "notes": "Changed/refreshed: the old 'CacheBlend 4.5x RAG (2024-10)' blog bar is superseded by the peer-reviewed numbers. CacheBlend won ACM EuroSys'25 Best Paper (Mar 2025) and the ACM TOCS journal version (2025) reports TTFT reduced 2.2-3.3x and throughput increased 2.8-5x vs full recompute, with ~15% selective recomputation and near-100% KV-cache hit rate in RAG. The 4.5x figure was the original blog's TTFT speedup; the consolidated/peer-reviewed headline is now the 2.8-5x throughput band, so best_value=5 (top of the throughput range). Distinct from prefix caching: CacheBlend reuses KV for chunks that are NOT the prefix, the gap fak's cache-value thesis also targets.",
      "in_scope": true
    },
    {
      "id": "kv-offloading-tiered-capacity",
      "category": "kv-cache",
      "dimension": "KV-cache offloading to CPU/host/NVMe/remote (multi-tier hierarchy depth)",
      "why_it_matters": "GPU HBM holds only a small KV working set. Offloading cold KV to CPU DRAM, local SSD/NVMe, and remote object/RDMA stores expands effective cache capacity by orders of magnitude, raising hit rates and enabling 'trade storage for compute'. The depth of the tier hierarchy, supported backends, and load bandwidth from each tier are what separate a production KV layer from a single-GPU cache.",
      "sota_systems": [
        "LMCache (vLLM, production Jan 2026)",
        "llm-d native KV offload (57x TTFT)",
        "Mooncake Store"
      ],
      "sota_bar": "LMCache (graduated to production Jan 2026, OSDI/arXiv paper Oct 2025) delivers up to 15x throughput improvement over vanilla vLLM by tiering KV cache across HBM/DRAM/NVMe with a global prefix index; llm-d production benchmarks report up to 57x faster TTFT and 2x throughput vs round-robin under high prefix reuse on 16 H100s.",
      "best_value": 15,
      "metric_unit": "x throughput vs vanilla vLLM (tiered HBM/DRAM/NVMe offload)",
      "source_url": "https://arxiv.org/abs/2510.09665",
      "source_date": "2025-10",
      "notes": "Old bar (LMCache 7x/100x, 2026-01) refreshed. The arXiv paper (2510.09665) states 'up to 15x improvement in throughput' as the headline; the older 7x/100x figures were earlier marketing splits (7x latency, 100x for fully-cached recompute avoidance). LMCache is now the de-facto tiered-offload standard, integrated into vLLM production and used by Google Cloud GKE Inference, CoreWeave, and Cohere; MP architecture shipped Apr 2026. llm-d reports the more aggressive 57x TTFT figure under high prefix reuse (https://llm-d.ai/blog/native-kv-cache-offloading-to-any-file-system-with-llm-d).",
      "in_scope": true
    },
    {
      "id": "kv-quantization-fp8-int4",
      "category": "quantization",
      "dimension": "KV-cache quantization (FP8 / INT8 / INT4 KV)",
      "why_it_matters": "Quantizing KV from BF16 to FP8 halves bytes-per-cached-token, directly doubling concurrency or context length at fixed HBM. INT8/INT4 push further. The buyer cares about the precision/accuracy/throughput trade: which formats are supported, whether attention runs in low precision (not just storage), and the measured accuracy delta. It is one of the cheapest capacity multipliers available.",
      "sota_systems": [
        "FP8 e4m3 KV (vLLM production, Apr 2026)",
        "INT4 KV (research: ~75% reduction)",
        "FP8 e5m2 (vLLM)"
      ],
      "sota_bar": "FP8 (e4m3) KV-cache quantization is the production standard: ~46% KV memory reduction (per-token cost ~54% of BF16), +14.9% output throughput / -14.8% median ITL on Llama-3.1-8B, recovering 97-98% of baseline long-context AUC at 128k and fully recovering aggregated AUC@1M. INT4 KV gives ~75% reduction but is research-stage; vLLM ships FP8 only.",
      "best_value": 54,
      "metric_unit": "% of BF16 per-token KV cost (FP8 e4m3, ~46% memory reduction)",
      "source_url": "https://vllm-project.github.io/2026/04/22/fp8-kvcache.html",
      "source_date": "2026-04",
      "notes": "Old bar (FP8 2x KV, 2025-01) refreshed with the vLLM team's own measured numbers (blog Apr 22 2026): the '2x' headline is the theoretical FP16->FP8 halving; the measured per-token cost is 54% of BF16 (best case), i.e. ~46% real reduction, with +14.9% throughput and 97-98% long-context AUC recovery. INT4 KV (75% reduction) exists in research (e.g. Kitty 2-bit, arXiv 2511.18643; INT8 GPU-accelerated arXiv 2601.04719) but is NOT production-default. The new datum is the validated FP8 accuracy/throughput profile, not a higher compression ratio.",
      "in_scope": true
    },
    {
      "id": "kv-compression-eviction",
      "category": "kv-cache",
      "dimension": "KV compression / token eviction under a fixed cache budget",
      "why_it_matters": "For very long contexts, even quantized KV is too large. Eviction/sparsity methods (keep attention-heavy or recent/sink tokens) shrink the resident KV to a fixed token budget, trading a controlled accuracy loss for memory and decode speed. Operators serving long-context or reasoning workloads must evaluate how aggressively a system can compress before quality breaks, and which algorithm it uses.",
      "sota_systems": [
        "R-KV (redundancy-aware, reasoning, arXiv 2505.24133)",
        "EpiCache (Apple, 2026)",
        "SnapKV / H2O / Quest (baselines now superseded)"
      ],
      "sota_bar": "Reasoning-aware eviction now beats the old H2O/Quest budget cliff: R-KV achieves lossless compression at 34% KV budget on MATH-500 and 10% on AIME-2024, and at 16% budget exceeds FullKV at 105% accuracy; Apple's EpiCache (2026) gets near-full-cache accuracy at 4-6x compression with up to 40% higher accuracy than eviction baselines and 3.5x lower peak memory.",
      "best_value": 10,
      "metric_unit": "% KV budget at lossless accuracy (R-KV, AIME-2024)",
      "source_url": "https://arxiv.org/pdf/2505.24133",
      "source_date": "2025-05",
      "notes": "Old bar (H2O/Quest budget cliff, 2024-06) refreshed: H2O/SnapKV/Quest are now the BASELINES that newer redundancy/reasoning-aware methods beat. R-KV (arXiv 2505.24133) reports lossless at 10% budget on AIME-2024 and 105% of FullKV at 16% -- a genuine break past the old budget cliff for CoT reasoning. EpiCache (Apple 2026) adds episodic clustering for near-full-cache accuracy at 4-6x compression, +40% over eviction baselines, 3.5x lower peak memory, ~2.4x lower latency. The field reframed eviction around reasoning/redundancy rather than raw attention-score thresholds.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "kv-transfer-disaggregation",
      "category": "kv-cache",
      "dimension": "KV-cache transfer for prefill/decode disaggregation (NIXL / NCCL / UCX)",
      "why_it_matters": "Disaggregated serving splits prefill and decode onto separate GPUs, requiring the prefill KV to be shipped to the decode engine. The transport library, supported fabrics (NVLink/InfiniBand/RoCE/NVMe-oF/S3), and whether transfer is non-blocking and overlapped layer-by-layer with compute decide whether disaggregation adds TTFT or hides the cost. This is the connective tissue that makes large-scale disaggregation viable.",
      "sota_systems": [
        "NVIDIA Dynamo 1.0 + NIXL (GA Mar 2026)",
        "Mooncake Transfer Engine",
        "SGLang EPD disaggregation"
      ],
      "sota_bar": "NVIDIA Dynamo 1.0 (GA March 16, 2026 at GTC) delivers up to 7x higher throughput with disaggregated prefill/decode serving combined with wide expert parallelism, measured on DeepSeek-R1 on GB200 NVL72; KV blocks move prefill->decode over RDMA via NIXL at wire speed.",
      "best_value": 7,
      "metric_unit": "x throughput (disaggregated serving + wide-EP, DeepSeek-R1 on GB200 NVL72)",
      "source_url": "https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/",
      "source_date": "2026-03",
      "notes": "Old bar (NIXL/Dynamo, 2025-04) refreshed: that was the original Dynamo preview announcement; Dynamo 1.0 went GA March 16, 2026 with the 7x-on-Blackwell number. NIXL remains the underlying point-to-point RDMA KV transfer library. The 7x is hardware+parallelism-coupled (GB200 NVL72 + wide-EP on an MoE model), not transfer-library throughput in isolation. Mooncake Transfer Engine is the open-source peer (FlexKV/Tencent+NVIDIA integration Jan 2026; SGLang EPD Dec 2025).",
      "in_scope": true
    },
    {
      "id": "kv-cache-aware-routing",
      "category": "kv-cache",
      "dimension": "KV-cache-aware / prefix-aware request routing across a fleet",
      "why_it_matters": "A cache is only as good as the routing that lands a request on the GPU that already holds its KV. Across a multi-node fleet, a cache-aware router computes prefix/block overlap and balances it against load to maximize cluster-wide hit rate. Without it, replicas duplicate KV and recompute prefill. This is the dimension that turns per-engine caching into fleet-level economics.",
      "sota_systems": [
        "Baseten + NVIDIA Dynamo KV router",
        "llm-d v0.5 (GAIE/EPP prefix-cache-aware)",
        "GKE Inference Gateway prefix-cache-aware routing"
      ],
      "sota_bar": "KV-cache-aware routing across a replica fleet now delivers ~50% lower TTFT, ~34% lower TPOT, ~61% more RPS and ~62% more output tok/s (Baseten on NVIDIA Dynamo, Qwen3 480B), with llm-d reporting 3x throughput / 2x TTFT vs round-robin and GKE Inference Gateway 13.9x faster TTFT than a 3rd-party service.",
      "best_value": 62,
      "metric_unit": "% more output tok/s (also -50% TTFT, -34% TPOT, +61% RPS)",
      "source_url": "https://www.baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/",
      "source_date": "2025-10",
      "notes": "SOTA moved well past the old 'Dynamo 2x' bar. Baseten/Dynamo (2025-10-16, Qwen3 480B Coder, ~50k-token inputs): -50% TTFT, -34% TPOT, +61% RPS, +62% output TPS, -48% P95 / -49% P99 latency, 89% cache hit across 4 replicas; separately 3x TTFT and 2x lower avg latency on 100k real queries. llm-d v0.5 (2026-02): 3x output throughput / 2x faster TTFT vs round-robin, plus 13.9x throughput with hierarchical KV offload at 250 concurrency vs GPU-only on 4xH100, and 5x lower ITL / 100% success when load-awareness is combined with prefix routing. GKE Inference Gateway (benchmark 2026-06-09): 13.93x faster mean TTFT vs leading 3rd-party managed K8s service (188ms vs 2625ms) on Llama-3.1-8B shared-prefix, 15.7% higher throughput, 62.6% lower ITL; Snap reports 75-80% prefix-cache hit in production. The single headline number depends on framing; I report Baseten/Dynamo's +62% output tok/s as best_value, with the larger TTFT multipliers (3x-13.9x) noted.",
      "in_scope": true
    },
    {
      "id": "attention-arch-kv-footprint",
      "category": "kv-cache",
      "dimension": "Attention architecture impact on KV footprint (MQA/GQA/MLA bytes-per-token)",
      "why_it_matters": "The model's attention design sets the floor on KV size before any serving trick applies. MQA/GQA share K/V heads; MLA compresses K/V into a low-rank latent. A 3-5x smaller KV footprint per token compounds with every other lever (paging, offload, quant) to multiply servable concurrency and context. Operators selecting models for serving cost must weigh this, and engines must implement the architecture's KV layout efficiently.",
      "sota_systems": [
        "MLA (DeepSeek-V3)",
        "MLA + Native Sparse Attention (NSA, DeepSeek 2025)",
        "Qwen-Latent / MiniMax-Text-01 MLA-style"
      ],
      "sota_bar": "Multi-head Latent Attention (MLA, DeepSeek-V3) holds ~70 KB/token KV footprint vs 192-328 KB/token for GQA models (a 2.7-4.7x reduction, ~9x vs MHA, ~2x vs 8-group GQA) at higher quality; the frontier is now MLA + learned sparsity (DeepSeek NSA, 2025) pushing toward sub-100 floats/token for long contexts.",
      "best_value": 70,
      "metric_unit": "KB per token KV cache (MLA, DeepSeek-V3)",
      "source_url": "https://arxiv.org/pdf/2502.14837",
      "source_date": "2025-02",
      "notes": "Old bar (MLA 70KB vs GQA 192-328KB, 2025-02) STILL STANDS as the per-token arithmetic baseline; the 70KB MLA vs 192-328KB GQA figures are unchanged and remain the canonical comparison (re-confirmed by 2025/2026 surveys, e.g. arXiv 2502.14837 and pyimagesearch DeepSeek-V3 MLA build Mar 2026). What MOVED since: MLA is now a standard architecture option (adopted by Qwen-Latent, MiniMax-Text-01), and the new frontier is layering learned sparsity on MLA -- DeepSeek NSA (2025) targets sub-100 floats/token for very long contexts. Date refreshed via adoption; the headline footprint number did not change.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "effective-capacity-slo-gain",
      "category": "cost-efficiency",
      "dimension": "End-to-end effective request capacity / cost gain from the KV system under SLO",
      "why_it_matters": "All the mechanisms above only matter if they raise the requests an operator can serve within latency SLOs on fixed hardware. The integrated, trace-driven gain (effective capacity, cost per token) is the bottom-line metric a buyer uses to compare whole KV-centric architectures rather than individual features. It captures the 'trade storage for compute' thesis at the system level.",
      "sota_systems": [
        "Mooncake (Moonshot AI / Kimi)",
        "NVIDIA Dynamo (KVBM + NIXL KV transfer)"
      ],
      "sota_bar": "Mooncake's KVCache-centric disaggregated architecture increases effective request capacity by 59%-498% on real traces (up to 525% in simulation) while meeting SLOs; this remains the headline effective-capacity-under-SLO bar, now republished in ACM Transactions on Storage (2026).",
      "best_value": 498,
      "metric_unit": "% effective request capacity gain vs baseline under SLO (real traces; up to 525% simulated)",
      "source_url": "https://arxiv.org/abs/2407.00079",
      "source_date": "2024-07",
      "notes": "No newer source beats Mooncake's 59-498% (real traces) / 525% (simulated) effective-capacity-under-SLO range; old bar stands. The figure is now corroborated and republished in ACM Transactions on Storage (10.1145/3773772, 2026) and in production reports (Kimi handles 75% more requests; 115%/107% more on A800/H800 clusters; >100B tokens/day). KV-centric disaggregation (the Mooncake idea) is now standard infrastructure (NIXL KV transfer in vLLM and NVIDIA Dynamo), but no system has published a higher effective-capacity-under-SLO percentage on real traces.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "persistent-cross-instance-cache-coherence",
      "category": "operability",
      "dimension": "Cross-instance / persistent KV cache sharing and coherence",
      "why_it_matters": "In production, the same KV should be reusable across engine instances, restarts, and prefill/decode roles, not trapped in one process's HBM. A shared, persistent, deduplicated KV store with peer-to-peer access raises fleet hit rate and survives autoscaling churn. Operators evaluate whether caching is per-process-ephemeral or a first-class shared substrate with consistent addressing.",
      "sota_systems": [
        "LMCache",
        "Mooncake Store"
      ],
      "sota_bar": "LMCache enables KV reuse across different vLLM engine instances plus multi-GPU peer-to-peer KV sharing and disaggregated prefill/decode; Mooncake Store exposes a global, deduplicated KVCache pool addressable across the cluster",
      "best_value": null,
      "metric_unit": "qualitative (cross-instance reuse, dedup, persistence)",
      "source_url": "https://arxiv.org/html/2510.09665v2",
      "source_date": "2025-10",
      "notes": "LMCache enterprise KV layer paper. Distinguishes a real KV 'layer/store' from a single-engine cache; key for autoscaling, blue/green deploys, and shared multi-tenant fleets where ephemeral caches lose all warmth on restart.",
      "in_scope": true
    },
    {
      "id": "fp8-weight-activation-accuracy-retention",
      "category": "quantization",
      "dimension": "FP8 (W8A8, E4M3) accuracy retention vs bf16 reference",
      "why_it_matters": "FP8 W8A8 is the production default at vLLM/SGLang/TRT-LLM on Hopper+ and is the baseline every buyer assumes is 'free.' Operators need proof it is effectively lossless before turning it on fleet-wide, because it roughly halves memory and adds 1.4-1.8x serving throughput only if quality holds.",
      "sota_systems": [
        "FP8 W8A8 via LLM Compressor (vLLM)",
        "NVIDIA TensorRT Model Optimizer FP8",
        "AMD Quark FP8"
      ],
      "sota_bar": "FP8 (W8A8, E4M3) weight+activation quantization is effectively lossless: ~99.9% accuracy recovery on HumanEval and <=1% degradation vs BF16 across all model scales, including Llama 3.1 405B fully recovering accuracy.",
      "best_value": 99.9,
      "metric_unit": "% accuracy recovery vs BF16 (HumanEval)",
      "source_url": "https://developers.redhat.com/articles/2024/10/17/we-ran-over-half-million-evaluations-quantized-llms",
      "source_date": "2024-10",
      "notes": "Old bar was 99% recovery (2025-08). The large-scale Red Hat eval (>500k evaluations) reports 99.9% recovery on HumanEval and labels W8A8-FP lossless across all scales; the 2026 vLLM/Spheron framing calls FP8 'nearly indistinguishable from BF16' and the practical 2026 sweet spot. Number ticks slightly up (99 -> ~99.9 on HumanEval) but the qualitative bar (near-lossless, <=1% vs BF16) is the same, so this is essentially a date/confidence refresh rather than a new bar. The strongest cited primary value (99.9%) predates the old bar's stamp date; no 2026 source materially exceeds it, so changed_vs_old=false.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "int8-w8a8-smoothquant-accuracy",
      "category": "quantization",
      "dimension": "INT8 W8A8 (SmoothQuant) accuracy under activation outliers",
      "why_it_matters": "INT8 W8A8 runs on far more hardware than FP8 (no FP8 tensor cores required) and dominates throughput in async continuous-batching serving. Its hard problem is activation outliers; how well a system tames them (SmoothQuant migration, per-channel/per-token dynamic scales) determines whether INT8 is viable for a given model.",
      "sota_systems": [
        "INT8 W8A8 SmoothQuant via LLM Compressor (vLLM)",
        "SmoothQuant (Xiao et al.)"
      ],
      "sota_bar": "INT8 W8A8 (per-channel static weight + per-token dynamic activation, optionally SmoothQuant) recovers ~99% of BF16 accuracy, i.e. ~1% average loss; vLLM/LLM Compressor report negligible Open-LLM-leaderboard drop.",
      "best_value": 99,
      "metric_unit": "% accuracy recovery vs BF16 (~1% loss)",
      "source_url": "https://developers.redhat.com/articles/2025/03/03/deployment-ready-reasoning-quantized-deepseek-r1-models",
      "source_date": "2025-03",
      "notes": "Old bar was 1-3% loss (2025-08). Red Hat (Mar 2025) puts INT8 W8A8 at ~99% recovery (i.e. ~1% loss, the favorable end of the old 1-3% band); vLLM INT8 docs describe a tolerable 1-3% per-task average with dynamic-activation or SmoothQuant. Bar is unchanged in substance. NOTE vLLM-relevant caveat: INT8 W8A8 is NOT supported on Blackwell (compute capability >=10.0), so on Blackwell FP8/NVFP4 displace INT8. Returning the old ~1-3% band; only the source/date refresh.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "int4-weight-only-awq-gptq",
      "category": "quantization",
      "dimension": "INT4 weight-only (W4A16: AWQ, GPTQ) accuracy and calibration sensitivity",
      "why_it_matters": "W4A16 is the most cost-efficient choice for latency/memory-bound and synchronous single-stream serving, and is the dominant format for consumer/edge deployment. Buyers must know the accuracy floor and how sensitive it is to calibration data and algorithm choice (AWQ activation-aware vs GPTQ second-order), since results flip with implementation and calibration set size.",
      "sota_systems": [
        "AWQ (AutoAWQ) via vLLM",
        "GPTQ / GPTQModel + Marlin (vLLM)",
        "W4A16 LLM Compressor"
      ],
      "sota_bar": "INT4 weight-only (W4A16) via AWQ or GPTQ keeps accuracy degradation within negligible-to-moderate range (<4% typical, often <1-2% on HumanEval); AWQ is the de-facto best-practice INT4 format for vLLM in 2026 and generally edges out GPTQ on perplexity.",
      "best_value": 4,
      "metric_unit": "% max accuracy degradation vs FP16 (typical upper bound)",
      "source_url": "https://vrlatech.com/llm-quantization-explained-int4-int8-fp8-awq-and-gptq-in-2026/",
      "source_date": "2026-01",
      "notes": "Old bar carried only a date (2025-08), no number. 2026 sources confirm AWQ remains the recommended INT4 weight-only format for vLLM with <4% degradation (and AWQ < GPTQ perplexity at 4-bit). No SOTA method materially displaced AWQ/GPTQ for pure weight-only INT4; the action has moved to 4-bit weight+activation (NVFP4/FP4) which is a separate dimension. Best primary source is the Microsoft Data-Science GPTQ-vs-AWQ guide; the VRLA 2026 page is the freshest dated framing. changed_vs_old=false (refresh + a number attached, but the bar did not move).",
      "in_scope": true
    },
    {
      "id": "fp4-nvfp4-mxfp4-blackwell",
      "category": "quantization",
      "dimension": "4-bit microscaling float (NVFP4 / MXFP4 / FP4) accuracy vs FP8 on Blackwell",
      "why_it_matters": "FP4 on Blackwell doubles peak throughput vs FP8 and is the new frontier format MLPerf submissions now use for Llama-3.1-405B. The differentiator is NVFP4 (E4M3 per-16 block scale, FP32 second-level) vs MXFP4 (E8M0 per-32 power-of-two scale): block size and scale precision drive whether 4-bit holds MMLU/perplexity near FP8.",
      "sota_systems": [
        "NVFP4 via NVIDIA TensorRT Model Optimizer + LLM Compressor (vLLM, Blackwell B200/GB200)",
        "NVFP4 (vLLM Marlin)"
      ],
      "sota_bar": "NVFP4 (4-bit, 16-value blocks, FP8 E4M3 fractional block scales) now achieves <=1% accuracy drop vs FP8 on large reasoning models (DeepSeek-R1-0528: MMLU-PRO 85->84, GPQA 81->80) and ~99% recovery vs BF16 for 70B-235B models, at ~2.3x higher throughput than weight-only 4-bit; clearly the SOTA 4-bit format on Blackwell, well ahead of MXFP4 (32-value, power-of-two scales).",
      "best_value": 1,
      "metric_unit": "% MMLU-PRO drop NVFP4 vs FP8 (DeepSeek-R1-0528)",
      "source_url": "https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/",
      "source_date": "2025-09",
      "notes": "Old bar was 'NVFP4 0.1 MMLU drop, 2025-04'. The field moved materially: NVIDIA's NVFP4 launch + Red Hat's Feb-2026 NVFP4 eval now report <=1% drop vs FP8 on DeepSeek-R1 (MMLU-PRO 85->84%, ~1pt) and 95-99% recovery vs BF16 by model scale, plus 2.3x throughput vs weight-only AWQ. Multiple 2025Q4-2026 PTQ papers (Four-Over-Six arXiv 2512.02010, ScaleSweep 2606.07618, ARCQuant 2601.07475) push NVFP4 PTQ further, and QAD (NVFP4-QAD report, Mar 2026) recovers near-BF16. NVFP4 is now first-class in vLLM via Marlin. The old '0.1 MMLU drop' was a single optimistic point; the current honest bar across models is ~1% vs FP8 / 1-4% vs BF16. Best_value=1 (pp MMLU-PRO drop vs FP8). vLLM supports NVFP4 + MXFP4 today.",
      "in_scope": true
    },
    {
      "id": "4bit-activation-rotation-quarot-spinquant",
      "category": "quantization",
      "dimension": "Full 4-bit (W4A4 + KV4) via rotation/outlier removal",
      "why_it_matters": "The aggressive end of the design space: quantizing weights, activations, AND KV to 4 bits unlocks the largest memory/throughput wins but is gated entirely by activation-outlier handling. Hadamard/learned-rotation methods (QuaRot, SpinQuant) define how close W4A4 can get to full precision, which bounds what an operator can push without quality collapse.",
      "sota_systems": [
        "SpinQuant (learned rotations)",
        "Qronos (2025)",
        "ResQ PCA-rotation (2025)",
        "TurboQuant (2026, KV cache)",
        "QuaRot (Hadamard baseline)"
      ],
      "sota_bar": "Learned-rotation methods now lead 4-bit weight+activation: SpinQuant (ICLR 2025) closes the gap to full precision by up to 45.1% relative to QuaRot on hard-to-quantize Llama-3 8B, and 2025-2026 successors (Qronos, ResQ PCA-rotations, TurboQuant for KV cache, WUSH adaptive transforms) advance further; Hadamard/learned rotations are the standard outlier-removal primitive feeding NVFP4/W4A4.",
      "best_value": 45.1,
      "metric_unit": "% relative reduction of gap-to-FP16 (SpinQuant vs QuaRot, Llama-3 8B)",
      "source_url": "https://arxiv.org/abs/2405.16406",
      "source_date": "2025-04",
      "notes": "Old bar was QuaRot-only at 2024-04. The rotation line clearly advanced: SpinQuant (published/accepted ICLR 2025) supersedes QuaRot with up to 45.1% gap reduction on Llama-3 8B, and is now the reference learned-rotation method. Further successors landed in 2025-2026 (Qronos arXiv 2505.11695 'best overall with SpinQuant', ResQ PCA mixed-precision, TurboQuant 2026 random-orthogonal KV-cache rotations, WUSH 2512.00956 near-optimal adaptive transforms, HadaCore Tensor-Core Hadamard kernel). best_value 45.1 is SpinQuant's headline relative gap-reduction over QuaRot. vLLM does not implement learned rotation itself; rotated checkpoints are produced offline (LLM Compressor / model-opt) then served as W4A4/NVFP4 in vLLM. changed_vs_old=true. [second-source verified]",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "kv-cache-quantization-accuracy",
      "category": "kv-cache",
      "dimension": "KV-cache quantization (FP8 / INT8 / KIVI sub-4-bit) accuracy at long context",
      "why_it_matters": "KV-cache dominates memory at long context and high concurrency; halving it (FP8) or going to 2-4 bits (KIVI/KVQuant) directly buys longer contexts or more concurrent requests at fixed hardware. The buyer question is exactly where it stays lossless vs where it degrades (prefill-heavy, head_dim=256, hybrid sliding-window layers).",
      "sota_systems": [
        "OSCAR (Together AI, 2026)",
        "TurboQuant (Google/NYU, ICLR 2026)",
        "Kitty 2-bit dynamic channel-wise (arXiv 2511.18643)"
      ],
      "sota_bar": "Ultra-low-bit KV quant frontier (2026): Together AI's OSCAR reaches 2.28 effective bits within 1.42 points of BF16 (Qwen3-8B; ~0.02-pt gap on Qwen3-32B) with ~8x KV memory reduction at 100K context and up to 7.83x job-level throughput; TurboQuant (ICLR 2026) is near-lossless at 3-4 bits / full-precision NIAH recall at 4x.",
      "best_value": 2.28,
      "metric_unit": "effective bits per KV element at near-BF16 accuracy (OSCAR, within 1.42 pts)",
      "source_url": "https://www.marktechpost.com/2026/06/18/the-kv-cache-compression-race-turboquant-vs-oscar-vs-epicache/",
      "source_date": "2026-06",
      "notes": "Old bar (2026-04, generic) refreshed to the named 2026 ultra-low-bit results. OSCAR's 2.28 effective bits within 1.42 pts of BF16 (and ~8x memory / 7.83x throughput at 100K ctx) is the most aggressive near-lossless accuracy bar; TurboQuant (ICLR 2026) is the academically-bounded 3-4 bit near-lossless method (distortion ~2.7x of the information-theoretic floor). Confidence medium because the OSCAR/TurboQuant headline figures are reported via a survey aggregator (MarkTechPost) and the vendor/paper PDFs (arXiv) rather than a single peer-reviewed leaderboard; the bit/accuracy pairs are consistently cited but worth a primary-PDF confirm before quoting in a brief. [second-source verified]",
      "in_scope": true
    },
    {
      "id": "quantization-granularity-scheme",
      "category": "quantization",
      "dimension": "Quantization granularity (per-tensor / per-channel / per-group / microscaling block size)",
      "why_it_matters": "Granularity is the single biggest accuracy lever at a fixed bit-width and the hardest cost/quality knob: finer scales (per-channel, group-128, MX block-32, two-level) cut quantization error but add scale storage and GEMM overhead. It also determines hardware compatibility (MX needs block-32 tensor-core support). A scorecard that ignores granularity can't compare two 'INT4' or 'FP4' systems honestly.",
      "sota_systems": [
        "MX formats (OCP MXFP4/MXFP8, block-32)",
        "NVFP4 (block-16, FP32 second-level)",
        "group-128 INT4 (AWQ/GPTQ)",
        "MOSS two-level microscaling"
      ],
      "sota_bar": "Per-block (block-32) microscaling beats per-tensor/per-channel; measured SNR gains of 3.0-3.4 dB over group-128 and 9.2-9.4 dB over per-tensor for two-level microscaling. Common group sizes 32/64/128; group-128 is the de-facto 4-bit weight-only default. NVFP4's block-16 + FP32 scale is finer than MXFP4's block-32 power-of-two.",
      "best_value": 9.4,
      "metric_unit": "dB SNR improvement, two-level MX block-32 vs per-tensor",
      "source_url": "https://arxiv.org/html/2511.05811v2",
      "source_date": "2025-11",
      "notes": "MOSS (arXiv:2511.05811). Granularity taxonomy from Intel Neural Compressor MX docs and apxml granularity course. This is the dimension a naive bit-width-only scorecard omits.",
      "in_scope": true
    },
    {
      "id": "numerical-correctness-error-metric",
      "category": "numerical-correctness",
      "dimension": "Numerical-correctness error metric vs reference (perplexity, KL-divergence, recovery %)",
      "why_it_matters": "Aggregate task scores (MMLU) hide quantization distortion because rare tokens barely move perplexity. Serious evaluation reports the deviation of the quantized model's token distribution from the bf16/fp16 reference (KL-divergence) plus generation-similarity, not just downstream accuracy. The choice of error metric is itself a differentiator buyers must scrutinize.",
      "sota_systems": [
        "llama.cpp (perplexity + KL-divergence tooling)",
        "Neural Magic / vLLM recovery-% methodology",
        "ModelOpt accuracy harness"
      ],
      "sota_bar": "Best practice pairs perplexity with KL-divergence against the FP16 baseline (llama.cpp emits both); KLD captures per-token distribution shift perplexity hides. Empirically: 5-bit GGUF is a strong near-lossless trade-off, 3-bit shows clear degradation; imatrix is essential below 3-bit, marginal at Q4_K_M+. Generation text-similarity is reported as a supplementary correctness check.",
      "best_value": null,
      "metric_unit": "KL-divergence (nats) / perplexity delta vs fp16 (no single SOTA value)",
      "source_url": "https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md",
      "source_date": "2026-01",
      "notes": "Unified GGUF eval: arXiv:2601.14277. KLD-vs-perplexity rationale widely documented. Set best_value=null: the SOTA is the methodology (report KLD + ppl + recovery), not one number.",
      "in_scope": true
    },
    {
      "id": "deterministic-bitexact-inference",
      "category": "numerical-correctness",
      "dimension": "Deterministic / bitwise-reproducible inference (batch invariance)",
      "why_it_matters": "Quantized serving is system-nondeterministic: identical prompts give different outputs because reduction order changes with batch size (floating-point non-associativity in matmul/attention/norm). For eval reproducibility, RL training-inference consistency, debugging, and audit/compliance, operators increasingly require bitwise-identical outputs - a hard correctness property distinct from accuracy.",
      "sota_systems": [
        "Thinking Machines batch-invariant-ops (on vLLM FlexAttention)",
        "vLLM batch-invariance mode",
        "LLM-42 (deterministic speculation)"
      ],
      "sota_bar": "Batch-invariant kernels (invariant normalization, matmul, attention reductions) yield 1000/1000 bitwise-identical completions under dynamic batching, vs dozens of distinct outputs without; cost ~60% slower than default kernels. Now exposed as a vLLM batch-invariance feature.",
      "best_value": 100,
      "metric_unit": "% bitwise-identical completions across runs (deterministic mode)",
      "source_url": "https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/",
      "source_date": "2025-09",
      "notes": "Thinking Machines 'Defeating Nondeterminism' + vLLM batch_invariance docs. Root cause = non-associativity in reduction schedules, not RNG. LLM-42 (arXiv:2601.17768) adds verified-speculation determinism.",
      "in_scope": true
    },
    {
      "id": "mlperf-accuracy-constrained-throughput",
      "category": "quantization",
      "dimension": "Accuracy-constrained throughput on a neutral benchmark (MLPerf Inference)",
      "why_it_matters": "MLPerf Inference is the only vendor-neutral, audited benchmark that ties quantized throughput to a fixed accuracy bar (99% or 99.9% of FP reference), preventing quality-for-speed cheating. It is the buyer's apples-to-apples ground truth: a number only counts if it clears the accuracy gate, which is exactly how operators should read all quantization claims.",
      "sota_systems": [
        "NVIDIA GB300 NVL72 (Blackwell Ultra, TensorRT-LLM/Dynamo)",
        "NVIDIA GB200 NVL72 (Blackwell)",
        "CoreWeave / Nebius GB300 NVL72 submissions"
      ],
      "sota_bar": "Under MLPerf's accuracy- and latency-constrained server scenario (TTFT and TPS/user SLOs enforced at p99), MLPerf Inference v6.0 (2026-03-30) DeepSeek-R1 reaches 8,064 tokens/second/GPU on GB300 NVL72 (vs 9,821 tok/s/GPU unconstrained offline); DeepSeek-R1's server SLO is 2s TTFT with a 12.5 tok/s/user p99 target while holding the required accuracy bar, and the GB300 server result is ~25% higher per-GPU than GB200 and up to 2.7x the GB300 debut via software.",
      "best_value": 8064,
      "metric_unit": "tokens/second/GPU, DeepSeek-R1 671B, MLPerf v6.0 server scenario (p99 TTFT=2s, 12.5 TPS/user, accuracy-constrained)",
      "source_url": "https://www.storagereview.com/news/nvidia-sets-mlperf-inference-v6-0-records-with-blackwell-ultra-platform",
      "source_date": "2026-04-01",
      "notes": "Refreshed from 2025-04 (v5.0) to MLPerf v6.0 (2026-03-30). MLPerf's server scenario IS the accuracy+latency-constrained throughput number: a submission only counts if it meets both the per-model accuracy target (e.g. DeepSeek-R1's required quality) and the p99 latency SLO (2s TTFT, 12.5 TPS/user). v6.0 DeepSeek-R1 server = 8,064 tok/s/GPU on GB300; full-rack/288-GPU server aggregate scales accordingly (single GB300 NVL72 ~575,580 tok/s server per Nebius). Primary source is MLCommons' v6.0 result set and NVIDIA's Blackwell-Ultra blog; storagereview gives the clean per-GPU server figure. The constrained (server) number sits below the offline 9,821 tok/s/GPU, as expected. The model carrying the constraint shifted from v5.x's Llama 3.1 405B to DeepSeek-R1 671B.",
      "in_scope": true
    },
    {
      "id": "hardware-format-coverage",
      "category": "hardware-coverage",
      "dimension": "Hardware / format coverage and portability of quantized kernels",
      "why_it_matters": "A quantization format is only useful where the serving stack has fast kernels for it. FP8 needs Hopper+/Ada or MI300X; NVFP4 is Blackwell-exclusive; MXFP4 is the cross-vendor OCP standard but kernel support lags. Buyers running heterogeneous fleets (NVIDIA + AMD + edge) must know which precision actually accelerates on their silicon vs silently falls back.",
      "sota_systems": [
        "vLLM (FP8 on Hopper/Ada/Blackwell + MI300X via hipBLASLt/aiter)",
        "SGLang (FP8 MI300X)",
        "TensorRT-LLM (NVFP4 Blackwell)",
        "llama.cpp (broad CPU/GPU GGUF + FP4)"
      ],
      "sota_bar": "FP8 W8A8 is hardware-accelerated on NVIDIA Hopper/Ada/Blackwell and AMD MI300X (hipBLASLt FP8 GEMM + aiter attention); NVFP4 is Blackwell-only; MXFP4 acceleration on MI300X/MI355X is on the roadmap, not yet broadly shipped. GGUF k-quants give the widest CPU/consumer-GPU portability.",
      "best_value": null,
      "metric_unit": "qualitative format-x-accelerator support matrix",
      "source_url": "https://docs.vllm.ai/en/stable/features/quantization/fp8/",
      "source_date": "2026-01",
      "notes": "AMD coverage: vLLM AMD roadmap issue #44092 + Spheron FP8 hardware guide. SGLang MXFP4-on-MI300X still a discussion (#13611). best_value=null: this is a coverage matrix, not a scalar.",
      "in_scope": true
    },
    {
      "id": "spec-decode-speedup",
      "category": "speculative-decoding",
      "dimension": "Speculative-decoding wall-clock speedup (single-stream / low-concurrency)",
      "why_it_matters": "The headline reason operators turn on speculative decoding: how much faster each request finishes at interactive (batch~1) load. It directly sets the latency a chat/agent user feels, and is the number buyers compare across engines and draft methods.",
      "sota_systems": [
        "EAGLE-3 (SafeAILab)",
        "vLLM",
        "SGLang",
        "P-EAGLE (vLLM)"
      ],
      "sota_bar": "EAGLE-3 reports up to 4.79x speedup on LLaMA-3.3-70B with no quality loss and is the de-facto industrial standard supported in vLLM and SGLang; vLLM's P-EAGLE reaches ~4-5x over standard decoding (20-30% over EAGLE-3 alone) on coding benchmarks",
      "best_value": 4.79,
      "metric_unit": "x speedup vs autoregressive decoding (batch~1)",
      "source_url": "https://github.com/SafeAILab/EAGLE",
      "source_date": "2025-09",
      "notes": "EAGLE-3 accepted NeurIPS'25; P-EAGLE (parallel draft) figures from vLLM blog 2026-03-13 https://vllm.ai/blog/2026-03-13-p-eagle . Speedup is workload- and temperature-dependent.",
      "in_scope": true
    },
    {
      "id": "acceptance-length-rate",
      "category": "speculative-decoding",
      "dimension": "Draft acceptance length / acceptance rate (mean accepted tokens per verify step)",
      "why_it_matters": "Acceptance length tau is the upstream driver of every spec-decode speedup; it is the engine-independent quality metric of a draft method and the thing that determines whether the verify-step overhead pays off. Operators tune draft depth/tree against measured tau on their own traffic.",
      "sota_systems": [
        "EAGLE-3 (SafeAILab)",
        "DeepSeek-V3 MTP",
        "FastMTP / FastEagle (2025 successors)"
      ],
      "sota_bar": "EAGLE-3 reports per-task acceptance rate ~0.80-0.88 (coding/instruction) and mean accepted length ~4.5-5.0 tokens per draft-verify cycle, exceeding the older DeepSeek-V3 MTP ~0.85 second-token figure.",
      "best_value": 0.88,
      "metric_unit": "acceptance rate (fraction); mean accepted length ~4.5-5.0 tokens/verify",
      "source_url": "https://arxiv.org/abs/2503.01840",
      "source_date": "2025-03",
      "notes": "SOTA moved: old bar was DeepSeek-V3 MTP 0.85 (2024-12). EAGLE-3 (2025-03) reports ~0.80-0.88 acceptance and ~4.5-5.0 mean accepted tokens per verify step on coding/instruction tasks, a newer and higher published bar; EAGLE-3's training-time-test fused-layer draft head is the mechanism. best_value 0.88 is the upper end of the reported coding/instruction-following band; the mean-accepted-length is the complementary metric (~4.5-5.0). Confidence medium because the 0.80-0.88 figure is reported via secondary summaries of the EAGLE-3 paper; the primary arXiv is the citable source. [second-source verified]",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "draft-method-coverage",
      "category": "speculative-decoding",
      "dimension": "Breadth of speculative methods supported (draft-model, EAGLE/-2/-3, Medusa, MTP, lookahead, n-gram/prompt-lookup)",
      "why_it_matters": "No single draft method wins on every model/workload, so an engine's menu of methods is a portability and future-proofing axis. Buyers want one engine that covers self-speculative (MTP/Medusa), feature-level (EAGLE), retrieval (n-gram), and external-draft so they can pick per model without re-platforming.",
      "sota_systems": [
        "TensorRT-LLM",
        "vLLM",
        "SGLang"
      ],
      "sota_bar": "TensorRT-LLM supports draft-target, Medusa, ReDrafter, Lookahead, and EAGLE(-1/-2, EAGLE-3 via two-model/disaggregated); vLLM supports EAGLE/EAGLE-3, Medusa, MTP, n-gram/prompt-lookup; SGLang supports EAGLE/EAGLE3 and standard draft",
      "best_value": null,
      "metric_unit": "count of distinct speculative methods supported",
      "source_url": "https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html",
      "source_date": "2025-12",
      "notes": "Method support and quality vary by model and backend (PyTorch vs TRT engine); EAGLE-3 same checkpoints are cross-engine portable between vLLM and SGLang.",
      "in_scope": true
    },
    {
      "id": "ngram-prompt-lookup",
      "category": "speculative-decoding",
      "dimension": "Retrieval / prompt-lookup (n-gram) drafting for input-grounded tasks",
      "why_it_matters": "A zero-draft-model, zero-extra-VRAM acceleration that works whenever the output echoes the input (summarization, RAG, code edit/refactor, structured rewrite). It is the cheapest spec-decode to deploy and a key differentiator for grounded agent workloads where outputs reuse prompt substrings.",
      "sota_systems": [
        "SuffixDecoding (Snowflake Arctic Inference + vLLM)",
        "vLLM PLD/n-gram (prior bar)",
        "TensorRT-LLM Lookahead"
      ],
      "sota_bar": "SuffixDecoding (NeurIPS 2025, model-free suffix-tree drafting) is the new prompt-lookup-family SOTA: outperforms classic PLD/n-gram by 1.3-3x and delivers up to 5.3x speculative speedup (and ~4x end-to-end on SWE-Bench agentic tasks) over vanilla decoding.",
      "best_value": 5.3,
      "metric_unit": "x speculative speedup over vanilla decoding (agentic; vs ~2.8x for classic PLD on summarization)",
      "source_url": "https://www.snowflake.com/en/engineering-blog/suffixdecoding-arctic-inference-vllm/",
      "source_date": "2025-05",
      "notes": "SOTA moved: old bar was classic prompt-lookup (PLD/n-gram) 2.8x on summarization (vLLM blog, 2024-10). SuffixDecoding (NeurIPS 2025 Spotlight, arXiv 2411.04975) generalizes prompt-lookup to suffix trees over prompt + prior outputs and beats PLD by 1.3-3x, reaching up to 5.3x speculative speedup and ~4x end-to-end on SWE-Bench agentic workloads, shipped in vLLM via Snowflake Arctic Inference. This is directly adjacent to fak's cross-agent prefix-reuse thesis (input-grounded, draft-model-free).",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "spec-decode-lossless-correctness",
      "category": "speculative-decoding",
      "dimension": "Output-distribution losslessness of speculation (greedy and stochastic)",
      "why_it_matters": "Speculative decoding must not change what the model would have produced; rejection-sampling verification preserves the target distribution exactly at T=0 and T>0. An engine that silently alters outputs (or only matches at greedy) fails reproducibility, eval parity, and audit requirements - this is the honesty gate operators must verify, not assume.",
      "sota_systems": [
        "EAGLE 3.1 (vLLM + TorchSpec)",
        "EAGLE-3",
        "SpecForge training framework"
      ],
      "sota_bar": "EAGLE 3.1 (vLLM/EAGLE/TorchSpec, May 2026) is the current SOTA lossless speculative-decoding method: up to 2x longer acceptance length and up to ~2.03x per-user output throughput vs EAGLE-3 (Kimi-K2.6-NVFP4, GB200), fixing 'attention drift' robustness; correctness is preserved by rejection-sampling verification that matches the target distribution (greedy spec == greedy non-spec).",
      "best_value": 2.03,
      "metric_unit": "x per-user output throughput speedup vs EAGLE-3 (concurrency 1), lossless",
      "source_url": "https://vllm.ai/blog/2026-05-26-eagle-3-1",
      "source_date": "2026-05",
      "notes": "Old bar dated 2025-06 (EAGLE-3 era). EAGLE-3 HAS a successor: EAGLE 3.1, announced jointly by the EAGLE, vLLM, and TorchSpec teams (May 2026), supported in vLLM. Headline: up to 2x longer acceptance length and 2.03x/1.71x/1.66x throughput at concurrency 1/4/16 over EAGLE-3 on Kimi-K2.6, primarily by fixing attention-drift degradation under long context / OOD prompts. Losslessness is the standing guarantee of spec decoding in vLLM: the rejection sampler is algorithmically validated to match the target distribution (lossless up to fp numerics). changed_vs_old=true (new method supersedes EAGLE-3). vLLM note below.",
      "in_scope": true
    },
    {
      "id": "spec-decode-batch-scaling",
      "category": "speculative-decoding",
      "dimension": "Speculative-decoding behavior under high concurrency / large batch (break-even)",
      "why_it_matters": "Spec decode helps latency at low load but can REGRESS throughput once the GPU is compute-bound - the single most common production footgun. Operators must know the break-even batch and whether the engine auto-disables or has a batch-robust drafter, or they pay for slowdowns at peak QPS.",
      "sota_systems": [
        "EAGLE-3 (SGLang)",
        "MagicDec (fixed-window draft, prior reference)",
        "vLLM/SGLang continuous batching"
      ],
      "sota_bar": "EAGLE-3's measured high-concurrency behavior: positive throughput uplift sustained to batch ~64 in SGLang (1.38x at batch 64), with the speedup approaching break-even beyond that as baseline throughput rises; EAGLE-2 already regresses (0.93x at batch 24).",
      "best_value": 64,
      "metric_unit": "batch size at which spec-decode uplift approaches break-even (1.38x at batch 64)",
      "source_url": "https://arxiv.org/abs/2503.01840",
      "source_date": "2025-03",
      "notes": "SOTA refreshed: old bar was MagicDec break-even framing at batch 16 (arXiv 2408.11049, 2024-08). The current, more concrete measured number is EAGLE-3 in SGLang sustaining 1.38x to batch 64 (vs EAGLE-2's 0.93x regression at batch 24), with the break-even/diminishing point now around batch 64+. The dimension's measurable bar moved from 16 to 64. This presupposes continuous batching, which fak does not have on its live path (decode lane serial by construction).",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "constrained-decode-per-token-overhead",
      "category": "structured-output",
      "dimension": "Constrained/guided decoding per-token overhead (JSON-schema, regex, grammar/CFG)",
      "why_it_matters": "Guaranteed-valid JSON/regex/grammar output is table stakes for tool-calling and structured APIs, but a slow mask-compute step throttles tokens/sec. The per-token mask overhead is the dominant cost in steady-state constrained generation and the number that decides whether structured output is 'free' or a tax.",
      "sota_systems": [
        "XGrammar-2 (vLLM/SGLang/TensorRT-LLM default backend)",
        "XGrammar",
        "LLGuidance (Rust Earley)"
      ],
      "sota_bar": "XGrammar-2 (arXiv 2601.04426, Jan 2026) drops grammar-mask per-token overhead to ~12-20us (Llama tool-calling ~12-13us, OpenAI Harmony ~15-20us) with <6% end-to-end overhead vs unconstrained and ~10ms (>100x faster) dynamic compilation; it is the new SOTA, far below LLGuidance's ~250us (Harmony) / >1000us (tool-calling) on the agentic workloads.",
      "best_value": 12,
      "metric_unit": "us per-token mask overhead (XGrammar-2, Llama tool-calling)",
      "source_url": "https://arxiv.org/html/2601.04426",
      "source_date": "2026-01",
      "notes": "Old bar was 'XGrammar 40us, 2025-03'. XGrammar-2 (Jan 2026) materially improves it: ~12-13us/token for Llama tool-calling and ~15-20us for OpenAI Harmony (down from XGrammar's ~40us baseline), <6% end-to-end overhead, and ~10ms dynamic compilation (>100x faster than XGrammar's >1000ms) -> 7x faster end-to-end function-calling. LLGuidance remains ~50us/token on simple JSON but balloons to ~250us (Harmony) / >1000us (tool-calling) on agentic schemas, so XGrammar-2 is SOTA for the agentic case. best_value=12 (low end of XGrammar-2 per-token overhead). vLLM uses XGrammar as default structured-gen backend. changed_vs_old=true.",
      "in_scope": true
    },
    {
      "id": "grammar-compilation-latency",
      "category": "structured-output",
      "dimension": "Grammar/schema compilation latency and cache (cold-start for dynamic agentic schemas)",
      "why_it_matters": "Agentic workloads change the schema/grammar per request (different tool sets, varying response protocols), so the one-time compile cost - not per-token mask - becomes the bottleneck. Compilation latency and cross-grammar caching determine whether constrained decoding is viable for high-churn tool-calling.",
      "sota_systems": [
        "XGrammar-2",
        "XGrammar",
        "LLGuidance/Guidance"
      ],
      "sota_bar": "XGrammar-2 compiles in ~10 ms vs >1000 ms for XGrammar (~80x faster), with >6x tool-calling compilation speedup via Earley-based adaptive mask cache + cross-grammar cache + JIT; integrated into vLLM and SGLang",
      "best_value": 10,
      "metric_unit": "milliseconds to compile a grammar/schema (cold)",
      "source_url": "https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation",
      "source_date": "2026-05",
      "notes": "XGrammar-2 also in arXiv 2601.04426 and ACM 3786335.3813124. Original XGrammar 0.12-0.30s, Guidance <60ms; cache hit rate on shared substructures is the operator-relevant secondary metric.",
      "in_scope": true
    },
    {
      "id": "structured-output-validity-vs-quality",
      "category": "structured-output",
      "dimension": "Structured-output validity vs reasoning-quality tax (constraint-induced degradation)",
      "why_it_matters": "Constrained decoding can guarantee 100% schema-valid output yet DEGRADE task accuracy by forcing answer fields before chain-of-thought. Buyers must weigh validity rate against the reasoning hit; this is the dimension a naive 'we always emit valid JSON' scorecard hides.",
      "sota_systems": [
        "XGrammar (vLLM/SGLang/TRT-LLM default)",
        "LLGuidance (guidance-ai)",
        "OpenAI Structured Outputs"
      ],
      "sota_bar": "Constrained decoding guarantees ~100% syntactic/schema validity (JSON-Pass/Type-Safety >.95) but does NOT guarantee semantic quality and can REDUCE accuracy: ExtractBench (2026) shows structured mode dropping overall validity 51%->37% and GPT-5 credit-agreement pass rate 86.9%->70.0% vs prompt-based extraction. The honest SOTA bar is 'validity is solved; value accuracy is the open gap (13.7pt spread across models).'",
      "best_value": 70,
      "metric_unit": "% pass rate under structured mode (GPT-5, credit agreements) vs 86.9% unconstrained",
      "source_url": "https://arxiv.org/pdf/2602.12247",
      "source_date": "2026-02",
      "notes": "Old bar dated 2025-01. New 2026 evidence materially sharpens the bar from 'structured output ~ valid' to 'valid != correct, and constraints can hurt accuracy.' ExtractBench (arXiv 2602.12247) reports structured mode lowering validity 51->37% and GPT-5 pass 86.9->70.0%; the Structured Output Benchmark (arXiv 2604.25359) shows structural metrics >.95 but value-accuracy spanning 0.693-0.830 (13.7pt). This builds on Park et al. NeurIPS-2024 distribution-distortion. best_value=70.0 (the degraded GPT-5 structured pass rate, the load-bearing cautionary number). changed_vs_old=true: the field reframed the metric (validity vs quality split) rather than just nudging a number. [second-source verified]",
      "in_scope": true
    },
    {
      "id": "tool-call-parse-robustness",
      "category": "structured-output",
      "dimension": "Tool/function-call parsing coverage and robustness (per-model formats)",
      "why_it_matters": "Agents live or die on correctly extracting structured tool calls from model output; each model family uses a different format (Hermes, llama3_json, mistral, pythonic, qwen3_coder XML). Wrong/absent parser yields malformed args or silent empty calls - so parser coverage and graceful failure are a core agent-serving differentiator.",
      "sota_systems": [
        "vLLM tool-call parsers",
        "SGLang",
        "XGrammar-2 (grammar-backed tool calling)"
      ],
      "sota_bar": "vLLM ships per-family parsers (hermes, llama3_json, mistral, pythonic, internlm, granite, plus qwen3/qwen3_coder); when constraints aren't enforced it falls back to extracting from raw text, so args can be malformed - making grammar-backed (constrained) tool calling the robustness frontier",
      "best_value": null,
      "metric_unit": "count of model-family parsers + valid-call rate / malformed-arg rate",
      "source_url": "https://docs.vllm.ai/en/latest/features/tool_calling/",
      "source_date": "2025-12",
      "notes": "Known model-specific failure modes (Mistral-7B parallel calls, Hermes-2-Theta quality). Pairing the parser with constrained decoding (XGrammar-2) removes the raw-text-extraction failure path.",
      "in_scope": true
    },
    {
      "id": "sampler-coverage-correctness",
      "category": "structured-output",
      "dimension": "Sampler coverage and correctness (temperature, top-p, top-k, min-p)",
      "why_it_matters": "Sampling controls the quality/diversity/coherence tradeoff and must be implemented correctly and identically across engines for eval parity. min-p in particular keeps outputs coherent at high temperature and is now an ecosystem default; missing or buggy samplers force operators to over-constrain temperature and lose quality.",
      "sota_systems": [
        "vLLM SamplingParams + logits processors",
        "llama.cpp sampler chain",
        "SGLang sampler"
      ],
      "sota_bar": "Reference samplers cover the full mainstream set: temperature, top_p, top_k, min_p, typical, presence/frequency/repetition penalties, logit_bias, seed/n/best_of, plus pluggable custom logits processors (argmax-invariant classification). vLLM and llama.cpp converge on Temperature+Min-P as the 2026 default power-user combo.",
      "best_value": null,
      "metric_unit": "qualitative (sampler/parameter coverage + correctness)",
      "source_url": "https://docs.vllm.ai/en/latest/design/logits_processors/",
      "source_date": "2026-01",
      "notes": "Old bar dated 2024-07 and is qualitative (no number). Coverage is essentially saturated and unchanged in kind: vLLM exposes temp/top_p/top_k/min_p/penalties/logit_bias plus a custom-logits-processor API (min_p is implemented as an argmax-invariant processor). The notable 2025-2026 movement is community convergence on Temperature+Min-P as the default and a documented vLLM nuance: under speculative decoding vLLM historically sampled greedily from the draft regardless of temperature (a correctness caveat some 2026 work patches for true rejection sampling at non-zero temp). Bar substance unchanged -> changed_vs_old=false, date refreshed to 2026-01. best_value null (qualitative).",
      "in_scope": true
    },
    {
      "id": "expert-parallelism-large-moe",
      "category": "parallelism",
      "dimension": "Large-scale expert parallelism (EP) for giant MoE models",
      "why_it_matters": "Sparse MoE models (DeepSeek-V3/R1 671B, Qwen MoE, Mixtral) activate only a few experts per token, so naive tensor parallelism replicates expert weights and wastes memory bandwidth. Scaling EP across dozens of GPUs (one or few experts per device) is the only way to serve 600B+ MoE economically, and demands all-to-all expert dispatch, expert-load balancing, and overlap of comms with compute. EP scale and balance is the differentiating capability for frontier-MoE serving.",
      "sota_systems": [
        "Huawei CloudMatrix-Infer (CloudMatrix384, DeepSeek-R1)",
        "SGLang + DeepEP (96-GPU, prior bar)",
        "vLLM expert-parallel deployment"
      ],
      "sota_bar": "EP320: Huawei CloudMatrix-Infer runs DeepSeek-R1 with 320-way expert parallelism (one expert per NPU die: 32 shared + 256 router + 32 redundant experts) over the UB network, superseding the 96-GPU large-scale EP (effective EP~72) demonstrated by LMSYS/SGLang.",
      "best_value": 320,
      "metric_unit": "EP degree (expert-parallel ways in one served MoE instance)",
      "source_url": "https://arxiv.org/abs/2506.12708",
      "source_date": "2025-06",
      "notes": "SOTA moved up: old bar was 72 (LMSYS/SGLang large-scale EP on 96 H100s, 2025-05-05). CloudMatrix-Infer (arXiv 2506.12708, June 2025) demonstrates EP320 for DeepSeek-R1 on Huawei CloudMatrix384 -- each NPU die hosts exactly one expert, with PD+caching disaggregation over the UB network. This is the current largest published expert-parallel degree for a giant MoE. (SGLang's 96-GPU run remains the leading NVIDIA-GPU EP demonstration.) Out of scope for fak's single-box engine, which runs GLM-5.2 MoE experts on one device.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "data-parallel-attention",
      "category": "parallelism",
      "dimension": "Data-parallel (DP) attention / hybrid attention-FFN parallelism",
      "why_it_matters": "When EP fans the FFN out across many GPUs, replicating the KV cache under tensor parallelism for attention duplicates memory and caps batch size. DP attention runs attention data-parallel (each rank owns distinct sequences, no KV duplication) while the MoE FFN stays expert-parallel. This is what lets a deployment hold large batches and long contexts at high EP without exploding KV memory, and it is a distinct, non-obvious axis from raw TP/PP/EP degrees.",
      "sota_systems": [
        "SGLang",
        "vLLM"
      ],
      "sota_bar": "SGLang's DP Attention is the production technique that eliminates KV-cache duplication across devices for DeepSeek-V3, enabling the EP32/EP72 deployment to scale memory-efficiently; combined with PD disaggregation and MTP it delivers up to ~60% higher output throughput.",
      "best_value": null,
      "metric_unit": "qualitative (KV-duplication eliminated; ~60% output-throughput uplift in combined stack)",
      "source_url": "https://www.lmsys.org/blog/2025-07-17-mtp/",
      "source_date": "2025-07-17",
      "notes": "The ~60% figure is the combined MTP+EP+PD uplift, not DP attention alone. DP attention is the enabling memory technique.",
      "in_scope": true
    },
    {
      "id": "multi-node-scale-out",
      "category": "parallelism",
      "dimension": "Multi-node scale-out (combined TP x PP x EP x DP across hosts)",
      "why_it_matters": "Serving a frontier model means composing tensor, pipeline, expert, and data parallelism across many hosts and an InfiniBand/RoCE fabric. The relevant operator question is not 'does it run on 8 GPUs' but how many GPUs/nodes a single coherent deployment spans, what interconnect it requires, and how cleanly the parallelism dimensions compose. Datacenter-scale orchestration (rack-scale NVLink domains, cross-node fabric) is the table-stakes axis for frontier serving.",
      "sota_systems": [
        "NVIDIA GB300 NVL72 x4 (288 GPUs over Quantum-X800 InfiniBand, MLPerf v6.0)",
        "SGLang/TensorRT-LLM + NVIDIA Dynamo multi-node",
        "GB200 NVL72 (prior generation)"
      ],
      "sota_bar": "A single GB300 NVL72 is 72 Blackwell-Ultra GPUs in one NVLink domain; MLPerf Inference v6.0 (2026-03-30) composed FOUR of them -- 288 GPUs over Quantum-X800 InfiniBand, NVIDIA's largest-ever submission -- into one coherent DeepSeek-R1 671B MoE serving run at ~2.494M tok/s aggregate offline. Disaggregated serving via NVIDIA Dynamo composes the TPxPPxEPxDP dimensions across the fabric (up to 1.5x on Llama-3.1-405B interactive).",
      "best_value": 72,
      "metric_unit": "GPUs in one coherent NVLink domain (GB300 NVL72; v6.0 scaled to 4 domains / 288 GPUs over InfiniBand)",
      "source_url": "https://www.storagereview.com/news/nvidia-sets-mlperf-inference-v6-0-records-with-blackwell-ultra-platform",
      "source_date": "2026-04-01",
      "notes": "Refreshed to MLPerf Inference v6.0 (results 2026-03-30 / published 2026-04-01), which supersedes the v5.1 debut bar this row previously cited. The single-NVLink-domain count is unchanged (72 GPUs / GB300 NVL72), but the current multi-node SOTA datapoint is v6.0's 288-GPU (4x GB300 NVL72 over Quantum-X800 InfiniBand) DeepSeek-R1 671B submission at ~2.494M tok/s aggregate offline -- see absolute-tokens-per-second-mlperf for the aggregate number and expert-parallelism-large-moe for the EP dimension. The earlier 'no v6.x round found' note was stale: v6.0 is the current round and DeepSeek-R1 671B MoE is now its headline multi-node workload.",
      "in_scope": true
    },
    {
      "id": "max-model-and-moe-coverage",
      "category": "model-coverage",
      "dimension": "Maximum parameter count & frontier-MoE architecture coverage",
      "why_it_matters": "The first capability question for any serving stack is which models it can actually run. The hard end is the 671B-class sparse MoE (DeepSeek-V3/R1) plus Qwen MoE and Mixtral; supporting these requires MoE kernels, EP, FP8 weights, and MLA/MTP support, not just more memory. A stack that tops out at dense 70B is in a different market than one that serves 671B MoE at scale.",
      "sota_systems": [
        "Kimi K2 / K2.x (Moonshot AI, 1T MoE; vLLM + SGLang + KTransformers)",
        "DeepSeek-V3/R1 671B (prior bar)",
        "Qwen3 MoE / Llama-4 (other frontier MoE)"
      ],
      "sota_bar": "Kimi K2 (Moonshot AI) -- 1 trillion total parameters / 32B active, 384 experts (8+1 active) -- is the largest open-weight MoE routinely served by vLLM and SGLang, exceeding DeepSeek-V3/R1's 671B.",
      "best_value": 1000,
      "metric_unit": "B total params (largest open-weight MoE served by vLLM/SGLang)",
      "source_url": "https://huggingface.co/moonshotai/Kimi-K2-Instruct",
      "source_date": "2025-07",
      "notes": "SOTA moved: old bar was DeepSeek-V3 671B (LMSYS, 2025-05-05). Kimi K2 (released mid-2025, 1T total / 32B active, 384 experts) is now the largest open-weight MoE with official vLLM and SGLang serving support, surpassing 671B. (Note: this scorecard's fak-side row uses '72B on own engine' framing for llama.cpp CPU; the dim is best read as the field's max-served frontier-MoE, where the bar is now 1T via Kimi K2.) best_value 1000 = 1T total parameters.",
      "in_scope": true
    },
    {
      "id": "multimodal-vlm-coverage",
      "category": "model-coverage",
      "dimension": "Multimodal / vision-language (VLM) model coverage",
      "why_it_matters": "A growing share of production traffic is image/video/audio + text (VLMs like Qwen-VL, Llama Vision, InternVL, Pixtral). Serving these adds a vision encoder, modality-aware batching, and image-token KV/prefix handling on top of the LLM path. Whether a stack treats VLMs as first-class (and reuses vision embeddings across requests) is a real differentiator versus text-only engines.",
      "sota_systems": [
        "SGLang",
        "vLLM"
      ],
      "sota_bar": "SGLang is explicitly a serving framework 'for large language models and multimodal models' and extends RadixAttention prefix caching to multimodal inputs with vision-embedding reuse; vLLM also provides broad multimodal model support. Coverage spans the major open VLM families (Qwen-VL, Llama Vision, InternVL, Pixtral, etc.).",
      "best_value": null,
      "metric_unit": "qualitative (first-class VLM support + vision-embedding prefix reuse)",
      "source_url": "https://github.com/sgl-project/sglang",
      "source_date": "2026-01-01",
      "notes": "VLM long-context prefill is a distinct cost (see MMInference, arXiv 2504.16083). Exact per-model coverage lists are version-dependent.",
      "in_scope": true
    },
    {
      "id": "long-context-capacity",
      "category": "model-coverage",
      "dimension": "Long-context serving (128K-1M tokens) & KV/prefix-cache reuse",
      "why_it_matters": "Context windows have moved from 4K to 128K-1M. Serving long context is gated by KV-cache memory (linear in tokens), prefill cost (quadratic attention), and cache reuse. The operator-relevant axes are the max context the engine can hold per request, KV quantization / paging to fit it, and prefix-cache hit rates that recover the prefill cost on RAG / multi-turn / agent workloads.",
      "sota_systems": [
        "SGLang (RadixAttention)",
        "vLLM (PagedAttention/prefix caching)"
      ],
      "sota_bar": "Million-token windows are production-ready (Qwen2.5-1M open weights; 128K-1M is mainstream). SGLang's RadixAttention prefix caching delivers up to ~6.4x gains on prefix-heavy RAG/multi-turn workloads and a ~29% throughput edge over vLLM on H100 (16.2K vs 12.5K tok/s) on shared-prefix traffic.",
      "best_value": 6.4,
      "metric_unit": "x throughput gain on prefix-heavy workloads (RadixAttention)",
      "source_url": "https://particula.tech/blog/sglang-vs-vllm-inference-engine-comparison",
      "source_date": "2026-01-01",
      "notes": "1M-token open-weight = Qwen2.5-1M. The 6.4x / 29% figures are workload-dependent benchmark blog numbers, not a standardized result; treat as indicative.",
      "in_scope": true
    },
    {
      "id": "lora-multi-adapter-serving",
      "category": "model-coverage",
      "dimension": "LoRA / multi-LoRA hot-swap serving",
      "why_it_matters": "Multi-tenant and per-customer customization is delivered cheaply by serving many LoRA adapters over one shared base model instead of one full model per tenant. The differentiating axes are how many adapters can be resident on GPU concurrently, how many can be cached in CPU (LRU), and whether adapters can be hot-swapped/updated at runtime without restarting (critical for async-RL weight updates). This is the economics of fine-tune-once-serve-many.",
      "sota_systems": [
        "S-LoRA (MLSys'24, ~2000 adapters/A100-80GB)",
        "vLLM Multi-LoRA",
        "SGLang",
        "TensorRT-LLM"
      ],
      "sota_bar": "The peer-reviewed SOTA is S-LoRA (MLSys'24, arXiv:2311.03285): ~2000 LoRA adapters served over one base model on a single A100-80GB at up to 4x vLLM throughput, via unified paging of adapter weights + custom MBGMM/MBGMV kernels. vLLM Multi-LoRA integrated S-LoRA-style techniques as a follow-on (max_loras GPU-resident + max_cpu_loras LRU CPU cache + runtime hot-swap); SGLang/TensorRT-LLM offer comparable multi-adapter batching. Practical resident count is bounded by GPU memory and adapter rank.",
      "best_value": 2000,
      "metric_unit": "LoRA adapters served over one base model (single A100-80GB, S-LoRA)",
      "source_url": "https://www.lmsys.org/blog/2023-11-15-slora/",
      "source_date": "2025-08",
      "notes": "Refreshed from the generic vLLM LoRA doc (2025-01-01) to a concrete 2025 figure: ~200 concurrent adapters on one A100 with <5% overhead, dynamic load-from-CPU/disk + LRU eviction. Confidence medium because the 200-adapter number is from a vendor/SEO blog summarizing vLLM's capability rather than a primary vLLM doc page; the underlying LoRAManager dynamic-swap mechanism is documented in vLLM's official LoRA docs. 2025 research (InfiniLoRA, disaggregated multi-LoRA) pushes further but is not yet a production default. Out of scope for fak (no LoRA path). [VERIFIER OVERRIDE -> 2000: NOT CORROBORATED. The specific \"200 concurrent LoRA adapters on a single A100 with <5% latency overhead\" figure is single-sourced to the cited Spheron vendor blog. Every other search hit asserting it either IS that blog ]",
      "in_scope": true
    },
    {
      "id": "embeddings-rerankers-coverage",
      "category": "model-coverage",
      "dimension": "Embedding & reranker (non-generative) model coverage",
      "why_it_matters": "RAG and search stacks need an engine that also serves embedding and cross-encoder reranker models, ideally on the same serving fabric as the generative LLM (shared pooling, batching, an OpenAI-compatible /v1/embeddings + rerank API). A stack that only does autoregressive decode forces operators to run a second serving system for retrieval, so embedding/reranker coverage is a real platform-completeness axis.",
      "sota_systems": [
        "vLLM",
        "SGLang",
        "TensorRT-LLM"
      ],
      "sota_bar": "vLLM and SGLang both serve embedding and reranker/cross-encoder models alongside generative models with pooling support and OpenAI-compatible embeddings endpoints, covering the common open embedding/reranker families (BGE, E5, GTE, Qwen-embedding, etc.) on the same engine.",
      "best_value": null,
      "metric_unit": "qualitative (embeddings + rerank served on the same engine)",
      "source_url": "https://docs.vllm.ai/en/latest/",
      "source_date": "2026-01-01",
      "notes": "Pooling/embedding/score tasks are documented runner types in vLLM; SGLang lists multimodal + embedding support. Exact model lists are version-dependent.",
      "in_scope": true
    },
    {
      "id": "hardware-backend-breadth",
      "category": "hardware-coverage",
      "dimension": "Hardware backend breadth (NVIDIA, AMD, Intel, TPU, AWS, Apple, CPU)",
      "why_it_matters": "Vendor lock-in to a single accelerator is a top procurement risk. The buyer axis is how many silicon families a stack runs on with maintained, performant paths: NVIDIA Hopper/Blackwell, AMD ROCm (MI300X/MI355X), Intel Gaudi/XPU, Google TPU, AWS Inferentia/Trainium, Apple Metal, and CPU. Breadth determines portability and negotiating leverage; depth (kernel maturity per backend) determines whether the non-NVIDIA path is actually usable in production.",
      "sota_systems": [
        "vLLM",
        "llama.cpp",
        "SGLang",
        "MLC-LLM"
      ],
      "sota_bar": "vLLM runs on NVIDIA, AMD ROCm (MI300X/MI325X/MI350X/MI355X), x86/ARM/PowerPC CPU, plus plugin backends for Google TPU, Intel Gaudi 2/3, AWS Inferentia/Trainium (Neuron), Apple Silicon, Huawei Ascend, IBM Spyre/Z, and more; llama.cpp targets the widest portability surface (CUDA, HIP/ROCm, Metal, Vulkan, SYCL, CANN, OpenCL, CPU SIMD on x86 AVX-512/AMX and ARM Neon/SVE/SME). MLC-LLM uniquely adds WebGPU/browser via TVM compilation.",
      "best_value": null,
      "metric_unit": "count/qualitative (vLLM ~10+ accelerator families; llama.cpp ~10+ compute backends)",
      "source_url": "https://github.com/vllm-project/vllm",
      "source_date": "2026-01-01",
      "notes": "Some non-NVIDIA paths are Tech-Preview (e.g. Trainium/Inferentia in Red Hat AI Inference). Breadth != per-backend performance parity.",
      "in_scope": true
    },
    {
      "id": "quantization-datatype-coverage",
      "category": "quantization",
      "dimension": "Quantization format & low-precision datatype coverage",
      "why_it_matters": "Low precision is how a frontier model fits and runs fast: FP8/FP4 on tensor cores, plus weight-only INT4 (AWQ/GPTQ) and KV-cache quantization. The relevant axes are which formats the engine supports, which the target hardware accelerates natively, and how much memory/throughput each buys at what accuracy. Quantization coverage tied to hardware generation (e.g. NVFP4 on Blackwell, FP8 on Hopper/MI300X) directly sets the cost-per-token frontier.",
      "sota_systems": [
        "vLLM quantization (LLM Compressor, Marlin, AutoAWQ, GPTQModel, bitsandbytes, Quark)",
        "TensorRT Model Optimizer (FP8/INT4/NVFP4)"
      ],
      "sota_bar": "Reference serving stacks now cover the full ladder: vLLM (2026) supports FP8 (W8A8), INT8 (W8A8 + W4A8), INT4 (W4A16), NVFP4 and MXFP4 (via Marlin), AWQ, GPTQ, BitsAndBytes (4/8-bit), AMD Quark, and online runtime quantization, across Turing->Hopper->Blackwell plus x86/Arm CPU for INT8.",
      "best_value": 10,
      "metric_unit": "distinct quantization methods/datatypes supported in vLLM",
      "source_url": "https://docs.vllm.ai/en/latest/features/quantization/",
      "source_date": "2026-01",
      "notes": "Old bar dated 2025-01. Coverage materially broadened since: NVFP4 and MXFP4 are now first-class in vLLM (Marlin path) on top of FP8/INT8/INT4/AWQ/GPTQ/bnb/Quark, plus a new W4A8 INT variant and online quantization. best_value=10 counts the distinct method families currently listed on the vLLM quantization index. This dimension is fak-vs-vLLM positioning: vLLM is the breadth leader on datatype coverage; the count moved up (NVFP4/MXFP4/W4A8 added), so changed_vs_old=true. Note the Blackwell wrinkle: INT8 W8A8 is dropped on compute capability >=10.0, so the supported SET is hardware-conditional.",
      "in_scope": true
    },
    {
      "id": "fault-tolerance-resilience",
      "category": "operability",
      "dimension": "Multi-node fault tolerance & failure recovery (resilient serving)",
      "why_it_matters": "At 96+ GPU scale a single GPU/node failure is a when, not an if, and tightly-coupled TP/EP means one failed rank can halt the whole instance. Operators evaluating large disaggregated/EP deployments need to know mean-time-to-recovery, whether the system degrades gracefully (reroute traffic, replicate KV, take only a failed replica offline) versus full restart. Resilience is a fast-emerging differentiator as deployments cross the single-node boundary.",
      "sota_systems": [
        "FailSafe (research)",
        "KevlarFlow (research)",
        "ReviveMoE (research)",
        "NVIDIA Dynamo"
      ],
      "sota_bar": "Research systems define the bar: KevlarFlow reports ~20x lower mean-time-to-recovery via decoupled parallel init, dynamic traffic rerouting, and background KV-cache replication; FailSafe sustains high-performance tensor-parallel serving under irregular GPU availability; ReviveMoE targets fast recovery from hardware failures specifically in large-scale MoE inference. Production stacks (Dynamo) add dynamic GPU scheduling and rerouting but full resilient serving is still maturing.",
      "best_value": 20,
      "metric_unit": "x reduction in mean-time-to-recovery (KevlarFlow, research)",
      "source_url": "https://arxiv.org/html/2601.22438v1",
      "source_date": "2026-01-01",
      "notes": "20x MTTR is a single-paper research claim, not a productionized standardized number; included to mark resilience as an emerging operator axis.",
      "in_scope": true
    },
    {
      "id": "cross-request-prefix-cache-reuse",
      "category": "kv-cache",
      "dimension": "Automatic cross-request prefix / KV-cache reuse (shared system prompts, agent scaffolds, few-shot)",
      "metric_unit": "x throughput improvement from cross-request prefix KV reuse",
      "best_value": 6.4,
      "sota_bar": "RadixAttention's automatic cross-request KV reuse via an LRU radix tree gives up to 6.4x throughput on shared-prefix workloads with 50%-99% cache hit rates; this remains the canonical cross-request prefix-reuse bar in 2026.",
      "sota_systems": [
        "SGLang RadixAttention",
        "vLLM Automatic Prefix Caching",
        "Mooncake KV pool"
      ],
      "source_url": "https://www.lmsys.org/blog/2024-01-17-sglang/",
      "source_date": "2024-01",
      "why_it_matters": "Agent and fleet workloads repeat large identical prefixes (system prompts, tool schemas, few-shot exemplars). Reusing their KV across requests is the single largest lever on prefill cost and TTFT, and is now table-stakes; buyers compare hit-rate and the throughput gain it delivers.",
      "notes": "No newer source supersedes the 6.4x cross-request prefix-reuse headline; the LMSYS RadixAttention announcement (2024-01-17) remains the cited bar and 2026 sources still quote 'up to 6.4x on prefix-heavy workloads' verbatim. Old bar stands. The 2026 refinement is breadth of measured hit rates (50%->near-99% across benchmarks) and the move to multi-tier/disaggregated cross-request pools (Mooncake serving Kimi across thousands of nodes, integrated as a vLLM v1 KV connector and into TensorRT-LLM), which extend cross-request reuse across nodes rather than raising the single-node multiplier.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "kv-aware-routing-locality",
      "category": "request-routing",
      "dimension": "Cache-aware request routing across replicas (route to the worker that already holds the prefix)",
      "metric_unit": "x median TTFT reduction vs least-load (224.55ms vs 568.27ms)",
      "best_value": 2.5,
      "sota_bar": "Cross-region KV-locality-aware routing (GORGO-proxy) makes network latency an explicit routing term and cuts median TTFT ~2.5x vs least-load / prefix-trie baselines (224ms vs 568ms) while preserving prefix-cache locality across regions.",
      "sota_systems": [
        "GORGO / GORGO-proxy (cross-region, arXiv 2602.11688)",
        "SkyWalker (locality-aware cross-region LB, EuroSys 2026, arXiv 2505.24095)"
      ],
      "source_url": "https://arxiv.org/html/2602.11688v1",
      "source_date": "2026-02",
      "why_it_matters": "At fleet scale a naive load balancer scatters requests so the same prefix is recomputed on every replica. Locality-aware routing turns single-node prefix caching into a fleet-wide property, the difference between a cache that works at 1 GPU and one that works at 100.",
      "notes": "The locality dimension moved from single-cluster cache-aware routing (old 2025-03-18 framing) to cross-region locality-aware load balancing. GORGO (arXiv 2602.11688, 2026-02-11): GORGO-proxy gives 2.5x median TTFT reduction vs least-load, 2.5x vs prefix-trie, 2.4x vs distributed GORGO (224.55ms vs 539-568ms), and ~2.9x higher RPS (1.89 vs 0.65) by jointly optimizing network latency + prefix locality + continuous-batching admission state in a centralized router. SkyWalker (EuroSys 2026, arXiv 2505.24095) reserves on global rather than per-region peak demand while preserving KV-cache locality. The key insight: a centralized router that treats network latency as an explicit objective term beats both pure prefix-trie and pure least-load. Cache-reuse is folded into an additive cost model rather than reported as a standalone % in GORGO.",
      "in_scope": true
    },
    {
      "id": "peak-serving-throughput-per-gpu",
      "category": "throughput",
      "dimension": "Peak served throughput per accelerator (model-served-by-engine, MLPerf-grounded)",
      "metric_unit": "tokens/sec per GPU (offline, DeepSeek-R1 671B, MLPerf v6.0; per-rack aggregate cross-referenced)",
      "best_value": 9821,
      "sota_bar": "MLPerf Inference v6.0 (2026-03-30) is the current round and the audited per-accelerator peak: DeepSeek-R1 671B on GB300 NVL72 reaches 9,821 tok/s/GPU offline (8,064 tok/s/GPU accuracy+latency-constrained server); the full 288-GPU (4x GB300 NVL72) submission aggregates ~2.494M tok/s (see absolute-tokens-per-second-mlperf). The audited single-H200 Llama-2-70B peak of ~33,000 tok/s (40% over H100) still stands for that older/smaller workload. The prior v5.1/v5.0 ~1.1M tok/s/rack Azure GB300 figure was an unverified submission and is superseded.",
      "sota_systems": [
        "NVIDIA GB300 NVL72 (DeepSeek-R1 671B, MLPerf v6.0, TensorRT-LLM/Dynamo)",
        "NVIDIA H200 (Llama-2-70B audited)",
        "MangoBoost/AMD MI300X"
      ],
      "source_url": "https://www.storagereview.com/news/nvidia-sets-mlperf-inference-v6-0-records-with-blackwell-ultra-platform",
      "source_date": "2026-04-01",
      "why_it_matters": "Throughput per GPU is the denominator of cost-per-token and the headline capacity number every operator sizes a fleet against. MLPerf is the only neutral, audited cross-vendor source, so it anchors the SOTA bar buyers actually trust over vendor blogs.",
      "notes": "Refreshed from the v5.1/v5.0 framing to MLPerf Inference v6.0 (results 2026-03-30 / published 2026-04-01), the current round, whose headline model is DeepSeek-R1 671B MoE. The audited per-GPU peak is now 9,821 tok/s/GPU offline / 8,064 server on GB300 NVL72 (DeepSeek-R1); the prior ~1.1M tok/s/rack Azure figure was flagged unverified and is superseded by v6.0's audited 2.494M tok/s 288-GPU aggregate (absolute-tokens-per-second-mlperf). The single-H200 Llama-2-70B ~33,000 tok/s remains the audited datapoint for that workload. best_value is now the audited per-GPU offline figure, matching this dimension's per-accelerator intent.",
      "in_scope": true
    },
    {
      "id": "cost-per-million-tokens",
      "category": "cost-efficiency",
      "dimension": "Inference unit economics ($ per 1M tokens) at realistic utilization",
      "metric_unit": "USD per 1M tokens; GPU utilization %",
      "best_value": 0.4,
      "sota_bar": "GPT-4-equivalent quality now serves at ~$0.40 / 1M tokens vs ~$20 in late 2022 (\u00e2\u2030\u02c610x/yr decline). Real production GPU utilization is only 30-60%, so realistic cost-per-token runs 2-3x above spreadsheet estimates; FP8/FP4 quantization roughly halves cost-per-token on H100/H200. Self-hosting a 7B breaks even only above ~50% utilization.",
      "sota_systems": [
        "Hosted frontier APIs",
        "vLLM/SGLang self-host on H100/H200",
        "FP8/FP4 quantization"
      ],
      "source_url": "https://introl.com/blog/inference-unit-economics-true-cost-per-million-tokens-guide",
      "source_date": "2025-09-01",
      "why_it_matters": "Cost-per-million-tokens at the utilization you can actually sustain is the metric finance signs off on. It folds throughput, batching, quantization, and utilization into one number and exposes the gap between marketing throughput and the bill.",
      "notes": "Number is market-level, not a single audited bench; utilization caveat is the load-bearing part.",
      "in_scope": true
    },
    {
      "id": "energy-tokens-per-watt",
      "category": "cost-efficiency",
      "dimension": "Energy efficiency (Wh per token / tokens-per-watt) and power-capped throughput",
      "metric_unit": "Wh per output token",
      "best_value": null,
      "sota_bar": "Empirical inference energy is ~0.0001-0.002 Wh per output token depending on model size; a ~500-token GPT-4o-class query is ~0.3 Wh (~3e-4 Wh/token). FP4/FP8 and Blackwell-class hardware are the main levers. No neutral audited tokens-per-watt leaderboard yet exists, so the bar is a measured range, not a single SOTA number.",
      "sota_systems": [
        "NVIDIA Blackwell (GB200/GB300, FP4)",
        "AMD MI355X (FP4)"
      ],
      "source_url": "https://arxiv.org/pdf/2603.06630",
      "source_date": "2026-03-01",
      "why_it_matters": "Power, not GPUs, is becoming the binding constraint on fleet scale; tokens-per-watt sets the ceiling on how much serving a datacenter can host and increasingly drives both cost and siting decisions. Buyers with power budgets evaluate on it directly.",
      "notes": "best_value null: ranges only, no audited per-watt leaderboard. MLPerf Power offers a partial neutral source.",
      "in_scope": true
    },
    {
      "id": "openai-compatible-api-and-structured-output",
      "category": "operability",
      "dimension": "OpenAI-compatible API surface + reliable structured output / tool-calling",
      "metric_unit": "% schema-valid output (guaranteed by constrained decoding when a structured response is produced)",
      "best_value": 100,
      "sota_bar": "Constrained/guided decoding now guarantees 100% schema-valid structured output by construction (XGrammar-2 is the default engine across vLLM/SGLang/TensorRT-LLM, <40 us/token), with the OpenAI-compatible /v1/chat/completions + response_format json_schema surface as the de facto contract.",
      "sota_systems": [
        "vLLM + XGrammar-2 (default)",
        "SGLang",
        "TensorRT-LLM",
        "OpenAI Structured Outputs (response_format)"
      ],
      "source_url": "https://arxiv.org/pdf/2601.04426",
      "source_date": "2026-01",
      "why_it_matters": "Agents depend on machine-parseable tool calls; a malformed JSON object breaks the loop. OpenAI-compatibility plus near-guaranteed schema conformance is what lets an operator swap engines/models without rewriting clients, a primary procurement criterion.",
      "notes": "SOTA moved: old bar was 98.2% JSON conformance (squeezebits guided-decoding, 2025-01). The current framing is 100% structural schema validity BY CONSTRUCTION via constrained decoding -- XGrammar-2 (arXiv 2601.04426, the default backend across vLLM/SGLang/TensorRT-LLM) achieves 100% schema-valid tool-call arguments whenever a call is produced, at <40 us/token. Caveat preserved: 100% is structural validity, not extraction accuracy -- constrained decoding can still reduce reasoning quality (e.g. GPT-5 credit-agreement pass rate 86.9%->70.0%). best_value 100 reflects the by-construction structural guarantee, the lever vLLM/SGLang cite.",
      "in_scope": true
    },
    {
      "id": "observability-metrics-tracing",
      "category": "operability",
      "dimension": "Production observability: Prometheus metrics (TTFT/TPOT/queue) + OpenTelemetry tracing",
      "metric_unit": "metric-set coverage (TTFT/TPOT/queue/KV-cache/prefix-hit Prometheus series + OTel traces)",
      "best_value": null,
      "sota_bar": "vLLM v1 exposes an extensive Prometheus /metrics surface (vllm:time_to_first_token_seconds, TPOT/inter-token, vllm:num_requests_running, vllm:kv_cache_usage_perc, prefix-cache hit) plus native OpenTelemetry distributed tracing via --otlp-traces-endpoint -- the de facto serving-observability contract.",
      "sota_systems": [
        "vLLM v1 (Prometheus + OpenTelemetry)",
        "SGLang",
        "NVIDIA Dynamo / GenAI-Perf"
      ],
      "source_url": "https://docs.vllm.ai/en/latest/design/metrics/",
      "source_date": "2026-01",
      "why_it_matters": "You cannot meet an SLO you cannot see. Per-request TTFT/TPOT histograms and KV/cache-hit telemetry are what drive autoscaling, capacity planning, and incident response; their absence is disqualifying for a production buyer.",
      "notes": "Refreshed from the same vLLM metrics doc dated 2025-01-01 to the current v1 metrics design (full per-request TTFT/TPOT/queue-depth/KV-cache-usage/prefix-cache-hit Prometheus series) plus native OpenTelemetry OTLP tracing (--otlp-traces-endpoint, OTEL_SERVICE_NAME). changed_vs_old=true: the v1 engine's metric set is materially broader than the v0 contract the old bar referenced, and OTel tracing is now native. Qualitative dimension (no single headline number), so best_value is null. fak trails: adjudication-plane /metrics + TraceID but no serving TTFT/TPOT/queue/cache-hit series.",
      "in_scope": true
    },
    {
      "id": "kubernetes-autoscaling-scale-to-zero",
      "category": "operability",
      "dimension": "Kubernetes-native deployment + LLM-aware autoscaling (metric-driven, scale-to-zero)",
      "metric_unit": "qualitative (scale-to-zero, custom-metric scaling)",
      "best_value": null,
      "sota_bar": "KServe + KEDA autoscale vLLM on LLM-specific signals (e.g. vllm:num_requests_running) rather than CPU, supporting scale-from/to-zero with a default 5-min cooldown; the 2025-2026 production stack is vLLM + KServe/llm-d + Ray + Kueue + KEDA on K8s. Ray Serve and KServe provide canary, traffic-split, and multi-model endpoints. The SOTA bar is custom-LLM-metric autoscaling with scale-to-zero, not generic HPA.",
      "sota_systems": [
        "KServe + KEDA",
        "Ray Serve",
        "llm-d / vLLM on Kubernetes"
      ],
      "source_url": "https://developers.redhat.com/articles/2025/09/23/how-set-kserve-autoscaling-vllm-keda",
      "source_date": "2025-09-23",
      "why_it_matters": "Fleet economics live or die on elasticity: scaling on queue depth/running-requests (not CPU) and scaling idle models to zero is how operators avoid paying for idle GPUs. K8s-native lifecycle (canary, rollout, multi-model) is the deployment substrate buyers require.",
      "notes": "KEDA scale-to-zero is Raw-deployment-mode only in KServe.",
      "in_scope": true
    },
    {
      "id": "multi-tenancy-fairness-slo",
      "category": "operability",
      "dimension": "Multi-tenant fairness, per-tenant quotas, and SLO attainment under contention",
      "metric_unit": "per-tenant fairness / SLO attainment under contention (qualitative; per-request unfairness bound)",
      "best_value": null,
      "sota_bar": "VTC (the 2024 first fair scheduler) is now superseded by a 2025 family: Equinox (holistic fair scheduling), FairBatching (fairness-aware batch formation), PROSERVE (multi-priority SLO-aware), and DLPM (locality-aware fair scheduling) -- all addressing VTC's gaps on prefix-locality, SLO, and diverse workloads.",
      "sota_systems": [
        "Equinox (2025)",
        "FairBatching (2025)",
        "PROSERVE (2025)",
        "DLPM locality-aware fair scheduling (2025)",
        "VTC (2024, prior bar)",
        "LiteLLM virtual-keys/budgets"
      ],
      "source_url": "https://arxiv.org/pdf/2508.16646",
      "source_date": "2025-08",
      "why_it_matters": "A shared cluster serving many agents/tenants must stop a noisy neighbor from starving everyone else and must hold each tenant's latency SLO under load. Fairness, quotas, and SLO-attainment-under-contention are exactly the properties a multi-tenant platform buyer evaluates.",
      "notes": "SOTA moved: old bar was VTC (arXiv 2401.00588, 2024-01). The 2025 successor family explicitly improves on VTC -- Equinox (arXiv 2508.16646) holistic fair scheduling, FairBatching (2510.14392) fairness-aware batch formation, PROSERVE (2512.12928) multi-priority SLO scheduling, and DLPM (2501.14312) locality-aware fairness. The dimension remains qualitative (no single canonical number), so best_value is null. Still a real gap fak should measure given its fleet-of-agents target (no work-conserving fair scheduler, no per-tenant token budgets).",
      "in_scope": true
    },
    {
      "id": "prompt-injection-defense-agent-security",
      "category": "security",
      "dimension": "Prompt-injection defense and agent security (attack-success-rate vs utility-under-attack)",
      "metric_unit": "successful IPI attacks on AgentDojo under the defense (CaMeL = 0; ASR effectively ~0%)",
      "best_value": 0,
      "sota_bar": "On AgentDojo, design-by-construction defenses now reach provable security: CaMeL drives successful indirect-prompt-injection attacks to 0 (vs 8 for the next-best tool-filter) while still solving 77% of tasks (vs 84% undefended); MELON (ICML'25) likewise beats prior SOTA defenses. Undefended best agents still see ASR <25%.",
      "sota_systems": [
        "CaMeL (Defeating Prompt Injections by Design, 2025)",
        "MELON (ICML'25)",
        "Meta SecAlign (secure foundation LLM)",
        "tool-filter / SecAlign baselines"
      ],
      "source_url": "https://arxiv.org/abs/2503.18813",
      "source_date": "2025-03",
      "why_it_matters": "An agent that calls tools and reads untrusted content is an exfiltration surface; indirect prompt injection is the top agent-security risk. Buyers now require an ASR-vs-utility number, not a vibe, and adaptive attacks routinely break naive defenses, so the bar is benchmark-grounded.",
      "notes": "SOTA moved: old bar was 2024-09-25 vintage. The 2025 frontier is provable, design-by-construction defense -- CaMeL (arXiv 2503.18813) reduces successful AgentDojo attacks to 0 with 77% utility (provable), and MELON (arXiv 2502.05174, ICML'25) outperforms five prior SOTA defenses across GPT-4o/o3-mini/Llama-3.3-70B. best_value 0.0 = CaMeL's zero successful attacks (security guarantee), the strongest published result. This aligns with fak's own structural-containment thesis (capability/IFC floor rather than a detection rate). fak's detector is conceded ~100% evadable; its real claim is the structural floor.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "tool-sandboxing-structural-containment",
      "category": "security",
      "dimension": "Tool/agent sandboxing, structural containment, and PII/exfil prevention",
      "metric_unit": "structural containment that holds when the model is fooled (capability/IFC/quarantine; provable-defense class)",
      "best_value": null,
      "sota_bar": "Capability/IFC-based structural containment is the 2025 SOTA: CaMeL (Defeating Prompt Injections by Design) extracts control/data flow so untrusted data can never alter program flow and enforces capability policies at tool-call time (0 successful AgentDojo attacks); MELON adds masked re-execution + tool-comparison provable defense.",
      "sota_systems": [
        "CaMeL (capability + control/data-flow IFC, 2025)",
        "MELON (masked re-execution, ICML'25)",
        "container/microVM sandboxes",
        "IFC / capability allow-lists"
      ],
      "source_url": "https://arxiv.org/abs/2503.18813",
      "source_date": "2025-03",
      "why_it_matters": "Detectors are evadable, so the durable control is architectural: bound the blast radius so a successful injection still cannot read secrets, write outside scope, or exfiltrate PII. Structural containment is the property that survives an adaptive attacker and that security-conscious buyers actually demand.",
      "notes": "Refreshed: old source was MELON (arXiv 2502.05174, 2025-02-07). The current strongest structural-containment result is CaMeL (arXiv 2503.18813, 2025-03) -- explicit control/data-flow extraction + capability-gated tool calls giving provable security (0 attacks, 77% utility), the clearest capability/IFC exemplar. MELON remains a co-SOTA (masked re-execution). Qualitative dim (containment-capability presence, no audited ASR-under-containment leaderboard), so best_value null. fak is at parity here -- default-deny capability allow-list, IFC taint sink-gating (FIDES/CaMeL-class), kernel-authored provenance, plan-CFI, context-MMU quarantine -- the same family.",
      "in_scope": true,
      "last_reviewed": "2026-06-27"
    },
    {
      "id": "served-model-capability-quality",
      "category": "model-coverage",
      "dimension": "Capability/quality of the served model (SWE-bench Verified, GPQA, AIME, LiveCodeBench)",
      "metric_unit": "benchmark accuracy %",
      "best_value": 80.6,
      "sota_bar": "Frontier models served by these engines define the quality bar buyers cite: SWE-bench Verified ~80.6% (Gemini 3.1 Pro) / 79.2% (Claude Opus 4.5 + Live-SWE-agent, Nov 2025); GPQA Diamond effectively saturated ~93-94% (Gemini 3 Deep Think 93.8%, GPT-5.4 92.8%); AIME 2025 up to 100% (GPT-5.4). These are model-served-by-engine results, not engine claims.",
      "sota_systems": [
        "Gemini 3.x (SWE-bench/GPQA)",
        "Claude Opus 4.5 + agent scaffold",
        "GPT-5.x (AIME)"
      ],
      "source_url": "https://epoch.ai/benchmarks/swe-bench-verified",
      "source_date": "2025-11-24",
      "why_it_matters": "A serving stack only matters if it serves a model that can do the job; buyers pick the model on SWE-bench/GPQA/AIME first, then the engine to serve it. Engines must be framed as transport for these scores, and constrained decoding/scaffolding can move them, so the dimension belongs on the scorecard explicitly.",
      "notes": "Numbers are model+scaffold specific and saturating fast; GPQA Diamond is near-saturated, weakening its discriminating power for 2026.",
      "in_scope": true
    },
    {
      "id": "cross-agent-fleet-serving",
      "category": "agent-fleet",
      "dimension": "Cross-agent fleet serving time (N agents \u00c3\u2014 T turns): work eliminated by shared-prefix fusion",
      "why_it_matters": "fak's product category. When a fleet of agents shares a large system/context prefix, fusing that prefix once across all agents eliminates re-prefill work that even a tuned per-agent cache repeats. This is the axis fak is built to win.",
      "sota_systems": [
        "vLLM Automatic Prefix Caching",
        "SGLang RadixAttention",
        "OpenAI prompt caching (warm-KV reuse floor)"
      ],
      "sota_bar": "The competing floor remains tuned warm per-agent KV reuse (vLLM Automatic Prefix Caching / SGLang RadixAttention / OpenAI prompt caching); fak's own measured 50x5 fleet-serving result (19 min vs 78 min warm-KV baseline, 4.1x from cross-agent prefix fusion) is the live number on this dim and no external SOTA has displaced the warm-KV-reuse baseline class.",
      "metric_unit": "min (warm-KV baseline fleet-serving time; fak arm = 19 min, 4.1x)",
      "best_value": 78,
      "source_url": "https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/",
      "source_date": "2026-01",
      "in_scope": true,
      "notes": "No newer external SOTA found that displaces the warm-KV-reuse baseline class for this dim; old bar stands. The competitor here is the warm-per-agent-KV reuse floor (APC/RadixAttention/OpenAI prompt caching), unchanged in kind through mid-2026 -- both arms run on fak's own kernel held constant, so the 4.1x isolates cross-agent prefix fusion (work eliminated), not a faster kernel. best_value 78.0 is the warm-KV baseline arm (fak's measured row, commit 2bbda6f); a production vLLM/SGLang would shrink both arms' absolute minutes but is expected to leave the reuse ratio ~intact. This is a fak-authored measured row; only the baseline-class source date is refreshed."
    },
    {
      "id": "marginal-reuse-over-warm-cache",
      "category": "agent-fleet",
      "dimension": "Marginal value of cross-agent prefix fusion ON TOP of an already-warm per-agent cache",
      "why_it_matters": "The honest few-fold: not the headline-vs-naive number, but the gain over the tuned warm-cache a real operator runs. This is the conservative claim a skeptic should be shown.",
      "sota_systems": [
        "Tuned warm per-agent KV cache, already hot (vLLM APC / SGLang RadixAttention class)"
      ],
      "sota_bar": "The conservative SOTA point is fak's own already-warm per-agent KV arm (baseline 1.0x); cross-agent prefix fusion on top of it yields a measured ~2.4-2.7x marginal gain. No external benchmark redefines the already-warm-cache baseline for this dim.",
      "metric_unit": "x (warm per-agent KV baseline = 1.0; fak marginal = 2.4-2.7x on top)",
      "best_value": 1,
      "source_url": "https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/",
      "source_date": "2026-01",
      "in_scope": true,
      "notes": "No newer source found; old bar stands. This is a fak-authored measured row (SESSION-VALUE-STACK-RESULTS.md, commit 92896a4): the baseline is fak's OWN warm-KV arm B (most conservative SOTA point, bit-exact, kernel held constant), against which cross-agent prefix fusion gives the honest marginal ~2.4-2.7x (NOT the 60x-vs-naive headline). best_value 1.0 is the warm-cache baseline reference value. The external warm-KV-reuse class (APC/RadixAttention) has not changed in kind; only the source date is refreshed."
    },
    {
      "id": "marginal-reuse-vs-live-engine",
      "category": "agent-fleet",
      "dimension": "Cross-agent reuse marginal value vs a LIVE tuned shared-prefix engine (head-to-head)",
      "why_it_matters": "A live SGLang/vLLM ALSO fuses a shared prefix once. The marginal win vs a real competing process \u00e2\u20ac\u201d not vs fak's own warm-KV arm \u00e2\u20ac\u201d is the number a buyer comparing engines actually wants, and fak has not measured it.",
      "sota_systems": [
        "SGLang RadixAttention (live process)",
        "vLLM Automatic Prefix Caching (live process)"
      ],
      "sota_bar": "Head-to-head marginal value of cross-agent reuse vs a LIVE tuned shared-prefix engine (vLLM APC / SGLang RadixAttention, which also fuses a shared prefix once) remains UNMEASURED; fak's own WebVoyager stratification puts the marginal-vs-tuned win at only ~1.0-1.10x on that workload.",
      "metric_unit": "x over a live tuned engine (unmeasured; WebVoyager stratification ~1.0-1.10x)",
      "best_value": null,
      "source_url": "https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/",
      "source_date": "2026-01",
      "in_scope": true,
      "notes": "No newer source found; old bar stands as a named no-claim/gap. Every committed fak fleet multiplier (4.1x, 60.3x, 139x) is vs naive re-prefill or fak's own kernel held constant in warm-KV mode -- NOT a head-to-head vs a live SGLang/vLLM process, which also fuses a shared prefix once. The honest marginal-vs-a-real-tuned-process number is still UNMEASURED; the repo's own WebVoyager baseline-stratification puts it at only ~1.0-1.10x. best_value null (no measured number). Only the competing-engine source date is refreshed; the gap is unchanged."
    },
    {
      "id": "single-stream-decode-cpu",
      "category": "single-stream",
      "dimension": "Single-stream (one chat) decode throughput on CPU",
      "why_it_matters": "The regime local/edge stacks (llama.cpp, Ollama, MLC) compete hardest on: one user, one chat, raw tokens/s on commodity hardware. fak explicitly does NOT target this \u00e2\u20ac\u201d showing the loss keeps the scorecard honest.",
      "sota_systems": [
        "llama.cpp",
        "Ollama",
        "MLC-LLM"
      ],
      "sota_bar": "llama.cpp Metal/CPU leads single-stream decode on the same box (e.g. Qwen2.5-7B Q8 ~17.3 tok/s Metal, M3 Pro).",
      "metric_unit": "tok/s",
      "best_value": 17.3,
      "source_url": "https://github.com/ggml-org/llama.cpp",
      "source_date": "2026-06",
      "in_scope": true,
      "notes": ""
    },
    {
      "id": "single-stream-prefill-cpu",
      "category": "single-stream",
      "dimension": "Single-stream prefill throughput, apples-to-apples CPU-vs-CPU",
      "why_it_matters": "Prefill (prompt ingestion) speed on CPU is a core local-inference axis. fak's arm64 build lacks the register-blocked int8 GEMM tile that x86 already has \u00e2\u20ac\u201d a disclosed, narrowing loss.",
      "sota_systems": [
        "llama.cpp"
      ],
      "sota_bar": "llama.cpp CPU sets the prefill bar on the same box + Q8 weights (1\u00c3\u2014 reference); fak arm64 is ~0.12\u00c3\u2014 until the int8 tile lands.",
      "metric_unit": "\u00c3\u2014 of llama.cpp CPU",
      "best_value": 1,
      "source_url": "https://github.com/ggml-org/llama.cpp",
      "source_date": "2026-06",
      "in_scope": true,
      "notes": ""
    },
    {
      "id": "single-stream-decode-gpu",
      "category": "single-stream",
      "dimension": "Single-stream decode throughput on a consumer GPU",
      "why_it_matters": "Local GPU single-stream decode (the RTX-class laptop/desktop regime) is where llama.cpp's CUDA backend sets the bar; fak reaches parity here at higher precision via a reusable CUDA graph.",
      "sota_systems": [
        "llama.cpp (CUDA)"
      ],
      "sota_bar": "llama.cpp Q8_0 ~120 \u00c2\u00b1 15 tok/s on an RTX 4070 (-ngl 99).",
      "metric_unit": "tok/s",
      "best_value": 120,
      "source_url": "https://github.com/ggml-org/llama.cpp",
      "source_date": "2026-06",
      "in_scope": true,
      "notes": ""
    },
    {
      "id": "int8-simd-decode-vs-peer",
      "category": "quantization",
      "dimension": "int8 / Q8_0 SIMD decode throughput vs the same-rung int8 peer",
      "why_it_matters": "The execution speed of an int8-quantized model on CPU SIMD (AVX2/AVX-512) decides local-inference viability. fak's hand-written int8 GEMM beats the standard HuggingFace dynamic-int8 reference.",
      "sota_systems": [
        "HuggingFace dynamic-int8",
        "llama.cpp Q8_0"
      ],
      "sota_bar": "HuggingFace dynamic-int8 is the standard same-rung CPU int8 reference (1\u00c3\u2014); llama.cpp Q8_0 CPU is the harder peer (~parity).",
      "metric_unit": "\u00c3\u2014 vs HF int8",
      "best_value": 1,
      "source_url": "https://huggingface.co/docs/transformers/main/quantization/overview",
      "source_date": "2026-06",
      "in_scope": true,
      "notes": ""
    },
    {
      "id": "vendor-gemm-free-correctness",
      "category": "numerical-correctness",
      "dimension": "Dense GPU compute correctness with ZERO vendor GEMM (cuBLAS-free)",
      "why_it_matters": "Every mainstream stack depends on NVIDIA cuBLAS for its GEMMs. Running a real model's dense path on an owned, dependency-free kernel \u00e2\u20ac\u201d and proving it bit-matches the vendor oracle \u00e2\u20ac\u201d is a portability/auditability differentiator.",
      "sota_systems": [
        "cuBLAS (NVIDIA tuned vendor GEMM)"
      ],
      "sota_bar": "cuBLAS is the tuned vendor GEMM every other stack carries; matching it bit-for-bit (cosine=1.0) is parity by construction.",
      "metric_unit": "cosine vs vendor-GEMM oracle",
      "best_value": null,
      "source_url": "https://docs.nvidia.com/cuda/cublas/",
      "source_date": "2026-06",
      "in_scope": true,
      "notes": ""
    },
    {
      "id": "model-routing-granularity",
      "category": "model-routing",
      "dimension": "Model routing granularity: per-aspect + first-class ensemble routing vs whole-request single-model selection",
      "why_it_matters": "Every mainstream 'LLM router' (RouteLLM, Martian, NotDiamond, Unify, OpenRouter, Portkey, LiteLLM Router) answers one question \u00e2\u20ac\u201d which SINGLE model should serve this WHOLE request \u00e2\u20ac\u201d and the only shipped model ensemble is a single fixed recipe (OpenRouter Fusion). Routing a sub-request aspect (one tool call, one reasoning step) to its own model, or declaring a configurable per-aspect ensemble with a reduction, is a granularity layer no surveyed product exposes; it is the axis that decides whether routing is a per-request pick or a first-class, in-loop decision.",
      "sota_systems": [
        "RouteLLM (LMSYS)",
        "Martian",
        "NotDiamond",
        "Unify.ai",
        "OpenRouter (+ Fusion)",
        "Portkey",
        "LiteLLM Router"
      ],
      "sota_bar": "Surveyed 2025-2026 routers/gateways (RouteLLM, Martian, NotDiamond, Unify, OpenRouter, Portkey, LiteLLM Router) all route the WHOLE request to ONE model; OpenRouter Fusion is the only shipped model ensemble and it is a fixed parallel-synthesize recipe, not a configurable per-aspect reduction. None routes a sub-request aspect to its own model. (Aurelio Semantic-Router routes to an intent, not a model; vLLM/SGLang routers balance replicas of ONE model for KV locality \u00e2\u20ac\u201d a different layer.)",
      "best_value": null,
      "metric_unit": "qualitative (routing granularity: whole-request vs per-aspect; fixed vs configurable ensemble reduction)",
      "source_url": "https://github.com/anthony-chaudhary/fak/blob/main/docs/model-routing.md",
      "source_date": "2026-06",
      "notes": "Survey of 2025-2026 routers/gateways in docs/model-routing.md. The gap is categorical (a routing GRANULARITY no surveyed product exposes), not a benchmarked speed/quality win \u00e2\u20ac\u201d any '10x' is a target to be measured, never an inferred or borrowed number.",
      "in_scope": true
    },
    {
      "id": "compaction-cache-preservation",
      "category": "client-compaction",
      "dimension": "Long-session history compaction that preserves the provider prompt-cache prefix (drop-and-splice vs summarize-and-resend)",
      "why_it_matters": "An agent re-sends its whole transcript every turn; the provider prompt cache discounts the unchanged prefix ONLY while it stays byte-for-byte identical. Almost every built-in compaction (Aider, LangChain ConversationSummaryMemory, Codex CLI, Copilot CLI, Anthropic API context-editing) summarizes or clears old turns and re-sends a REWRITTEN prompt \u00e2\u20ac\u201d which by the providers' own docs breaks that exact-match prefix and re-bills it at full price on the compacting turn. Whether a tool can shrink a long conversation WITHOUT busting the cache is the axis that decides if compaction saves money on the turn it fires or costs it.",
      "sota_systems": [
        "Anthropic API context-editing (clear_tool_uses_20250919)",
        "OpenAI Codex CLI (summary compaction)",
        "GitHub Copilot CLI (cache-boundary compaction)",
        "Aider (ChatSummary)",
        "LangChain (trim_messages / ConversationSummaryMemory)"
      ],
      "sota_bar": "The field splits two ways. Summarizers (Aider, LangChain ConversationSummaryMemory, Codex CLI, Copilot CLI) and Anthropic's API context-editing REWRITE/clear content and break the prefix cache (OpenAI docs: 'when you drop, summarize or compact earlier turns ... you'll break the cache'; Anthropic context-editing docs: 'Invalidates cached prompt prefixes when content is cleared'). The cache-preserving sub-field is small: LangChain trim_messages (pure sliding-window drop, no rewrite) and Copilot's DELIBERATE cache-boundary reset reason about it; no surveyed tool ships a PROVEN byte-identity splice on by default. Context-editing pairs clearing with a memory tool for recovery (a different axis: recoverability), reporting up to ~84% token reduction on an Anthropic agent eval.",
      "best_value": null,
      "metric_unit": "qualitative (preserves byte-identical cache prefix while compacting: yes/no; lossy vs drop; on-by-default vs library/beta)",
      "source_url": "https://platform.claude.com/docs/en/docs/build-with-claude/context-editing",
      "source_date": "2026-01",
      "notes": "Industry mechanisms web-sourced 2026-06 (OpenAI prompt-caching cookbook, Anthropic context-editing docs, GitHub Copilot context blog, Aider/LangChain source). fak's drop-and-splice is verified in internal/agent/anthropic_compact.go (bytes.Equal prefix proof, 13 fail-safe identity exits). FENCE: fak proves byte-identity of the shipped prefix; whether the provider's cache CASCADE realizes the hit on the compacting turn is unmeasured (epic #745) \u00e2\u20ac\u201d so this is a cache-PRESERVATION-mechanism position, not a measured cost-saving number.",
      "in_scope": true
    }
  ]
}
