{
  "_about": "fak position rows for the 'numerics' group. dim_id links each to a _taxonomy.json dimension.",
  "rows": [
    {
      "id": "kv-quantization-fp8-int4",
      "dim_id": "kv-quantization-fp8-int4",
      "axis": "KV-cache quantization (FP8 / INT8 / INT4 KV)",
      "category": "quantization",
      "competitor": "TensorRT-LLM; vLLM",
      "competitor_class": "sota",
      "competitor_value": 2,
      "competitor_source": "https://blog.squeezebits.com/vllm-vs-tensorrtllm-8-kv-cache-quantization-35079 (2025-01)",
      "fak_value": null,
      "unit": "x KV memory reduction (KV dtype)",
      "higher_is_better": true,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "fak quantizes WEIGHTS (Q8_0 SIMD lane, CLAIMS line 95; Q4_K/Q8 device GEMV witnesses line 146) but there is no FP8/INT8/INT4 KV-CACHE quantization claim. The kernel-owned KVCache and KV-decode parity (line 85) are at f32; no per-token KV-byte reduction from a quantized KV dtype is measured.",
      "apples_to_apples": false,
      "comparison_note": "OUT OF SCOPE for the current kernel. fak's quantization work is on weight bytes (Q8/Q4_K), not on the KV cache dtype; it has no FP8/INT8/INT4 KV path and so claims none of TensorRT-LLM/vLLM's ~2-3x batch gain from FP8 KV. fak's KV story is exact eviction/reuse on an f32 cache, a different axis from KV-precision compression.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "fp8-weight-activation-accuracy-retention",
      "dim_id": "fp8-weight-activation-accuracy-retention",
      "axis": "FP8 (W8A8, E4M3) accuracy retention vs bf16 reference",
      "category": "quantization",
      "competitor": "vLLM (FP8 W8A8 dynamic/static); TensorRT-LLM (ModelOpt); SGLang; Red Hat / Neural Magic llm-compressor",
      "competitor_class": "sota",
      "competitor_value": 99,
      "competitor_source": "https://arxiv.org/abs/2411.02355 (2025-08)",
      "fak_value": null,
      "unit": "% accuracy recovery vs bf16",
      "higher_is_better": true,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "none — CLAIMS.md: compute HAL is f32-only; the only narrowing fak ships is Q8_0 (8-bit), and BENCHMARK-AUTHORITY.md carries no FP8 or accuracy-recovery number",
      "apples_to_apples": true,
      "comparison_note": "OUT OF SCOPE for a reuse kernel. fak has no FP8 (W8A8/E4M3) path at all — its compute HAL is f32-only and its sole quant rung is Q8_0 weight-narrowing on CPU SIMD/GPU. No accuracy-retention-vs-bf16 study exists, so fak makes no claim. Not a measurable gap fak should chase: FP8 lossless retention is a model-numerics property of vLLM/TensorRT-LLM, orthogonal to fak's prefix-reuse/adjudication differentiator.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "int8-w8a8-smoothquant-accuracy",
      "dim_id": "int8-w8a8-smoothquant-accuracy",
      "axis": "INT8 W8A8 (SmoothQuant) accuracy under activation outliers",
      "category": "quantization",
      "competitor": "SmoothQuant (MIT HAN Lab); vLLM INT8 W8A8; TensorRT-LLM; llm-compressor",
      "competitor_class": "sota",
      "competitor_value": null,
      "competitor_source": "https://arxiv.org/abs/2411.02355 (2025-08)",
      "fak_value": null,
      "unit": "% per-task degradation vs bf16",
      "higher_is_better": false,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "none — fak's int8 lane is Q8_0 WEIGHT-only (W8A16-style), argmax-exact vs the f32 oracle; there is no W8A8 activation-quant path and no SmoothQuant outlier-handling, so no accuracy-under-outliers number exists",
      "apples_to_apples": true,
      "comparison_note": "OUT OF SCOPE for a reuse kernel. fak's int8/Q8_0 SIMD lane (in-flight) is weight-only and proven argmax-exact against its own f32 oracle, NOT an activation-quantized W8A8 path with SmoothQuant. fak has no per-token activation scaling, no outlier-suppression study, and no per-task degradation measurement — it makes no SmoothQuant accuracy claim.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "int4-weight-only-awq-gptq",
      "dim_id": "int4-weight-only-awq-gptq",
      "axis": "INT4 weight-only (W4A16: AWQ, GPTQ) accuracy and calibration sensitivity",
      "category": "quantization",
      "competitor": "AWQ (LMDeploy, vLLM); GPTQ (vLLM, AutoGPTQ); TensorRT-LLM W4A16",
      "competitor_class": "sota",
      "competitor_value": null,
      "competitor_source": "https://arxiv.org/abs/2411.02355 (2025-08)",
      "fak_value": null,
      "unit": "% degradation vs bf16 (W4A16)",
      "higher_is_better": false,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "none — fak ships only Q8_0 (8-bit weight) and f32; no W4A16 / INT4 / AWQ / GPTQ path and no calibration-sensitivity study in CLAIMS.md or BENCHMARK-AUTHORITY.md",
      "apples_to_apples": true,
      "comparison_note": "OUT OF SCOPE for a reuse kernel. fak has no INT4 weight-only path (no AWQ, no GPTQ, no calibration-set tooling); its lowest precision is 8-bit Q8_0. Note BENCHMARK-AUTHORITY does report serving a q4_k_m GGUF (Qwen3.6-27B) but at 0.9 tok/s with only 2-token parity before drift — that is a GGUF-load capability, not an INT4-accuracy-retention claim, so no W4A16 accuracy number is asserted.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "fp4-nvfp4-mxfp4-blackwell",
      "dim_id": "fp4-nvfp4-mxfp4-blackwell",
      "axis": "4-bit microscaling float (NVFP4 / MXFP4 / FP4) accuracy vs FP8 on Blackwell",
      "category": "quantization",
      "competitor": "NVIDIA Blackwell + TensorRT-LLM (ModelOpt NVFP4); vLLM NVFP4; llama.cpp (NVFP4/MXFP4)",
      "competitor_class": "sota",
      "competitor_value": 0.1,
      "competitor_source": "https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/ (2025-04)",
      "fak_value": null,
      "unit": "MMLU-point drop vs FP8",
      "higher_is_better": false,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "none — no 4-bit microscaling-float path (NVFP4/MXFP4/FP4) anywhere in fak; compute HAL is f32-only, quant rung is Q8_0, no Blackwell-class hardware path",
      "apples_to_apples": true,
      "comparison_note": "OUT OF SCOPE for a reuse kernel. fak has no microscaling 4-bit float format, no NVFP4/MXFP4 kernels, and no Blackwell tensor-core path. This is squarely vendor-stack (TensorRT-LLM/ModelOpt) territory; fak asserts nothing and should not — it is orthogonal to its prefix-reuse differentiator.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "4bit-activation-rotation-quarot-spinquant",
      "dim_id": "4bit-activation-rotation-quarot-spinquant",
      "axis": "Full 4-bit (W4A4 + KV4) via rotation/outlier removal",
      "category": "quantization",
      "competitor": "QuaRot (Hadamard rotation); SpinQuant (learned rotation); QServe W4A8KV4",
      "competitor_class": "sota",
      "competitor_value": 99,
      "competitor_source": "https://arxiv.org/html/2404.00456v1 (2024-04)",
      "fak_value": null,
      "unit": "WikiText-2 perplexity delta / zero-shot retention",
      "higher_is_better": false,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "none — no W4A4, no KV4, no Hadamard/learned rotation (QuaRot/SpinQuant) path; fak is f32 compute with an 8-bit weight rung only",
      "apples_to_apples": true,
      "comparison_note": "OUT OF SCOPE for a reuse kernel. Full 4-bit (W4A4+KV4) via rotation/outlier removal is an aggressive model-numerics research line; fak has none of it (no activation quant, no KV quant, no rotation). It makes no perplexity-delta or zero-shot-retention claim here.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "quantization-granularity-scheme",
      "dim_id": "quantization-granularity-scheme",
      "axis": "Quantization granularity (per-tensor / per-channel / per-group / microscaling block size)",
      "category": "quantization",
      "competitor": "MX formats (OCP MXFP4/MXFP8, block-32); NVFP4 (block-16, FP32 second-level); group-128 INT4 (AWQ/GPTQ); MOSS two-level microscaling",
      "competitor_class": "sota",
      "competitor_value": 3,
      "competitor_source": "https://arxiv.org/html/2511.05811v2 (2025-11)",
      "fak_value": null,
      "unit": "SNR dB by granularity",
      "higher_is_better": true,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "none — fak consumes Q8_0 (the GGUF 8-bit block scheme) but has shipped no per-tensor/per-channel/per-group/microscaling granularity comparison or SNR measurement",
      "apples_to_apples": true,
      "comparison_note": "OUT OF SCOPE for a reuse kernel. fak's only granularity is whatever Q8_0/GGUF gives it; it has authored no microscaling block-size study and no SNR-vs-granularity numbers. The MX/NVFP4 granularity question is a format-design axis owned by the quant-format authors, not fak — no claim.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "numerical-correctness-error-metric",
      "dim_id": "numerical-correctness-error-metric",
      "axis": "Numerical-correctness error metric vs reference (perplexity, KL-divergence, recovery %)",
      "category": "numerical-correctness",
      "competitor": "HuggingFace transformers f32 reference (the oracle)",
      "competitor_class": "reference-oracle",
      "competitor_value": null,
      "competitor_source": "HuggingFace transformers reference forward (the oracle); fak IN-KERNEL-MODEL-RESULTS.md (R0–R14, argmax-exact).",
      "fak_value": null,
      "unit": "max|Δ| / cosine",
      "higher_is_better": false,
      "verdict": "parity",
      "status": "shipped",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "IN-KERNEL-MODEL-RESULTS.md · CLAIMS.md (In-kernel model)",
      "apples_to_apples": true,
      "comparison_note": "Embedding exact, per-layer cos=1.000000, final-logits max|Δ|≈4.4e-5, KV-decode + KV-quarantine-evict token-for-token identical (max|Δ|=0) vs the HF oracle. This correctness parity is the floor under every fak speed claim — a win on speed is meaningless if the tokens differ; they do not.",
      "measured_on": "2026-06-18"
    },
    {
      "id": "deterministic-bitexact-inference",
      "dim_id": "deterministic-bitexact-inference",
      "axis": "Deterministic / bitwise-reproducible inference (batch invariance)",
      "category": "numerical-correctness",
      "competitor": "Thinking Machines batch-invariant-ops (on vLLM FlexAttention); vLLM batch-invariance mode; LLM-42 (deterministic speculation)",
      "competitor_class": "sota",
      "competitor_value": 1000,
      "competitor_source": "https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ (2025-09)",
      "fak_value": null,
      "unit": "bitwise-identical completions under dynamic batching",
      "higher_is_better": true,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "CLAIMS.md: greedy decode is deterministic+input-driven and bit-identical serial-vs-batched (TestBatchedDecodeMatchesSerial, TestBatchFromPrefixMatchesIndependentPrefill, parallel matmul math.Float32bits-equal) — but these prove SAME-batch-shape determinism, not the cross-batch-SIZE invariance the SOTA bar measures",
      "apples_to_apples": false,
      "comparison_note": "REAL GAP, partially adjacent. fak DOES ship strong determinism witnesses: greedy decode is deterministic, and its batched decode is bit-identical (math.Float32bits equality) to the serial reference — the property that the cross-agent fusion preserves tokens. BUT the SOTA bar is BATCH-INVARIANCE: identical output regardless of dynamic batch SIZE/co-tenant composition (Thinking Machines / vLLM batch-invariant kernels, 1000/1000 identical). fak proves batched==serial for a FIXED shape, never tested across varying dynamic batch sizes, so it makes no batch-invariance claim. Honest verdict no-claim on the SOTA-defined dimension; the related fixed-shape bit-identity is real but a different property.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "mlperf-accuracy-constrained-throughput",
      "dim_id": "mlperf-accuracy-constrained-throughput",
      "axis": "Accuracy-constrained throughput on a neutral benchmark (MLPerf Inference)",
      "category": "quantization",
      "competitor": "NVIDIA GB300 NVL72 (Blackwell Ultra, TensorRT-LLM/Dynamo); AMD MI355X (ROCm); MLPerf Inference v6.0 server scenario (DeepSeek-R1 671B, accuracy+latency-constrained)",
      "competitor_class": "sota",
      "competitor_value": 8064,
      "competitor_source": "https://www.storagereview.com/news/nvidia-sets-mlperf-inference-v6-0-records-with-blackwell-ultra-platform (2026-04-01)",
      "fak_value": null,
      "unit": "accuracy-gated tok/s per GPU",
      "higher_is_better": true,
      "verdict": "no-claim",
      "status": "stub",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "none — fak has no MLPerf Inference submission and no accuracy-gated throughput run; its only live concurrent head-to-head (served-throughput-vs-sglang) is a raw tok/s race fak TRAILS, with no 99%/99.9%-accuracy gate applied",
      "apples_to_apples": true,
      "comparison_note": "OUT OF SCOPE BY DESIGN for a reuse kernel. The accuracy-constrained number is now MLPerf v6.0's (2026-03-30) server scenario, carried by DeepSeek-R1 671B MoE (8,064 tok/s/GPU on GB300 NVL72 under a 2 s TTFT / 12.5 tok/s/user p99 SLO while holding the accuracy bar), superseding v5.x's Llama-3.1-405B framing. fak has never submitted to MLPerf and runs no accuracy-constrained throughput benchmark. Its closest real number — 1085.6 vs 1451.6 tok/s vs raw SGLang on 8-GPU datacenter server (it TRAILS, the gateway tax) — is an ungated throughput race, not an MLPerf accuracy-gated result, so no MLPerf claim is made. fak does NOT intend an own-engine MLPerf-shaped DeepSeek-R1 number; on a 671B MoE it fronts SGLang/llama.cpp, and its in-kernel DeepSeek-class-MoE path is the GLM-5.2 work (#1010, #1026; single device, commit 498a4ab). See max-model-and-moe-coverage.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "quantization-datatype-coverage",
      "dim_id": "quantization-datatype-coverage",
      "axis": "Quantization format & low-precision datatype coverage",
      "category": "quantization",
      "competitor": "TensorRT-LLM; vLLM; NVIDIA Blackwell",
      "competitor_class": "sota",
      "competitor_value": null,
      "competitor_source": "https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_w4a4_fp4/ (2025-01-01)",
      "fak_value": null,
      "unit": "low-precision formats with a native compute path",
      "higher_is_better": true,
      "verdict": "no-claim",
      "status": "in-flight",
      "fak_commit": "",
      "fak_artifact": "",
      "fak_doc": "CLAIMS: GGUF Q8_0/Q4_K loaded but DEQUANT-to-f32 on own engine; int8/Q8_0 SIMD lane is in-flight (not [SHIPPED]); no FP8/NVFP4/MXFP4/AWQ/GPTQ",
      "apples_to_apples": true,
      "comparison_note": "REAL but partial gap. fak LOADS GGUF Q8_0/Q4_K and witnesses Q8 device GEMM (Vulkan 24.6 tok/s 1.49x vs f32; datacenter GPU k_q8_gemm cosine~1.0), and has an in-flight AVX2/AVX-512 int8 SIMD decode lane (~2.97x vs HF dynamic-int8, int8-simd-vs-hf row) — but its own engine still largely DEQUANTS to f32, and it has NO FP8 W8A8, NVFP4/MXFP4 W4A4, AWQ/GPTQ, or KV-cache quantization. The TensorRT-LLM/vLLM Blackwell-tensor-core NVFP4 path is entirely out of fak's reach. No shipped fak coverage-breadth number; the int8 lane is deliberately not yet [SHIPPED]. Verdict no-claim on coverage; the speed sub-claim lives in int8-simd-vs-hf.",
      "measured_on": "2026-06-23"
    },
    {
      "id": "int8-simd-vs-hf",
      "dim_id": "int8-simd-decode-vs-peer",
      "axis": "int8 / Q8_0 SIMD decode throughput vs the same-rung int8 peer",
      "category": "quantization",
      "competitor": "HuggingFace dynamic-int8 (the standard same-rung int8 reference)",
      "competitor_class": "sota",
      "competitor_value": 1.0,
      "competitor_source": "fak experiments/model-baseline/INT8-RUNG-VERIFICATION.md — same-rung HF dynamic-int8 peer, argmax-exact vs the f32 oracle.",
      "fak_value": 2.97,
      "unit": "× vs HF int8",
      "higher_is_better": true,
      "fak_commit": "",
      "fak_doc": "MODEL-BASELINE-RESULTS.md (Act 3) · experiments/model-baseline/INT8-RUNG-VERIFICATION.md",
      "apples_to_apples": true,
      "comparison_note": "The int8/Q8_0 SIMD lane (AVX2/AVX-512 Go assembly) is the ACTIVE IN-FLIGHT increment — deliberately NOT a [SHIPPED] CLAIMS row until the lane commits. vs the same-rung HF dynamic-int8 peer fak is ~2.97× faster (argmax-exact); vs llama.cpp Q8_0 CPU it is near-parity (~0.90×, 7.7 vs 6.9 ms/tok), a separate harder peer.",
      "measured_on": "2026-06-23",
      "verdict": "lead",
      "status": "in-flight"
    },
    {
      "id": "glm52-pure-kernel-vs-cublas",
      "dim_id": "vendor-gemm-free-correctness",
      "axis": "Dense GPU compute correctness with ZERO vendor GEMM (cuBLAS-free)",
      "category": "numerical-correctness",
      "competitor": "cuBLAS (NVIDIA's tuned vendor GEMM — the standard dependency every other stack carries)",
      "competitor_class": "reference-oracle",
      "competitor_value": null,
      "competitor_source": "cpu-ref bit-exact witness (TestGLMMoeDsaBackendGEMMMatchesCPU) + datacenter GPU TestCUDAGLMMoeDsaBackendForward (cosine=1.000000); docs/notes/GLM52-PURE-KERNEL-ON-GPU-SERVER-2026-06-21.md.",
      "fak_value": null,
      "unit": "cosine",
      "higher_is_better": true,
      "fak_commit": "498a4ab",
      "fak_doc": "docs/notes/GLM52-PURE-KERNEL-ON-GPU-SERVER-2026-06-21.md",
      "apples_to_apples": true,
      "comparison_note": "GLM-5.2's MoE/FFN experts + router + vocab head (the bulk of its params) run on fak's OWN k_q8_gemm kernel — cosine=1.000000, argmax-exact vs the CPU Q8 forward on a real datacenter GPU, with ZERO cuBLAS dependency. Honest fence: the DSA sparse-attention + DSA-KV stay host-side (the next #86/#413 slice); the dense path is pure-GPU, the sparse-attention is not yet.",
      "measured_on": "2026-06-22",
      "verdict": "parity",
      "status": "shipped"
    }
  ]
}