# Tiny-LLM Speedrun Prior Art (25M–100M, single RTX 4070-class)

Date: 2026-05-17
Scope: state-of-the-art techniques for training a 25M–100M-param hybrid-attention
LLM from scratch on a single 16 GB Ada GPU (sm_89) as fast as possible. Goal:
fold into ARLE to drop a 25M run from 10 h → 2 h. Sources cited inline.

## Axis 1 — Single-GPU speedrun research / leaderboards

| Source | What it provides | How we adopt | Risk / cost |
|---|---|---|---|
| KellerJordan/modded-nanogpt (Apr 2026, record #80, 1.406 min on 8×H100; sub-3 min Jan 2025) [gh](https://github.com/KellerJordan/modded-nanogpt) [leloy](https://x.com/leloykun/status/1880301753213809016) | Reference recipe stack: Muon + value embeddings + U-net skip + FP8 head + FlexAttention long-short window + YaRN warmup + paired-head Newton-Schulz + cautious WD. | Mine recipe step-by-step; port the *training-loop tricks* (value embeddings, U-net skips, ReLU², trapezoidal/WSD LR, no grad clip, batch+seq-len schedule) into ARLE training crate. Skip FP8 head (sm_89 has fp8 tensor cores but the asymmetric-rescale trick is fragile on small dense models). | Many tricks are 8×H100-shape-tuned; need to re-validate at 25M+single-GPU. |
| Tyler Romero worklog (attribution table) [tylerromero](https://www.tylerromero.com/posts/nanogpt-speedrun-worklog/) | Speedup attribution per technique: Muon ~46%, FlexAttention+doc-mask ~36%, arch modernizations ~8%, logit soft-cap ~6%, micro-batch loader ~6%. | Use as priority ordering — go after Muon first, then attention/doc-mask + sequence packing, before kernel-level work. | Numbers are for GPT-2 124M @ 8×H100; transfer to 25M+4070 is plausible but unmeasured. |
| nanochat scaling-law sweep (Karpathy, Jan 2026) [discussion #420](https://github.com/karpathy/nanochat/discussions/420) [deepwiki](https://deepwiki.com/karpathy/nanochat/3.2-hyperparameter-scaling-and-auto-configuration) | Single-dial depth scaling: optimal D:N ≈ 8 for Muon (vs 9.6 Adam, 20 Chinchilla) ⇒ ~20% fewer tokens at same loss; Muon for matrices + AdamW for embeddings/scalars. | Adopt the hybrid Muon/AdamW partition (we already have AdamW in `crates/autograd/`); lower our default D:N target to ~8 for the from-scratch sweeps in `scaling_laws.sh`. | Muon-favorable D:N only holds when Muon is correctly tuned — under-tuned Muon makes this worse, not better. |
| Prime Intellect auto-nanogpt + Claude/Opus self-research entry (May 2026, 2930 steps) [primeintellect](https://www.primeintellect.ai/auto-nanogpt) | Confirms the recipe stack is still the SOTA frontier and that hyperparameter tuning (not arch) is the remaining ~1-3% ceiling. | Treat this as evidence that *we don't need novel ideas* — we need to faithfully port the recipe. | None — just calibration. |

## Axis 2 — Optimizers (highest leverage)

| Source | What it provides | How we adopt | Risk / cost |
|---|---|---|---|
| Muon (Keller Jordan, 2024) + "Muon is Scalable" (Liu et al., 2502.16982) [blog](https://kellerjordan.github.io/posts/muon/) [arxiv](https://arxiv.org/abs/2502.16982) | ~1.5× sample efficiency vs AdamW for hidden matrices; Newton-Schulz orthogonalization in bf16; layerwise sharding recovers throughput within 8%. Wallclock cost per step modest if NS iters batched. | **First adoption.** Implement Muon in `crates/autograd/` (alongside AdamW). NS iter is a few bf16 matmuls — already a strength of our CUDA stack. Apply only to 2D hidden weights; keep AdamW for embeddings + norms + scalars (the nanochat partition). | NS is sensitive to spectral norm; need warmup or it diverges. Cost: ~3-5% extra per-step compute. |
| "Fantastic Pretraining Optimizers" (Wen et al., 2509.02046) [arxiv](https://arxiv.org/abs/2509.02046) | Honest re-benchmark: matrix optimizers give **1.4× wallclock at 0.1B → only 1.1× at 1.2B**; Muon wins at 1–4× Chinchilla, Soap/Kron win at ≥8×; under-tuned alternatives often lose to AdamW. | Sets expectations: Muon is the right pick at our 25–100M scale (1.4× wallclock target = 5× → 3.6×; we still need kernel-level wins to hit 5×). | The 1.4× depends on properly tuned LR/WD; cargo-cult Muon under-performs. |
| Schedule-Free AdamW / SF-Muon (Defazio et al., 2405.15682) + "Through the River" (2507.09846) [arxiv](https://arxiv.org/abs/2405.15682) [pypi](https://pypi.org/project/schedulefree/) | Removes need for fixed step horizon; matches or beats cosine on 124M LLaMA-style; LR 1–10× higher than scheduled AdamW; won MLCommons 2024 AlgoPerf self-tuning. | Optional: gate behind a feature flag in `crates/autograd/`. Useful when we don't know step count up-front (continual pretraining). For fixed-budget speedrun runs, trapezoidal/WSD is fine and simpler. | Not a free perf win — main value is hyperparameter robustness, not raw speed. |
| SOAP (Vyas et al., 2409.11321) [openreview](https://openreview.net/forum?id=IDxZhXrpNf) | Adam in Shampoo eigenbasis; 35% wallclock and 40% iter reduction vs AdamW at *large batch*; ~20% over Shampoo. | **Don't adopt at 25–100M.** SOAP/Shampoo wins emerge at ≥8× Chinchilla + large batch; the Wen-et-al paper shows it underperforms Muon at our regime. Note for later (300M+). | Preconditioner memory + complexity vs Muon. |
| AdEMAMix / D-Muon / MARS (Semenov et al., 2509.01440) [arxiv](https://arxiv.org/pdf/2509.01440) | AdEMAMix has consistent edge as iterations grow; sign-based methods (Lion, MARS) benefit from large batch. | Keep on the radar; not first-cut. | Crowded space, only ~5% wins over Muon. |

## Axis 3 — Hybrid linear-attention training

| Source | What it provides | How we adopt | Risk / cost |
|---|---|---|---|
| Qwen3-Next tech blog [blog](https://01.me/en/2025/09/qwen3-next/) + vLLM intro [vllm](https://blog.vllm.ai/2025/09/11/qwen3-next.html) + NVIDIA Dev blog | Confirms **3:1 Gated-DeltaNet:Full-Attention** layer ratio (75/25) is sufficient to retain ICL + needle-in-haystack; this is the architecture we already target with Qwen3.5. | We already match this — main action: keep the 3:1 pattern in our from-scratch 25M (don't go pure-linear, the 25% softmax matters for downstream eval). | None — alignment check. |
| Mamba2 SSD recipe (Gu/Dao, goombalab; Bamba) [goombalab](https://goombalab.github.io/blog/2024/mamba2-part1-model/) [hf-bamba](https://huggingface.co/blog/bamba) | Stable training tricks: WSD scheduler (warmup-stable-decay), fp16 compute + fp32 accumulation works due to SSD reformulation, gradient checkpointing, high-quality data in decay phase. | Adopt **WSD over cosine** for the from-scratch run (matches nanochat's trapezoidal); enable selective grad-checkpoint only on Mamba2 layers; bf16 compute + fp32 master weights (we already do this). | WSD vs cosine is wash at fixed budget — value is restart-friendliness. |
| LightOn "Passing the Torch" Mamba post [lighton](https://lighton.ai/lighton-blogs/passing-the-torch-training-a-mamba-model-for-smooth-handover) | Distillation + scratch-train hybrids; concrete loss curves. | Reference only — we're scratch-training, not distilling. | — |

## Axis 4 — Kernel-level speed (≥1.5× tok/s)

| Source | What it provides | How we adopt | Risk / cost |
|---|---|---|---|
| FlashAttention-3 (Dao et al.) [tridao](https://tridao.me/blog/2024/flash3/) [openreview](https://openreview.net/forum?id=tVConYid20) | 1.5–2× over FA2 on H100 via WGMMA async + fp8. **Hopper-only path.** On sm_89 / 4070, FA3 falls back to FA2-class kernels (no WGMMA). | **Don't port FA3.** Stick with FA2 / our TileLang prefill on sm_89. FA3 is a no-op on Ada. | Wasted effort if pursued. |
| FlashAttention-2 on Ada + cuDNN 9 attention [gh-issue-533](https://github.com/Dao-AILab/flash-attention/issues/533) [nvidia](https://developer.nvidia.com/blog/next-generation-of-flashattention/) | FA2 is supported on sm_89; cuDNN 9 attention is competitive with FA2 on Ada and is shipped via cudarc-friendly C API. | We already have TileLang BF16 prefill. For *training backward*, evaluate cuDNN fused MHA backward vs writing a TileLang backward — cuDNN is cheaper to integrate. | cuDNN dispatch adds runtime dep; numerical parity with TileLang fwd must be tested. |
| Liger Kernel (LinkedIn, 2410.10989) [gh](https://github.com/linkedin/Liger-Kernel) [arxiv](https://arxiv.org/abs/2410.10989) | Triton fused kernels: **FusedLinearCrossEntropy 4× memory reduction**, RMSNorm/RoPE/CE up to 7.5×; +20% overall throughput, –60% memory at LLaMA-3-8B. Works on Ada (Triton-compiled). | **Highest-ROI kernel adoption.** Specifically port (or wrap) **FusedLinearCrossEntropy**, **RMSNorm fused fwd+bwd**, **fused RoPE**, and **fused SwiGLU** into `crates/cuda-kernels/csrc/`. These are LLaMA-shape kernels; Qwen3.5 layout matches. | Triton vs our native CUDA C: either compile Triton-via-TileLang or hand-port. Hand-port is ~2-3 d each but matches our stack. |
| Dataset Decomposition / SlimPack (Apple ML, 2405.13226; 2509.26246) [apple](https://machinelearning.apple.com/research/dataset-decomposition) [arxiv](https://arxiv.org/html/2509.26246) | Variable-length packing with per-document masking; fewer optimization steps for same perplexity; SlimPack fixes fwd/bwd asymmetry. | Adopt **packed-sequence training with doc-mask** (we already use FlexAttention-style additive masks on Metal — port the masking convention to CUDA training path). Combine with FlexAttention sliding-window mask from speedrun. | Doc-mask correctness gate; perplexity baseline must be regenerated. |
| torch.compile reduce-overhead [pytorch-blog](https://pytorch.org/blog/accelerating-generative-ai-2/) [hf](https://huggingface.co/docs/transformers/en/perf_torch_compile) | 40–78% speedups on small GPT-2 training; CUDA-graph capture removes per-step launch overhead. | **N/A — we're Rust-native, no PyTorch.** But the *principle* (CUDA-graph capture of the training step) is directly applicable: ARLE already does CUDA-graph capture on the inference path; lift that to training step capture. | Graph capture of training step with dynamic shapes is the hard part — fix shape via sequence packing first. |
| FP8 training on Ada [nvidia](https://developer.nvidia.com/blog/floating-point-8-an-introduction-to-efficient-lower-precision-ai-training/) [lmsys](https://www.lmsys.org/blog/2025-11-25-fp8-rl/) | Ada (sm_89) has 4th-gen TC with fp8 E4M3/E5M2. Training-stable recipes (TWEO, per-tensor scaling) emerged in 2025. NVIDIA's official recipe still gates on Hopper/Blackwell Transformer Engine. | **Defer.** sm_89 fp8 GEMM works but fp8 training stability + scaling is a multi-week side-quest with marginal wallclock win at 25M (we're memory-bandwidth-bound on Ada, not flops-bound). | High implementation cost, low ROI at 25M. |

## Synthesis — what to adopt first, what to skip

**Adopt now (in order of leverage, all evidence-anchored):**

1. **Muon for hidden-matrix params + AdamW for embeddings/norms/scalars** — Tyler Romero attributes ~46% of the modded-nanogpt speedup to Muon alone, Wen-et-al confirms ~1.4× wallclock at 100M-class. Implement Newton-Schulz in `crates/autograd/`; we already have the bf16 matmul kernels in `crates/cuda-kernels/`. This single change plausibly delivers ~3-4 h of the 10 h → 2 h gap.
2. **Liger-style fused kernels: FusedLinearCrossEntropy + RMSNorm + RoPE + SwiGLU** — Liger reports +20% throughput / –60% memory at LLaMA-3-8B; on a 16 GB 4070 the memory headroom directly enables larger batch/seq, compounding with Muon. Hand-port to `crates/cuda-kernels/csrc/`; biggest single kernel win is FusedLinearCE (skips materializing logits).
3. **Packed-sequence training with document-masking FlexAttention + WSD LR + ReLU² + no-grad-clip** — these are the *cheap* speedrun tricks that together attribute ~50% of modded-nanogpt's post-Muon speedup (~36% from FlexAttention+doc-mask, plus the arch/LR bundle). Packing also unblocks training-step CUDA-graph capture (fixed shapes).

**Don't bother with (for the 25M-on-4070 target):**

- **FlashAttention-3 port and fp8 training**. FA3's WGMMA path is Hopper-only; on sm_89 it falls back to FA2-class kernels. fp8 training on Ada is feasible but the stability work (TWEO/per-tensor scaling, master-bf16 fallback) is multi-week with marginal wallclock win at 25M where we are bandwidth-bound, not flops-bound. Revisit only when we move to ≥1B-class or Hopper hardware.
- **SOAP / Shampoo**. Wen-et-al + Semenov-et-al both show SOAP only beats Muon at ≥8× Chinchilla + large batch — outside our 25M single-GPU regime. Note for the 300M+ track.

End. Expected combined wallclock impact of items 1–3: ≈3.5–5× on a 25M from-scratch run, putting the 10 h → 2 h target in reach without resorting to fp8 or Hopper-only kernels.
