Llm Opd Tutorial En

§0 TL;DR Cheat Sheet

1. OPD = On-Policy Distillation: the student samples trajectories using its own current policy; the teacher provides per-token supervision (KL / log-prob / soft label) on the states that the student itself visits. It is not a typo for DPO, and not Online Preference Distillation — in the LLM context this term refers specifically to "on-policy distillation". Representative papers: MiniLLM (Gu 2024 ICLR, arXiv 2306.08543), GKD (Agarwal 2024 ICLR, arXiv 2306.13649), Thinking Machines blog (Lu 2025-10-27), Qwen3 Technical Report (May 2025, arXiv 2505.09388), Survey (Song & Zheng 2026, arXiv 2604.00626).
2. Core loss (writeable in one line): sample $y \sim \pi_\theta(\cdot|x)$, compute reverse KL per token, defined as $L_{\text{OPD}}(\theta) = \mathbb{E}_{x \sim D,\, y \sim \pi_\theta}[\sum_t D_{\text{KL}}(\pi_\theta(\cdot|x, y_{student samples on its own), and the KL direction is $\pi_\theta \| \pi_T$ (reverse / mode-seeking) — these two points are the essential distinction between OPD and vanilla KD. Full formula: $$L_{\text{OPD}}(\theta) = \mathbb{E}_{x \sim D,\, y \sim \pi_\theta}\!\left[\sum_{t=1}^{|y|} D_{\text{KL}}\!\big(\pi_\theta(\cdot|x, y_{

3. Three distillation paradigms in a nutshell:

   • SFT / Hard distillation: teacher generates $\hat y$, student does cross-entropy on $\hat y$ (off-policy + hard label)
   • Vanilla KD / Soft distillation (Hinton 2015): teacher generates $\hat y$, student matches teacher's soft logits (off-policy + soft label, forward KL, mode-covering)
   • OPD: student generates $y$ itself, teacher computes logits on $y$ to give a KL signal (on-policy + soft label, reverse KL, mode-seeking)
4. Why on-policy is crucial: off-policy distillation suffers from exposure bias — during training the teacher's prefixes are "perfect", but at inference the student encounters its own erroneous prefixes which it has never seen, so errors compound (cumulative error roughly $O(L^2)$ with sequence length $L$, see §1.3). OPD aligns the training distribution with the inference distribution, squeezing the compound error from $O(L^2)$ down to $O(L)$.
5. OPD vs RL (the headline selling point): the Thinking Machines blog provides an empirical rule of thumb — RL teaches $O(1)$ bits per trajectory (one outcome reward), OPD teaches $O(N)$ bits (each token gets supervised by the teacher's soft label). On Qwen3-8B-Base + Qwen3-32B-teacher math reasoning experiments, OPD matches RL's AIME'24 gain, but compute drops to roughly 1/9-1/30. The Qwen3 Tech Report independently reports OPD ≈ RL in performance but only 1/10 GPU hours.
6. Relation to DPO / GRPO: (a) DPO uses offline preference pairs for closed-form RLHF, no teacher logits, no student rollouts — almost orthogonal to OPD; (b) GRPO uses group-relative advantage for on-policy RL, has student rollouts but reward is sparse outcome; (c) OPD = GRPO's "dense teacher KL replacing sparse outcome reward" version. The Survey (Song 2026) gives a unifying view: OPD ≈ "KL-constrained RL with $\beta \to \infty$ and token-level reward from teacher log-prob".
7. 2025-2026 industrial adoption: Qwen3 (off-policy + on-policy two-stage distillation of small models), DeepSeek-R1 distillation series (off-policy SFT primarily, but follow-ups use OPD), Gemma 2/3, MiMo-V2, Kimi distill series. Thinking Machines (Murati's team, 2025) packages OPD as "the cheap RL alternative" route.
8. Three most-tested footguns: (a) Length inflation / truncation collapse — student rollouts grow longer over training, hitting truncation triggers gradient bias and validation plummets (Demystifying OPD 2026, arXiv 2604.08527); (b) Reverse KL mode collapse — student converges to a single mode of the teacher, generation diversity collapses; (c) Teacher / student gap too large: the student cannot sample states where the teacher has high probability, OPD signal becomes near zero ("prefix teach, suffix fade" phenomenon, arXiv 2605.13643).

§1 Intuition: Why On-Policy Distillation Is Needed

1.1　A Brief History of Knowledge Distillation (Hinton → DistilBERT → MiniLLM → OPD)

The core idea of distillation has not changed since 2015: train the student with the teacher's soft labels. But for LLM autoregressive generation, there is a hidden assumption that needs to be re-examined — what distribution does the prefix seen during training come from.

The key evolution: off-policy → on-policy is the watershed step that upgrades LLM distillation from "imitating the dataset" to "imitating the decision process".

1.2　Forward KL vs Reverse KL: mode-covering vs mode-seeking

Vanilla KD (Hinton) uses forward KL:

$$D_{\text{KL}}(\pi_T \,\|\, \pi_\theta) = \sum_y \pi_T(y) \log \frac{\pi_T(y)}{\pi_\theta(y)}$$

The expectation is taken under $\pi_T$; only places where the teacher assigns high probability contribute to the loss. If $\pi_T(y) > 0$ but $\pi_\theta(y) = 0$, the loss explodes → the student is forced to cover every mode of the teacher (mode-covering / mass-covering), even modes that are unreachable for the student. The result is that the student spreads probability across many tokens, generating "safe but mediocre" answers.

MiniLLM switches to reverse KL:

$$D_{\text{KL}}(\pi_\theta \,\|\, \pi_T) = \sum_y \pi_\theta(y) \log \frac{\pi_\theta(y)}{\pi_T(y)}$$

The expectation is taken under $\pi_\theta$; only places the student itself samples contribute to the loss. If $\pi_T(y) = 0$ but $\pi_\theta(y) > 0$, the loss explodes → the student is forced to avoid tokens the teacher considers impossible (mode-seeking / zero-forcing). The result is that the student converges to a single high-quality mode of the teacher and generates "sharp and coherent" answers.

• Forward KL makes the student learn a trimodal averaged distribution, where each mode has some probability but none is sharp enough
• Reverse KL makes the student lock onto one of the modes (which mode is determined by init + sampling), producing the most confident path closest to the teacher's decision

For LLM generation tasks this is usually a good thing — we want "fluent, confident, correct" output, not "averaged, vague, mediocre" output. This is also the key insight behind MiniLLM being the first to apply reverse KL to LLM distillation.

1.3　Exposure bias: the structural flaw of off-policy distillation

Consider an autoregressive generation task with sequence length $L$. Off-policy distillation (including SFT / vanilla KD / seq-level KD) uses prefixes from the teacher or dataset during training; at inference, prefixes come from the student itself.

Assume the per-step error rate (the probability that the student deviates from the teacher on some prefix) is $\epsilon$. Off-policy training only "teaches" the student on teacher prefixes, so at inference once the student takes one wrong step, subsequent steps enter a state distribution never seen during training — the student's error rate is no longer $\epsilon$, but $\epsilon' > \epsilon$ (often jumping to 0.5+ random level).

Bagnell 2010 proved for imitation learning: the cumulative error of off-policy imitation is $O(L^2 \epsilon)$ (compound error), whereas on-policy supervision (such as DAgger) compresses cumulative error to $O(L \epsilon)$. This is precisely OPD's theoretical motivation on LLMs — align the training state distribution with the inference state distribution.

The Survey (Song & Zheng 2026 arXiv 2604.00626) repeats this result in §3:

"The exposure bias of off-policy distillation grows roughly quadratically with sequence length; on-policy distillation linearizes the cumulative error."

This theoretical guarantee is the fundamental reason OPD is especially effective on long sequence + high complexity tasks like long-CoT, agent, and code.

1.4　Bit-rate perspective: dense teacher signal vs sparse RL reward

The Thinking Machines blog (Lu 2025-10) gives a clean information-theoretic intuition:

In other words, OPD "teaches" the student a whole distribution at every token, while RL only "teaches" a single number at the end of the sequence. On tasks like math reasoning, code, and long-form QA that need dense supervision, OPD's sample efficiency is significantly higher than RL's. This also explains why Qwen3 / Thinking Machines both report OPD reaching RL performance with an order of magnitude less compute.

§2 Precise Definition and Core Formula of OPD

2.1　Formalization

Given:

• prompt distribution $D$ (e.g., distribution of instructions in the dataset)
• teacher policy $\pi_T$ (fixed, frozen, parameter count $\gg$ student)
• student policy $\pi_\theta$ (trainable)

The OPD objective (general form, from Survey arXiv 2604.00626 §2.1):

$$\boxed{\;L_{\text{OPD}}(\theta) = \mathbb{E}_{x \sim D}\,\mathbb{E}_{y \sim \pi_\theta(\cdot|x)}\!\left[\sum_{t=1}^{|y|} D_f\!\big(\pi_\theta(\cdot|x, y_{

where $D_f$ is some $f$-divergence, with common choices:

Key point: regardless of which $D_f$ is chosen, the expectation subscript must contain $y \sim \pi_\theta$ — the student samples on its own; this is the definition of "on-policy".

2.2　Reverse KL expansion + token-level loss

Expanding reverse KL on each prefix $s_t = (x, y_{

$$D_{\text{KL}}\!\big(\pi_\theta(\cdot|s_t)\,\|\,\pi_T(\cdot|s_t)\big) = \sum_{v=1}^{V} \pi_\theta(v|s_t) \left[\log \pi_\theta(v|s_t) - \log \pi_T(v|s_t)\right]$$

This is the full KL ("full-distribution" form), requiring the teacher to do one forward pass per prefix to obtain logits over the entire vocab. Computational cost: $O(B \cdot L \cdot V)$ teacher forward + softmax.

In practice, the "sampled-token" approximation is common — only compute on the token $v = y_t$ that the student actually sampled:

$$\hat L_t = \log \pi_\theta(y_t|s_t) - \log \pi_T(y_t|s_t)$$

This is a single-sample Monte-Carlo estimate of reverse KL ($\mathbb{E}_{v \sim \pi_\theta}[\log \pi_\theta(v) - \log \pi_T(v)] \approx \log \pi_\theta(y_t) - \log \pi_T(y_t)$ when $y_t \sim \pi_\theta$). This form is called per-token reverse KL and is the version adopted by the Thinking Machines blog and Tinker cookbook implementations.

• Full KL: high information density, low variance, but requires a vocab-level forward by the teacher at every step (expensive, especially if the teacher is large)
• Sampled-token KL: cheap (the teacher only computes one number $\log \pi_T(y_t)$), but high variance; a control-variate baseline can reduce variance (see §4.4)

2.3　Two implementation routes: full-vocab supervised KL vs REINFORCE

OPD's "theoretical objective" is for the student to minimize the per-state KL on its own state visitation distribution $\rho_{\pi_\theta}$:

$$L_{\text{OPD}}(\theta) = \mathbb{E}_{s \sim \rho_{\pi_\theta}}\!\left[D_{\text{KL}}\!\big(\pi_\theta(\cdot|s)\,\|\,\pi_T(\cdot|s)\big)\right]$$

But $\theta$ appears simultaneously in (a) the inner KL and (b) the state visitation $\rho_{\pi_\theta}$. Production implementations choose to decouple these two $\theta$-dependencies — that is what MiniLLM / GKD / Thinking Machines / Qwen3 actually do:

Route A: Full-vocab supervised KL + stop-grad rollouts (the textbook-clean path, simplest when teacher full logits are available)

Treat the rollout as data produced by a "behavior policy" (stop-grad on $\theta$), and backprop only through the inner KL:

$$\boxed{\;L_{\text{OPD}}^{\text{A}}(\theta) = \mathbb{E}_{s_t \sim \text{rollout}(\pi_{\theta^-})}\!\left[\sum_t \sum_{v \in V} \pi_\theta(v|s_t) \log\frac{\pi_\theta(v|s_t)}{\pi_T(v|s_t)}\right]\;}$$

Here $\pi_{\theta^-}$ denotes that $\theta$ is stop-grad during the rollout phase (same idea as PPO's old policy); the inner full-vocab sum is directly differentiable, autograd handles it, no REINFORCE needed. This is the form adopted by the §4.1 code below.

Route B: Sampled-token REINFORCE / importance-sampling estimator (Tinker default / MiniLLM trajectory PG form)

Route B first requires clarifying two distinct gradient computations that are often conflated in the literature:

(B1) Fixed-state inner KL — assume prefix $s_t$ is given, do REINFORCE estimation only on the inner $D_{\text{KL}}(\pi_\theta(\cdot|s_t)\,\|\,\pi_T(\cdot|s_t))$: $$\nabla_\theta D_{\text{KL}}(s_t) = \mathbb{E}_{y_t \sim \pi_\theta(\cdot|s_t)}\!\big[\nabla_\theta \log \pi_\theta(y_t|s_t) \cdot G_t^{\text{detach}}\big],\quad G_t = \log\tfrac{\pi_\theta(y_t|s_t)}{\pi_T(y_t|s_t)}.$$ This is the form used by §4.4's vOPD estimator, unrelated to state visitation along the trajectory.

(B2) Trajectory objective with state visitation — to strictly retain $\rho_{\pi_\theta}$'s dependence on $\theta$, the policy gradient requires the return-to-go (sum of all future KL costs) as the score-function weight: $$\nabla_\theta\,\mathbb{E}_\tau\!\Big[\sum_t D_{\text{KL}}(s_t)\Big] = \mathbb{E}_\tau\!\Big[\sum_u \nabla_\theta \log\pi_\theta(a_u|s_u)\cdot \underbrace{\sum_{t \ge u} D_{\text{KL}}(s_t)^{\text{detach}}}_{\text{return-to-go}}\Big] + \mathbb{E}_\tau\!\Big[\sum_t \nabla_\theta D_{\text{KL}}(s_t)\Big].$$ Note the score-function weight is not the same-token $G_t$, but the sum of KLs over all subsequent steps.

Unified practical rules:

• Route A (pedagogically clear, §4.1): full-vocab KL + stop-grad rollouts, autograd directly, no REINFORCE needed. Zero variance, but requires teacher full logits.
• Route B1 (§4.4 vOPD-style, Tinker / MiniLLM default): sampled-token REINFORCE / IS estimator + control variate. Equivalent in expectation to A, with higher variance but compatible with the sampled-token interface, saves teacher vocab memory, supports black-box teachers.
• You cannot directly .backward() on the sampled-token $\log(\pi_\theta/\pi_T)$ — that is pathwise gradient + fixed sampled index, dropping the score-function term, and is not the reverse-KL gradient (neither MLE nor a KL descent direction)

2.4　Relation to KL-constrained RL

Consider RL with a reverse-KL constraint (the standard RLHF objective):

$$\max_\theta\, \mathbb{E}_{y \sim \pi_\theta}\!\big[R(x, y)\big] - \beta\, \mathbb{E}_{y \sim \pi_\theta}\!\big[D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}})\big]$$

OPD is the special case of this formula when $R \equiv 0$ (no external reward) and $\pi_{\text{ref}} = \pi_T$ (reference replaced with the teacher):

$$\min_\theta\, \mathbb{E}_{y \sim \pi_\theta}\!\big[D_{\text{KL}}(\pi_\theta \,\|\, \pi_T)\big]$$

This means OPD is RLHF with "pure KL term, teacher as reference". From this perspective, GRPO + KL term is equivalent to OPD + GRPO-style group baseline; this is why the Survey classifies OPD as "a special case of KL-constrained RL", and it is also the formal basis for integrating OPD into GRPO in §3.

2.5　Forward KL form ("Sampled-Token OPD")

Some works (e.g., parts of Qwen3, TRL GKD trainer) use forward KL on student samples:

$$L_{\text{OPD-fwd}}(\theta) = \mathbb{E}_{y \sim \pi_\theta}\!\left[\sum_t D_{\text{KL}}\!\big(\pi_T(\cdot|s_t)\,\|\,\pi_\theta(\cdot|s_t)\big)\right]$$

Note that the sample comes from $\pi_\theta$, but the KL direction is forward. Expanded to the token level:

$$L_{\text{OPD-fwd}} = \mathbb{E}_{y \sim \pi_\theta}\!\left[\sum_t \sum_v \pi_T(v|s_t)\big(\log \pi_T(v|s_t) - \log \pi_\theta(v|s_t)\big)\right]$$

Since $\theta$ appears only in $\log \pi_\theta$ and not in the expectation subscript (although $y \sim \pi_\theta$ still depends on $\theta$, after stop-gradient treatment the $\theta$-dependence of the forward term simplifies), training degenerates into cross-entropy on the teacher distribution at each student-visited prefix — formally, this is soft-label cross-entropy on student rollouts. This is the engineering-simplest OPD implementation (no REINFORCE / control variate needed), at the cost of losing reverse KL's mode-seeking property.

§3 The Relationship of OPD with DPO / GRPO / RLHF (Unified View)

3.1　Big-picture comparison table

• vs SFT: OPD trains on student rollouts (SFT trains on teacher/dataset)
• vs vanilla KD: OPD is on-policy + reverse KL (vanilla KD is off-policy + forward KL)
• vs RL (PPO/GRPO): OPD uses teacher log-prob as dense reward (RL uses RM as sparse reward)

3.2　OPD vs DPO

DPO (Rafailov 2023 NeurIPS, arXiv 2305.18290) is closed-form RLHF: plug the optimal solution of KL-regularized RL back into Bradley-Terry, eliminate the partition $\log Z$, and obtain a pairwise preference loss:

$$L_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)}\!\left[\log \sigma\!\big(\beta \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\big)\right]$$

DPO is completely offline: no sampling, no teacher, no RM, no critic, only preference pairs.

The two can be stacked: first DPO to align human preferences, then OPD to compress to a small student with a stronger teacher. Production pipelines like Qwen3 / R1-Distill all use similar recipes.

3.3　OPD vs GRPO

GRPO (DeepSeekMath 2024, arXiv 2402.03300) replaces PPO's critic with group-normalized advantage:

$$\hat A_i = \frac{r_i - \text{mean}(\mathbf r)}{\text{std}(\mathbf r)}, \quad r_i = R(x, y_i),\; i \in [1..G]$$

$$L_{\text{GRPO}}(\theta) = \mathbb{E}_x \frac{1}{G}\sum_i \frac{1}{|y_i|}\sum_t \min\!\big(\rho_t^i \hat A_i,\, \text{clip}(\rho_t^i, 1{-}\epsilon, 1{+}\epsilon) \hat A_i\big) - \beta\, D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}})$$

where $\rho_t^i = \pi_\theta / \pi_{\theta_{\text{old}}}$ is the importance ratio.

The relation between OPD and GRPO: replace GRPO's sparse outcome reward $R(x, y_i)$ with per-token teacher KL reward $r_t^i = \log\pi_T(y_t^i|s_t^i) - \log\pi_\theta^{\text{old}}(y_t^i|s_t^i) = -\log\!\frac{\pi_\theta^{\text{old}}(y_t^i|s_t^i)}{\pi_T(y_t^i|s_t^i)}$ (note: this must be a log-ratio, not the LaTeX-prone misreading $-\log\pi_\theta/\pi_T$), and set the reference $\pi_{\text{ref}} \leftarrow \pi_T$; the GRPO loss degenerates to the policy-gradient form of OPD (see §2.3).

• outcome reward from the verifier (math correct / test pass)
• dense reward from per-token teacher KL
• Total reward $r_t = \alpha \cdot r_{\text{outcome}} \cdot \mathbb{1}[t = T] + (1-\alpha) \cdot r_{\text{teacher-KL}}$

This gives both the task-aligned signal from outcome and dense supervision from the teacher — it is the standard for small-model post-training in the Qwen3 / DeepSeek series.

3.4　OPD vs vanilla RL distillation (R1-Distill)

The DeepSeek-R1-Distill series (Qwen / Llama 1.5B-70B) uses off-policy distillation:

1. The R1 teacher generates 800K reasoning trajectories (math + code + general)
2. The small student does token-level cross-entropy SFT
3. No on-policy rollout, no KL signal

This is hard-label seq-level KD (the LLM version of Kim & Rush 2016), not OPD. The success of R1-Distill is mainly due to extremely high-quality teacher trajectories (R1 was RL-tuned) + large data volume; structurally it remains off-policy.

OPD and R1-Distill are complementary:

• R1-Distill trajectory data can serve as OPD's init / SFT warm-start
• Running OPD after R1-Distill can further eliminate exposure bias

Academically / engineeringly this combination is called "Off-policy SFT + On-policy distillation", which is the standard two-stage Qwen3 small-model recipe (see §6.1).

§4 Implementation: Core Code Blocks

4.1　Per-token reverse KL (Route A, full-vocab conceptual implementation)

import torch
import torch.nn.functional as F


def per_token_reverse_kl_loss(
    student,        # student model, requires_grad
    teacher,        # teacher model, frozen
    input_ids,      # [B, L]  prompt + student rollout (rollout done externally, stop_grad)
    action_mask,    # [B, L]  1 for student-generated tokens, 0 for prompt / pad
):
    """
    Route A — full-vocab per-token reverse KL on student rollouts (concept-clean form).
    Requires teacher full logits to be available. Route B (Tinker / MiniLLM production default)
    is equivalent in expectation but uses sampled-token + IS / REINFORCE estimator,
    see §2.3 callout and §4.4.

        L(θ) = E_{s_t ~ rollout(π_{θ⁻})} [ Σ_t D_KL(π_θ(·|s_t) || π_T(·|s_t)) ]
             = E_{s_t} [ Σ_t Σ_v  π_θ(v|s_t) · ( log π_θ(v|s_t) − log π_T(v|s_t) ) ]

    Rollout is done externally with stop_grad(θ) (vLLM / TGI returns input_ids);
    this function **only** computes the differentiable full-vocab KL, autograd
    backprops directly, **no REINFORCE needed**.
    """
    # Student full-vocab log-probs (requires_grad on θ)
    s_logits = student(input_ids).logits                   # [B, L, V]
    s_log_probs = F.log_softmax(s_logits[:, :-1], dim=-1)  # [B, L-1, V]
    s_probs = s_log_probs.exp()                            # [B, L-1, V]

    # Teacher full-vocab log-probs (frozen, no grad)
    with torch.no_grad():
        t_logits = teacher(input_ids).logits
        t_log_probs = F.log_softmax(t_logits[:, :-1], dim=-1)  # [B, L-1, V]

    # Full-vocab reverse KL per state s_t: Σ_v π_θ(v) (log π_θ(v) − log π_T(v))
    kl_per_token = (s_probs * (s_log_probs - t_log_probs)).sum(dim=-1)  # [B, L-1]

    # Mask to student-generated positions (KL on prompt tokens is meaningless)
    mask = action_mask[:, 1:].float()
    loss = (kl_per_token * mask).sum() / mask.sum().clamp_min(1.0)

    # Diagnostic: at sampled positions, is the student more "confident" than the teacher (monitor over-confidence)
    with torch.no_grad():
        targets = input_ids[:, 1:].unsqueeze(-1)
        s_logp_t = s_log_probs.gather(-1, targets).squeeze(-1)
        t_logp_t = t_log_probs.gather(-1, targets).squeeze(-1)
        overconf_frac = ((s_logp_t > t_logp_t).float() * mask).sum() / mask.sum().clamp_min(1.0)
    return loss, {
        "full_vocab_kl": loss.item(),
        "overconf_frac": overconf_frac.item(),
    }

A few production-implementation details to note:

• action_mask must only mask student rollout tokens; KL on prompt tokens is meaningless (the teacher is only conditioning)
• Use torch.no_grad() for the teacher forward; otherwise memory doubles
• Full-vocab (Route A) vs sampled-token (Route B): the loss above is Route A (summed over $V$ tokens, autograd directly backprops). Route B (sampled-token + REINFORCE/IS, §4.4) is the production-default form in implementations like Tinker / MiniLLM, sharing the sampled-token interface with PPO/GRPO and supporting black-box teachers; A and B are equivalent in expectation. Regardless of which you choose, you cannot directly .backward() on sampled-token log π_θ − log π_T — that is equivalent to taking the pathwise $\nabla\log\pi_\theta(y_t)$ at a fixed sampled index, losing the score-function term, and is neither the reverse-KL gradient nor MLE. Route B must use a REINFORCE estimator (detached reward + score-function trick)
• Batched teacher inference is typically asynchronous in production: spin up a teacher server with vLLM, send the student rollout batch and fetch log-probs; only run student forward + backward locally. This is the standard for Tinker / vLLM-based frameworks
• Mixed precision: student in bf16, but compute teacher log-softmax in fp32 (avoid numerical instability)

Shared helpers (used throughout §4.2 - §4.6)

def per_token_logp(model, input_ids):
    """
    Returns log π(y_{t+1}|s_t) per position. Shape [B, L-1]. Grad behavior decided by the caller
    (want grad: call directly; want detached behavior log-prob: wrap in torch.no_grad()).
    """
    logits = model(input_ids).logits[:, :-1]                     # [B, L-1, V]
    log_probs = F.log_softmax(logits, dim=-1)
    targets = input_ids[:, 1:].unsqueeze(-1)                     # [B, L-1, 1]
    return log_probs.gather(-1, targets).squeeze(-1)             # [B, L-1]


def teacher_kl_reward(student, teacher, input_ids, action_mask):
    """
    The teacher-vs-behavior KL used as dense token reward in OPD-GRPO:
        r_t = log π_T(y_t|s_t) - log π_θ_old(y_t|s_t)
    Input action_mask shape [B, L] (same as input_ids); internally shifted to [B, L-1].
    All under no_grad; returns scalar reward per trajectory (sum over generated tokens).
    """
    with torch.no_grad():
        s_old = per_token_logp(student, input_ids)               # [B, L-1]
        t_lp  = per_token_logp(teacher, input_ids)               # [B, L-1]
        token_mask = action_mask[:, 1:].float()                  # aligned to [B, L-1]
        per_tok = (t_lp - s_old) * token_mask
    return per_tok.sum(dim=-1)                                   # [B]

4.2　Full OPD training loop (with student rollout)

@torch.no_grad()
def student_rollout(student, prompts, max_new_tokens=512, temperature=1.0):
    """Student samples trajectory on its own (the critical on-policy step)"""
    out = student.generate(
        prompts.input_ids,
        attention_mask=prompts.attention_mask,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=temperature,
        return_dict_in_generate=True,
    )
    full_ids = out.sequences           # [B, L_prompt + L_gen]
    prompt_len = prompts.input_ids.shape[1]
    action_mask = torch.zeros_like(full_ids)
    action_mask[:, prompt_len:] = 1
    # mask out pad
    action_mask = action_mask * (full_ids != student.config.pad_token_id).long()
    return full_ids, action_mask


def opd_train_step(student, teacher, optimizer, batch, kl_coef=1.0):
    """one step of OPD: sample rollout, compute teacher KL, update student"""
    # 1. student samples (on-policy)
    input_ids, action_mask = student_rollout(student, batch.prompts)

    # 2. compute per-token reverse KL with teacher
    loss, stats = per_token_reverse_kl_loss(
        student, teacher, input_ids, action_mask
    )

    # 3. backward
    optimizer.zero_grad()
    (kl_coef * loss).backward()
    torch.nn.utils.clip_grad_norm_(student.parameters(), max_norm=1.0)
    optimizer.step()
    return stats

This is the simplest OPD training loop. Production code adds advantage / baseline / clipping in step 2 (see §4.4), and group sampling (GRPO-style, see §4.5) in step 1.

4.3　Off-policy KD vs OPD comparison code (one line apart)

# ── Off-policy KD (vanilla, Hinton-style) ──
def off_policy_kd_loss(student, teacher_outputs, dataset_y):
    """
    teacher_outputs computed by forwarding on dataset_y
    student also forwards on dataset_y
    forward KL: teacher → student
    """
    s_logits = student(dataset_y).logits
    t_logits = teacher_outputs                 # already computed
    return F.kl_div(
        F.log_softmax(s_logits, dim=-1),
        F.softmax(t_logits, dim=-1),
        reduction="batchmean",
    )


# ── On-Policy Distillation (OPD) ──
def opd_loss(student, teacher, prompts):
    student_y, mask = student_rollout(student, prompts)         # ← key diff: student samples on its own
    loss, _ = per_token_reverse_kl_loss(student, teacher, student_y, mask)
    return loss

The only key difference: OPD computes loss on its own sampled trajectory; off-policy computes it on dataset / teacher trajectory. The code differs by only one student_rollout call, but the overall benefit of aligning training distribution with inference distribution is enormous.

4.4　Control Variate Baseline (vOPD-style token-level KL)

Note: the vOPD ("KL for a KL") control-variate idea is a recurring pattern in the OPD literature (Survey 2026 / Tinker blog etc. all have equivalent discussions). Below gives the correct detach + sign + estimator form.

REINFORCE-form sampled-token estimator: for a single sampled token $y_t \sim \pi_\theta(\cdot|s_t)$, the unbiased estimate of the gradient of reverse KL is

$$\nabla_\theta D_{\text{KL}}(\pi_\theta\,\Vert\,\pi_T)(s_t) = \mathbb{E}_{y_t \sim \pi_\theta}\!\big[\,\nabla_\theta \log \pi_\theta(y_t|s_t) \cdot \underbrace{(\log \pi_\theta(y_t|s_t) - \log \pi_T(y_t|s_t))}_{\hat r_t,\;\textbf{detached}}\,\big].$$

Note $\hat r_t$ only appears as a scalar reward (must be detached); the gradient comes entirely from the outer $\nabla_\theta \log\pi_\theta$. Control variate to reduce variance:

$$\hat r_t \leftarrow \hat r_t - B(s_t),\quad B(s_t) = \mathbb{E}_{y_t \sim \pi_\theta}[\hat r_t] = D_{\text{KL}}(\pi_\theta\,\Vert\,\pi_T)(s_t).$$

$B(s_t)$ is the full-vocab KL at that step (with positive sign), computable in closed form from the student forward. Its detached value does not change the unbiasedness of the estimator (because $\mathbb{E}_{y_t}[\nabla \log\pi_\theta \cdot B(s_t)] = B(s_t)\cdot \mathbb{E}[\nabla\log\pi_\theta] = 0$), but significantly reduces variance.

def vopd_token_kl_estimator(student, teacher, input_ids, action_mask):
    """
    Sampled-token REINFORCE estimator of ∇_θ D_KL(π_θ || π_T) with closed-form baseline.

    Returns surrogate loss L_surr, whose ∇L_surr in expectation = ∇ E[D_KL]. Note L_surr
    itself is NOT a numerical estimate of D_KL — for diagnostics use full-vocab D_KL.
    """
    s_logits = student(input_ids).logits[:, :-1]                    # [B, L-1, V]
    s_log_probs = F.log_softmax(s_logits, dim=-1)
    s_probs     = s_log_probs.exp()
    with torch.no_grad():
        t_logits = teacher(input_ids).logits[:, :-1]
        t_log_probs = F.log_softmax(t_logits, dim=-1)

    targets = input_ids[:, 1:].unsqueeze(-1)                        # [B, L-1, 1]
    s_logp_t = s_log_probs.gather(-1, targets).squeeze(-1)          # grad-aware: log π_θ(y_t|s_t)

    # ── reward signal (must detach: scalar weight, not part of grad path) ──
    with torch.no_grad():
        t_logp_t = t_log_probs.gather(-1, targets).squeeze(-1)
        r_hat    = s_logp_t.detach() - t_logp_t                     # scalar per token

        # closed-form baseline B(s_t) = D_KL(π_θ || π_T)(s_t), full vocab
        baseline = (s_probs.detach() * (s_log_probs.detach() - t_log_probs)).sum(-1)
        advantage = r_hat - baseline                                # both detached

    # ── REINFORCE-style surrogate: ∇ = E[ ∇log π_θ(y_t|s_t) · advantage_detached ] = +∇D_KL ──
    mask = action_mask[:, 1:].float()
    surrogate = (s_logp_t * advantage * mask).sum() / mask.sum().clamp_min(1.0)
    #            ↑ Sign note: advantage = (logπ_θ − logπ_T) − B is the KL cost. To minimize D_KL,
    #            the optimizer step is θ ← θ − η·∇L_surr, and ∇L_surr = +E[∇logπ·advantage] = +∇D_KL,
    #            so the surrogate is **positive**; treat surrogate as the loss to .backward().

    # diagnostic (no grad): monitor the actual full-vocab KL
    with torch.no_grad():
        full_kl = (baseline * mask).sum() / mask.sum().clamp_min(1.0)
    return surrogate, {"full_vocab_kl_diag": full_kl.item()}

4.5　Integrating OPD into GRPO (multi-sample group baseline)

GRPO's ratio is new student policy vs old behavior student policy (record s_logp_old during rollout, compute s_logp_new during update); the teacher enters in two independent positions: (i) reward via the token-level KL reward $r_t^{\text{kl}} = \log\pi_T(y_t|s_t) - \log\pi_\theta^{\text{old}}(y_t|s_t)$ (dense token reward form), weighted with the outcome reward to give the total reward; (ii) KL regularization via the explicit $D_{\text{KL}}(\pi_\theta\,\Vert\,\pi_T)$ constraint on the student to stay close to the teacher. The teacher never enters the denominator of the PPO ratio.

def opd_grpo_step(student, teacher, batch, G=8, alpha=0.5, kl_coef=0.1, clip=0.2):
    """
    OPD + GRPO hybrid (per-prompt group baseline):
      - sample G rollouts per prompt, record behavior-policy log-prob
      - reward = α * outcome_reward + (1-α) * sum_t [log π_T(y_t|s_t) - log π_θ_old(y_t|s_t)]
      - GRPO group-relative advantage (group-normalized)
      - ratio = exp(s_logp_new - s_logp_old.detach())  ← critical
      - KL penalty term uses student vs teacher (not the PPO ratio)

    Shape conventions:
      input_ids / action_mask:   [B, L]   (prompt + generated)
      per_token_logp output:     [B, L-1] (predict y_{t+1} from prefix up to t)
      token_mask = action_mask[:, 1:]: aligned with per-token logp
    """
    ids_all, action_mask_all, s_old_all, adv_all = [], [], [], []

    for prompt in batch.prompts:
        rollouts = []
        for _ in range(G):
            ids, action_mask = student_rollout(student, prompt)    # [B', L]; B'=1 for single prompt
            with torch.no_grad():
                s_old = per_token_logp(student, ids)               # [B', L-1] behavior log-prob
                t_lp  = per_token_logp(teacher, ids)               # [B', L-1]
            rollouts.append((ids, action_mask, s_old, t_lp))

        # total reward across G rollouts in the group
        rewards = []
        for (ids, action_mask, s_old, t_lp) in rollouts:
            r_outcome = math_verifier(ids)                         # scalar ∈ {0,1}
            token_mask = action_mask[:, 1:].float()                # [B', L-1]
            r_kl_dense = ((t_lp - s_old) * token_mask).sum().item()
            r_total = alpha * r_outcome + (1.0 - alpha) * r_kl_dense
            rewards.append(r_total)
        # Ensure same device/dtype as s_old / s_logp_new, avoid CPU/GPU mismatch
        rewards = torch.tensor(rewards, device=s_old.device, dtype=s_old.dtype)
        adv_per_traj = (rewards - rewards.mean()) / (rewards.std() + 1e-8)  # [G]

        for i, (ids, action_mask, s_old, _t_lp) in enumerate(rollouts):
            ids_all.append(ids); action_mask_all.append(action_mask)
            s_old_all.append(s_old)
            adv_all.append(adv_per_traj[i].view(1, 1).expand_as(s_old))     # broadcast to [B', L-1]

    ids         = torch.cat(ids_all, dim=0)                                  # [BG, L]
    action_mask = torch.cat(action_mask_all, dim=0)                          # [BG, L]
    token_mask  = action_mask[:, 1:].float()                                 # [BG, L-1]
    s_logp_old  = torch.cat(s_old_all, dim=0).detach()                       # [BG, L-1]
    A           = torch.cat(adv_all, dim=0).detach()                         # [BG, L-1]

    # ── new student log-prob (this step has grad) ──
    s_logp_new = per_token_logp(student, ids)                                # [BG, L-1]

    # ── PPO clipped ratio: new student vs old student (NOT student vs teacher) ──
    ratio = torch.exp(s_logp_new - s_logp_old)
    pg = torch.min(ratio * A, torch.clamp(ratio, 1 - clip, 1 + clip) * A)
    pg_loss = -(pg * token_mask).sum() / token_mask.sum().clamp_min(1.0)

    # ── KL penalty: student vs teacher (OPD reverse KL, full-vocab closed-form) ──
    # Pass [B, L] action_mask; §4.1 internally shifts it to [B, L-1]
    kl_loss, _ = per_token_reverse_kl_loss(student, teacher, ids, action_mask)

    return pg_loss + kl_coef * kl_loss

Practical tip: alpha = 0.5 is an empirical starting point; alpha → 1 degenerates to pure GRPO (outcome only), alpha → 0 degenerates to pure OPD (KL only). Qwen3 / R1-Distill-style recipes usually stage-wise schedule: early-stage small alpha (pull student toward teacher first), late-stage large alpha (push for outcome SOTA).

4.6　Synthetic-data + distillation pipeline (top-level pseudocode)

def synthetic_distillation_pipeline(
    teacher,             # large teacher (e.g., R1-671B / Qwen3-32B)
    student,             # small student (e.g., Qwen3-8B-Base)
    prompt_pool,         # massive prompt pool (can be unlabeled instructions)
    n_stage1=800_000,    # off-policy SFT data volume
    n_stage2_steps=5_000,
    n_stage3_steps=1_000,
    verifiable_prompts=None,
):
    # ─── Stage 1: use teacher to generate synthetic trajectories, do off-policy SFT ───
    synthetic_data = []
    for prompt in random_sample(prompt_pool, n_stage1):           # placeholder sampler
        with torch.no_grad():
            y_teacher = teacher.generate(prompt, max_new_tokens=4096)
        synthetic_data.append((prompt, y_teacher))
    run_sft(student, synthetic_data, epochs=2)                    # cross-entropy on (prompt, y_teacher)

    # ─── Stage 2: do OPD on student's own rollouts ───
    optimizer = torch.optim.AdamW(student.parameters(), lr=1e-5)
    for step in range(n_stage2_steps):
        batch = make_batch(prompt_pool, batch_size=64)
        opd_train_step(student, teacher, optimizer, batch, kl_coef=1.0)

    # ─── Stage 3 (optional): OPD + outcome reward (math/code verifier) ───
    if verifiable_prompts is not None:
        for step in range(n_stage3_steps):
            batch = make_batch(verifiable_prompts, batch_size=64)
            loss  = opd_grpo_step(student, teacher, batch, G=8, alpha=0.5)
            optimizer.zero_grad(); loss.backward(); optimizer.step()

    return student

This is the standard three-stage form of Qwen3 / R1-Distill / Thinking Machines-style recipes. Stage 1 provides cold start (student learns to mimic teacher's style and format), Stage 2 uses OPD to eliminate exposure bias, Stage 3 uses a verifier for task-aligned fine-tuning.

§5 OPD's Position in the Knowledge Distillation Family

5.1　LLM distillation method lineage (from Hinton to OPD)

   2015 ─────────── 2019 ─────── 2023 ────── 2024 ─────── 2025/2026
   Hinton          DistilBERT   MiniLLM     GKD          Thinking
   soft target     (Sanh)       (Gu)        (Agarwal)    Machines
                                                          blog
     │              │            │            │            │
     ↓              ↓            ↓            ↓            ↓
   classification   encoder-     first        generalized   "cheap RL"
   forward KL      level KD     reverse KL   JSD + mix     paradigm + large-
                                + on-policy  on/off        scale empirical
                                              policy        results (Qwen3/
                                                            Gemma/Kimi)

Core evolution directions:

• off-policy → on-policy (handles exposure bias)
• forward KL → reverse KL / JSD (handles mode-covering)
• hard label → soft logit (preserves dark knowledge)
• single-stage → multi-stage hybrid (off-policy SFT cold start + on-policy OPD + outcome RL)

5.2　One-sentence index of key papers

5.3　OPD's position in Reasoning Model distillation (the R1-Distill era)

The two main lines of reasoning model distillation:

When to use which:

• If teacher trajectories are extremely high quality and the teacher-student capability gap is small → off-policy SFT alone is sufficient
• If the student is much smaller than the teacher (>5×), the task is long horizon, and the inference-time distribution is sensitive → OPD is the more stable choice
• In practice, the two are typically stacked: off-policy SFT for cold start, then OPD to eliminate exposure bias

§6 Engineering Case Studies: OPD in Production

6.1　Qwen3 Recipe (arXiv 2505.09388 §3.2)

The post-training of Qwen3 small models (1.7B / 4B / 8B) uses two-stage distillation:

Qwen3-235B (teacher, dual /think + /no_think mode)
    │
    │ Stage 1: Off-policy distillation
    │   - teacher generates trajectories (mix of /think and /no_think)
    │   - student SFT on teacher trajectories (cross-entropy)
    │   - goal: basic reasoning + mode switching
    ↓
Qwen3-8B-mid
    │
    │ Stage 2: On-policy distillation (OPD)
    │   - student samples on its own → teacher gives per-token logits
    │   - per-token reverse KL loss
    │   - goal: eliminate exposure bias + push reasoning depth
    ↓
Qwen3-8B-final

Reported results:

• Saves ~10× GPU hours vs pure RL
• Significant pass@64 improvement on AIME'24 / AIME'25 (showing OPD does not collapse diversity)
• +3-5pp over off-policy SFT on long-CoT tasks

6.2　Thinking Machines Blog (Lu 2025-10-27)

Experimental setting:

• Student: Qwen3-8B-Base
• Teacher: Qwen3-32B
• Task: math reasoning (AIME'24 as primary benchmark)
• Init: one SFT round with OpenThoughts data
• OPD: implemented with Tinker, per-token reverse KL, no outcome reward

Reported results (blog content):

• Training Qwen3-8B-Base with OPD reaches almost identical AIME'24 performance to RL
• Total FLOPs saved by ~9-30×, depending on batch / lr settings
• Emphasizes "OPD is the cheap alternative to RL", not a supplement to RL

6.3　Tinker Cookbook implementation details

GitHub: thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/distillation

Key design points:

• Environment is designed to be no-reward: the only supervision is the teacher KL
• Reverse KL implementation: (student_logp - teacher_logp) * mask
• Advantage computation: treat KL as negative reward fed into advantage (advantage = -kl_penalty_coef * reverse_kl)
• LoRA: rank 128 + lr 1e-4
• Batch: groups_per_batch = 64 (GRPO-style grouping)

This "inject KL as reward into advantage" design exactly corresponds to the OPD-GRPO equivalence in §3.3: replace the sparse outcome reward with dense teacher KL reward.

6.4　DeepSeek-V4 report (multi-teacher OPD replaces RL)

DeepSeek-V4 in the model consolidation stage completely replaces mixed RL with multi-teacher OPD — the logits of multiple teachers (specialists: math / code / reasoning / chat) are weighted-ensembled as the OPD target for the student. This is the representative case of extending the OPD paradigm to multi-teacher and specialist consolidation.

6.5　Black-Box OPD (no teacher logits required)

arXiv 2511.10643 "Black-Box On-Policy Distillation": when the teacher is a closed-source API (e.g., GPT-4 / Claude) from which you can only sample but cannot extract logits, how do you do OPD?

Core idea: use teacher samples as trajectory-level reward, combined with token-level student log-prob to approximate reverse KL. This extends OPD from "requires teacher logits" to "only needs teacher API". The trade-off is losing per-token dense supervision and falling back to trajectory-level reward — essentially a hybrid between OPD and RL.

§7 Failure Modes and Mitigation Strategies

7.1　Truncation Collapse / Length Inflation

Phenomenon (Demystifying OPD 2026, arXiv 2604.08527): during training, the average length of student rollouts keeps growing, eventually hitting max_length truncation; the truncated trajectories contribute most of the gradient (because they have many tokens), causing gradient bias and validation performance crash.

Root cause: the mode-seeking nature of reverse KL biases the student toward "keep emitting high-teacher-probability tokens", and the teacher naturally favors long output on long-CoT. This forms a positive feedback loop.

Mitigation:

• Length normalization: divide the loss by trajectory length (per-token average, not sum) — this is the default in Tinker / most production implementations
• Length penalty: add $-\lambda \cdot L$ to the reward to suppress overly long outputs
• Max-length much larger than sample max-length: avoid truncation entering training
• Early stop on val loss spike: stop immediately when val crashes

7.2　Mode Collapse / Diversity Loss

Phenomenon: reverse KL is mode-seeking, so in theory the student should lock onto one mode of the teacher; but on some tasks (e.g., open-ended generation) the student converges to a single answer template, losing diversity, with pass@N ($N > 1$) dropping.

Mitigation:

• Use forward KL / JSD instead of reverse KL (GKD's $\beta$ interpolation)
• Add entropy bonus (similar to PPO)
• Mixed loss: reverse KL × $\alpha$ + forward KL × $(1-\alpha)$
• MiniLLM's stabilization trick: mixed policy $\pi_{\text{mix}} = (1-\alpha)\pi_\theta + \alpha\pi_T$ with $\alpha = 0.2$ to prevent early mode collapse

7.3　Reward Hacking (teacher-game)

Phenomenon: the student learns to hack — outputs some tokens that have high teacher probability but are semantically garbage (e.g., stopword sequences, repeated phrases), with KL loss extremely low but task performance poor.

Mitigation:

• Mix in outcome reward (the OPD-GRPO hybrid in §4.5)
• Trajectory-level filter: filter out trajectories where the teacher itself is uncertain ("teacher-improbable filter")
• Token diversity regularizer: penalize overly low token entropy

7.4　Prefix Teach, Suffix Fade (local teachability collapse)

arXiv 2605.13643 reports: in late training, the teacher has almost "nothing left to teach" in the latter half of trajectories — the student's and teacher's log-probs on the suffix are already close, and the loss comes almost entirely from the prefix. This leads to poor quality of the second-half reasoning in long-CoT tasks.

Mitigation:

• Token-level importance weighting: weight suffix tokens more
• Teacher upgrade: use a stronger teacher (multi-teacher ensemble / specialists for different tasks)
• Curriculum: start with short trajectories, gradually increase max_length

7.5　Teacher / Student Gap too large

Phenomenon: when the student is much smaller than the teacher (e.g., 1B vs 70B), the student's samples cannot reach regions of high teacher probability, and the OPD signal becomes near zero ("student visits states the teacher never thought of").

Mitigation:

• off-policy SFT warm start: let the student first imitate the teacher for a while to bring the state distributions closer, then run OPD
• Temperature scheduling: high student temperature early in training to enlarge exploration; reduce to 1.0 later
• Use a closer teacher: control the teacher-student gap to 4-8× (Qwen3 choosing a 32B teacher → 8B student reflects this consideration)

7.6　Catastrophic Forgetting (loss of old capabilities)

Phenomenon: OPD is mainly trained on math / reasoning, but the student's general capabilities (chat / safety / instruction-following) degrade.

Mitigation:

• Replay buffer: retain some general SFT data and mix into OPD training in proportion
• Regularization to SFT init: add a $\beta \cdot D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{SFT-init}})$ penalty
• Multi-task OPD: distill multiple domain teachers simultaneously (math + chat + code)

§8 Complexity and Resources

8.1　Per-step training cost comparison

OPD's extra overhead mainly comes from: (1) student rollout (sampling is slower than forward due to KV cache accumulation); (2) teacher forward (teacher is large). But compared to RL, OPD does not need reward model training + RM forward, so the total cost is usually less than a pure RL pipeline.

8.2　Memory

Memory reduction tricks:

• Run the teacher on a separate inference server (vLLM), so the student training machine does not store teacher weights
• Teacher uses fp8 / int8 quantization (teacher only forwards, no backprop; precision loss is small)
• Student uses LoRA (rank 128) + full teacher weights
• Async teacher inference: while the student trains batch $N$, the teacher server computes batch $N+1$

8.3　Sample Efficiency

The core data point from the Thinking Machines blog:

OPD is roughly 9-30× more sample-efficient at the same accuracy — this is the core number behind OPD's rise.

§9 Comparison and Positioning with Related Methods

9.1　Big table (the main interview targets in a cheat sheet)

9.2　Decision tree: which method for which scenario

Do you have a strong teacher model?
├── No
│   ├── have preference dataset → DPO
│   ├── have verifier (math/code) → GRPO
│   └── have RM → PPO RLHF
└── Yes
    ├── teacher-student gap small (< 4×) → off-policy SFT KD is enough
    ├── gap moderate (4-10×) + long CoT task → OPD (first choice)
    ├── gap large (> 10×) + complex task → off-policy SFT warm start + OPD
    ├── also have verifier → OPD + GRPO hybrid (production standard)
    └── teacher only API-accessible → Black-Box OPD

§10 25 Frequently-Asked Interview Questions

L1 must-know (10 questions, post-training engineer / LLM RL roles)

L2 advanced (10 questions, senior post-training / paper reproduction)

L3 top labs (5 questions, research / algorithm lead)

§A Appendix

A.1　Sanity-check: verifying your OPD implementation via convergence

After implementing OPD, do these three micro-tests to confirm the loss is correct:

# Test 1: student == teacher → loss should be ~0
student.load_state_dict(teacher.state_dict())
loss, _ = per_token_reverse_kl_loss(student, teacher, ids, mask)
assert abs(loss.item()) < 1e-4, f"identical model should give zero KL, got {loss}"

# Test 2: student random init → loss > 0
student = init_random_model(...)
loss, _ = per_token_reverse_kl_loss(student, teacher, ids, mask)
assert loss.item() > 0.5, f"random student should give positive KL, got {loss}"

# Test 3: loss should decrease over training
losses = []
for step in range(100):
    loss = opd_train_step(student, teacher, ...)
    losses.append(loss)
assert losses[-1] < losses[0], "OPD should reduce KL over training"

A.2　Common mistakes and correct practices

A.3　Core paper and resource list

Core OPD papers:

• MiniLLM (Gu et al. 2023, ICLR 2024) — arXiv 2306.08543
• GKD: On-Policy Distillation of Language Models (Agarwal et al. 2023, ICLR 2024) — arXiv 2306.13649
• A Survey of On-Policy Distillation for LLMs (Song & Zheng 2026) — arXiv 2604.00626
• Rethinking On-Policy Distillation (2026) — arXiv 2604.13016
• KL for a KL (vOPD, 2026) — arXiv 2605.07865
• Black-Box On-Policy Distillation (2026) — arXiv 2511.10643
• Decoupling KL and Trajectories (2026) — arXiv 2605.16826

Industry tech reports:

• Qwen3 Technical Report (Qwen Team 2025-05) — arXiv 2505.09388 (§3.2 OPD recipe)
• DeepSeek-R1 (DeepSeek 2025-01) — arXiv 2501.12948 (off-policy distillation series)

Blogs / code:

• Thinking Machines Lab — "On-Policy Distillation" blog (Lu et al. 2025-10-27) — thinkingmachines.ai/blog/on-policy-distillation/
• Tinker Cookbook — github.com/thinking-machines-lab/tinker-cookbook/tree/main/tinker_cookbook/recipes/distillation
• TRL GKD Trainer — huggingface.co/docs/trl/gkd_trainer
• Awesome OPD list — github.com/thinkwee/AwesomeOPD, github.com/nick7nlp/Awesome-LLM-On-Policy-Distillation

Related foundations:

• Hinton, Vinyals, Dean — "Distilling the Knowledge in a Neural Network" (2015) — arXiv 1503.02531
• Sanh et al. — "DistilBERT" (2019) — arXiv 1910.01108
• Kim & Rush — "Sequence-Level Knowledge Distillation" (EMNLP 2016) — arXiv 1606.07947
• Bagnell — "Reinforcement Learning and Imitation Learning" (theoretical foundation for exposure bias)
• DeepSeekMath GRPO (2024) — arXiv 2402.03300
• DPO (Rafailov et al. NeurIPS 2023) — arXiv 2305.18290

A.4　[needs-verify] index

The following items in this cheat sheet are marked [needs-verify]; we recommend cross-checking against the original papers / tech reports before and after interviews:

1. L3-2 "OPD surpasses teacher": the specific setting and magnitude reported by Rethinking OPD (arXiv 2604.13016)
2. L3-3 DeepSeek-V4 multi-teacher OPD implementation details: weight choice strategy, whether token-level routing
3. L3-4 "OPD + PRM fusion": as of 2026-05 active research; no single authoritative paper that integrates the two yet
4. §6.2 Thinking Machines numbers: "9-30× FLOPs savings" is from blog secondary sources; original blog numbers and specific settings should be checked
5. §5.2 timeline: 2025-2026 multiple OPD-related arXiv paper IDs (such as the 2604.* series) are from 2026 Q1-Q2 submissions/preprints; some IDs may be updated to new versions or renumbered after submission
6. OPD specific adoption details on Qwen3 / Gemma 2 / MiMo: most information comes from the Thinking Machines blog and the Qwen3 paper §3.2, but the distillation specifics of Gemma 2 / MiMo need cross-checking in their respective tech reports

A.5　Terminology quick reference