Rlhf Dpo Grpo Ppo Tutorial En

§0 TL;DR Cheat Sheet

1. RLHF pipeline (Ouyang 2022 InstructGPT): SFT → RM (Bradley-Terry pairwise) → PPO + per-token KL; the value model (value head) is trained separately, and the policy is anchored to the reference policy by KL.
2. PPO core (Schulman 2017): clipped surrogate $L^{\text{CLIP}}(\theta) = \mathbb{E}[\min(r_t A_t,\; \text{clip}(r_t, 1-\epsilon, 1+\epsilon) A_t)]$, importance ratio $r_t = \pi_\theta / \pi_{\theta_\text{old}}$; advantages use GAE $A_t^{\text{GAE}} = \sum_{l \ge 0} (\gamma\lambda)^l \delta_{t+l}$ to balance bias/variance.
3. DPO closed-form (Rafailov 2023 NeurIPS): the optimal policy of KL-regularized RLHF is $\pi^*(y|x) \propto \pi_\text{ref}(y|x)\exp(r(x,y)/\beta)$; inverting gives $r = \beta\log(\pi^*/\pi_\text{ref}) + \beta\log Z(x)$; substituting into Bradley-Terry, $\log Z$ cancels in the pairwise difference, leaving a pure SFT-style loss $-\log\sigma(\beta\log\frac{\pi(y_w)}{\pi_\text{ref}(y_w)} - \beta\log\frac{\pi(y_l)}{\pi_\text{ref}(y_l)})$.
4. GRPO (DeepSeekMath 2024, R1 2025): for each prompt sample a group of $G$ responses, advantage uses within-group normalization $\hat{A}_i = (r_i - \text{mean}(\mathbf{r}))/\text{std}(\mathbf{r})$; drops the value model, saving half the memory, especially suitable for LLM math/code RL.
5. DPO variant ecosystem: KTO (only needs thumbs up/down, no pairs), IPO ($\ell_2$ form preventing reward overfit), SimPO (reference-free, length-normalized), ORPO (one-stage SFT + odds-ratio fusion), RLOO (leave-one-out baseline over multiple samples, value-free), ReMax (greedy baseline, further savings).
6. PRM vs ORM (Lightman 2023 arXiv / 2024 ICLR): process-reward supervises each reasoning step, Math-Shepherd (Wang 2024 ACL) auto-labels PRM; outcome-reward only looks at the final answer. PRM consistently beats ORM on math reasoning, but annotation cost is high.
7. Reward hacking is the central pain point: the model over-optimizes the proxy reward, producing answers that get longer, sycophantic, with abnormal style; mitigations = KL penalty, reward clipping, length penalty, ensemble RM, Constitutional AI / RLAIF (Bai 2022, Lee 2023).

§1 Post-Training Alignment Intuition

LLM training splits into three stages:

• Pretraining: next-token prediction over trillion-token corpora — learns world knowledge and language patterns
• SFT (Supervised Fine-Tuning): next-token on instruction-response pairs — learns instruction format and basic capability
• Alignment / RL post-training: make outputs align with human preferences (helpful / harmless / honest) — learns "which response is better"

Why isn't SFT enough? Because SFT can only mimic positive examples ("what a good response looks like") and cannot explicitly learn contrastive signal ("A is better than B"). RL post-training provides three paradigms:

1.1　Special characteristics of language-task RL

Classic game RL (Atari, Go) and LLM RL differ significantly, a common interview probe:

Because there is only a terminal reward, LLM RL advantage estimation often uses coarse schemes: PPO+GAE distributes advantage over each token (but reward is 0 on most tokens); GRPO directly uses the whole response's reward, assigning all tokens the same within-group normalized advantage.

§2 PPO Core

2.1　Vanilla policy gradient recap

Policy gradient theorem:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\!\left[\sum_t \nabla_\theta \log \pi_\theta(a_t | s_t)\, A^{\pi_\theta}(s_t, a_t)\right]$$

REINFORCE uses $A \leftarrow R_t$ (return), which has high variance; Actor-Critic uses $A^{\pi}(s, a) = Q^\pi(s,a) - V^\pi(s)$ to reduce variance.

2.2　PPO clipped surrogate (must-know formula)

Define importance ratio:

$$r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_\text{old}}(a_t | s_t)}$$

PPO-Clip objective:

$$\boxed{\;L^{\text{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\!\Big(r_t(\theta) A_t,\; \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\Big)\right]\;}$$

Intuition:

• If $A_t > 0$ (this action beats baseline), we want to increase $\pi_\theta(a_t|s_t)$, but only up to $r_t = 1+\epsilon$ (preventing too aggressive a single update).
• If $A_t < 0$ (this action is worse than baseline), we want to decrease $\pi_\theta(a_t|s_t)$, but only down to $r_t = 1-\epsilon$.
• min picks the smaller of the two → pessimistic estimate (pessimistic bound): when we want to "add points," clip the upper bound; when we want to "subtract points," clip the lower bound.

Typical $\epsilon = 0.1 \sim 0.2$. In LLM RLHF practice, $0.1 \sim 0.2$ is common; too large easily blows up the KL.

2.3　GAE: generalized advantage estimation

Define TD residual:

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

GAE is the exponentially weighted average of advantages over different step lengths:

$$\boxed{\;A_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}\;}$$

Boundary cases:

• $\lambda = 0$ → $A_t = \delta_t$, pure TD(0), high bias low variance
• $\lambda = 1$ → $A_t = \sum_l \gamma^l r_{t+l} - V(s_t)$, pure Monte Carlo (advantage equals actual return minus baseline), low bias high variance
• Typical $\lambda = 0.95$, $\gamma = 0.99$

Degeneration of GAE in LLMs: in RLHF typically $\gamma = 1$ (no discount), and there is only a terminal reward, so $\delta_t = -V(s_t) + V(s_{t+1})$ for intermediate tokens, $\delta_T = R_T - V(s_T)$ for the terminal token. In this case GAE is equivalent to "value baseline + reward backflow."

2.4　Full PPO objective in RLHF

In LLMs, each timestep $t$ corresponds to generating the $t$-th token, state = $(x, y_{\lt t})$, action = $y_t$. The reward usually adds a KL penalty:

$$\tilde{r}_t = \mathbb{1}[t = T] \cdot R(x, y) - \beta \log \frac{\pi_\theta(y_t | x, y_{\lt t})}{\pi_\text{ref}(y_t | x, y_{\lt t})}$$

Final objective:

$$L^{\text{PPO}}(\theta) = L^{\text{CLIP}}(\theta) - c_v \cdot \underbrace{\mathbb{E}_t (V_\phi(s_t) - V_t^\text{target})^2}_{\text{value loss}} + c_e \cdot \underbrace{\mathbb{E}_t \mathcal{H}[\pi_\theta(\cdot | s_t)]}_{\text{entropy bonus}}$$

Typical $c_v = 0.5$, $c_e = 0.01$ (in LLMs the entropy bonus is usually very small or 0, because the vocab is large and entropy is already high).

2.5　Code (core 60 lines)

import torch
import torch.nn as nn
import torch.nn.functional as F

def compute_gae(rewards, values, dones, gamma=1.0, lam=0.95):
    """
    rewards: [B, T]    per-token reward (with KL penalty)
    values:  [B, T+1]  value at s_0...s_T (s_T = terminal, V=0)
    dones:   [B, T]    1 if terminal, 0 otherwise
    returns advantages [B, T], returns [B, T]
    """
    B, T = rewards.shape
    advantages = torch.zeros_like(rewards)
    gae = 0.0
    for t in reversed(range(T)):
        non_term = 1.0 - dones[:, t]
        delta = rewards[:, t] + gamma * values[:, t + 1] * non_term - values[:, t]
        gae = delta + gamma * lam * non_term * gae
        advantages[:, t] = gae
    returns = advantages + values[:, :T]
    return advantages, returns


def ppo_step(policy, value, batch, eps_clip=0.2, c_v=0.5, c_e=0.01):
    """
    batch: dict with keys
      input_ids:    [B, L]     prompt + response tokens
      action_mask:  [B, L]     1 for response tokens, 0 for prompt/pad
      old_log_probs:[B, L]     log π_θold(y_t | s_t) at sample time
      advantages:   [B, L]     GAE advantages (already normalized)
      returns:      [B, L]     GAE returns (V target)
    """
    logits = policy(batch["input_ids"]).logits          # [B, L, V]
    # log-prob of taken action y_t at position t (shifted by 1: predict t+1 from t)
    log_probs = F.log_softmax(logits[:, :-1], dim=-1)
    targets   = batch["input_ids"][:, 1:].unsqueeze(-1)
    new_log_probs = log_probs.gather(-1, targets).squeeze(-1)  # [B, L-1]
    new_log_probs = F.pad(new_log_probs, (1, 0))               # align to [B, L]

    mask = batch["action_mask"].float()
    ratio = torch.exp(new_log_probs - batch["old_log_probs"])  # r_t

    # ── PPO-Clip core ──
    A = batch["advantages"]
    surr1 = ratio * A
    surr2 = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip) * A
    policy_loss = -((torch.min(surr1, surr2) * mask).sum() / mask.sum())

    # ── Value loss ──
    V = value(batch["input_ids"]).squeeze(-1)                  # [B, L]
    value_loss = (((V - batch["returns"]) ** 2) * mask).sum() / mask.sum()

    # ── Entropy bonus ──
    probs = log_probs.exp()                                    # [B, L-1, V]
    entropy = -(probs * log_probs).sum(-1)                     # [B, L-1]
    entropy = F.pad(entropy, (1, 0))
    entropy_bonus = (entropy * mask).sum() / mask.sum()

    loss = policy_loss + c_v * value_loss - c_e * entropy_bonus

    # Monitoring (does not participate in backprop)
    with torch.no_grad():
        approx_kl = ((ratio - 1) - torch.log(ratio.clamp_min(1e-8)))
        approx_kl = (approx_kl * mask).sum() / mask.sum()
        clip_frac = (((ratio < 1 - eps_clip) | (ratio > 1 + eps_clip)).float() * mask).sum() / mask.sum()

    return loss, {"policy": policy_loss.item(), "value": value_loss.item(),
                  "entropy": entropy_bonus.item(),
                  "approx_kl": approx_kl.item(), "clip_frac": clip_frac.item()}

• approx_kl uses $\mathbb{E}[r - 1 - \log r] \ge 0$ (Schulman 2020 blog), more stable than $\mathbb{E}[\log r]$ and never negative.
• Multiple PPO updates per epoch (typically 4), but monitor approx_kl: early stop if it exceeds target_kl (e.g. 0.02).
• Advantage must do batch-level normalization (subtract mean divide std), otherwise scale will make learning rate ineffective.
• When computing KL penalty with reference policy, KL is token-level (not sequence-level); writing wrong dimensions easily causes gradient explosion.
• c_v=0.5 is just an empirical value; if value loss is far greater than policy loss, it "eats" the gradient. You can use a separate optimizer or a separate lr for value.

§3 RLHF Pipeline (InstructGPT Paradigm)

Ouyang et al. 2022 NeurIPS, Training language models to follow instructions with human feedback, is the predecessor of ChatGPT and defines today's standard three-stage RLHF:

3.1　Stage 1 — SFT (Supervised Fine-Tuning)

Given instruction-response pairs $\{(x, y)\}$ (high-quality human-written), do next-token:

$$\mathcal{L}_\text{SFT}(\phi) = -\mathbb{E}_{(x, y) \sim \mathcal{D}_\text{SFT}}\left[\sum_t \log \pi_\phi(y_t | x, y_{\lt t})\right]$$

Output is $\pi_\text{SFT}$ (also called $\pi_\text{ref}$, since downstream RL uses it as KL anchor).

3.2　Stage 2 — RM (Reward Model)

For the same prompt $x$, sample multiple responses with $\pi_\text{SFT}$ and have humans pairwise compare to obtain preference pairs $(x, y_w, y_l)$, $y_w \succ y_l$.

Bradley-Terry preference model:

$$P(y_w \succ y_l | x) = \sigma(r^*(x, y_w) - r^*(x, y_l))$$

where $r^*$ is the unknown "true" reward function. We fit it with $r_\psi$ (a transformer with a scalar head on the last layer):

$$\boxed{\;\mathcal{L}_\text{RM}(\psi) = -\mathbb{E}_{(x, y_w, y_l)} \log \sigma\!\big(r_\psi(x, y_w) - r_\psi(x, y_l)\big)\;}$$

Implementation details:

• RM is usually initialized from SFT model (shared backbone), with the last token's hidden state going through a linear layer to scalar reward.
• After training, RM parameters are frozen.

import torch
import torch.nn.functional as F

def rm_loss(reward_model, batch):
    """
    batch:
      chosen_ids:   [B, L]     prompt + y_w
      rejected_ids: [B, L]     prompt + y_l
    """
    r_w = reward_model(batch["chosen_ids"])      # [B]  scalar at last token
    r_l = reward_model(batch["rejected_ids"])    # [B]
    # Bradley-Terry NLL
    loss = -F.logsigmoid(r_w - r_l).mean()
    accuracy = (r_w > r_l).float().mean()
    return loss, accuracy

3.3　Stage 3 — PPO + KL (Policy Optimization)

Objective:

$$\boxed{\;\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)} \big[r_\psi(x, y)\big] - \beta\, \mathbb{E}_x\, \text{KL}\!\big(\pi_\theta(\cdot|x) \,\big\|\, \pi_\text{ref}(\cdot|x)\big)\;}$$

In implementation, split the KL across each token and merge it with the RM reward into a per-token reward $\tilde{r}_t$ (see §2.4), then run PPO.

prompt x
   │
   │   π_θ generates response y
   ↓
(x, y)
   │
   │   r_ψ(x, y) → scalar reward       (Stage 2 RM, frozen)
   │   KL(π_θ || π_ref)               (per-token, on the fly)
   ↓
r̃_t = R · 1[t=T] − β log(π_θ/π_ref)
   │
   │   GAE on r̃_t → advantages
   ↓
PPO update on π_θ and V_φ

3.4　In practice: 4 models resident simultaneously

PPO RLHF training has 4 models in memory simultaneously:

1. Policy $\pi_\theta$ (trainable)
2. Reference policy $\pi_\text{ref}$ (frozen, for KL)
3. Reward model $r_\psi$ (frozen, for reward)
4. Value model $V_\phi$ (trainable, required by PPO)

This is the root cause of RLHF's memory consumption; 4× base model + optimizer state + gradient = easy to blow up. This is also the fundamental reason DPO / GRPO are widely accepted in LLM RL — they each chop off some of the models.

§4 DPO: Closed-Form Direct Preference Optimization

Rafailov et al. 2023 NeurIPS, Direct Preference Optimization: Your Language Model is Secretly a Reward Model, is the milestone in RLHF simplification. Core observation: the KL-regularized RL problem has a closed-form optimum, which can be inverted to give an implicit reward, replacing PPO with pure supervised learning.

4.1　KL-regularized optimal policy (key step)

Consider the RLHF objective (formula §3.3):

$$\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot|x)}\big[r(x, y)\big] - \beta\, \text{KL}\!\big(\pi(\cdot|x) \| \pi_\text{ref}(\cdot|x)\big)$$

Solve for the optimal $\pi^*(\cdot | x)$ for a single $x$ (Lagrangian):

$$\mathcal{L}_x[\pi] = \sum_y \pi(y|x) r(x, y) - \beta \sum_y \pi(y|x) \log \frac{\pi(y|x)}{\pi_\text{ref}(y|x)} + \mu\!\left(1 - \sum_y \pi(y|x)\right)$$

Take partial derivative with respect to $\pi(y|x)$ and set = 0:

$$r(x, y) - \beta\!\left(\log \frac{\pi(y|x)}{\pi_\text{ref}(y|x)} + 1\right) - \mu = 0$$

Rearranging:

$$\log \pi^*(y|x) = \log \pi_\text{ref}(y|x) + \frac{r(x, y)}{\beta} - \frac{\mu + \beta}{\beta}$$

Let $\log Z(x) = (\mu + \beta)/\beta$ (log of partition function), getting:

$$\boxed{\;\pi^*(y|x) = \frac{1}{Z(x)} \pi_\text{ref}(y|x) \exp\!\left(\frac{r(x, y)}{\beta}\right), \quad Z(x) = \sum_y \pi_\text{ref}(y|x) \exp\!\left(\frac{r(x, y)}{\beta}\right)\;}$$

4.2　Inversion: implicit reward

Taking the log of the above and solving for $r$:

$$r(x, y) = \beta \log \frac{\pi^*(y|x)}{\pi_\text{ref}(y|x)} + \beta \log Z(x)$$

Key insight: the reward is the log-ratio of $\pi^*$ to $\pi_\text{ref}$ (plus a term $\beta \log Z(x)$ that does not depend on $y$). So once we have $\pi^*$, we have the reward; conversely as well.

4.3　Plug into Bradley-Terry to get DPO loss

Bradley-Terry gives preference probability:

$$P(y_w \succ y_l | x) = \sigma\!\big(r(x, y_w) - r(x, y_l)\big)$$

Substituting §4.2's inversion (note $\beta \log Z(x)$ does not depend on $y$ and cancels in $r(x, y_w) - r(x, y_l)$!):

$$r(x, y_w) - r(x, y_l) = \beta \log \frac{\pi^*(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_\text{ref}(y_l|x)}$$

Replacing the target $\pi^*$ with the learnable $\pi_\theta$, we get the DPO loss:

$$\boxed{\;\mathcal{L}_\text{DPO}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_\text{ref}(y_l|x)}\right)\;}$$

• It has no explicit reward model, but the implicit reward = $\beta \log(\pi/\pi_\text{ref})$.
• It needs no sampling (unlike PPO requiring on-policy rollouts), the entire training is offline contrastive learning.
• The KL constraint is written implicitly into the loss (via $\pi_\text{ref}$ in the log-ratio denominator).
• The whole pipeline becomes SFT-style: one pass of supervised learning and you're done.

4.4　Implicit meaning of the DPO gradient

Take the gradient with respect to $\theta$ (derivation details in original paper Appendix):

$$\nabla_\theta \mathcal{L}_\text{DPO} = -\beta \mathbb{E}\big[\sigma(\hat{r}_l - \hat{r}_w)\big(\nabla_\theta \log \pi_\theta(y_w|x) - \nabla_\theta \log \pi_\theta(y_l|x)\big)\big]$$

where $\hat{r} = \beta \log(\pi_\theta/\pi_\text{ref})$ is the implicit reward. Intuition:

• The coefficient $\sigma(\hat{r}_l - \hat{r}_w)$ is "the current model's confidence in the wrong direction $y_l \succ y_w$" — the more confidently wrong, the bigger the weight (hard-example mining effect).
• Then increase $\log \pi_\theta(y_w | x)$ and decrease $\log \pi_\theta(y_l | x)$.

4.5　Code (core DPO loss in 30 lines)

import torch
import torch.nn.functional as F

def dpo_loss(policy, ref_policy, batch, beta=0.1):
    """
    batch:
      chosen_ids:    [B, L]     prompt + y_w
      rejected_ids:  [B, L]     prompt + y_l
      chosen_mask:   [B, L]     1 for y_w response tokens, 0 elsewhere (prompt + pad)
      rejected_mask: [B, L]
    """
    # log π(y | x) = sum_t log π(y_t | x, y_<t) over response tokens
    def log_prob_sum(model, ids, mask):
        logits = model(ids).logits[:, :-1]                  # [B, L-1, V]
        logp = F.log_softmax(logits, dim=-1)
        tgt = ids[:, 1:].unsqueeze(-1)                      # [B, L-1, 1]
        token_logp = logp.gather(-1, tgt).squeeze(-1)       # [B, L-1]
        token_mask = mask[:, 1:]                            # align with next-token prediction
        return (token_logp * token_mask).sum(dim=-1)        # [B]

    pi_w  = log_prob_sum(policy,     batch["chosen_ids"],   batch["chosen_mask"])
    pi_l  = log_prob_sum(policy,     batch["rejected_ids"], batch["rejected_mask"])
    with torch.no_grad():
        ref_w = log_prob_sum(ref_policy, batch["chosen_ids"],   batch["chosen_mask"])
        ref_l = log_prob_sum(ref_policy, batch["rejected_ids"], batch["rejected_mask"])

    # log-ratios
    logits_w = pi_w - ref_w        # = log(π/π_ref)(y_w)
    logits_l = pi_l - ref_l
    diff = beta * (logits_w - logits_l)

    loss = -F.logsigmoid(diff).mean()
    # implicit reward margin (for monitoring)
    chosen_reward   = beta * logits_w.detach()
    rejected_reward = beta * logits_l.detach()
    margin = (chosen_reward - rejected_reward).mean()
    accuracy = (chosen_reward > rejected_reward).float().mean()
    return loss, {"loss": loss.item(), "margin": margin.item(),
                  "accuracy": accuracy.item()}

• Likelihood decreases for both $y_w$ and $y_l$ (Pal et al. 2024, Saeidi et al.): DPO only requires that $\log\pi(y_w) - \log\pi(y_l)$ increases, not that $\log\pi(y_w)$ increases. In practice both often decrease together, with $y_l$ decreasing more — which can make the model "unwilling to say anything."
• Sensitive to $\beta$: too small → loses KL constraint → reward hacking; too large → fails to learn. Common range $0.05 \sim 0.5$.
• Data quality determines everything: when preference pairs are noisy, DPO is more brittle than PPO (PPO buffers via RM ensemble, DPO eats raw data directly).
• Off-policy bias: preference data is sampled by $\pi_\text{SFT}$, but $\pi_\theta$ drifts during training, creating distribution shift.

§5 GRPO: Group Relative Advantage

DeepSeekMath (Shao et al. 2024 arXiv 2402.03300) proposes Group Relative Policy Optimization, and DeepSeek-R1 (DeepSeek-AI 2025 arXiv 2501.12948) pushes it to the flagship of reasoning.

5.1　Motivation

PPO's value model is hard to train for LLMs:

• LLM token-level value has no clear semantics (intermediate tokens usually have reward = 0, only terminal has reward)
• Value model is the same size as policy → doubled memory

GRPO's core idea: for each prompt $x$, sample a group of $G$ responses $\{y_1, \dots, y_G\}$, score them with RM giving $G$ rewards $\{r_1, \dots, r_G\}$, advantage directly uses within-group normalization:

$$\boxed{\;\hat{A}_{i, t} = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\}) + \epsilon}\;}$$

Within the entire response, all tokens share the same $\hat{A}_i$ (since there is no value model providing per-token baseline, only sequence-level reward and group-relative advantage).

5.2　GRPO objective

GRPO keeps PPO-Clip's form but puts the KL penalty into the loss (not into the reward):

Let importance ratio $\rho_{i, t}(\theta) = \pi_\theta(y_{i,t}|x_i, y_{i,\lt t}) / \pi_{\theta_\text{old}}(y_{i,t}|x_i, y_{i,\lt t})$ (avoiding confusion with sequence-level reward $r_i$):

$$L^\text{GRPO}(\theta) = \mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \Big(\min(\rho_{i, t} \hat{A}_{i, t},\, \text{clip}(\rho_{i, t}, 1\!-\!\epsilon, 1\!+\!\epsilon) \hat{A}_{i, t}) - \beta\, \text{KL}_{i, t}(\pi_\theta \| \pi_\text{ref})\Big)\right]$$

where KL uses the K3 estimator (Schulman blog 2020):

$$\text{KL}_{i, t} = \frac{\pi_\text{ref}(y_{i,t}|\cdot)}{\pi_\theta(y_{i,t}|\cdot)} - \log\frac{\pi_\text{ref}(y_{i,t}|\cdot)}{\pi_\theta(y_{i,t}|\cdot)} - 1$$

This estimator is guaranteed non-negative and low-variance, more stable than directly using $\log(\pi_\theta/\pi_\text{ref})$.

5.3　DeepSeek-R1's key modifications

DeepSeek-R1 (Jan 2025) does two things on top of GRPO:

1. R1-Zero: directly run GRPO from pretrain base, without an SFT stage, purely with rule-based reward (math correctness + format reward). Emergent long CoT.
2. R1: add a small SFT cold-start + multi-stage RL (reasoning RL → SFT → general RL). The open-source 32B/70B compete with o1 on math reasoning.

• Rule-based reward: math has unique answers, code has unit tests, bypassing the neural RM, reward hacking risk is significantly reduced (reward signal is close to ground truth, but the policy can still find loopholes in the grader, e.g. guessing answers, writing only "42" without reasoning, exploiting test coverage blind spots, etc., so "no hacking at all" is too strong a claim).
• Within-group normalization: same problem sampled $G=16$ solutions, automatically figuring out "which reasoning paths are better," without needing absolute scale.
• Save value model: math prompts are many and reasoning is long, saving half the memory enables larger batches.

5.4　Code (core GRPO advantage + loss in 50 lines)

import torch
import torch.nn.functional as F

def grpo_loss(policy, ref_policy, batch, eps_clip=0.2, beta=0.04):
    """
    batch:
      input_ids:     [N, L]       N = sum_b G_b samples in the batch
      action_mask:   [N, L]
      old_log_probs: [N, L]
      rewards:       [N]          sequence-level reward (rule-based or RM)
      group_id:      [N]          which prompt each sample belongs to (long tensor)
    """
    rewards = batch["rewards"]
    gid = batch["group_id"].long()
    N = rewards.shape[0]

    # ── Within-group normalization (GRPO core) ──
    # Use scatter to aggregate mean / std per group, no assumption that group_id is sorted or equal-sized.
    num_groups = int(gid.max().item()) + 1
    counts = torch.zeros(num_groups, device=rewards.device).scatter_add_(
        0, gid, torch.ones_like(rewards))                              # [num_groups]
    sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
        0, gid, rewards)                                                # [num_groups]
    group_mean = sums / counts.clamp_min(1.0)                          # [num_groups]
    diff_sq = (rewards - group_mean[gid]) ** 2
    sq_sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
        0, gid, diff_sq)                                                # [num_groups]
    group_var = sq_sums / counts.clamp_min(1.0)
    group_std = group_var.sqrt()                                       # [num_groups]

    A = (rewards - group_mean[gid]) / (group_std[gid] + 1e-8)          # [N]
    A = A.unsqueeze(-1)                                                # [N, 1] shared across the whole sequence

    # ── log-prob ratio ──
    logits = policy(batch["input_ids"]).logits[:, :-1]
    log_probs = F.log_softmax(logits, dim=-1)
    tgt = batch["input_ids"][:, 1:].unsqueeze(-1)
    new_log_probs = log_probs.gather(-1, tgt).squeeze(-1)
    new_log_probs = F.pad(new_log_probs, (1, 0))        # [B*G, L]
    mask = batch["action_mask"].float()

    ratio = torch.exp(new_log_probs - batch["old_log_probs"])

    # ── PPO-Clip surrogate (advantage shared across the sequence) ──
    surr1 = ratio * A                                    # [N, L]
    surr2 = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip) * A

    # ── KL penalty (K3 estimator: KL ≈ exp(Δ) - Δ - 1) ──
    with torch.no_grad():
        ref_logits = ref_policy(batch["input_ids"]).logits[:, :-1]
        ref_log_probs = F.log_softmax(ref_logits, dim=-1)
        ref_token_lp = ref_log_probs.gather(-1, tgt).squeeze(-1)
        ref_token_lp = F.pad(ref_token_lp, (1, 0))
    delta = ref_token_lp - new_log_probs                 # log(π_ref / π_θ)
    kl_per_token = torch.exp(delta) - delta - 1.0        # K3, non-negative [N, L]

    # Formula: (1/G) Σ_i (1/|y_i|) Σ_t [ min(...) - β·KL_{i,t} ]
    # Implementation: each sample first normalized by response length |y_i|, then averaged over batch
    token_obj = torch.min(surr1, surr2) - beta * kl_per_token   # [N, L]
    seq_len = mask.sum(dim=-1).clamp_min(1.0)            # [N] = |y_i|
    per_seq = (token_obj * mask).sum(dim=-1) / seq_len   # [N]
    loss = -per_seq.mean()                               # Negative sign: maximize -> minimize

    # Monitoring
    with torch.no_grad():
        policy_term = -(torch.min(surr1, surr2) * mask).sum(dim=-1) / seq_len
        kl_term = (kl_per_token * mask).sum(dim=-1) / seq_len
    return loss, {"policy": policy_term.mean().item(),
                  "kl": kl_term.mean().item(),
                  "reward_mean": rewards.mean().item(),
                  "advantage_std": A.squeeze(-1).std().item()}

• $G$ is usually 8/16/32: too small means noisy within-group statistics, too large blows up memory.
• The degenerate case where std is 0 (all rewards in group are the same, all-pass / all-fail): add $\epsilon$ or drop the group.
• DAPO (ByteDance Seed, 2025 arXiv 2503.14476) makes several improvements to GRPO: clip higher (breaking symmetric clip), dynamic sampling (drop all-correct/all-wrong groups), token-level loss, overlong shaping — it is the industrial-grade evolution of GRPO.
• DeepSeek also discusses GRPO's limitations in the R1 report: within-group baseline fails when reward variance is small or group size is small; subsequent work (e.g. VAPO, CISPO) tries to improve.

§6 DPO Variant Ecosystem

After DPO, 2024–2025 saw a surge of improvements, and interviews frequently ask "what's the difference between X and Y." The table below compares core loss differences; notation is unified as: $\pi$ = current policy, $\pi_\text{ref}$ = SFT model, $\beta$ = inverse temperature, $r$ = implicit reward.

6.1　IPO: prevent reward overfit

Azar et al. 2024 A General Theoretical Paradigm to Understand Learning from Human Preferences points out: Bradley-Terry + DPO will blow up the implicit reward to infinity when preferences are deterministic ("$y_w$ always wins"), causing overfit. IPO replaces sigmoid with squared loss:

$$\mathcal{L}_\text{IPO} = \mathbb{E}\!\left[\left(\log\frac{\pi(y_w)/\pi_\text{ref}(y_w)}{\pi(y_l)/\pi_\text{ref}(y_l)} - \frac{1}{2\beta}\right)^2\right]$$

Intuition: aim for log-ratio equal to a fixed margin $1/(2\beta)$, not increasingly large.

6.2　KTO: no need for pairs

Ethayarajh et al. 2024 ICML KTO: Model Alignment as Prospect Theoretic Optimization. Production data is often a single label ("this is good" / "this is bad"), not pairs. KTO is inspired by Prospect Theory (Kahneman-Tversky) and symmetrically pushes desirable / undesirable samples away from a reference point:

Let implicit reward $\hat{r}_\theta(x, y) = \beta \log \tfrac{\pi_\theta(y|x)}{\pi_\text{ref}(y|x)}$, reference point $z_0 = \mathbb{E}_{x',\, y' \sim \pi_\theta(\cdot | x')}\!\big[\beta \cdot \text{KL}\!\big(\pi_\theta(\cdot | x') \,\|\, \pi_\text{ref}(\cdot | x')\big)\big]$ (estimated via mismatched pairs within the batch, does not participate in backprop — treated as a constant). The KTO loss is:

$$\boxed{\;\mathcal{L}_\text{KTO}(\theta) = \mathbb{E}_{(x, y) \sim \mathcal{D}} \big[\, w(y)\big(1 - v(x, y)\big)\,\big]\;}$$

where piecewise definitions ($\hat r$ and $z_0$ already contain $\beta$, no additional $\beta$ in the sigmoid):

$$v(x, y) = \begin{cases}\sigma\!\big(\hat{r}_\theta(x, y) - z_0\big) & y \text{ desirable}\\[2pt] \sigma\!\big(z_0 - \hat{r}_\theta(x, y)\big) & y \text{ undesirable}\end{cases}$$

$$w(y) = \begin{cases}\lambda_D & y \text{ desirable}\\ \lambda_U & y \text{ undesirable}\end{cases}$$

Intuition: desirable samples want $\hat{r} > z_0$ (push up), undesirable samples want $\hat{r} < z_0$ (push down); $\sigma$ gives a probability score of "how far from the reference point." $\lambda_D, \lambda_U$ are weights for imbalanced data (e.g. if desirable is more, $\lambda_U > \lambda_D$). When data only has single thumbs up/down, KTO is more practical than DPO. See Ethayarajh et al. 2024 ICML, HuggingFace TRL KTOTrainer for details.

6.3　SimPO: drop the reference

Meng et al. 2024 NeurIPS SimPO: Simple Preference Optimization with a Reference-Free Reward. SimPO observes: DPO does not use $\pi_\text{ref}$ at inference, but stores it during training (occupies memory). SimPO directly changes the implicit reward to length-normalized log-prob:

$$r_\text{SimPO}(x, y) = \frac{\beta}{|y|}\log\pi(y|x)$$

Loss:

$$\boxed{\;\mathcal{L}_\text{SimPO} = -\mathbb{E}\log\sigma\!\left(\frac{\beta}{|y_w|}\log\pi(y_w|x) - \frac{\beta}{|y_l|}\log\pi(y_l|x) - \gamma\right)\;}$$

where $\gamma$ is the target reward margin (encourages $r_w - r_l \ge \gamma$, no loss past that).

• No $\pi_\text{ref}$ needed during training, saves a copy of model weights in memory.
• Length-normalization mitigates DPO's length bias (DPO favors long answers since their log-probs are more negative, with larger differences).
• Experimentally on AlpacaEval-2 / Arena-Hard, SimPO often outperforms DPO; but the reward-free form lacks a KL anchor, requiring more careful tuning of $\beta, \gamma$.

6.4　ORPO: single-stage SFT + preference

Hong et al. 2024 EMNLP ORPO: Monolithic Preference Optimization without Reference Model. ORPO removes the two-stage "SFT then DPO," doing SFT and preference simultaneously in one training pass:

$$\mathcal{L}_\text{ORPO} = \mathcal{L}_\text{SFT}(y_w) - \lambda \cdot \log\sigma\!\left(\log\frac{\text{odds}_\theta(y_w|x)}{\text{odds}_\theta(y_l|x)}\right)$$

where odds = $p / (1 - p)$. No reference model needed, but it requires the chosen response to be high quality (driven by the SFT part).

6.5　RLOO / ReMax

Ahmadian et al. 2024 ACL Back to Basics: Revisiting REINFORCE Style Optimization. On LLM RLHF, simple REINFORCE + multi-sample baseline empirically outperforms PPO:

• RLOO: for each prompt sample $k$ responses, the baseline for each is the average reward of the other $k-1$ (leave-one-out), $\hat{A}_i = r_i - \frac{1}{k-1}\sum_{j\ne i} r_j$.
• ReMax (Li et al. 2024 ICML): baseline directly uses the greedy decode reward for the same prompt, only needing 1 sample + 1 greedy per step.

Both drop the value model, in the same lineage as GRPO — fundamentally all using multi-sample or greedy baselines to replace the critic; the difference is only in baseline form.

• GRPO: within-group mean+std normalization ($z$-score) + PPO-clip.
• RLOO: within-group leave-one-out mean as baseline + REINFORCE.
• ReMax: greedy decode as baseline + REINFORCE.
• The three share a philosophy: bypass the value model, use sample baselines; differing only in baseline form.

§7 Advanced Reward Modeling

7.1　PRM vs ORM

Lightman et al. Let's Verify Step by Step (OpenAI arXiv 2305.20050, 2023; ICLR 2024) compared ORM and PRM on the MATH dataset:

• PRM800K: 800K step-level annotations (each step labeled ✓ / ⚠️ / ✗).
• On GPT-4 generator, PRM rerank over 1860 candidates gives MATH accuracy 78.2%, significantly higher than ORM's 72.4%.

Math-Shepherd (Wang et al. 2024 ACL Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations) uses rollout-based automatic annotation of PRM: from each step run $K$ subsequent generations, automatically giving soft labels per step based on correctness rate, removing the need for human labels.

7.2　Constitutional AI / RLAIF

Constitutional AI (Bai et al. 2022 Anthropic) Constitutional AI: Harmlessness from AI Feedback:

1. SL stage: have the AI rewrite harmful responses itself according to "constitution" principles.
2. RL stage: use AI (rather than humans) for preference annotation → train RM → PPO (i.e. RLAIF).

The entire harmless training needs no human preference annotation, relying on LLM self-evaluation.

RLAIF (Lee et al. 2023 Google) RLAIF: Scaling RLHF with AI Feedback: on tasks like summarization, helpful dialogue, systematically verifies AI feedback ≈ human feedback in effect. Conditions: the LLM judge must be strong enough (GPT-4 / Claude-3 level), and the prompt must be well designed.

7.3　Reward model ensemble & uncertainty

Recent years (Coste 2024, Eisenstein 2024) have found: single RM is easily hacked, and ensemble of RMs (5-7 RMs with different seeds) combined with conservative aggregation (min / mean - $k\cdot$std) can significantly mitigate reward hacking without losing performance. The cost is doubled memory — hence industrial deployment uses lightweight RM (e.g. 6B evaluating 70B policy).

§8 Reward Hacking & Engineering Experience

Reward hacking is the central pain point of RL post-training: the model finds blind spots in the RM to get high scores, but humans find the result worse. Common symptoms:

8.1　Core mitigation mechanisms

1. KL penalty ($\beta \log\pi/\pi_\text{ref}$): the most basic anchor.
2. Early stopping by KL budget: stop when $\text{KL}(\pi || \pi_\text{ref}) > K_\text{target}$.
3. Reward model ensemble (Coste 2024 ICLR Reward Model Ensembles Help Mitigate Overoptimization): take min or mean - std of multiple RMs.
4. Reward shaping: decompose reward into multiple terms (helpfulness + length + diversity), capping each individually.
5. Offline + online mix: first DPO / KTO to get to ~70 points, then PPO + strong RM for the last mile.
6. Composite reward: weighted combination of rule-based + RM-based, the rule part cannot be hacked.

8.2　Gao 2023 scaling law

Gao, Schulman, Hilton 2023 ICML Scaling Laws for Reward Model Overoptimization:

• Proxy reward (RM score) monotonically rises with KL, but gold reward rises then falls (inverted-U).
• Best-of-N's over-optimization differs slightly in form from PPO: BoN's gold reward is close to linear-minus-plus-linear in $\sqrt{\text{KL}}$, PPO has more higher-order terms.
• Provides a scaling law that larger RM → slower over-optimization, also discussing different functional forms for fitting.

Interview takeaway: reward hacking is not a bug, it is the essence of RL — the inevitable phenomenon of proxy and true objective being separated.

§9 Complexity / Resource Comparison

9.1　Resource comparison of four paradigms

9.2　Rough LLM RLHF training memory estimation

For 7B base, fp16 forward + fp32 master, AdamW (typical RLHF):

Plus activation, KV cache, generation buffer, a single 80GB card is hard to fit; typically using ZeRO-3 + offload or multi-machine sharding.

DPO with same base: only needs $\pi_\theta$ + $\pi_\text{ref}$ = 70 + 14 = 84 GB (half).

GRPO with same base: $\pi_\theta$ + $\pi_\text{ref}$ + RM = 70 + 14 + 14 = 98 GB (saves one value copy).

9.3　Training throughput comparison

Measured (open-source 7B, 8×H100, TRL/OpenRLHF report):

• PPO: ~50-100 token/s/GPU (bottleneck: on-policy generation + 4 copies)
• DPO: ~500-1000 token/s/GPU (pure supervised, no generation)
• GRPO: ~100-200 token/s/GPU (needs generation, but one fewer model)
• SimPO: ~600-1200 token/s/GPU (DPO further drops reference)

DPO is fast because it is fully offline, almost SFT speed; PPO is slow mainly due to rollout (generate a response before one gradient).

§10 25 Frequently-Asked Interview Questions

Divided into 3 tiers by difficulty: L1 = asked at any LLM engineering position; L2 = asked by research/alignment teams; L3 = hardcore questions from top labs / DeepSeek-class teams. Each question expands to answer points + common pitfalls.

L1 Must-Know (10 questions)

L2 Advanced (10 questions)

L3 Top Lab (5 questions)

§A Appendix: Reference List

Organized by section, all verified by codex (gpt-5.5 xhigh) reviewer for correct author-year-venue:

PPO / RL fundamentals

• Schulman et al. 2017 arXiv 1707.06347 Proximal Policy Optimization Algorithms
• Schulman et al. 2016 ICLR High-Dimensional Continuous Control Using GAE
• Schulman 2020 blog Approximating KL Divergence (source of K3 estimator)

RLHF

• Christiano et al. 2017 NeurIPS Deep Reinforcement Learning from Human Preferences (first paper on pairwise preference + RM)
• Stiennon et al. 2020 NeurIPS Learning to Summarize from Human Feedback (OpenAI summarization report)
• Ouyang et al. 2022 NeurIPS Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
• Bai et al. 2022 Anthropic arXiv 2204.05862 Training a Helpful and Harmless Assistant with RLHF
• Bai et al. 2022 Anthropic arXiv 2212.08073 Constitutional AI
• Lee et al. 2023 Google arXiv 2309.00267 RLAIF: Scaling RLHF with AI Feedback

DPO family

• Rafailov et al. 2023 NeurIPS Direct Preference Optimization
• Azar et al. 2024 AISTATS A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO)
• Ethayarajh et al. 2024 ICML KTO: Model Alignment as Prospect Theoretic Optimization
• Meng et al. 2024 NeurIPS SimPO: Simple Preference Optimization with a Reference-Free Reward
• Hong et al. 2024 EMNLP ORPO: Monolithic Preference Optimization without Reference Model
• Tang et al. 2024 ICML Generalized Preference Optimization (unifying DPO/IPO/SLiC framework)

Critic-free RL

• Ahmadian et al. 2024 ACL Back to Basics: Revisiting REINFORCE Style Optimization (RLOO)
• Li et al. 2024 ICML ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method
• Shao et al. 2024 arXiv 2402.03300 DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (proposes GRPO)
• DeepSeek-AI 2025 arXiv 2501.12948 DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Yu et al. 2025 ByteDance arXiv 2503.14476 DAPO: An Open-Source LLM Reinforcement Learning System at Scale

Reward Modeling

• Lightman et al. 2024 ICLR / OpenAI arXiv 2305.20050 (2023) Let's Verify Step by Step (PRM800K, PRM vs ORM)
• Wang et al. 2024 ACL Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations
• Coste et al. 2024 ICLR Reward Model Ensembles Help Mitigate Overoptimization
• Eisenstein et al. 2024 COLM (arXiv 2312.09244, 2023) Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
• Gao, Schulman, Hilton 2023 ICML Scaling Laws for Reward Model Overoptimization

Iterative / Self-rewarding

• Xu et al. 2023 arXiv 2312.16682 Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss
• Yuan et al. 2024 ICML Self-Rewarding Language Models
• Pal et al. 2024 arXiv 2402.13228 Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive