Agentic Rl Tutorial En

§0 TL;DR Cheat Sheet

1. The fundamental difference between Agentic RL and RLHF: RLHF is single-turn preference alignment with reward from an RM scoring an entire response; Agentic RL is multi-turn decision-making, state is (obs, history), action is (thought, tool_call), and reward comes from the external environment (test-pass, task success, verifier) rather than an RM. Trajectory length grows from RLHF's hundreds of tokens to an agent's thousands or even tens of thousands of tokens, taking credit assignment difficulty up a notch.
2. Key modifications PPO/GRPO need on agents (must memorize): the token mask must restrict the loss to tokens the agent itself generates — observation tokens (the stdout / search snippet returned by tools) belong to the environment, the policy gradient must not flow there; otherwise the model will try to "teach the tool how to respond" and behavior collapses. GRPO's advantage is more pronounced: on long-horizon trajectories, a value model can hardly learn per-token V (almost all middle rewards are 0); intra-group normalization is a more stable baseline.
3. Three-tier reward design pyramid: (a) Outcome reward is cheapest and sparsest — final answer/task success 0/1; (b) Process reward scores each step, requiring a PRM or step verifier; (c) Hybrid / shaping — tool-call shaping (encourage calling the right tool), length penalty (prevent the agent from dragging too long), format reward (strictly constrain output schema). The R1 line uses rule-based outcome reward (math correctness + format), SWE-RL uses test-pass, WebRL uses task success — rule-based outcome reward + dense format shaping is the most stable combination empirically in 2025 industry.
4. Representative early work: AgentTuning (Zeng et al. 2023 arXiv 2310.12823 THU) — agent SFT dataset + multi-task training; Agent-FLAN (Chen et al. 2024 ACL Findings arXiv 2403.12881) — splits agent corpus into multi-turn / formatted / negative example three classes; ReFT (Trung et al. 2024 ACL arXiv 2401.08967) — SFT warm-start + online RL on math reasoning, PPO gains +9pp on GSM8K. These three are the "SFT first, then RL" standard three-stage of Agentic RL.
5. Tool-augmented reasoning RL: ToolRL (Qian et al. 2025 arXiv 2504.13958) — embeds tool calls into GRPO with reward = correctness + format + tool-use efficiency; ReSearch (Chen et al. 2025 arXiv 2503.19470) — treats search calls as first-class actions and learns multi-hop search with rule-based reward; RAGEN / StarPO (Wang et al. 2025 arXiv 2504.20073) — a multi-turn RL training framework with state-action token-level loss + critic-free GRPO variant. Common thread: outcome-only reward + format shaping + token-mask loss + GRPO.
6. Web / GUI agent RL: WebRL (Qi et al. 2024 ICLR-25 arXiv 2411.02337) — self-evolving curriculum + ORM + retrospective rollout, pushing 8B Llama to 43% on WebArena; AgentQ (Putta et al. 2024 arXiv 2408.07199) — MCTS search + AI critique + DPO offline training; Computer-Use (Anthropic Claude 3.5/3.7/4 Sonnet, since 2024-10-22) — RLHF + RL on screenshot + mouse/keyboard action space for GUI control (public knowledge: training details undisclosed, but system card mentions extensive human + AI feedback).
7. Code agent RL: CodeRL (Le et al. 2022 NeurIPS arXiv 2207.01780) was the first to use unit tests as reward signal + actor-critic; PPOCoder (Shojaee et al. 2023 arXiv 2301.13816) adds a composite reward of compilable + functional correctness; SWE-RL (Wei et al. 2025 Meta FAIR arXiv 2502.18449) does RL with rule-based reward (patch similarity + test-pass) on GitHub PR data, with Llama-3.3-70B pushing SWE-bench Verified to 41%.
8. Self-rewarding & exploration: Self-Rewarding LM (Yuan et al. 2024 Meta arXiv 2401.10020) has the policy serve as judge for iterative DPO with LLM-as-judge; but self-rewarding is more dangerous in agents than in single-turn alignment — the judge is also the agent itself, and reward drift / model collapse easily occur. In production the common trio is LLM-as-judge ensemble + rule-based grounding (test-pass, math checker) + human spot check.
9. Three weapons for long-horizon credit assignment: (a) GAE + γ < 1 propagates credit along the trajectory but degenerates to MC return under sparse outcome reward; (b) Hindsight relabeling (the agent-side counterpart to HER) — failed trajectories are relabeled with "intermediate state as goal"; (c) subgoal decomposition + process reward — slice a 50-step trajectory into 5 subgoals × 10 steps and have a PRM score each subgoal. The L3 interview question "why is GRPO more sample efficient than PPO on long-horizon agents" — the answer is trace-level reward directly matches trace-level credit, bypassing the pain that a value model can hardly learn anything on long CoT.

§1 Intuition: from RLHF to Agentic RL

1.1　Upgrading an LLM from "policy that writes" to "agent that acts"

RLHF trains an LLM into a policy that "writes per human preference"; but an RLHF policy is still fragile on tool calling / multi-turn interaction / long-horizon tasks:

• single-turn preferences do not directly transfer to multi-turn task success
• The RM learns "which style humans prefer", not "which call order solves the problem"
• Reward is over the entire response, with no way to distinguish "the first 100 tokens reasoned correctly but token 101 chose the wrong tool"

The essence of Agentic RL is to hang the RL signal on objective trajectory-terminal outcomes (test pass, math correct, web task completed) rather than the RM's subjective preference. This step upgrades alignment-style RL into decision-making RL.

1.2　Mental model: MDP / POMDP formulation

      ┌──────────────────────────────┐
      │   Policy π_θ (LLM)           │  agent
      └──────────────┬───────────────┘
                     │ action a_t = (thought, tool_call)
                     ↓
      ┌──────────────────────────────┐
      │   Environment / Tool         │
      │   - search / shell / browser │
      │   - Python / unit test       │
      └──────────────┬───────────────┘
                     │ observation o_t
                     ↓
      ┌──────────────────────────────┐
      │   History buffer             │
      │   (concat back to prompt)    │
      └──────────────┬───────────────┘
                     │
                     └─→ back to π_θ

1.3　Agentic RL relative to three RL neighbors

• (1) Strict definition: multi-turn + tool I/O + outcome reward RL
• (2) Boundary vs RLHF: RLHF is single-turn alignment, Agentic is multi-turn decision-making
• (3) Boundary vs reasoning RL: reasoning RL only checks answer correctness; Agentic RL evaluates task success on a real environment

These three sentences anchor the interviewer's expectations within 30 seconds.

§2 Key modifications PPO / GRPO need for agents

2.1　Token mask: loss only on agent tokens

This is the first iron law of Agentic RL. Tokens in an agent trajectory split into two classes:

• Agent token: generated by the policy $\pi_\theta$ (thought, action JSON, final answer)
• Environment token: observation returned by the tool (search snippet, stdout, screenshot caption)

PPO/GRPO log-prob ratios and loss must be computed only on agent tokens. If loss is also applied on observation tokens:

• The policy will try to "teach the tool how to respond" (meaningless and reward-hackable)
• The gradient will be diluted by huge amounts of low-information observation tokens
• KL penalty will mistakenly estimate the policy's own distribution from environment text

The implementation is an action_mask: [B, L] tensor: 1 marks agent-generated tokens, 0 marks prompt / observation / padding. Loss computation: (loss * action_mask).sum() / action_mask.sum().

2.2　Trajectory-level GAE for agents

Agent trajectories are long (50-200 steps) and rewards are extremely sparse (only at the terminal). Let:

• $r_t \in \mathbb{R}$: reward at step $t$ (mostly $r_t = 0$, terminal $r_T = R \in \{0, 1\}$)
• $V_\phi(s_t)$: critic estimate

TD residual:

$$\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$$

GAE:

$$A_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}$$

How to choose $\gamma$ for LLM agents? Depends on the definition of "step":

• If step = single token: $\gamma$ close to 1 (token-level discount has no meaning)
• If step = one thought-action-obs cycle: $\gamma \in [0.95, 0.99]$ is reasonable, controlling long-horizon discounting

How GAE degenerates in LLM agents: under sparse terminal reward, $\lambda = 1$ + $\gamma = 1$ is equivalent to sequence-level MC return minus baseline. This is exactly the implicit explanation of GRPO directly using trace-level reward.

2.3　PPO loss adapted to agents

Masked PPO-Clip (outer expectation over the whole trajectory, inner sum over tokens):

$$\boxed{\;L^{\text{CLIP-agent}}(\theta) = \mathbb{E}_{\tau \sim \pi_\text{old}}\!\left[\frac{\sum_{t=1}^{T} m_t \cdot \min\!\big(\rho_t A_t,\, \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon) A_t\big)}{\sum_{t=1}^{T} m_t}\right]\;}$$

where $\tau$ is the trajectory (containing all $T$ tokens / steps), $m_t \in \{0, 1\}$ is the agent action_mask (1 for agent-generated tokens, 0 for observation/system tokens), and $\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\theta_\text{old}}(a_t \mid s_t)$ is the token-level importance ratio. Note that the outer expectation index is the trajectory $\tau$ while the inner sum index is the token $t$; do not confuse them.

Per-token KL penalty (written into the reward):

$$\tilde{r}_t = m_t \cdot \big(\text{rule}\_\text{reward}_t - \beta \log \tfrac{\pi_\theta(a_t \mid s_t)}{\pi_\text{ref}(a_t \mid s_t)}\big)$$

Note: KL is computed only on agent tokens; the $\pi$ on observation tokens is meaningless (those are emitted by the environment, not sampled by the model).

2.4　GRPO for agents: trace-level group-relative advantage

GRPO fits agents better because:

1. Saves the critic — agent value is hard to learn (long horizon + sparse reward)
2. Trace-level reward directly corresponds to trace-level advantage — no per-step value needed
3. Multiple rollouts per prompt automatically reduce variance — agent tasks are usually deterministic-env, multi-rollout gives the true reward variance

The formula (keep the PPO-Clip structure, advantage switched to intra-group normalization):

$$\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\}) + \epsilon}$$

All agent tokens in the whole trajectory share the same $\hat{A}_i$ (observation tokens are still masked out):

$$L^{\text{GRPO-agent}}(\theta) = \mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{\sum_t m_{i,t}}\sum_{t=1}^{T_i} m_{i,t} \cdot \Big(\min(\rho_{i,t} \hat{A}_i, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_i) - \beta\, \text{KL}_{i,t}\Big)\right]$$

KL typically uses the K3 estimator (Schulman 2020 blog): $\text{KL}_{i,t} = \exp(\log\pi_\text{ref} - \log\pi_\theta) - (\log\pi_\text{ref} - \log\pi_\theta) - 1$.

• Saves the value model (one model's VRAM)
• Saves per-token credit assignment (trace-level advantage)
• Saves reward shaping (rule-based terminal is enough)
• Saves hyperparameters ($c_v, \lambda$ are not needed)

• On long trajectories the trace-level advantage is too coarse (all agent tokens share the same $\hat{A}$) — credit dilution on long trajectories
• All-success / all-fail groups have $\text{std} = 0$ degeneracy (agent task reward is binary, this happens frequently)
• On-policy rollout is slow (agent rollout includes tool I/O latency, far slower than chat completion)

§3 Reward design for agents (core)

Reward is the lifeline of Agentic RL. If the reward is wrong, no model size or new algorithm will save you; if the reward is right, plain GRPO can hit SOTA.

3.1　Outcome reward vs process reward

Interview take: reasoning RL (R1) chose outcome reward because math/code can be programmatically verified; mainstream agent RL also chose outcome reward because agent task ground truth is clearer (task completed yes/no). Process reward is mainly used for reasoning-heavy tasks (PRM on math steps); it is rare in tool-use agents.

3.2　Verifier-based reward (rule-based)

This is the cleanest reward form for Agentic RL: write the reward as an executable verifier function.

def verifier_reward(trajectory) -> float:
    """
    trajectory: list of (thought, action, observation)
    Returns an outcome reward of 0 or 1
    """
    final_answer = trajectory[-1].final_answer

    # 1. Math: exact-match ground truth
    if task_type == "math":
        return 1.0 if normalize_math(final_answer) == ground_truth else 0.0

    # 2. Code: run unit tests
    if task_type == "code":
        code = extract_code(final_answer)
        pass_count = run_unit_tests(code, test_cases)
        return pass_count / len(test_cases)   # partial credit

    # 3. SWE-bench: apply patch + run test
    if task_type == "swe":
        try:
            apply_patch(repo, final_answer)
            return 1.0 if run_test(repo, expected_test) else 0.0
        except PatchError:
            return 0.0

    # 4. Web agent: task-specific verifier
    if task_type == "webshop":
        return webshop_grader(final_state)    # provided by the benchmark

Key advantages of verifier-based reward:

• Close to ground truth, avoiding the main failure mode of reward hacking on learned RMs
• Repeatable (the same trajectory gets the same reward), making advantage estimates well-defined
• Memory / compute overhead is minimal (executing a verifier is hundreds of times cheaper than one LLM forward pass)

Core limitation: usable only on "verifiable tasks" — math, code, formal verification, grader-able web/GUI tasks. Open-ended tasks (writing, dialogue) still need an RM.

3.3　Format reward / shaping reward

With outcome reward alone, agents often learn "format-broken but happens to be correct" trajectories — e.g. skipping the <think> block and going straight to Action: answer(42). A format reward adds a lightweight format constraint signal:

def format_reward(trajectory) -> float:
    """
    Check whether the trajectory matches the expected format schema
    Returns a continuous score in [0, 1]
    """
    score = 0.0

    # Must contain a <think>...</think> block
    if "<think>" in trajectory.text and "</think>" in trajectory.text:
        score += 0.3

    # Tool calls must be valid JSON
    for action in trajectory.actions:
        if is_valid_json(action.tool_call):
            score += 0.1
        else:
            score -= 0.2   # serious error

    # The final answer must be wrapped in \boxed{...} (math task)
    if has_boxed_answer(trajectory.final):
        score += 0.2

    return max(0.0, min(1.0, score))

A typical composite reward formulation:

$$r_\text{total} = \alpha \cdot r_\text{outcome} + \beta \cdot r_\text{format} + \gamma \cdot r_\text{shaping}$$

R1 / R1-Zero use a simple accuracy_reward + format_reward sum; ToolRL / RAGEN etc. add tool-call efficiency shaping.

3.4　Length penalty (prevent the agent from dragging too long)

An emergent failure mode of agent RL: the model learns that "dragging out the trajectory raises the probability of a correct answer" — for a 5-step task the agent goes 50 steps. This is a form of reward hacking.

Mitigation:

$$r_\text{adjusted} = r_\text{outcome} - \lambda \cdot \max(0, T - T_\text{target})$$

or a softer sigmoid form:

$$r_\text{adjusted} = r_\text{outcome} \cdot \sigma\!\big(-(T - T_\text{target}) / \tau\big)$$

DAPO reports "overlong shaping" — exponential reward decay after exceeding the length budget, preventing the agent from extending to the context limit.

3.5　Tool-call shaping reward

Reward "calling the right tool", penalize "calling the wrong tool / repeated calls":

def tool_shaping(trajectory) -> float:
    score = 0.0

    # Calling the right tool (task-relevance heuristic)
    if task_needs_search and any(a.tool == "search" for a in trajectory.actions):
        score += 0.1

    # Penalize consecutive duplicate calls
    consecutive_dup = count_consecutive_duplicate_calls(trajectory.actions)
    score -= 0.05 * consecutive_dup

    # Penalize calls to non-existent tools
    invalid_calls = sum(1 for a in trajectory.actions if a.tool not in TOOL_REGISTRY)
    score -= 0.3 * invalid_calls

    return score

3.6　RLAIF for agents: LLM as reward

When an external verifier is hard to write (open-ended tasks), use a strong LLM as judge:

def llm_judge_reward(trajectory, judge_model) -> float:
    """
    Use a judge LLM to score the trajectory
    """
    prompt = f"""
    Judge whether the agent completed this task successfully.
    Task: {trajectory.task}
    Final state: {trajectory.final_state}
    Return JSON: {{"success": bool, "reasoning": str}}
    """
    judgment = judge_model(prompt)
    return 1.0 if judgment["success"] else 0.0

Risks:

• The judge LLM has its own biases (length preference, sycophancy) → amplified into the student
• The judge itself may be prompt-injected (the agent trajectory contains malicious instructions)
• High compute cost (one judge call per trajectory)

Mainstream mitigations: (a) judge ensemble (3-5 different models judge, majority vote); (b) judge with rubric (strictly constrain output structure); (c) rule + LLM hybrid (verifiable parts use rule, open-ended parts use LLM).

§4 Long-horizon credit assignment

4.1　Sparse reward is the core pain of agent RL

Classical game RL (Atari, Mujoco) has dense reward; an LLM agent on a long trajectory gets reward 0 almost everywhere and a signal only at the terminal. This causes:

• The value model can barely learn anything (what should middle V be?)
• Per-token policy gradient has huge variance
• The causal chain from early steps to reward is diluted — did "step 3 picking search tool" cause "step 50 being correct"?

4.2　Degeneration of discount + GAE on agents

Recall GAE:

$$A_t = \sum_l (\gamma\lambda)^l \delta_{t+l}, \quad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

Under sparse terminal reward ($r_t = 0$ for $t \lt T$, $r_T = R$):

$$A_t = \gamma^{T-t} R - V(s_t) + \text{value correction terms}$$

That is, advantage ≈ discounted return - baseline. If $V_\phi$ is inaccurate (often the case in agents), this degenerates into raw MC return; GAE's bias-variance trade-off fails.

Practical take: on agent RL, GAE is less stable than group-relative advantage (GRPO); this is one of the fundamental reasons why GRPO is more sample efficient than PPO on agents.

4.3　Hindsight relabeling for agents

Hindsight Experience Replay (Andrychowicz et al. 2017 NeurIPS) originated in robot manipulation: failed trajectories aren't thrown away; instead the state actually reached is treated as the goal and reward is relabeled.

LLM agent version:

def hindsight_relabel(trajectory):
    """
    Turn a failed trajectory into a successful trajectory for an "alternative task"
    """
    if trajectory.outcome == 1:
        return [trajectory]   # successful trajectories untouched

    # Suppose the agent meant to buy item A on the web but ended on item B's page
    # → relabel as "find item B"; reward = 1
    alt_task = describe_terminal_state(trajectory.final_state)
    relabeled = trajectory.with_task(alt_task)
    relabeled.outcome = 1
    return [trajectory, relabeled]

4.4　Subgoal decomposition + process reward

Slicing a long trajectory into subgoals is another credit-assignment route:

• Slice a 100-step trajectory into 5 subgoals × 20 steps
• Each subgoal terminal gets a process reward (whether the subgoal is completed)
• Subgoal rewards sum to the trajectory reward

Implementations:

• Hand-crafted subgoals: humans write the criterion for each subgoal (e.g. "reaching the cart page" triggers subgoal-1 reward)
• LLM-decomposed subgoals: a planner LLM splits the task into subgoals; a verifier judges each
• PRM-style step reward: train a PRM to score each step (Math-Shepherd line)

4.5　Per-step KL penalty to prevent policy collapse

Over long agent trajectories the policy easily "all-ins" (generates high-confidence tokens at every step), causing entropy collapse:

$$\tilde{r}_t = m_t \cdot \big(r_t - \beta \cdot \text{KL}(\pi_\theta(\cdot \mid s_t) \| \pi_\text{ref}(\cdot \mid s_t))\big)$$

Key: KL must be per-step + only on agent tokens. Once KL is computed on observation tokens, the policy gets penalized for "mimicking environment text" — but this penalty is meaningless (the policy shouldn't mimic the environment, only use observations as conditioning).

R1-Zero uses $\beta = 0.001$; agent tasks commonly use $\beta \in [0.001, 0.05]$.

§5 Self-rewarding & exploration in agent RL

5.1　Self-Rewarding LM (Yuan et al. 2024 Meta arXiv 2401.10020)

Core idea: have the policy serve as judge, iterative DPO:

  Iteration k:
    1. policy_k generates multiple responses per prompt
    2. policy_k self-scores via LLM-as-judge (also a policy_k prompt)
    3. High vs low scores form a preference pair
    4. policy_{k+1} = DPO(policy_k, preference_pair)

Effect: on AlpacaEval, iterative self-scoring continuously improves.

5.2　Risks of self-rewarding on agents

Biggest difference between agent task and alignment task: alignment has an "objective preference distribution" (humans find it helpful, polite) suitable for LLM judging; agent tasks have objective ground truth (test pass / task success) — self-rewarding then:

• The judge itself may be wrong (the agent got it wrong but self-evaluates correct) → training collapse
• Iterative drift: each round reinforces trajectories "I thought were correct", drifting away from ground truth
• Exploration degeneration: high self-scores prefer known patterns, suppressing exploration of new tools

Mainstream practice: agent RL prefers rule-based ground truth, with self-rewarding only as an auxiliary signal (e.g. open-ended task fallback).

5.3　Exploration in agent RL

Agent action space includes:

• Token-level exploration: sampling temperature, controlling token-choice diversity
• Tool-call-level exploration: each thought-action cycle picks different tools / different queries
• Trajectory-level exploration: entirely different trajectory plans

Classical practices:

The key insight of RAGEN / StarPO (Wang et al. 2025 arXiv 2504.20073): in multi-turn agent RL rollout diversity is the "firebreak" for collapse — single-trajectory training causes the policy to degenerate into deterministic mode.

5.4　Curriculum & difficulty scheduling

Sort agent tasks from easy to hard, schedule based on the model's current ability:

• WebRL (Qi et al. 2024 ICLR-25 arXiv 2411.02337) uses a self-evolving curriculum — failed tasks are recorded and added to the buffer in the next round
• Absolute Zero / R-Zero use a learnability reward: pick tasks where the model's success rate is ≈ 50% (maximum learning signal)

§6 Specific algorithms (representative Agentic RL papers)

Sorted by "time + data/task type", each with one sentence + key formula.

6.1　VPT (Baker et al. 2022 NeurIPS OpenAI, arXiv 2206.11795)

Setting: Minecraft; first pretrain an inverse dynamics model (IDM) on 70k hours of YouTube videos, then have the IDM auto-label action → behavior clone → RL fine-tune.

Key: the first large-scale "video → action label → policy → RL" pipeline. The early blueprint for agent RL, proving scaling RL with imitation pretrain is feasible.

6.2　AgentTuning (Zeng et al. 2023 THU arXiv 2310.12823)

Setting: construct the AgentInstruct dataset (demonstrations from 6 agent tasks), multi-task SFT.

Key: not RL, but agent SFT — yet this is the standard warm-start step for Agentic RL. After AgentTuning, Llama-2 averages +50% on agent tasks.

6.3　Agent-FLAN (Chen et al. 2024 ACL Findings, arXiv 2403.12881)

Setting: Agent SFT data split into three classes — multi-turn dialogue / formatted tool calls / negative examples (refusal / failure cases).

Key: negative examples significantly reduce hallucinated tool calls — SFT doesn't just see "how to do it right", it also sees "why this is wrong". An important milestone for agent SFT engineering.

6.4　ReFT (Trung et al. 2024 ACL arXiv 2401.08967)

Setting: math reasoning agent; SFT warm-start, then PPO with rule-based outcome reward (answer correctness).

Key formula (standard PPO + verifier):

$$r(\tau) = \mathbb{1}[\text{answer}(\tau) = y^*] - \beta \cdot \text{KL}(\pi_\theta \| \pi_\text{SFT})$$

Result: GSM8K +9pp over SFT, MathQA +7pp. Proves PPO + outcome reward is stable and feasible on reasoning agents — the predecessor experiment for R1.

6.5　DeepSeek-R1 (DeepSeek-AI 2025 arXiv 2501.12948) extension on agents

R1 itself isn't an agent paper, but R1's GRPO + rule-based reward + format reward methodology was directly inherited by ToolRL / ReSearch / RAGEN. R1 = the algorithmic baseline template of Agentic RL.

Review of GRPO on agents:

• $G$ rollouts per prompt
• Trace-level reward (rule-based)
• Group-relative advantage
• Per-step KL with K3 estimator
• Agent token mask

6.6　ToolRL (Qian et al. 2025 arXiv 2504.13958)

Setting: GRPO on a tool-augmented LLM with reward = correctness + format + tool-use efficiency.

Key formula:

$$r(\tau) = r_\text{correct} + \alpha \cdot r_\text{format} + \gamma \cdot r_\text{tool-eff}$$

where $r_\text{tool-eff}$ penalizes redundant/invalid tool calls.

Result: 7B model approaches GPT-4 on BFCL (Berkeley Function Calling Leaderboard). Open-source validation of GRPO + tool shaping stability.

6.7　ReSearch (Chen et al. 2025 arXiv 2503.19470)

Setting: search-augmented agent treating search as a first-class action; reward = answer correctness only (rule-based).

Key idea: multi-hop search can be learned without process reward; outcome-only + GRPO is enough — provided the base model, after SFT warm-start, can already emit legal search queries.

6.8　RAGEN / StarPO (Wang et al. 2025 arXiv 2504.20073)

Paper: "Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning".

Setting: multi-turn RL agent training framework with state-action token-level loss + critic-free.

Key contributions:

1. StarPO (State-Thinking-Actions-Reward Policy Optimization): critic-free, the whole trajectory shares the advantage, with a strict token mask that limits loss to agent tokens. The StarPO-S variant introduces fine-grained reasoning-aware reward + optional critic incorporation, further mitigating multi-turn reward sparsity (the paper abstract's exact wording).
2. Rollout diversity = collapse firebreak: empirically, group size $G = 16$ is significantly more stable than $G = 4$
3. Trajectory length signal: when failed-trajectory length is large, reward shaping should add a length penalty

6.9　WebRL (Qi et al. 2024 ICLR-25 arXiv 2411.02337)

Setting: web agent (WebArena), self-evolving curriculum + ORM + retrospective rollout.

Key components:

• ORM (Outcome Reward Model) trained from task success → gives reward online
• Failed tasks enter the curriculum buffer; the next round increases their weight
• Retrospective rollout: failed trajectories are rewritten into "correct trajectories" by an LLM and re-SFT'd

Result: Llama-3.1-8B reaches 43% on WebArena (vs GPT-4 14.4%), proving that small model + good RL pipeline > large model + zero-shot.

6.10　AgentQ (Putta et al. 2024 arXiv 2408.07199)

Setting: web agent, MCTS + AI critique + offline DPO.

Key idea: MCTS searches for "reward-balanced" preference pairs (high-score trajectory vs low-score trajectory) and DPO offline-trains them. No online RL infra needed; practical for compute-limited scenarios.

6.11　WebGUM (Furuta et al. 2024 ICLR arXiv 2305.11854)

Setting: HTML + screenshot multimodal web agent, offline SFT from demonstrations (not RL fine-tune — the paper is the imitation/supervised paradigm).

Key: feeds DOM + screenshot together into the model, improving grounding accuracy over pure text. It's the dataset + base stage for Computer-Use / web agent RL (e.g. WebRL, AgentQ). Subsequent web agent RL often does RL fine-tune on top of WebGUM-class bases.

6.12　CodeRL (Le et al. 2022 NeurIPS arXiv 2207.01780)

Setting: code generation + actor-critic, reward = unit-test pass.

Key formula: classical actor-critic with the critic as a token-level baseline; the PG signal comes from final test-pass.

6.13　PPOCoder (Shojaee et al. 2023 arXiv 2301.13816)

Setting: code generation + PPO with composite reward = compilable + functional correctness.

Key: early attempt at PPO for code-gen. Proves multi-component reward is more stable than single test-pass training.

6.14　SWE-RL (Wei et al. 2025 Meta FAIR arXiv 2502.18449)

Setting: SWE-bench / GitHub PR data, rule-based reward = patch similarity + test-pass, GRPO.

Key features:

• Data scale: Meta constructs 76M+ context-issue-patch triples from GitHub PR commit history
• Reward: edit similarity (oracle patch ↔ predicted patch) + binary test pass
• Algorithm: pure GRPO + format reward

Result: Llama-3.3-70B + SWE-RL reaches 41% on SWE-bench Verified (no scaffold), proving rule-based RL on real PR data can let the model learn emergent reasoning behavior — e.g. file-level retrieval planning, root-cause analysis, test self-validation.

6.15　OpenVLA (Kim et al. 2024 arXiv 2406.09246)

Setting: a 7B vision-language-action model for robot manipulation, fine-tuned on 970K real robot demonstrations (open-source base + LoRA fine-tuning recipe); the paper focuses on imitation learning + parameter-efficient adaptation, not task-specific RL fine-tune.

Key: a robot agent rather than an LLM agent; often used to compare "LLM agent vs robot agent". Subsequent work (OpenVLA-OFT, π-RL family, etc.) does RL on top of the OpenVLA base, but the OpenVLA paper itself does not.

6.16　Anthropic Computer-Use (public knowledge — training details undisclosed)

Anthropic Claude 3.5 Sonnet (new) supports Computer-Use since 2024-10-22; 3.7 / 4.0 / 4.5 / Opus 4.x continue to iterate.

Public materials describe only capability + safety guardrails; the training algorithm / reward form is undisclosed:

• Action space = screen coordinates + keyboard events + mouse events (screenshot as observation)
• Safety: constitutional-AI-style guardrails + red-teaming + prompt-injection defense
• The system card mentions training involves human demonstrations + synthetic data, but does not disclose specifics like RLHF / GRPO / verifier reward

Community speculation (speculative only, not official): possibly GRPO/PPO + verifier-based reward + Constitutional-AI-style AI feedback, but with no official confirmation. When discussing Computer-Use training in interviews, clearly distinguish "public capability" from "speculated internals".

§7 Code patterns (PyTorch / pseudocode)

The most easily mis-implemented pieces when writing Agentic RL. Each block is independently readable.

7.1　Agent rollout (state, action, reward collection)

import torch
from dataclasses import dataclass

@dataclass
class Step:
    obs: str                   # previous-step observation or prompt
    thought_tokens: list[int]  # thought generated by the agent
    action_tokens: list[int]   # tool_call JSON generated by the agent
    tool_name: str
    tool_args: dict
    observation: str           # tool's returned result
    done: bool

def rollout(policy, env, prompt, max_steps=20, max_tokens_per_step=512):
    """
    One agent trajectory rollout
    Returns: trajectory (list of Step), final reward
    """
    trajectory = []
    history = prompt
    for step_idx in range(max_steps):
        # ── Agent generates (thought, action) ──
        agent_output = policy.generate(
            prompt=history,
            stop_tokens=["</action>"],
            max_new_tokens=max_tokens_per_step,
            temperature=0.7,
        )
        thought, action_json = parse_thought_action(agent_output)

        # ── Call the tool ──
        tool_name = action_json["tool"]
        tool_args = action_json["args"]
        if tool_name == "final_answer":
            obs = action_json["answer"]
            done = True
        else:
            obs = env.call(tool_name, tool_args)
            done = False

        # Update history (append back to the prompt)
        history = history + agent_output + f"\n<obs>{obs}</obs>\n"

        trajectory.append(Step(
            obs=history,                    # history up to this point
            thought_tokens=tokenize(thought),
            action_tokens=tokenize(action_json),
            tool_name=tool_name,
            tool_args=tool_args,
            observation=obs,
            done=done,
        ))

        if done or step_idx == max_steps - 1:
            break

    # ── Terminal reward ──
    final_reward = env.compute_reward(trajectory)
    return trajectory, final_reward

7.2　Trajectory-level GAE advantage

import torch

def trajectory_gae(rewards, values, dones, gamma=0.99, lam=0.95):
    """
    Compute step-level GAE advantages
    rewards: [T]    per-step reward (mostly = 0, terminal = R)
    values:  [T+1]  V(s_0)...V(s_T); V(s_T) should be 0 (terminal)
    dones:   [T]    1 if terminal else 0
    Returns: advantages [T], returns [T]
    """
    T = rewards.shape[0]
    advantages = torch.zeros_like(rewards)
    gae = 0.0
    for t in reversed(range(T)):
        non_term = 1.0 - dones[t]
        delta = rewards[t] + gamma * values[t + 1] * non_term - values[t]
        gae = delta + gamma * lam * non_term * gae
        advantages[t] = gae
    returns = advantages + values[:T]
    return advantages, returns

7.3　PPO loss adapted to agents (with action_mask)

This is the most important piece of Agentic RL code. Difference from RLHF PPO: observation tokens must be explicitly masked out.

import torch
import torch.nn.functional as F

def ppo_agent_step(policy, value, batch, eps_clip=0.2, c_v=0.5, c_e=0.01):
    """
    batch:
      input_ids:       [B, L]    full trajectory tokens (prompt + thought + action + obs ...)
      action_mask:     [B, L]    1 = agent-generated token, 0 = prompt/observation/pad
      old_log_probs:   [B, L]    log π_θ_old at sample time, 0 at masked positions
      advantages:      [B, L]    step-level GAE advantages (broadcast to all agent tokens of that step)
      returns:         [B, L]    GAE returns for value loss
    """
    logits = policy(batch["input_ids"]).logits           # [B, L, V]
    log_probs = F.log_softmax(logits[:, :-1], dim=-1)    # [B, L-1, V]
    targets = batch["input_ids"][:, 1:].unsqueeze(-1)
    new_log_probs = log_probs.gather(-1, targets).squeeze(-1)  # [B, L-1]
    new_log_probs = F.pad(new_log_probs, (1, 0))               # align to [B, L]

    # ── Key: action_mask ──
    mask = batch["action_mask"].float()
    # Compute ratio / loss only on agent-generated tokens
    ratio = torch.exp((new_log_probs - batch["old_log_probs"]) * mask)
    # At observation positions: ratio = exp(0) = 1, does not affect surr1/surr2

    A = batch["advantages"]
    surr1 = ratio * A
    surr2 = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip) * A
    # Mean over masked positions (avoid observation tokens lowering loss scale)
    policy_loss = -((torch.min(surr1, surr2) * mask).sum() / mask.sum().clamp_min(1.0))

    # Value loss: also only on agent tokens (V of observation tokens is meaningless)
    V = value(batch["input_ids"]).squeeze(-1)            # [B, L]
    value_loss = (((V - batch["returns"]) ** 2) * mask).sum() / mask.sum().clamp_min(1.0)

    # Entropy bonus: only on agent tokens
    probs = log_probs.exp()
    entropy = -(probs * log_probs).sum(-1)               # [B, L-1]
    entropy = F.pad(entropy, (1, 0))
    entropy_bonus = (entropy * mask).sum() / mask.sum().clamp_min(1.0)

    loss = policy_loss + c_v * value_loss - c_e * entropy_bonus

    # Monitoring
    with torch.no_grad():
        approx_kl = ((ratio - 1) - torch.log(ratio.clamp_min(1e-8))) * mask
        approx_kl = approx_kl.sum() / mask.sum().clamp_min(1.0)
    return loss, {
        "policy": policy_loss.item(),
        "value": value_loss.item(),
        "entropy": entropy_bonus.item(),
        "approx_kl": approx_kl.item(),
    }

• Must cover prompt tokens: prompts aren't agent-generated; mask = 0
• Must cover all observation tokens: every token returned by the tool (including the <obs> tag itself) mask = 0
• Must cover all padding: right-pad positions mask = 0
• Separator tokens (e.g. <action>, </thought>) count as agent tokens; mask = 1
• Multi-turn batches have different mask patterns for different trajectories; must be computed per-sample

7.4　GRPO group-relative reward on agent trajectories

import torch
import torch.nn.functional as F

def grpo_agent_loss(policy, ref_policy, batch, eps_clip=0.2, beta=0.04):
    """
    batch:
      input_ids:     [N, L]   N = sum_b G samples in batch
      action_mask:   [N, L]   agent-generated token mask
      old_log_probs: [N, L]   detached log probs at rollout time
      rewards:       [N]      trajectory-level outcome reward
      group_id:      [N]      same prompt → same group_id
    """
    rewards = batch["rewards"]
    gid = batch["group_id"].long()

    # ── Intra-group normalization ──
    num_groups = int(gid.max().item()) + 1
    counts = torch.zeros(num_groups, device=rewards.device).scatter_add_(
        0, gid, torch.ones_like(rewards))
    sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
        0, gid, rewards)
    group_mean = sums / counts.clamp_min(1.0)
    diff_sq = (rewards - group_mean[gid]) ** 2
    sq_sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
        0, gid, diff_sq)
    group_std = (sq_sums / counts.clamp_min(1.0)).sqrt()

    A = (rewards - group_mean[gid]) / (group_std[gid] + 1e-8)   # [N]
    A = A.unsqueeze(-1)                                          # [N, 1] shared across the trajectory

    # ── log-prob ratio ──
    logits = policy(batch["input_ids"]).logits[:, :-1]
    log_probs = F.log_softmax(logits, dim=-1)
    tgt = batch["input_ids"][:, 1:].unsqueeze(-1)
    new_log_probs = log_probs.gather(-1, tgt).squeeze(-1)
    new_log_probs = F.pad(new_log_probs, (1, 0))                 # [N, L]
    mask = batch["action_mask"].float()

    ratio = torch.exp((new_log_probs - batch["old_log_probs"]) * mask)

    # ── PPO-Clip surrogate (advantage broadcast to the whole trajectory) ──
    surr1 = ratio * A
    surr2 = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip) * A

    # ── KL with K3 estimator (Schulman 2020 blog) ──
    with torch.no_grad():
        ref_logits = ref_policy(batch["input_ids"]).logits[:, :-1]
        ref_log_probs = F.log_softmax(ref_logits, dim=-1)
        ref_token_lp = ref_log_probs.gather(-1, tgt).squeeze(-1)
        ref_token_lp = F.pad(ref_token_lp, (1, 0))
    delta = ref_token_lp - new_log_probs                          # log(π_ref / π_θ)
    kl_per_token = torch.exp(delta) - delta - 1.0                 # K3, non-negative

    token_obj = torch.min(surr1, surr2) - beta * kl_per_token     # [N, L]
    seq_len = mask.sum(dim=-1).clamp_min(1.0)                     # [N]
    per_seq = (token_obj * mask).sum(dim=-1) / seq_len            # [N]
    loss = -per_seq.mean()

    return loss, {
        "reward_mean": rewards.mean().item(),
        "advantage_std": A.squeeze(-1).std().item(),
        "kl": (kl_per_token * mask).sum().item() / mask.sum().clamp_min(1.0).item(),
    }

7.5　Outcome + step reward combination

def composite_reward(trajectory, weights=None):
    """
    Combine outcome / format / shaping rewards into the final reward
    weights: dict[str, float]
    """
    if weights is None:
        weights = {"outcome": 1.0, "format": 0.2, "tool_eff": 0.1, "length": -0.05}

    r = {}
    r["outcome"] = outcome_verifier(trajectory)                  # in {0, 1}
    r["format"]  = format_score(trajectory)                      # in [0, 1]
    r["tool_eff"] = tool_efficiency_score(trajectory)            # in [-1, 1]
    r["length"]  = max(0, len(trajectory.steps) - target_len)    # excess steps

    total = sum(weights[k] * r[k] for k in r)
    return total, r

7.6　Verifier-based reward (code test / math match)

import re
import subprocess

def verifier_reward(trajectory, task_type, ground_truth):
    """
    Rule-based outcome reward
    """
    final = trajectory.final_answer

    if task_type == "math":
        return float(extract_boxed_answer(final) == normalize(ground_truth))

    if task_type == "code":
        code = extract_python_code(final)
        if code is None:
            return 0.0
        passed = 0
        for test in ground_truth["tests"]:
            try:
                result = subprocess.run(
                    ["python", "-c", code + "\n" + test],
                    capture_output=True, timeout=5,
                )
                if result.returncode == 0:
                    passed += 1
            except subprocess.TimeoutExpired:
                continue
        return passed / len(ground_truth["tests"])

    if task_type == "swe":
        patch = extract_unified_diff(final)
        if patch is None:
            return 0.0
        success = apply_patch_and_run_test(
            repo=ground_truth["repo"],
            patch=patch,
            test=ground_truth["test"],
        )
        return float(success)

    if task_type == "webshop":
        return webshop_grader(trajectory.final_state, ground_truth)

    raise ValueError(f"Unknown task type: {task_type}")

§8 Frontier (key 2024-2026 trends)

8.1　Critic-free RL is the new default

After DeepSeek-R1 (Jan 2025), critic-free RL (the GRPO family) became the open-source agent RL mainstream. Reasons:

• The value model can hardly learn anything on long-horizon agents
• One model's worth of VRAM saved → bigger batches / group sizes
• Easier tuning (no $c_v$, value lr to tune)

verl (ByteDance 2024+), OpenRLHF, TRL have all made GRPO / RLOO / ReMax first-class trainers.

8.2　The victory of rule-based reward on agents

What WebRL / ReSearch / SWE-RL / RAGEN have in common: outcome reward + format reward, all rule-based, avoiding the main failure mode of reward hacking on learned RMs.

Why has rule-based suddenly become viable:

• Agent tasks are more "programmatically verifiable" than alignment tasks — test pass / answer match / task complete
• DeepSeek-R1 proved rule-based reward is stable and scalable for LLM RL
• Learned RMs are more easily hacked in multi-turn settings (the trajectory space is large)

8.3　The infra challenges of long-horizon training

Agent RL rollout is slow because every trajectory includes tool I/O latency. Infra trends:

• vLLM / SGL async rollout: decouple generation from training; pool rollouts
• Sandboxed execution: tool execution inside isolated containers, parallelizable
• Trajectory queue: rollout workers / training workers are async; trajectories flow through a message queue
• Off-policy correction: there's lag between rollout and update; use IS clip or V-trace to correct

Representative implementations: verl (ByteDance Seed open source), AReaL (Ant Group + Tsinghua, async RL system arXiv 2505.24298), OpenRLHF v0.5+.

8.4　Tool-RL on TAU-bench / SWE-bench / OSWorld

Industry-benchmark SOTA trends (public numbers in 2025-2026):

Note: benchmark contamination risk is high; as of 2026Q1, OpenAI has retired SWE-bench Verified (reasons: contamination + test flaws); more credible benchmarks now are SWE-Lancer, SWE-bench Multilingual, private holdouts.

8.5　Anthropic Computer-Use (Claude 3.5 → 4.5 → Opus 4.x)

Public knowledge:

• 2024-10-22 Claude 3.5 Sonnet (new) first shipped Computer-Use beta
• 2025: Claude 3.7 / 4.0 / 4.5 continued iterating with speed + accuracy improvements
• Training involves many human demonstrations + AI-generated rollouts + RLHF + safety red-teaming
• Action space = screenshots + mouse + keyboard + filesystem access
• Public system cards mention Constitutional AI + Computer-Use specific safety filter

Algorithm-level inferred information (academic speculation, not officially confirmed):

• Training likely involves RLHF on screenshot trajectories
• Reward likely includes a task-completion grader (LLM-as-judge) + safety classifier
• Possibly GRPO / RLOO style critic-free RL (multiple public papers mention critic-free)

8.6　2025-2026 papers panorama

§9 Failure modes & engineering experience

9.1　The "seven deadly sins" of Agentic RL

9.2　Online vs offline RL trade-offs

Practical advice: start with offline RL (agent SFT + DPO) to establish a baseline; once compute is available, migrate to online RL (PPO/GRPO + verifier).

9.3　Rollout optimization tips

• vLLM PagedAttention is 5-10× faster than HF generate; a must-install for agent rollout
• Tool sandbox with Docker + gVisor; 64-128 trajectories per machine in parallel is fine
• Async rollout pipeline: rollout worker doesn't block the trainer worker
• Trajectory replay buffer: mix FIFO + priority; weight replay by reward
• Batch size = group_size × prompt_per_batch: typical $G = 16$, prompt = 32 → 512 trajectories / batch

9.4　Debug checklist

When agent RL crashes, troubleshoot in order:

1. Look at the reward: do most trajectories have reward = 0? Is the rule wrong?
2. Look at the length distribution: are most trajectories truncated at max_steps? Means the agent can't finish
3. Look at the action_mask: does it correctly cover prompt + observation + padding?
4. Look at KL: is approx_kl blowing up? Increase β; is entropy collapsing? Increase entropy_bonus
5. Look at group_std: are many groups all 0 / all 1? Need a data filter
6. Look at the tool-call distribution: over-reliance on a single tool? Add diversity bonus
7. Look at sample efficiency: is the reward variance over multiple rollouts per prompt reasonable? Too large → base is too weak

§10 25 frequently-asked interview questions (L1 must-know / L2 advanced / L3 top-lab)

Sorted into 3 tiers by difficulty: L1 = asked at any agent / LLM RL role; L2 = asked by research / alignment teams; L3 = hardcore questions for top labs. Each links to answer points + footguns.

L1 must-know (10 questions)

L2 advanced (10 questions)

L3 top-lab (5 questions)

§A Appendix: reference list

Grouped by direction; papers verified via web search + arXiv for authors / year / venue. A few 2025-2026 papers without a fixed conference are listed by arXiv.

Agent SFT / fundamentals

• Zeng et al. 2023 arXiv 2310.12823 AgentTuning: Enabling Generalized Agent Abilities for LLMs (THU)
• Chen et al. 2024 ACL Findings arXiv 2403.12881 Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for LLMs
• Trung et al. 2024 ACL arXiv 2401.08967 ReFT: Reasoning with Reinforced Fine-Tuning

RL for agent / reasoning (fundamental algorithms)

• Schulman et al. 2017 arXiv 1707.06347 Proximal Policy Optimization Algorithms
• Schulman et al. 2016 ICLR High-Dimensional Continuous Control Using GAE
• Shao et al. 2024 arXiv 2402.03300 DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO proposed)
• DeepSeek-AI 2025 arXiv 2501.12948 DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Yu et al. 2025 ByteDance arXiv 2503.14476 DAPO: An Open-Source LLM Reinforcement Learning System at Scale

Tool-augmented RL

• Qian et al. 2025 arXiv 2504.13958 ToolRL: Reward is All Tool Learning Needs (arXiv preprint; no formal venue as of 2026-05)
• Chen et al. 2025 arXiv 2503.19470 ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (accepted to NeurIPS 2025)
• Wang et al. 2025 arXiv 2504.20073 Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning (RAGEN; StarPO = State-Thinking-Actions-Reward Policy Optimization)

Web / GUI agent RL

• Qi et al. 2024 ICLR-25 arXiv 2411.02337 WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• Putta et al. 2024 arXiv 2408.07199 Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
• Furuta et al. 2024 ICLR arXiv 2305.11854 Multimodal Web Navigation with Instruction-Finetuned Foundation Models (WebGUM)

Code agent RL

• Le et al. 2022 NeurIPS arXiv 2207.01780 CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• Shojaee et al. 2023 arXiv 2301.13816 Execution-Based Code Generation Using Deep Reinforcement Learning (PPOCoder)
• Wei et al. 2025 Meta FAIR arXiv 2502.18449 SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution

Embodied / robot agent

• Baker et al. 2022 NeurIPS arXiv 2206.11795 Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
• Kim et al. 2024 arXiv 2406.09246 OpenVLA: An Open-Source Vision-Language-Action Model

Self-rewarding / exploration

• Yuan et al. 2024 ICML arXiv 2401.10020 Self-Rewarding Language Models
• Andrychowicz et al. 2017 NeurIPS arXiv 1707.01495 Hindsight Experience Replay

RLHF / DPO fundamentals (cross-reference)

• Ouyang et al. 2022 NeurIPS Training Language Models to Follow Instructions with Human Feedback
• Rafailov et al. 2023 NeurIPS Direct Preference Optimization
• Bai et al. 2022 Anthropic arXiv 2212.08073 Constitutional AI
• Lee et al. 2023 Google arXiv 2309.00267 RLAIF: Scaling RLHF with AI Feedback

Reward model / verification

• Lightman et al. 2024 ICLR arXiv 2305.20050 (OpenAI 2023) Let's Verify Step by Step (PRM800K)
• Wang et al. 2024 ACL arXiv 2312.08935 Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations
• Coste et al. 2024 ICLR Reward Model Ensembles Help Mitigate Overoptimization

Infrastructure / frameworks

• TRL (HuggingFace): https://github.com/huggingface/trl — standard PPO / DPO / GRPO trainer
• OpenRLHF: https://github.com/OpenRLHF/OpenRLHF — industrial-grade PPO / GRPO / RLOO implementation
• verl (ByteDance): https://github.com/volcengine/verl — mainstream framework for GRPO / DAPO / agent RL
• ReaLHF / AReaL (Ant Group + Tsinghua, async RL system, arXiv 2505.24298)

SOTA benchmarks (2024-2026)

• Jimenez et al. 2024 ICLR arXiv 2310.06770 SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Yao et al. 2024 arXiv 2406.12045 τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• Xie et al. 2024 NeurIPS arXiv 2404.07972 OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• Zhou et al. 2024 ICLR arXiv 2307.13854 WebArena: A Realistic Web Environment for Building Autonomous Agents
• Mialon et al. 2024 ICLR GAIA: A Benchmark for General AI Assistants
• Chan et al. 2024 arXiv 2410.07095 MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (OpenAI)

Anthropic Computer-Use (public knowledge)

• Claude 3.5 Sonnet (new) 2024-10-22 Computer Use beta launch (Anthropic blog + system card)
• Claude 3.7 / 4.0 / 4.5 / Opus 4.x system cards (Anthropic public)

Code framework recommendations:

• Get started with HF TRL's GRPOTrainer + your own verifier
• For industrial use, use verl (GRPO/RLOO/DAPO all supported, includes agent rollout)
• For custom multi-turn use OpenRLHF v0.5+ agent examples + add a tool sandbox