在线蒸馏论文工具库

Preference, Reward-Guided, and Hybrid RL+KD

• Direct Preference Knowledge Distillation (2024) — Preference-aware KD combining reverse-KL with implicit reward objectives.
• Online Knowledge Distillation with Reward Guidance (2025) — Sequential KD via preference optimization; offline and online variants.
• KDRL (2025) — Unified reverse-KL KD with RL in a single post-training objective.
• RLTF-SD: Expanding RL via Text Feedback (2026) — Internalizes text feedback via self-distillation.
• RLAD: Reinforcement-aware KD for LLM Reasoning (2026) — Trust-region ratio distillation on student rollouts.
• Multi-Token Prediction via Self-Distillation (2026) — Online self-distillation for multi-token prediction and faster inference.
• ORPO-Distill: Mixed-Policy Preference Optimization for Cross-Architecture LLM Distillation (2025) — Mixed-policy preference distillation with student-generated outputs; black-box cross-architecture transfer.
• SRPO: Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing (2026) — Routes correct student rollouts to reward-based RL and failed ones to self-distillation.
• KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation (2025) — K-step Bellman return replaces high-variance single-step REINFORCE in sequence-level OPD.
• Rethinking LLM Distillation: A Constrained MDP Perspective (2025) — Maximizes task reward under hard KL constraint against the teacher; avoids manual Lagrangian tuning.
• RLKD: Distilling LLMs' Reasoning via Reinforcement Learning (2025) — Generative Structure Reward Model on student rollouts; outperforms SFT-RL pipelines on 0.1% data.
• LUFFY: Learning to Reason under Off-Policy Guidance (2025) — Mixed-policy GRPO combining on-policy rollouts with off-policy teacher traces via regularized importance sampling.
• BOND: Aligning LLMs with Best-of-N Distillation (2024) — RL mimicking best-of-N via Jeffreys-divergence matching; eliminates inference-time BoN cost.
• Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignment (2024) — Game-theoretic iterative BoN as self-play; win-rate dominance optimization (AISTATS 2025).
• AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation (2025) — Casts RLHF as token-level distillation by injecting DPO rewards (ACL 2025).
• KEPO: Knowledge-Enhanced Preference Optimization for Reinforcement Learning with Reasoning (2026) — Quality-gated OPD on high-quality trajectories plus knowledge-enhanced exploration via teacher hints.
• 𝒳-KD: General Experiential Knowledge Distillation for Large Language Models (2026) — Jointly models teacher reward and policy-distills so the student learns inside the teacher's original environment.
• ExGRPO: Probing to Refine — Reinforcement Distillation of LLMs via Explanatory Inversion (2026) — Explanatory probes plus dialogue-structure utility bonus reward coherent reasoning over memorized answers.
• NPO: Near-Future Policy Optimization (2026) — Later checkpoint of same policy as teacher; AutoNPO adaptively triggers switch to maximize RLVR signal.
• CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization (2026) — Dual GRPO with on-policy KD reward between large/small models; matches standard GRPO with 18% speedup.

Domain Extensions

OPD applied to non-text-reasoning settings — agents, multimodal models, diffusion, audio, robotics — and to inference acceleration via speculative decoding. These pass the inclusion criterion (student rollouts central to the learning signal) but on substrates beyond LLM text reasoning.

Agent, Multimodal, and Other Extensions

• Structured Agent Distillation (2025) — Queries teacher online to avoid distribution drift in agent settings.
• From Deferral to Learning: Online In-Context KD for LLM Cascades (2025) — Teacher-student cascade with reusable online knowledge store.
• AllMem (2026) — Offline on-policy distillation for long-context modeling.
• Video-OPD (2026) — OPD for temporal video grounding in multimodal LLMs.
• Reinforced Attention Learning (2026) — On-policy attention distillation for multimodal models.
• SCoRe: From Correction to Mastery via Reinforced Distillation of LLM Agents (2025) — Teacher intervenes at first critical error in student agent trajectories for corrective distillation.
• VOLD: Reasoning Transfer from LLMs to Vision-Language Models via OPD (2025) — Text-only teacher distills reasoning into VLM via student-generated traces with combined GRPO and OPD.
• X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs (2026) — Student on-policy rollouts with token-level teacher feedback for cross-modal speech-LLM distillation.
• VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via OPD (2026) — Reverse-KL OPD bridging offline SFT and online RL for robotic manipulation.
• TCOD: Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents (2026) — Short-to-long trajectory-depth curriculum mitigating multi-turn KL instability.
• LLM4Teach: Large Language Model as a Policy Teacher for Training RL Agents (2023) — LLM teacher distills into small RL agent that surpasses teacher through environment interaction.
• RPD: Refined Policy Distillation — From VLA Generalists to RL Experts (2025) — Teacher VLA actions guide student during RL exploration; combines RL with behavioral cloning (IROS 2026).
• π-Flow: Policy-Based Few-Step Generation via Imitation Distillation (2025) — Imitation distillation aligns student flow-model trajectories with teacher under standard flow matching (ICLR 2026).
• Step-Audio-R1 Technical Report (2025) — Modality-Grounded Reasoning Distillation produces audio reasoning grounded in acoustic features.
• OPD-AVMP: On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning (2026) — Generalized OPD for LLM-based driving planners; 5× compression at near-teacher performance.
• CORD: Bridging the Audio–Text Reasoning Gap via Weighted On-policy Cross-modal Distillation (2026) — Audio-conditioned rollouts; text-conditioned same model as teacher; importance-weighted reverse KL plus GRPO.
• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (2026) — Plain-prompt student; skill-augmented same model as token-level self-teacher for multi-turn agent training.
• CoPD: Co-Evolving Policy Distillation (2026) — Parallel expert training with bidirectional OPD; experts co-evolve as mutual teachers during RLVR.
• PRISM: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL (2026) — Black-box OPD pre-alignment between SFT and RLVR for VLMs; MoE discriminator supplies adversarial signals.
• D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models (2026) — OPSD ported to few-step T2I diffusion; text-only student vs. text+image teacher with velocity-MSE on rollouts.
• Flow-OPD: On-Policy Distillation for Flow Matching Models (2026) — Per-domain Flow-GRPO experts supervise student SDE rollouts via reverse-KL with Manifold Anchor Regularization.
• VISD: Enhancing Video Reasoning via Structured Self-Distillation (2026) — Structured video judge feeds EMA teacher with privileged feedback; direction-magnitude decoupling stabilizes RL+supervision.
• TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM (2026) — Partitions masked positions by remaining decoding steps into near (CE) and distant (KL) subsets.
• DiMO: Distilling Masked Diffusion Models into One-step Generator (2025) — First OPD for masked discrete diffusion image generation; Generalized Jeffrey divergence with DMD-style auxiliary (ICCV 2025).
• SDAR: Self-Distilled Agentic Reinforcement Learning (2026) — Sigmoid-gated OPSD auxiliary on top of GRPO for multi-turn agents; amplifies positive-gap, attenuates negative-gap tokens (COLM 2026).
• GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation (2026) — Reverse-KL between student and ground-truth-conditioned same-model teacher gives token saliency; KL-initiated entropy-terminated segmentation propagates credit within segments to sign-aware reweight GRPO advantages.
• Revisiting DAgger in the Era of LLM-Agents (2026) — Turn-level (DAgger) and trajectory-prefix (AggreVaTe) student/teacher rollout mixtures with teacher actions queried at every visited state.
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models (2026) — Lifts OPD from autoregressive tokens to diffusion denoising via closed-form reverse-KL along student rollouts; unifies SDE and ODE samplers.
• Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning (2026) — Alternates GRPO with offline self-distillation on correct-fewest-search / divergent-sibling pairs mined from the converged rollout pool.
• Healthcare AI GYM for Medical Agents (2026) — Clinical-agent gymnasium plus Turn-level Truncated OPD: an EMA teacher conditioned on outcome-privileged hints that are stripped before logprob comparison.
• HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents (2026) — Parallel multimodal search agent that applies OPD only to failed rollouts to salvage correct intermediate tool calls from GRPO's uniform negative advantage.