---
title: "Agent Optimization Methods — A Field Inventory, Survey & Index"
description: "A comprehensive, indexed survey of the publicly known methods for optimizing LLM-based AI agents: 370 distinct techniques across 16 families spanning the whole stack — reasoning and prompting, test-time compute, self-improvement, decoding-time control, context and memory, retrieval, KV-cache reuse, tool use, planning, multi-agent orchestration, context-engineering patterns, routing and cost, serving-layer inference, training and adaptation, and reliability/evaluation-as-optimization. Each method gets a one-line definition, how it optimizes, what it improves, a representative reference, and a maturity tag, with cross-cutting views, a map of where the fak kernel's own work sits, and an alphabetical master index."
slug: agent-optimization-methods-survey
keywords:
  - agent optimization methods
  - LLM agent optimization
  - test-time compute scaling
  - context engineering
  - KV cache reuse
  - prompt caching
  - retrieval augmented generation
  - multi-agent orchestration
  - speculative decoding
  - agent memory
  - tool use optimization
  - inference optimization
  - chain of thought
  - model routing
date: 2026-06-23
---

# Agent Optimization Methods — A Field Inventory, Survey & Index

Date: 2026-06-23

> **What this is.** A field-wide inventory of *publicly known* techniques for making LLM-based
> agents better, faster, cheaper, or more reliable — from prompt-level reasoning tricks down to
> the serving kernels they ride on. It is built as three things at once: an **index** (a
> 16-family taxonomy you can jump from), a **survey** (each method gets a definition, a
> mechanism, what it optimizes, a representative reference, and a maturity tag), and an
> **inventory** (per-family tables plus one alphabetical master index you can grep). It
> deliberately spans layers most single surveys keep apart: a context-pruning heuristic and a
> PagedAttention kernel are both "agent optimizations" — they just optimize different terms of
> the same cost.
>
> **What "agent optimization" means here.** An agent turn is not `prompt in → tokens out`; it is
> `memory view → model proposal → tool/effect check → world read/write → result admission →
> memory update → cache/invalidation → audit`. Every stage has its own optimization literature.
> Methods are grouped by *what they act on* and tagged by *what they improve*: accuracy/quality,
> latency, cost/tokens, memory, throughput, reliability, autonomy, or context-capacity.
>
> **Honesty line — read before citing.** References are *representative canonical pointers*
> ("FirstAuthor et al., YEAR"), not a verified bibliography — treat them as search anchors, not
> as a claim that a specific identifier is correct. Method *names* are the durable, checkable
> part; confirm exact identifiers at the source before relying on them. Inclusion is not
> endorsement, and maturity tags are judgment calls as of mid-2026.

**How this was built.** The inventory was assembled by a structured fan-out — one researcher per
family enumerating named methods, then a completeness-critic pass over the union to surface what
was missing — then deduplicated (453 raw entries → 370 distinct methods) and re-homed into the
taxonomy below. It is a *knowledge* survey: the methods and what they do are drawn from the public
literature and practice through early 2026, not from new measurement. No number in this document
is a fak benchmark; the families fak has actually measured link out to their own witnessed result
docs.

## How this maps to fak

This survey is the umbrella; the repo already has depth on several families. The fak thesis is
*not* "we invented these" — a 29-claim prior-art audit scored 0/29 novel. The contribution is the
*assembly* into one in-process gate where the tool call is the checkpoint, and where cache reuse
becomes a kernel verdict (identity · state · freshness · authority · integrity · causality ·
economics) rather than a boolean. The field-to-repo map:

| Family (this survey) | fak's deeper treatment |
|---|---|
| Inference / serving optimizations | [SOTA serving optimizations](../explainers/sota-optimizations.md) — the 10 already-shipped serving optimizations that define the honest baseline ("tuned SOTA"), each with fak's status. |
| KV cache & prefix reuse | [Agentic caching SOTA](AGENTIC-CACHING-SOTA-2026-06-19.md) — nine named cache layers mapped to SOTA + fak; [Addressable KV cache](../explainers/addressable-kv-cache.md) — bit-exact mid-run span eviction (`max\|Δ\|=0`); [KV cache as agentic context grows](../explainers/kv-cache-agentic-context.md). |
| Context-window management | [O(1) context window economics](../explainers/o1-context-window-economics.md); [ctxwin baseline](CTXWIN-CONTEXT-WINDOW-BASELINE-2026-06-23.md); [the O(1) turn context planner](O1-TURN-CONTEXT-PLANNER-2026-06-23.md) (`internal/ctxplan`). |
| Multi-agent / orchestration | [Scaling laws of agents](SCALING-LAWS-OF-AGENTS-2026-06-19.md) — `work ≈ agents × turns × working-set × reread-rate × legality`; [ultra-long-context levels & levers](RESEARCH-ultra-long-context-levels-and-naming-2026-06-22.md). |
| Tool use & function calling | [Grammar-constrained tool-call decoding](RESEARCH-grammar-constrained-tool-call-decoding-2026-06-22.md); the `internal/vdso` tool-result fast path. |
| Reliability / eval-as-optimization | [Policy in the kernel](../explainers/policy-in-the-kernel.md) (default-deny floor); trajectory-replay / offline policy evaluation (`internal/turnbench`). |
| Training (quantization) | [AWQ quantization](../explainers/awq-quantization.md). |

The honest read: fak owns the **governance band** (which effects are allowed, which results may
enter context, when reuse is still *legal*) plus a few **reuse/coherence** levers on top of an
engine it fronts. It does not try to beat vLLM/SGLang/llama.cpp at raw throughput. Most rows in
this survey are levers fak *sits on top of*, not levers it claims.

---

## The index — 16 families

**370 distinct methods**, grouped into 16 families across 5 parts. Counts are post-deduplication (a method that spans families is filed under its primary home and cross-noted in its row).

**Part I — Reasoning & inference-time problem solving**
- [1. Prompting & reasoning strategies (20)](#1-prompting--reasoning-strategies)
- [2. Test-time compute scaling & search over reasoning (23)](#2-test-time-compute-scaling--search-over-reasoning)
- [3. Self-improvement & feedback loops at inference (19)](#3-self-improvement--feedback-loops-at-inference)
- [4. Decoding-time control (5)](#4-decoding-time-control)

**Part II — Context, memory & knowledge**
- [5. Context-window management & compression (26)](#5-context-window-management--compression)
- [6. Agent memory architectures (23)](#6-agent-memory-architectures)
- [7. Retrieval & knowledge augmentation (31)](#7-retrieval--knowledge-augmentation)
- [8. KV cache & prefix reuse (25)](#8-kv-cache--prefix-reuse)

**Part III — Action, planning & orchestration**
- [9. Tool use & function calling optimization (26)](#9-tool-use--function-calling-optimization)
- [10. Planning algorithms & world models for agents (26)](#10-planning-algorithms--world-models-for-agents)
- [11. Multi-agent orchestration patterns (27)](#11-multi-agent-orchestration-patterns)
- [12. Context engineering & agent-design patterns (5)](#12-context-engineering--agent-design-patterns)

**Part IV — Cost, serving & training**
- [13. Model routing, cascades & cost optimization (22)](#13-model-routing-cascades--cost-optimization)
- [14. Inference / serving optimizations beneath agents (31)](#14-inference--serving-optimizations-beneath-agents)
- [15. Training & adaptation for agentic capability (34)](#15-training--adaptation-for-agentic-capability)

**Part V — Reliability & evaluation**
- [16. Reliability, structured generation & evaluation-as-optimization (27)](#16-reliability-structured-generation--evaluation-as-optimization)

## Cross-cutting views

**By what they optimize** (a method may count under several):

| Objective | # methods |
|---|--:|
| accuracy/quality | 266 |
| reliability | 180 |
| cost/tokens | 150 |
| latency | 103 |
| context-capacity | 72 |
| throughput | 69 |
| autonomy | 62 |
| memory | 59 |

**By maturity:**

| Maturity | # methods |
|---|--:|
| production-standard | 137 |
| emerging | 117 |
| research | 116 |

**By where they apply:**

| Applies to | # methods |
|---|--:|
| single-agent | 128 |
| any | 92 |
| serving | 76 |
| training | 41 |
| multi-agent | 33 |

---

# Part I — Reasoning & inference-time problem solving

## 1. Prompting & reasoning strategies

The oldest and cheapest lever: change what you ask, not the model. These methods restructure the *prompt* to elicit better reasoning — surfacing intermediate steps, decomposing the problem, abstracting before answering, or offloading computation to code. They cost only tokens and work on any model, which is why they stay the first thing to try.

*20 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Algorithm of Thoughts** <br/><sub>AoT</sub> | Guides the model through algorithmic, in-context exploration (mimicking search like DFS/BFS) within a single generation rather than many external queries. *Embedding algorithmic exploration examples in the prompt lets the model internalize search-and-backtrack behavior with far fewer model calls than tree-search frameworks.* | accuracy/quality, cost/tokens | research | Sel et al., 2023, 'Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models' |
| **Analogical prompting** <br/><sub>analogical reasoning prompting, self-generated exemplars</sub> | Prompting the model to self-generate relevant exemplars or related solved problems before tackling the target problem. *Generating tailored, problem-specific analogues replaces fixed few-shot examples with on-the-fly relevant demonstrations, improving accuracy without manual labeling.* | accuracy/quality, cost/tokens | emerging | Yasunaga et al., 2023, 'Large Language Models as Analogical Reasoners' |
| **Automatic Chain-of-Thought** <br/><sub>Auto-CoT</sub> | Automatically constructing CoT demonstrations by clustering questions and generating reasoning chains for representative samples, removing manual exemplar authoring. *Diversity-based sampling plus zero-shot-CoT-generated rationales builds a varied demonstration set, avoiding the error-propagation of similar exemplars.* | accuracy/quality, cost/tokens | research | Zhang et al., 2022, 'Automatic Chain of Thought Prompting in Large Language Models' |
| **Chain-of-Thought prompting** <br/><sub>CoT</sub> | Prompting the model to produce intermediate reasoning steps before the final answer, typically via few-shot exemplars that show step-by-step worked solutions. *Eliciting an explicit token-level reasoning trace gives the model more serial compute and a scaffold to decompose multi-step problems, raising answer accuracy on reasoning tasks.* <br/><sub>↔ also: Test-time compute scaling & search over reasoning</sub> | accuracy/quality, reliability | production-standard | Wei et al., 2022, 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models' |
| **Chain-of-Thought without prompting (CoT-decoding)** | Eliciting reasoning paths by altering the decoding (inspecting top-k first-token branches) rather than by any prompt, since CoT paths already exist in the distribution. *Branch over alternative first tokens and select the path whose answer tokens carry the highest confidence margin; surfaces latent reasoning without a 'think step by step' instruction.* | accuracy/quality | research | Wang & Zhou, 2024 (Chain-of-Thought Reasoning without Prompting) |
| **Complexity-based prompting** <br/><sub>complex CoT</sub> | Selects few-shot CoT exemplars with more reasoning steps (higher complexity) and votes among high-complexity sampled chains at decoding time. *Conditioning on and voting over longer reasoning chains biases the model toward the more thorough reasoning that correlates with correct answers on hard tasks.* | accuracy/quality | research | Fu et al., 2022, 'Complexity-Based Prompting for Multi-Step Reasoning' |
| **Emotion Prompting (EmotionPrompt)** | Appending emotional stimulus phrases (e.g. 'This is very important to my career') to a prompt to elicit higher-quality model responses. *Emotional / psychological-stimulus sentences appended to the instruction shift the model toward more careful, complete generations, measurably improving zero-shot performance.* | accuracy/quality | research | Li et al., 2023 (Large Language Models Understand and Can Be Enhanced by Emotional Stimuli) |
| **Faithful Chain-of-Thought** <br/><sub>Faithful CoT</sub> | Translates a query into an interleaved natural-language and symbolic (code/logic) reasoning chain whose final answer is deterministically computed from the symbolic part. *Binding the answer to a deterministic solver over the symbolic chain guarantees the stated reasoning actually produced the answer, improving faithfulness and accuracy.* | accuracy/quality, reliability | research | Lyu et al., 2023, 'Faithful Chain-of-Thought Reasoning' |
| **Few-shot prompting** <br/><sub>in-context learning, ICL</sub> | Providing several input-output exemplars in the prompt so the model infers the task format and pattern from demonstrations. *Demonstrations condition the model's next-token distribution toward the demonstrated mapping, improving format adherence and task accuracy without weight updates.* | accuracy/quality, reliability | production-standard | Brown et al., 2020, 'Language Models are Few-Shot Learners' (GPT-3) |
| **Maieutic prompting** | Generates a tree of abductive explanations (recursively justified propositions and their negations) and resolves contradictions to infer a logically consistent answer. *Building a maieutic tree of self-justifying statements and using a satisfiability solver over their logical relations yields answers robust to inconsistent model beliefs.* | accuracy/quality, reliability | research | Jung et al., 2022, 'Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations' |
| **Program-Aided Language models** <br/><sub>PAL</sub> | An approach where the model reads a problem and emits intermediate reasoning as code that an interpreter runs to produce the answer. *The LLM handles natural-language understanding and produces program steps while the interpreter guarantees correct execution of the computation.* | accuracy/quality, reliability | emerging | Gao et al., 2022, 'PAL: Program-aided Language Models' |
| **Program-of-Thoughts** <br/><sub>PoT</sub> | Expressing the reasoning steps as an executable program (e.g., Python) and delegating the computation to an interpreter rather than to the language model. *Offloading arithmetic and logic to a code interpreter separates reasoning from computation, eliminating the model's calculation errors.* | accuracy/quality, reliability | emerging | Chen et al., 2022, 'Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks' |
| **Rephrase and Respond** <br/><sub>RaR</sub> | Prompts the model to first rephrase and expand the user's question, then answer the clarified version, optionally in one pass. *Self-rephrasing resolves ambiguity and aligns the question with the model's understanding before answering, reducing misinterpretation errors.* | accuracy/quality, reliability | research | Deng et al., 2023, 'Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves' |
| **Self-Ask** | A prompting pattern where the model explicitly poses and answers follow-up sub-questions before composing the final answer, often invoking a search tool per sub-question. *Decomposing into explicit follow-up questions makes the compositional structure visible and lets each sub-question be answered (or retrieved) independently, narrowing the compositionality gap.* | accuracy/quality, reliability | emerging | Press et al., 2022, 'Measuring and Narrowing the Compositionality Gap in Language Models' |
| **Step-Back prompting** <br/><sub>abstraction prompting</sub> | Prompting the model to first derive a high-level concept, principle, or abstraction from the question before reasoning toward the specific answer. *Abstracting to governing principles grounds subsequent reasoning, reducing errors from getting lost in low-level details.* | accuracy/quality, reliability | emerging | Zheng et al., 2023, 'Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models' |
| **System 2 Attention (S2A)** | A two-pass prompting method where the model first regenerates the context to remove irrelevant/biasing content, then answers over the cleaned context. *An explicit context-regeneration step filters distractors and opinion before the answering pass, reducing sycophancy and irrelevant-context errors.* | accuracy/quality, reliability | research | Weston & Sukhbaatar, 2023 (System 2 Attention) |
| **Tab-CoT / Structured (tabular) reasoning** <br/><sub>Tabular Chain-of-Thought, Tab-CoT</sub> | Elicits reasoning in a structured table format with explicit columns (step, subquestion, operation, result) rather than free-form prose. *Imposing a tabular schema on the reasoning trace makes intermediate steps explicit and self-aligned across rows and columns, improving structured task accuracy.* | accuracy/quality, reliability | research | Jin & Lu, 2023, 'Tab-CoT: Zero-shot Tabular Chain of Thought' |
| **Take a Deep Breath / optimized meta-prompts (OPRO-discovered instructions)** | Using LLM-discovered instruction phrases (e.g. 'Take a deep breath and work on this problem step by step') that empirically outperform hand-written CoT triggers. *An optimizer LLM searches the instruction space; the best-found natural-language instruction is reused as a zero-shot trigger, lifting reasoning accuracy.* | accuracy/quality | research | Yang et al., 2023 (Large Language Models as Optimizers / OPRO) |
| **Thread of Thought (ThoT)** | A prompting strategy for chaotic / long contexts that walks the context piece-by-piece, summarizing and carrying a running analysis before answering. *An instruction to 'analyze this part by part, summarizing as we go' segments long, disordered context into manageable steps, improving retrieval-style QA.* | accuracy/quality, context-capacity | research | Zhou et al., 2023 (Thread of Thought Unraveling Chaotic Contexts) |
| **Zero-shot Chain-of-Thought** <br/><sub>Zero-shot CoT, Let's think step by step</sub> | Eliciting step-by-step reasoning without exemplars by appending a trigger phrase such as 'Let's think step by step' to the prompt. *A single instruction phrase induces the model to generate a reasoning trace before answering, capturing most CoT gains with no hand-crafted examples.* | accuracy/quality, cost/tokens | production-standard | Kojima et al., 2022, 'Large Language Models are Zero-Shot Reasoners' |

## 2. Test-time compute scaling & search over reasoning

Spend more inference compute for a better answer: sample many times, search over reasoning states, or let a verifier pick the winner. A fixed model gets smarter when it thinks longer or wider — the regime behind o1/R1-style reasoning models. The trade is latency and tokens for accuracy.

*23 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Adaptive / early-stopping sampling** <br/><sub>adaptive consistency, early-stopping self-consistency, dynamic sample allocation</sub> | Dynamically decides how many samples to draw per question, stopping early once the answer distribution is confidently converged to save compute. *Monitors agreement among accumulated samples with a stopping criterion (e.g. Bayesian/beta confidence) and halts sampling when further samples are unlikely to change the vote.* | cost/tokens, latency, accuracy/quality | emerging | Aggarwal et al., 2023 (Let's Sample Step by Step: Adaptive-Consistency) |
| **Best-of-N sampling** <br/><sub>BoN, rejection sampling at inference, repeated sampling</sub> | Generate N independent candidate outputs and select the single best one according to a scorer (reward model, verifier, or self-evaluation). *Trades inference compute for quality by sampling N completions and returning argmax under an external selector rather than relying on one decode.* <br/><sub>↔ also: Reliability, structured generation & evaluation-as-optimization</sub> | accuracy/quality, reliability | production-standard | Stiennon et al., 2020; Nakano et al., 2021 (WebGPT); Cobbe et al., 2021 (verifier-reranked BoN on GSM8K) |
| **Budget forcing / long-thinking (s1-style)** <br/><sub>s1, test-time budget forcing, wait-token forcing</sub> | Explicitly controlling the length of the model's reasoning by forcing it to continue thinking (e.g. appending 'Wait') or to stop, to spend a target compute budget. *Manipulates the decode trajectory by suppressing the end-of-thinking token and injecting continuation cues so the model extends or truncates its chain to a budget.* | accuracy/quality, cost/tokens | emerging | Muennighoff et al., 2025 (s1: Simple Test-Time Scaling) |
| **Compute-optimal test-time scaling** <br/><sub>test-time compute-optimal allocation, FLOPs-matched test-time scaling</sub> | Choosing per-prompt how to spend a fixed inference budget (which strategy, how many samples, how deep a search) to maximize accuracy, framed as compute-optimal allocation. *Adapts strategy and sample/search depth to problem difficulty (often via a verifier/difficulty estimate), shifting between sequential refinement and parallel search.* | accuracy/quality, cost/tokens | emerging | Snell et al., 2024 (Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters) |
| **Forest-of-Thoughts / ensemble-of-trees** <br/><sub>Forest of Thought, FoT, boosting-style tree ensembling</sub> | Runs multiple reasoning trees in parallel and aggregates across them (with consensus/voting and dynamic self-correction) for sparse-activation collective reasoning. *Ensembles several independent ToT-style trees and combines their leaf answers with consensus-guided decision and selective re-exploration of weak trees.* | accuracy/quality, reliability | research | Bi et al., 2024 (Forest-of-Thought: Scaling Test-Time Compute for Enhanced LLM Reasoning) |
| **Graph of Thoughts** <br/><sub>GoT</sub> | Generalizes tree-structured reasoning to an arbitrary graph where thoughts can be aggregated, refined, and looped, with self-scoring at vertices. *Models reasoning as a graph whose transformations (aggregate/refine/generate) and volume/latency-aware scheduling let the model merge and re-improve thoughts beyond a strict tree.* <br/><sub>↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference; Planning algorithms & world models for agents</sub> | accuracy/quality | research | Besta et al., 2024, 'Graph of Thoughts: Solving Elaborate Problems with Large Language Models' |
| **Least-to-Most / Decomposed Prompting** <br/><sub>DecomP, decomposed prompting, Least-to-Most prompting</sub> | Decomposes a complex task into a sequence of sub-tasks, each handled by a dedicated prompt or sub-handler that can itself be further decomposed or call tools. *A modular, recursive decomposer routes sub-tasks to specialized prompt 'modules', enabling reuse, hierarchical decomposition, and tool delegation.* <br/><sub>↔ also: Prompting & reasoning strategies</sub> | accuracy/quality, reliability, autonomy | research | Khot et al., 2022, 'Decomposed Prompting: A Modular Approach for Solving Complex Tasks' |
| **Long chain-of-thought RL reasoning (o1/R1-style)** <br/><sub>inference-time scaling via RL, long-CoT, test-time thinking models</sub> | Models trained (typically via RL) to produce very long internal chains of thought at inference, scaling accuracy with the number of thinking tokens spent. *RL on reasoning traces teaches the model to allocate many sequential 'thinking' tokens to self-correct and explore before answering, monotonically improving with test-time budget.* | accuracy/quality, reliability, autonomy | production-standard | OpenAI o1 system card, 2024; DeepSeek-AI, 2025 (DeepSeek-R1: Incentivizing Reasoning Capability via RL) |
| **Lookahead / rollout-guided decoding** <br/><sub>lookahead search, rollout scoring</sub> | At each step, simulate one or more short continuations (lookahead rollouts) and use their estimated value to choose the current step. *Performs limited forward simulation per candidate step and backs up a value estimate to bias the immediate choice toward promising branches.* | accuracy/quality | research | Snell et al., 2024 (lookahead variant); Zhang et al., 2023 (Planning with lookahead) |
| **Outcome Reward Model reranking** <br/><sub>ORM, outcome-supervised verifier</sub> | A model trained to predict whether a full solution is ultimately correct, used to rerank or filter candidate completions. *Scores each complete candidate with a learned correctness predictor and selects/weights by that final-answer reward.* | accuracy/quality, reliability | production-standard | Cobbe et al., 2021 (Training Verifiers to Solve Math Word Problems); Uesato et al., 2022 |
| **Quiet-STaR** <br/><sub>token-level rationale generation</sub> | Teaches a model to generate implicit rationales at every token to predict future text, generalizing STaR beyond QA to free-form reasoning. *Inserts learned 'thought' tokens between text tokens and rewards thoughts that improve next-token prediction, trained with a REINFORCE-style objective.* | accuracy/quality | research | Zelikman et al., 2024 (Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking) |
| **Repeated sampling / coverage scaling** <br/><sub>large language monkeys, sampling-based scaling, pass@k scaling</sub> | Drawing a very large number of samples per problem so that at least one correct solution appears, exploiting that coverage (pass@k) rises log-linearly with sample count. *Amplifies the chance of hitting a correct answer by sampling hundreds to thousands of attempts, paired with a verifier or oracle to pick winners.* | accuracy/quality, cost/tokens | emerging | Brown et al., 2024 (Large Language Monkeys: Scaling Inference Compute with Repeated Sampling) |
| **Self-consistency confidence (logit/entropy-weighted) aggregation** <br/><sub>confidence-informed self-consistency, CISC, entropy-weighted voting</sub> | Weights sampled answers by the model's internal confidence (token probabilities / answer entropy) so that fewer, higher-confidence samples suffice. *Derives a confidence score per sample from logits/answer entropy and performs a confidence-weighted vote, improving sample efficiency over uniform voting.* | accuracy/quality, cost/tokens | research | Taubenfeld et al., 2025 (Confidence Improves Self-Consistency in LLMs / CISC) |
| **Self-Consistency early termination by entropy / agreement (ESC, Adaptive-Consistency adaptive sampling)** | Stopping self-consistency sampling early once a confidence/agreement criterion is met instead of drawing a fixed N samples. *After each batch, a Dirichlet/agreement stopping rule estimates whether more samples could change the majority vote; sampling halts when the answer is locked in.* | cost/tokens, latency | emerging | Aggarwal et al., 2023 (Adaptive Consistency); Li et al., 2024 (Early-Stopping Self-Consistency) |
| **Self-consistency over tool-augmented / program reasoning** <br/><sub>PAL voting, PoT self-consistency, code-execution majority voting</sub> | Applies repeated sampling and majority voting to program-aided / code-generating reasoning, where each sample is an executable program whose run yields the answer. *Samples multiple programs, executes them to obtain answers, and votes/verifies over execution results to offload arithmetic and reduce reasoning errors.* | accuracy/quality, reliability | emerging | Gao et al., 2022 (PAL: Program-Aided Language Models); Chen et al., 2022 (Program of Thoughts) |
| **Self-Verification / self-evaluation selection** <br/><sub>LLM self-grading, self-evaluation guided decoding</sub> | Using the model itself (rather than a separate trained verifier) to evaluate or rank its candidate solutions for selection or pruning. *Prompts the model to check/score its own candidate answers (forward-backward verification or stepwise self-eval) and selects the best-scored one.* | accuracy/quality, reliability | emerging | Weng et al., 2022 (Large Language Models are Better Reasoners with Self-Verification); Xie et al., 2023 (Self-Evaluation Guided Beam Search) |
| **Skeleton-of-Thought** <br/><sub>SoT</sub> | The model first generates a concise skeleton (outline of points) for the answer, then expands each point in parallel. *Producing the outline first exposes independent sub-answers that can be decoded concurrently, cutting end-to-end latency while preserving structure.* <br/><sub>↔ also: Prompting & reasoning strategies; Planning algorithms & world models for agents</sub> | latency, accuracy/quality, throughput | research | Ning et al., 2023, 'Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding' |
| **Speculative reasoning / draft-then-verify reasoning skeletons (Skeleton-of-Thought parallel decode)** | Generating a reasoning/answer skeleton first and then expanding the points in parallel to cut end-to-end latency of long generations. *A short skeleton outline is produced, then each bullet is expanded concurrently (batched), reducing sequential decode depth for the final long answer.* | latency, throughput | research | Ning et al., 2023 (Skeleton-of-Thought) — parallel-expansion variant |
| **Step-level Beam Search over reasoning** <br/><sub>reasoning beam search, PRM beam search</sub> | Beam search applied at the granularity of reasoning steps, keeping the top-k partial solutions ranked by a process verifier rather than token log-prob. *Expands each beam by sampling candidate next steps, scores them with a PRM/value model, and retains the top-k partial trajectories.* | accuracy/quality, cost/tokens | emerging | Snell et al., 2024 (Scaling LLM Test-Time Compute Optimally); Yu et al., 2023 (OVM) |
| **Tree of Thoughts** <br/><sub>ToT</sub> | A deliberate problem-solving framework that explores a tree of intermediate 'thoughts', evaluating and backtracking among partial solutions via BFS or DFS search. *The LLM generates multiple thought branches at each step and self-evaluates their promise, enabling lookahead and backtracking instead of a single left-to-right chain.* <br/><sub>↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference; Planning algorithms & world models for agents</sub> | accuracy/quality, reliability | research | Yao et al., 2023, 'Tree of Thoughts: Deliberate Problem Solving with Large Language Models' |
| **Universal Self-Consistency** <br/><sub>USC, LLM-as-aggregator voting, LLM-as-aggregator self-consistency</sub> | An extension of self-consistency to free-form/open-ended generations where exact-match voting is impossible, by having the LLM itself select the most consistent answer among samples. *Instead of string-level majority vote, the model is prompted to read all sampled responses and pick the one most consistent with the rest, generalizing voting to non-extractable outputs.* <br/><sub>↔ also: Self-improvement & feedback loops at inference; Reliability, structured generation & evaluation-as-optimization</sub> | accuracy/quality, reliability | emerging | Chen et al., 2023, 'Universal Self-Consistency for Large Language Model Generation' |
| **Verifier-guided / reward-guided decoding** <br/><sub>value-guided decoding, reward-guided search, VAS / value-augmented sampling</sub> | Steering token- or step-level generation using a value/reward signal so that high-reward continuations are preferred during decoding. *Reweights or prunes candidate continuations at each decoding/search step by a learned value or process-reward estimate.* | accuracy/quality, reliability | emerging | Yang & Klein, 2021 (FUDGE); Mudgal et al., 2023 (Controlled Decoding); Liu et al., 2024 (value-guided / VAS) |
| **Weighted Self-Consistency** <br/><sub>weighted majority voting, verifier-weighted voting, confidence-weighted voting</sub> | A variant of self-consistency where votes from sampled paths are weighted by a score (verifier/PRM score, model confidence, or path probability) instead of counted equally. *Aggregates answers as an argmax over the sum of per-path weights, so high-quality or high-confidence paths dominate the vote.* | accuracy/quality, reliability | emerging | Li et al., 2022 (On the Advance of Making Language Models Better Reasoners / 'DiVeRSe'); Uesato et al., 2022 |

## 3. Self-improvement & feedback loops at inference

Let the model critique and revise its own output within a session, with no weight updates. A generator proposes, an evaluator (the same model, a tool, or a peer) finds faults, and the loop refines. Strong when an external signal — tests, execution, a verifier — grounds the critique; weak when the model grades itself unaided.

*19 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Best-of-N with reward/verifier model** <br/><sub>BoN sampling, rejection sampling, verifier reranking</sub> | Generate N candidate outputs and select the highest-scoring one according to a reward model or verifier (which may be the model itself). *A scoring model ranks the N samples and the top-scored candidate is returned, converting sampling diversity plus an evaluation signal into a quality gain at the cost of extra inference.* | accuracy/quality, reliability | production-standard | Cobbe et al., 2021 (verifiers for GSM8K); Nakano et al., 2021 (WebGPT BoN); widely used since |
| **CRITIC (tool-augmented self-correction)** <br/><sub>CRITIC</sub> | A framework where the LLM verifies and corrects its outputs by interacting with external tools (search engines, code interpreters) rather than relying solely on intrinsic judgment. *The model generates an output, queries external tools to get grounded critique/feedback, and revises iteratively, compensating for the unreliability of pure self-critique.* | accuracy/quality, reliability | emerging | Gou et al., 2024, 'CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing' |
| **Iterative refinement / generate-evaluate-refine loop** <br/><sub>draft-and-revise, refinement loop</sub> | The general inference-time pattern of repeatedly generating, evaluating against a criterion, and revising an output until a quality threshold or iteration budget is reached. *A scoring or critique function gates each revision, trading extra inference compute for higher output quality via a closed loop.* | accuracy/quality, reliability | production-standard | Madaan et al., 2023 (Self-Refine) and Welleck et al., 2023, 'Generating Sequences by Learning to Self-Correct' (Self-Correction / corrector model) |
| **LLM-as-judge debate / multi-agent evaluation** <br/><sub>debate-based evaluation, peer-review among agents, ChatEval</sub> | Multiple LLM judges discuss/debate to reach an evaluation verdict, improving the reliability of automated assessment over a single judge. *Judge agents exchange and contest their assessments across rounds, reducing single-judge bias and yielding a more human-aligned evaluation signal that can feed back into selection.* | accuracy/quality, reliability | emerging | Chan et al., 2023, 'ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate' |
| **Process Reward Model guided refinement** <br/><sub>PRM-guided search, step-level verification</sub> | A verifier that scores intermediate reasoning steps (not just final answers) is used to guide search, prune, or refine the reasoning trajectory at inference. *Step-level reward signals steer tree/beam search or trigger re-reasoning at the first low-scoring step, providing denser feedback than outcome-only verification.* | accuracy/quality, reliability | emerging | Lightman et al., 2023, 'Let's Verify Step by Step' (OpenAI PRM) |
| **Recursive Criticism and Improvement** <br/><sub>RCI</sub> | A prompting scheme where the model first generates an output, then is prompted to criticize it, then to improve it based on the criticism, recursively. *Explicit critique-then-improve prompt steps are chained recursively so each round conditions on the prior round's self-identified faults, improving grounded computer-control and reasoning tasks.* | accuracy/quality, autonomy, reliability | emerging | Kim et al., 2023, 'Language Models can Solve Computer Tasks' (RCI prompting) |
| **Reflection / verifier-in-the-loop agent reflection** <br/><sub>actor-critic LLM loop, generator-discriminator loop</sub> | A two-role pattern where a generator agent produces and a separate critic/reflector agent evaluates and returns actionable feedback for revision, looping until acceptance. *Decoupling generation from critique into distinct prompts/agents yields more independent error detection than single-pass self-critique, with the critic's feedback driving each revision.* | accuracy/quality, reliability, autonomy | production-standard | AutoGen reflection pattern (Wu et al., 2023) and LangGraph reflection agents; descends from Shinn et al., 2023 |
| **Reflective / metacognitive prompting** <br/><sub>metacognitive prompting, step-back-and-reflect</sub> | Prompting the model to reason about its own reasoning process (assess understanding, plan, monitor, evaluate) before and after answering. *Explicit metacognitive stages make the model surface assumptions and self-monitor, catching errors that a direct answer would miss.* | accuracy/quality, reliability | research | Wang & Zhao, 2024, 'Metacognitive Prompting Improves Understanding in Large Language Models' |
| **Reflexion (verbal self-reflection memory)** <br/><sub>verbal reinforcement learning, self-reflection, verbal reinforcement / self-reflection</sub> | An agent that maintains an episodic memory of self-generated verbal reflections on prior failed attempts, using them as feedback to improve on subsequent trials of the same task. *After a failed trajectory, the agent verbalizes what went wrong and stores that reflection; the reflection is prepended on the next attempt, acting as gradient-free reinforcement through language.* <br/><sub>↔ also: Prompting & reasoning strategies; Test-time compute scaling & search over reasoning; Agent memory architectures; Multi-agent orchestration patterns; Planning algorithms & world models for agents; Reliability, structured generation & evaluation-as-optimization</sub> | accuracy/quality, reliability, autonomy | emerging | Shinn et al., 'Reflexion: Language Agents with Verbal Reinforcement Learning', 2023 |
| **Reflexion with self-generated unit tests** <br/><sub>test-time self-test generation, agentic test-then-fix</sub> | An agent writes its own tests/checks for a task, runs them, and reflects on failures to drive correction, common in coding agents. *Self-authored executable checks provide concrete, grounded failure signals that are reflected on and used to revise the solution, sharpening the otherwise-vague self-critique.* | accuracy/quality, reliability, autonomy | emerging | Common in SWE-agent / coding-agent literature; descends from Shinn et al., 2023 and Chen et al., 2024 (Self-Debugging) |
| **Self-Consistency as feedback / verifier-guided refinement** <br/><sub>agreement-as-reward, consistency feedback</sub> | Using the (dis)agreement among multiple self-generated samples as an explicit feedback signal to trigger or guide a refinement step rather than just voting once. *Inter-sample inconsistency is detected and surfaced to the model as a critique ('your answers disagree on X'), prompting targeted re-reasoning on the contested portion.* | accuracy/quality, reliability | research | Derived from Wang et al., 2023 (self-consistency) and Madaan et al., 2023 (Self-Refine); a recognized composite pattern |
| **Self-Contrast (multi-perspective divergent self-checking)** | Self-correction that first generates diverse solving perspectives, contrasts their discrepancies, and uses the differences as a checklist for revision. *Explore-then-contrast: multiple solver views are diffed to surface inconsistencies that a single self-critique pass misses, then resolved into a revised answer.* | accuracy/quality, reliability | research | Zhang et al., 2024 (Self-Contrast) |
| **Self-Correction with learned corrector (Self-Correct)** <br/><sub>Self-Correct, learned corrector model</sub> | Pairs a fixed generator with a separately trained corrector module that iteratively edits the generator's output toward higher reward at inference. *A corrector is trained on (hypothesis, improved-hypothesis) value-improving pairs and applied repeatedly at decode time to refine outputs.* | accuracy/quality, reliability | research | Welleck et al., 2023, 'Generating Sequences by Learning to Self-Correct' |
| **Self-critique / self-correction** <br/><sub>self-critiquing, intrinsic self-correction</sub> | The general technique of prompting a model to critique its own answer and then produce a corrected version, without external tools or ground truth. *A critique prompt elicits the model's identification of errors, and a revision prompt conditions a new answer on that critique within the same context.* | accuracy/quality, reliability | production-standard | Saunders et al., 2022, 'Self-critiquing models for assisting human evaluators'; see also Huang et al., 2024 on the limits of intrinsic self-correction |
| **Self-Debugging** <br/><sub>self-debug, rubber-duck debugging for code</sub> | A code-generation technique where the model explains and debugs its own generated program using execution results or its own line-by-line explanation, then fixes it. *Execution feedback (errors, unit-test results) and self-generated code explanations are fed back to the model so it can localize and repair bugs without human guidance.* | accuracy/quality, reliability, autonomy | emerging | Chen et al., 2024, 'Teaching Large Language Models to Self-Debug' |
| **Self-Discover** <br/><sub>self-composed reasoning structures</sub> | A framework where the model self-composes a task-specific reasoning structure from a set of atomic reasoning modules before solving, then follows and can revise it. *The model selects, adapts, and implements reasoning modules into an explicit structure that scaffolds the solution, improving over fixed prompting templates.* <br/><sub>↔ also: Prompting & reasoning strategies</sub> | accuracy/quality, cost/tokens | research | Zhou et al., 2024, 'Self-Discover: Large Language Models Self-Compose Reasoning Structures' (Google DeepMind) |
| **Self-evaluation / self-assessment** <br/><sub>self-eval, confidence self-estimation</sub> | The model evaluates the quality or correctness of its own outputs (e.g., assigning a confidence or pass/fail), and that self-estimate gates or weights the answer. *An evaluation prompt elicits a calibrated judgment of the model's own answer, which is used to decide whether to accept, retry, or branch.* | accuracy/quality, reliability | emerging | Kadavath et al., 2022, 'Language Models (Mostly) Know What They Know'; Ren et al., 2023 on self-evaluation for guided decoding |
| **Self-Refine** <br/><sub>iterative self-refinement, self-feedback refinement, tool error recovery</sub> | A single model iteratively produces an output, generates feedback on its own output, and refines it, repeating until a stopping condition. *The same LLM alternates between a generate step and a self-feedback-then-revise step in a closed loop, using only its own critiques (no external supervision) to improve the draft.* <br/><sub>↔ also: Prompting & reasoning strategies; Test-time compute scaling & search over reasoning; Tool use & function calling optimization; Planning algorithms & world models for agents; Reliability, structured generation & evaluation-as-optimization</sub> | accuracy/quality, reliability, autonomy | production-standard | Madaan et al., 2023, 'Self-Refine: Iterative Refinement with Self-Feedback' (NeurIPS 2023) |
| **Self-verification** <br/><sub>backward verification, self-verifier, generate-and-verify</sub> | The model checks the correctness of its own candidate answer (often by reasoning backward or re-deriving), using the verification signal to accept, reject, or re-rank candidates. *A separate verification pass scores or validates each candidate output (e.g., backward verification of conditions), and only verified candidates are kept or weighted higher.* <br/><sub>↔ also: Prompting & reasoning strategies</sub> | accuracy/quality, reliability | emerging | Weng et al., 2023, 'Large Language Models are Better Reasoners with Self-Verification' |

## 4. Decoding-time control

Steer generation at the token level by reshaping the output distribution itself — contrasting a strong model against a weak one, contrasting layers, amplifying the context, or reshaping the sampling tail. Model-internal knobs that lift factuality or quality without touching the prompt or the weights.

*5 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Context-aware / Context-Aware Decoding (CAD, PMI context amplification)** | Amplifying the influence of provided context by contrasting the model's output distribution with and without the context (a pointwise-mutual-information adjustment). *Logits-with-context minus logits-without-context up-weight tokens the context makes more likely, reducing reliance on stale parametric memory in RAG/summarization.* | reliability, accuracy/quality | research | Shi et al., 2023 (Trusting Your Evidence / Context-Aware Decoding) |
| **Contrastive Decoding (expert vs amateur)** | A decoding method that subtracts a small 'amateur' model's log-probabilities from a large 'expert' model's to favor tokens the expert prefers more strongly. *The contrast amplifies the capability gap between models, suppressing generic/degenerate continuations and improving reasoning/open-ended quality at decode time.* | accuracy/quality | research | Li et al., 2022 (Contrastive Decoding); O'Brien & Lewis, 2023 (CD improves reasoning) |
| **DoLa (Decoding by Contrasting Layers)** | A factuality decoding method that contrasts the next-token distributions of later vs earlier transformer layers to surface factual knowledge. *Mature (later-layer) logits are contrasted against premature (earlier-layer) logits, amplifying factual tokens and reducing hallucination with no extra model.* | reliability, accuracy/quality | research | Chuang et al., 2023 (DoLa) |
| **Guidance / constrained generation interleaving (template-guided + acceleration)** | A programming model that interleaves fixed template text, control flow, and constrained model generations, reusing KV across the fixed parts. *By emitting deterministic template tokens directly and only sampling the variable slots, it cuts generated tokens and guarantees format while keeping cache warm.* | cost/tokens, latency, reliability | production-standard | Microsoft Guidance (Lundberg et al., 2023) |
| **Min-p sampling** | A truncation sampling scheme that keeps tokens whose probability is at least a fraction p of the top token's probability, adapting the candidate set to confidence. *The dynamic threshold widens when the model is uncertain and narrows when confident, giving better quality/diversity trade-offs than fixed top-p/top-k at high temperature.* | accuracy/quality | emerging | Nguyen et al., 2024 (Min-p sampling) |

# Part II — Context, memory & knowledge

## 5. Context-window management & compression

The context window is a scarce, expensive resource that grows every turn. These methods keep it bounded: summarize or compact old turns, compress the prompt, prune low-salience tokens, window long observations, or offload to external state. fak's measured work here: [O(1) context economics](../explainers/o1-context-window-economics.md), the [ctxwin baseline](CTXWIN-CONTEXT-WINDOW-BASELINE-2026-06-23.md), and the [context planner](O1-TURN-CONTEXT-PLANNER-2026-06-23.md).

*26 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **A-MEM (agentic memory / Zettelkasten-style)** <br/><sub>episodic memory, reflection, memory stream</sub> | A self-organizing agent memory that, on each new note, generates structured attributes (keywords, tags, context) and dynamically links it to related existing memories, forming an evolving interconnected knowledge network. *Inspired by the Zettelkasten method, each new memory triggers LLM-driven link generation and memory evolution that updates the attributes of connected older notes, so the graph reorganizes itself over time.* <br/><sub>↔ also: Agent memory architectures</sub> | context-capacity, accuracy/quality, autonomy, cost/tokens, reliability | research | Xu et al., 'A-MEM: Agentic Memory for LLM Agents', 2025 |
| **Activation Beacon (sliding context condensation)** | Condensing raw activations into compact 'beacon' tokens at intervals so a short-context model effectively reads a much longer window. *Beacon tokens summarize past activations and are kept while raw tokens are dropped, extending effective context with bounded memory.* | context-capacity, memory | research | Zhang et al., 2024 (Soaring from 4K to 400K: Extending LLM Context with Activation Beacon) |
| **Attention sinks / StreamingLLM** <br/><sub>attention sink, streaming LLM, sink tokens</sub> | An inference technique that keeps the first few tokens (attention sinks) plus a sliding window of recent tokens in the KV cache to enable stable streaming over effectively unbounded inputs. *Retaining initial sink tokens preserves the attention distribution while a rolling recent window bounds memory, avoiding the collapse seen when early KV entries are evicted.* | context-capacity, memory, latency, throughput | production-standard | Xiao et al., 'Efficient Streaming Language Models with Attention Sinks (StreamingLLM)', ICLR 2024 |
| **Contextual Compression / Retrieved-Context Pruning** <br/><sub>cross-encoder reranking, context distillation filter, RECOMP</sub> | Compresses or filters retrieved passages down to the query-relevant content before feeding them to the LLM, cutting tokens while keeping evidence. *An extractor or abstractor (rule-based, LLM-based, or a trained compressor like RECOMP/LLMLingua) prunes sentences/tokens with low query relevance, shrinking the context window load.* <br/><sub>↔ also: Retrieval & knowledge augmentation; Model routing, cascades & cost optimization</sub> | accuracy/quality, cost/tokens, reliability, context-capacity, latency | emerging | Xu et al., 2023 (RECOMP); Jiang et al., 2023 (LLMLingua) |
| **Conversation summarization / compaction** <br/><sub>context compaction, history summarization, conversation summary memory</sub> | Periodically replacing earlier turns of a conversation or agent trajectory with a model-generated summary so the working context stays within the window. *An LLM condenses prior messages/tool-results into a shorter abstractive summary that is reinserted in place of the raw history.* | cost/tokens, context-capacity, latency, memory | production-standard | LangChain ConversationSummaryMemory; Anthropic/Claude Code auto-compact; widely deployed pattern |
| **Deduplication / redundancy elimination in context** <br/><sub>context dedup, near-duplicate removal, repetition collapse</sub> | Detecting and collapsing repeated or near-duplicate content (boilerplate, re-pasted files, repeated tool outputs) within the context. *Exact/near-duplicate spans are hashed or embedding-matched and replaced by a single instance plus references, removing redundant tokens.* | cost/tokens, context-capacity | emerging | Retrieval/context-engineering practice; near-duplicate detection (MinHash/embedding) applied to prompts |
| **DMC / Dynamic Memory Compression (learned KV merging at decode)** | A retrofit that lets the model learn, per step, whether to append a new KV pair or accumulate it into the previous one, compressing the cache online. *A learned gate decides append-vs-merge for each token's KV, achieving multi-x KV compression with retained quality after light continued training.* | memory, throughput, context-capacity | research | Nawrot et al., 2024 (Dynamic Memory Compression) |
| **Generative / parametric memory (memory tokens & soft prompts)** <br/><sub>Gist tokens, gisting, AutoCompressor</sub> | Encoding past context into a small set of learned memory vectors or summary tokens stored and re-injected, rather than keeping natural-language text. *A model compresses long history into compact soft tokens / gist vectors that are appended to later contexts, recovering needed information at a fraction of the token cost.* <br/><sub>↔ also: Agent memory architectures</sub> | cost/tokens, memory, context-capacity, latency | research | Mu et al., 'Learning to Compress Prompts with Gist Tokens', 2023; Chevalier et al., 'AutoCompressors', 2023 |
| **Hierarchical / recursive summarization** <br/><sub>recursive summarization, tree summarization, multi-level summary</sub> | Building summaries at multiple levels of granularity (e.g., summaries of summaries) so very long histories or documents can be compressed in a structured tree. *Chunks are summarized, then groups of summaries are recursively summarized up a tree until the result fits the budget.* | context-capacity, cost/tokens, accuracy/quality | production-standard | Wu et al. 'Recursively Summarizing Books with Human Feedback', 2021; RAPTOR (Sarthi et al., 2024) |
| **In-context autoencoding / 500x context compression (ICAE — explicit), and prompt-distillation** | Training a learnable compressor to encode a long context into a handful of memory slots a frozen LLM can decode/condition on. *A LoRA-adapted encoder compresses context into soft memory tokens; the target model attends to those slots instead of the raw text, slashing prompt length.* | cost/tokens, context-capacity, latency | research | Ge et al., 2023 (In-context Autoencoder for Context Compression) |
| **LLMLingua** <br/><sub>prompt compression, coarse-to-fine prompt compression</sub> | A prompt-compression method that uses a small language model to drop low-information tokens from a long prompt while preserving the answer. *A budget controller plus token-level iterative perplexity estimation from a small LM prunes tokens whose removal least raises model uncertainty.* | cost/tokens, latency, context-capacity | emerging | Jiang et al., 'LLMLingua', EMNLP 2023 |
| **LLMLingua-2** <br/><sub>task-agnostic prompt compression</sub> | A task-agnostic prompt-compression model trained via data distillation to classify each token as keep-or-drop. *A bidirectional Transformer token-classifier, trained on GPT-4-distilled compression labels, predicts token retention faster and more robustly than perplexity pruning.* | cost/tokens, latency, context-capacity | emerging | Pan et al., 'LLMLingua-2', ACL 2024 (findings) |
| **LongLLMLingua** <br/><sub>long-context prompt compression</sub> | An extension of LLMLingua targeting long-context RAG that compresses and reorders retrieved documents to mitigate lost-in-the-middle and reduce tokens. *Question-aware coarse-to-fine compression plus document reordering by relevance, with a contrastive perplexity to keep question-relevant tokens.* | cost/tokens, accuracy/quality, context-capacity, latency | emerging | Jiang et al., 'LongLLMLingua', ACL 2024 |
| **Lost-in-the-middle mitigation via reordering** <br/><sub>context reordering, relevance reranking placement, primacy/recency placement</sub> | Placing the most important retrieved/context items at the beginning and end of the prompt to counter the model's degraded use of mid-context information. *Documents are reranked and positioned so high-relevance content occupies the high-attention head and tail positions rather than the neglected middle.* | accuracy/quality, reliability | production-standard | Liu et al., 'Lost in the Middle: How Language Models Use Long Contexts', TACL 2024 |
| **MemGPT (paged/virtual context memory)** <br/><sub>MemGPT, Letta, context paging</sub> | An agent memory architecture that treats the LLM context window like RAM and external stores like disk, paging information in and out via self-issued function calls so the agent can reason over data far larger than its context. *The agent uses tool calls to read/write a tiered hierarchy (main context, recall storage, archival storage) and an interrupt-driven controller evicts and recalls memory pages as the working context fills.* <br/><sub>↔ also: Agent memory architectures</sub> | context-capacity, memory, autonomy, reliability, accuracy/quality, cost/tokens, latency | production-standard | Packer et al., 'MemGPT: Towards LLMs as Operating Systems', 2023 |
| **RAPTOR** <br/><sub>tree-organized memory, hierarchical index, recursive abstractive processing</sub> | Builds a tree of recursively clustered and summarized chunks, enabling retrieval at multiple levels of abstraction so both detailed and high-level questions are served. *Recursively embeds, clusters, and summarizes text into a multi-level tree, then retrieves across tree nodes (collapsed-tree or traversal) to supply context at the right granularity.* <br/><sub>↔ also: Retrieval & knowledge augmentation</sub> | context-capacity, accuracy/quality, cost/tokens | emerging | Sarthi et al., 2024, 'RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval' |
| **Recurrent / segment-level memory transformers** <br/><sub>RMT, Recurrent Memory Transformer, Transformer-XL</sub> | Architectures that carry information across segments of a long input via recurrent memory tokens or cached/compressed past activations. *Special memory slots or a (optionally compressed) recurrent state pass summarized context from one segment to the next, extending effective range beyond a single window.* | context-capacity, memory, accuracy/quality | research | Dai et al., 'Transformer-XL', 2019; Rae et al., 'Compressive Transformers', 2020; Bulatov et al., 'Recurrent Memory Transformer', 2022 |
| **Retrieval-based context selection** <br/><sub>RAG context selection, retrieve-then-read, just-in-time retrieval</sub> | Selecting only the most relevant external passages to inject into the window per query instead of stuffing the whole corpus. *A retriever (dense/sparse/hybrid) scores and returns top-k passages by relevance to the query, bounding injected tokens.* | cost/tokens, accuracy/quality, context-capacity, latency | production-standard | Lewis et al., 'Retrieval-Augmented Generation', NeurIPS 2020 |
| **Scratchpad reasoning** <br/><sub>scratchpad, filesystem as memory, external memory</sub> | Training or prompting the model to emit intermediate computation steps to a 'scratchpad' before the final output, e.g., showing carries in multi-digit arithmetic. *Externalizing intermediate state into generated tokens gives the model working memory it can attend to, enabling multi-step algorithmic computation.* <br/><sub>↔ also: Prompting & reasoning strategies</sub> | accuracy/quality, reliability, context-capacity, cost/tokens, autonomy | research | Nye et al., 2021, 'Show Your Work: Scratchpads for Intermediate Computation with Language Models' |
| **Selective Context** <br/><sub>self-information pruning</sub> | A prompt-pruning method that removes redundant lexical units (tokens, phrases, sentences) judged low-information by a model. *A base LM computes self-information (negative log-prob) per unit and prunes the least informative units to fit a budget.* | cost/tokens, context-capacity, latency | emerging | Li et al., 'Selective Context' / 'Compressing Context to Enhance Inference Efficiency', 2023 |
| **Semantic chunking** <br/><sub>semantic splitting, embedding-based chunking, proposition chunking</sub> | Splitting source text into coherent units by meaning rather than fixed token length, so each chunk carries a self-contained idea. *Sentence/segment embeddings detect topic boundaries (similarity drops) to cut the text where semantic shifts occur, yielding cleaner retrieval/compression units.* | accuracy/quality, cost/tokens, context-capacity | production-standard | Greg Kamradt 'Levels of Text Splitting'; LlamaIndex/LangChain SemanticChunker; Chen et al. 'Dense X Retrieval (propositions)', 2023 |
| **Sliding-window / segment context processing** <br/><sub>chunked long-document processing, map-reduce over context, windowed inference</sub> | Processing an over-long input in overlapping windows or map-reduce passes and combining the partial results, rather than fitting it all at once. *The document is split into windows the model can handle; per-window outputs are aggregated (map-reduce) or carried forward (refine) into a final answer.* | context-capacity, cost/tokens | production-standard | LangChain map-reduce / refine summarization chains; long-document QA practice |
| **Structured / running state notes** <br/><sub>agent scratchpad state, running summary, todo/plan file</sub> | Maintaining a compact, structured note (plan, decisions, open items) that is updated each step and replaces re-reading the full trajectory. *The agent keeps a small canonical state object it rewrites incrementally, so each turn conditions on a dense summary instead of raw history.* | context-capacity, cost/tokens, reliability, autonomy | production-standard | Anthropic context-engineering / 'effective agents'; Cognition 'Don't Build Multi-Agents' (single thread + state) |
| **Sub-agent / context isolation (sub-context offload)** <br/><sub>context isolation, sub-agent delegation, spawned context</sub> | Delegating a bounded subtask to a separate agent/context that does heavy reading and returns only a distilled result to the parent's window. *A child context absorbs the token-heavy exploration and reports a small summary, keeping the orchestrator's window lean.* | context-capacity, cost/tokens, reliability | production-standard | Anthropic 'How we built our multi-agent research system', 2025; context-engineering practice |
| **Token / context pruning by saliency** <br/><sub>context distillation, saliency token dropping, input token pruning</sub> | Dropping input tokens deemed unimportant to the task before or during processing to shrink the context the model must handle. *Importance scores (attention, gradients, or a learned scorer) rank tokens and prune low-saliency ones from the prompt or intermediate sequence.* | cost/tokens, latency, context-capacity, memory | research | Goyal et al., 'PoWER-BERT', 2020; LazyLLM (Fu et al., 2024); general token-pruning literature |
| **Tool-result / observation truncation and windowing** <br/><sub>observation pruning, tool-output truncation, result head+tail windowing</sub> | Trimming verbose tool outputs and stale observations in an agent trajectory to a compact head/tail plus a pointer, keeping only what later steps still need. *Large or superseded tool results are replaced with truncated excerpts or references so the running context does not accumulate dead bytes.* | cost/tokens, context-capacity, latency | production-standard | Agent context-engineering practice; ReAct observation management (Yao et al., 2022); Anthropic context-engineering guidance |

## 6. Agent memory architectures

When a fact must outlive the context window, it goes to memory. These methods give agents typed, persistent stores — episodic, semantic, procedural — with policies for what to write, when to recall, and when to forget. Cross-cut: fak's [four layers of agent memory](../MEMORY-LAYERS-EXPLAINER.md) and [context-is-not-memory](../CONTEXT-IS-NOT-MEMORY.md).

*23 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Associative graph-traversal recall (HippoRAG-style)** <br/><sub>HippoRAG, neurobiologically-inspired memory, personalized PageRank recall</sub> | A long-term memory that builds a knowledge graph index and performs single-step multi-hop associative retrieval inspired by the hippocampal indexing theory. *Passages are indexed into a schemaless KG; a Personalized PageRank pass over the graph from query-anchored nodes retrieves multi-hop-relevant memories in one shot rather than iterative search.* <br/><sub>↔ also: Retrieval & knowledge augmentation</sub> | accuracy/quality, latency, cost/tokens, context-capacity | emerging | Jimenez Gutierrez et al., 'HippoRAG: Neurobiologically Inspired Long-Term Memory for LLMs', 2024 |
| **Cognitive memory typing (working / episodic / semantic / procedural)** <br/><sub>multi-store memory, CoALA memory taxonomy, typed agent memory</sub> | An organizing framework that partitions agent memory into working memory (current context), episodic memory (past experiences/events), semantic memory (facts/knowledge), and procedural memory (skills/how-to), mirroring human memory systems. *Each memory type gets its own store, write policy, and retrieval path so the agent reads procedural skills, episodic precedents, and semantic facts through specialized channels rather than one undifferentiated buffer.* | accuracy/quality, autonomy, context-capacity, reliability | emerging | Sumers et al., 'Cognitive Architectures for Language Agents (CoALA)', 2023 |
| **Conversational summary-buffer memory** <br/><sub>summary buffer, recursive conversation summarization, rolling summary</sub> | A memory that keeps recent turns verbatim while progressively summarizing older turns into a running condensed summary to fit a bounded context. *When the buffer exceeds a token budget, the oldest spans are folded into an LLM-generated running summary, preserving gist while discarding token-heavy detail.* | cost/tokens, context-capacity, latency | production-standard | LangChain ConversationSummaryBufferMemory; recursive summarization (Wu et al., 2021) |
| **Editable core / persona memory blocks** <br/><sub>core memory, memory blocks, in-context persistent memory</sub> | A small, always-in-context, agent-editable region holding durable facts (persona, user profile, key constraints) that the agent rewrites via dedicated functions. *Reserved context slots are exposed to the model as editable blocks; the agent calls append/replace functions to keep this high-priority state current without re-retrieving it each turn.* | accuracy/quality, reliability, context-capacity, cost/tokens | production-standard | MemGPT/Letta core memory (Packer et al., 2023) |
| **Entity / profile memory** <br/><sub>entity memory, user profile memory, knowledge-base memory</sub> | A structured memory keyed by entities (people, projects, preferences) that accumulates and updates per-entity facts across a long-running interaction. *Mentioned entities are extracted and their attribute records are read into context and updated after each turn, giving the agent stable, queryable per-entity state.* | accuracy/quality, reliability, context-capacity | production-standard | LangChain entity memory; conversational-personalization literature |
| **Experiential memory / case-based experience reuse** <br/><sub>experience replay for agents, Expel, trajectory memory</sub> | An episodic store of past task trajectories and the lessons/insights extracted from them, retrieved to guide the agent on analogous new tasks without weight updates. *After a batch of tasks, the agent compares successes and failures to distill natural-language insights and stores representative trajectories; at inference it retrieves the most similar cases and insights as in-context exemplars.* | accuracy/quality, autonomy, reliability | emerging | Zhao et al., 'ExpeL: LLM Agents Are Experiential Learners', 2023 |
| **Generative / RAG-free memory via Cache-Augmented Generation distinction — RecurrentGPT (language-based recurrence)** | An interpretable long-form agent loop that keeps a natural-language short-term and long-term memory state updated each step like an RNN written in prose. *The model reads/writes plain-text memory + a plan each turn, simulating recurrence to generate arbitrarily long, coherent outputs without hidden state.* | context-capacity, reliability | research | Zhou et al., 2023 (RecurrentGPT) |
| **Graph-based agent memory** <br/><sub>knowledge-graph memory, temporal knowledge graph memory, Zep/Graphiti</sub> | An agent memory represented as a (often temporally-aware) knowledge graph of entities and relations extracted from the interaction history, queried via graph traversal plus semantic search. *Conversations/observations are parsed into entity-relation triples with valid-time edges; retrieval combines graph traversal with embedding search to assemble a relevant, temporally-consistent subgraph for context.* | accuracy/quality, context-capacity, reliability | production-standard | Rasmussen et al., 'Zep: A Temporal Knowledge Graph Architecture for Agent Memory', 2025; Graphiti |
| **Importance scoring / memory prioritization** <br/><sub>salience scoring, memory importance weighting</sub> | Assigning each memory an importance/poignancy score at write time that biases later retrieval, consolidation, and forgetting decisions. *An LLM (or heuristic) rates how significant an observation is on write; the score is folded into retrieval ranking and reflection triggers, so consequential memories surface more readily and survive pruning.* | accuracy/quality, memory, cost/tokens | emerging | Generative Agents importance score (Park et al., 2023) |
| **Mem0 (extraction-and-update memory layer)** <br/><sub>Mem0, scalable memory layer</sub> | A memory layer that extracts salient facts from conversations and maintains them through an explicit add/update/delete/no-op decision over existing memories, with an optional graph variant. *An LLM extraction stage pulls candidate facts; a second update stage compares each against retrieved existing memories and chooses to insert, merge/update, or discard, keeping the store consolidated and non-redundant.* | accuracy/quality, cost/tokens, latency, context-capacity | production-standard | Chhikara et al., 'Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory', 2025 |
| **Memory consolidation** <br/><sub>memory summarization, memory compaction, consolidation pass</sub> | A background or threshold-triggered process that merges, deduplicates, and abstracts raw memories into compact higher-level representations to control store growth and improve retrieval signal. *Periodically the system clusters or summarizes related raw memories into consolidated entries and may retire the originals, trading fine detail for density and reduced retrieval noise.* | cost/tokens, context-capacity, accuracy/quality, memory | emerging | MemGPT (Packer et al., 2023); generative-agents reflection lineage |
| **Memory forgetting / decay** <br/><sub>temporal decay, memory eviction, TTL forgetting</sub> | Policies that downweight or remove memories over time or by disuse so stale, low-value entries do not dominate retrieval or bloat the store. *Each memory carries a decay factor (e.g., exponential recency or access-count-based) and/or an eviction threshold, so unaccessed low-importance memories fade or are pruned while reinforced ones persist.* | cost/tokens, memory, accuracy/quality, latency | emerging | Generative Agents recency decay (Park et al., 2023); broader memory-management literature |
| **Memory retrieval re-ranking / relevance gating** <br/><sub>memory reranker, salience gating, two-stage memory retrieval</sub> | A second-stage filter/reranker applied to candidate memories after first-pass retrieval to keep only the most salient, non-redundant entries before injection. *Cheap recall fetches a broad candidate set, then a cross-encoder or LLM judge reorders and prunes them for relevance, recency, and diversity, raising precision of what enters context.* | accuracy/quality, cost/tokens, context-capacity | emerging | RAG reranking practice applied to agent memory (cross-encoder rerankers; HippoRAG lineage) |
| **Memory writeback policy** <br/><sub>write gate, memory update policy, ADD/UPDATE/DELETE/NOOP decision</sub> | The decision policy that governs when and how new information is committed to long-term memory, including whether to insert, update an existing entry, delete, or ignore. *Incoming candidate facts are compared against retrieved existing memories and an LLM (or rules) selects an operation, preventing duplication and drift while keeping the store consistent.* | accuracy/quality, cost/tokens, reliability, memory | emerging | Mem0 (Chhikara et al., 2025); MemGPT write functions (Packer et al., 2023) |
| **MemoryBank with Ebbinghaus forgetting curve** | A long-term memory module that strengthens or decays stored memories per an Ebbinghaus-style forgetting schedule based on recall and recency. *Each memory carries a strength that decays over time and is refreshed on access, prioritizing durable, frequently-recalled facts at retrieval.* | memory, accuracy/quality | research | Zhong et al., 2023 (MemoryBank: Enhancing LLMs with Long-Term Memory) |
| **Procedural memory / learned skills writeback** <br/><sub>agent workflow memory, procedural skill induction, AWM</sub> | A mechanism that induces reusable procedures or workflows from an agent's past successful trajectories and writes them back into memory for retrieval on similar future tasks. *The agent mines recurring action patterns from solved tasks, abstracts them into named workflows/macros, and stores them so future planning retrieves the procedure instead of re-deriving the steps.* | accuracy/quality, cost/tokens, autonomy, latency | emerging | Wang et al., 'Agent Workflow Memory', 2024 |
| **Recurrent / Memorizing-Transformer memory** <br/><sub>kNN memory, Memorizing Transformers, Transformer-XL recurrence</sub> | Architectural long-term memory that stores past key-value pairs (or recurrent states) and retrieves them via attention/kNN to extend effective context beyond the attention window. *Past hidden states or KV pairs are cached in a non-differentiable external store; a kNN-augmented attention layer attends over retrieved entries, giving the model long-range recall at sublinear cost.* | context-capacity, memory, accuracy/quality | research | Wu et al., 'Memorizing Transformers', 2022; Dai et al., 'Transformer-XL', 2019 |
| **Retrieval-augmented memory (retrieval-over-memory / RAG)** <br/><sub>RAG over memory, vector-store memory, long-term memory retrieval</sub> | Treating the long-term memory store as a retrieval corpus, embedding entries and fetching the top-k semantically relevant memories into the working context at each step. *Memories are embedded and indexed (e.g., in a vector DB); a query embedding (often the current task or last turn) retrieves nearest neighbors that are injected into the prompt, bounding context to relevant content.* | context-capacity, cost/tokens, accuracy/quality, latency | production-standard | Lewis et al., 'Retrieval-Augmented Generation', 2020 (applied to agent memory) |
| **Retrieval-scored memory (recency-importance-relevance)** <br/><sub>RIR scoring, weighted memory retrieval</sub> | A retrieval policy over a memory store that ranks candidate memories by a weighted sum of recency (exponential time decay), importance, and semantic relevance. *Each memory carries a recency decay, an LLM-assigned importance score, and an embedding similarity to the query; the normalized weighted sum selects the top-k memories injected into context.* | accuracy/quality, cost/tokens, context-capacity | emerging | Park et al., 'Generative Agents', 2023 |
| **Self-editing / learning-to-memorize agents** <br/><sub>self-edit memory, memory-as-learning, MemoryBank / SiliconFriend</sub> | Agents that learn or are explicitly trained to decide what to remember and how to update their own memory store over long horizons, including human-memory-inspired update curves. *The agent applies (sometimes Ebbinghaus-curve-based) update and reinforcement rules to its memory, strengthening repeatedly-recalled items and letting others decay, optimizing the store as a learned policy.* | accuracy/quality, autonomy, memory, reliability | research | Zhong et al., 'MemoryBank: Enhancing LLMs with Long-Term Memory', 2023 |
| **Shared / collective multi-agent memory** <br/><sub>shared memory blackboard, team memory, collective experience pool</sub> | A memory store shared across multiple agents in a system, letting one agent's learned facts, skills, or experiences be retrieved and reused by peers. *Agents write experiences/skills to a common indexed pool with provenance and access policies; teammates retrieve relevant shared entries, propagating knowledge without retraining.* | accuracy/quality, throughput, autonomy, cost/tokens | research | Multi-agent collaboration literature (e.g., generative-agents communities; MetaGPT/AutoGen shared context) |
| **Typed multi-view memory** <br/><sub>multi-view memory, multi-perspective memory store</sub> | Storing the same underlying experience under several typed views (e.g., raw episodic, summarized, entity-indexed, intent/goal-indexed) so different retrieval needs hit the most suitable representation. *On write, an event is projected into multiple parallel representations/indices; retrieval routes the query to the view that best matches its granularity or facet, improving recall precision.* | accuracy/quality, context-capacity, reliability | research | Generative-agents/CoALA-inspired multi-view designs (Sumers et al., 2023) |
| **Voyager skill library** <br/><sub>skill library, lifelong skill acquisition, code-as-skill memory</sub> | A growing, retrievable library of executable skills (programs) that an agent authors, verifies, and reuses, enabling lifelong learning and compositional capability in an open-ended environment. *Successful self-generated code routines are embedded by their description and stored; later tasks retrieve relevant skills as building blocks, so capability compounds and complex skills are composed from simpler ones.* <br/><sub>↔ also: Planning algorithms & world models for agents</sub> | accuracy/quality, autonomy, reliability, cost/tokens | research | Wang et al., 'Voyager: An Open-Ended Embodied Agent with Large Language Models', 2023 |

## 7. Retrieval & knowledge augmentation

Bring the right external knowledge into context at the right moment. RAG and its descendants vary on *what* to retrieve (chunks, graphs, hypothetical docs), *when* (statically, iteratively, on-demand), and *how to trust it* (re-ranking, correction, conflict resolution).

*31 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Adaptive-RAG (query-complexity-routed retrieval depth)** | A router that classifies query complexity and chooses between no-retrieval, single-step, and multi-step iterative retrieval accordingly. *A trained complexity classifier directs simple queries to direct answering and hard ones to iterative retrieval, balancing accuracy against retrieval cost/latency.* | cost/tokens, latency, accuracy/quality | emerging | Jeong et al., 2024 (Adaptive-RAG) |
| **Agentic RAG** <br/><sub>iterative RAG, agent-driven retrieval, ReAct-style RAG</sub> | Treats retrieval as a tool an agent invokes adaptively across multiple reasoning steps, deciding when, what, and how often to retrieve rather than retrieving once up front. *An LLM agent interleaves reasoning and retrieval actions (often via ReAct), issuing follow-up queries based on intermediate findings until it has enough evidence to answer.* | accuracy/quality, autonomy, reliability | emerging | Builds on Yao et al., 2022 (ReAct) and IRCoT (Trivedi et al., 2023); 'agentic RAG' as a 2024-2025 design pattern |
| **Astute RAG / knowledge-conflict resolution** | Explicitly reconciling the model's internal/parametric knowledge against retrieved passages when they conflict, instead of blindly trusting retrieval. *Internal knowledge is elicited, consolidated with external passages by source, and conflicts adjudicated before final answer generation.* | accuracy/quality, reliability | emerging | Wang et al., 2024 (Astute RAG) |
| **Chain-of-Note (CoN)** | A robust-RAG method where the model writes per-document reading notes before answering, so noisy/irrelevant retrieved docs are explicitly down-weighted. *Generating evidence notes per passage lets the model judge relevance and abstain ('unknown') when retrieval is unhelpful, cutting hallucination from bad context.* | accuracy/quality, reliability | research | Yu et al., 2023 (Chain-of-Note) |
| **ColBERT Late-Interaction Retrieval** <br/><sub>ColBERT, ColBERTv2, late interaction</sub> | A retrieval model that stores per-token embeddings and scores documents via fine-grained token-level MaxSim interaction, giving cross-encoder-like precision at near bi-encoder speed. *Encodes query and document into multiple token vectors and computes relevance as the sum of maximum cosine similarities (MaxSim) between query tokens and document tokens, enabling precise yet scalable search.* | accuracy/quality, latency | emerging | Khattab & Zaharia, 2020, 'ColBERT'; Santhanam et al., 2022, 'ColBERTv2' |
| **Contextual Retrieval** <br/><sub>contextual embeddings, contextual chunking</sub> | Prepends a chunk-specific explanatory context (generated from the whole document) to each chunk before embedding/indexing, so isolated chunks retain document-level meaning. *An LLM writes a short situating context for each chunk (and a BM25 variant), which is indexed with the chunk, sharply reducing retrieval failures from context-stripped chunks.* | accuracy/quality, reliability | emerging | Anthropic, 2024, 'Introducing Contextual Retrieval' |
| **Corrective RAG** <br/><sub>CRAG</sub> | Adds a lightweight retrieval evaluator that grades retrieved documents and triggers corrective actions (e.g. web search or knowledge refinement) when retrieval quality is poor. *A retrieval evaluator labels results as correct/incorrect/ambiguous; on low confidence it decomposes-then-recomposes knowledge or falls back to web search before generation.* | accuracy/quality, reliability | research | Yan et al., 2024, 'Corrective Retrieval Augmented Generation' |
| **Cross-Encoder Re-ranking** <br/><sub>re-ranking, neural reranker, two-stage retrieval</sub> | A second-stage scorer that jointly encodes the query and each retrieved candidate to produce a precise relevance score, reordering an initial candidate set. *Feeds query-passage pairs through a cross-attention transformer (e.g. a fine-tuned BERT or a hosted reranker) so each candidate is scored with full query-document interaction, unlike the bi-encoder used for first-stage recall.* | accuracy/quality, reliability | production-standard | Nogueira & Cho, 2019, 'Passage Re-ranking with BERT'; commercial rerankers (Cohere Rerank) |
| **Dense Passage Retrieval** <br/><sub>DPR, bi-encoder dense retrieval</sub> | A dual-encoder dense retriever that embeds queries and passages separately into a shared space and retrieves by nearest-neighbor similarity, the workhorse first-stage retriever for RAG. *Trains two BERT encoders with contrastive learning so relevant query-passage pairs are close in embedding space, enabling fast ANN search over precomputed passage vectors.* | accuracy/quality, latency | production-standard | Karpukhin et al., 2020, 'Dense Passage Retrieval for Open-Domain Question Answering' |
| **FLARE (Forward-Looking Active Retrieval)** <br/><sub>active retrieval, FLARE</sub> | An active retrieval strategy that anticipates upcoming content and triggers retrieval mid-generation whenever the model is about to produce low-confidence tokens. *Generates a tentative next sentence, and if its token probabilities fall below a confidence threshold, uses that sentence as a query to retrieve and regenerate, repeating throughout generation.* | accuracy/quality, reliability | research | Jiang et al., 2023, 'Active Retrieval Augmented Generation' |
| **Generate-then-Read / GenRead (LLM-as-retriever)** | Replacing or supplementing a retriever by having the LLM generate background documents that are then used as context to answer. *The model produces several synthetic context documents (optionally clustered for diversity), which are fed back as evidence, leveraging parametric knowledge.* | accuracy/quality | research | Yu et al., 2022 (Generate rather than Retrieve / GenRead) |
| **GraphRAG** <br/><sub>graph-based RAG, knowledge-graph RAG</sub> | Builds a knowledge graph (entities, relations, and community summaries) from the corpus and retrieves over graph structure to answer global/holistic questions a flat vector index cannot. *Extracts entities and relationships into a graph, clusters into communities with hierarchical summaries, and answers by traversing/aggregating relevant graph elements rather than just top-k passages.* | accuracy/quality, context-capacity, reliability | emerging | Edge et al., 2024, 'From Local to Global: A Graph RAG Approach to Query-Focused Summarization' (Microsoft GraphRAG) |
| **Hybrid Retrieval (sparse + dense)** <br/><sub>BM25 + dense, hybrid search, lexical-semantic fusion</sub> | Combines a lexical/sparse retriever (e.g. BM25) with a dense embedding retriever and fuses their results to capture both exact-term matches and semantic similarity. *Runs both retrievers in parallel and merges ranked lists, commonly via Reciprocal Rank Fusion or weighted score combination, before passing candidates downstream.* | accuracy/quality, reliability | production-standard | Reciprocal Rank Fusion, Cormack et al., 2009; widely operationalized in modern RAG stacks |
| **Hypothetical Document Embeddings** <br/><sub>HyDE</sub> | A zero-shot dense retrieval technique that has the LLM generate a hypothetical answer document for the query, then retrieves real documents similar to that synthetic document. *Embeds an LLM-generated hypothetical answer (rather than the raw query) and searches the corpus with it, bridging the lexical/semantic gap between question phrasing and answer phrasing.* | accuracy/quality | emerging | Gao et al., 2022, 'Precise Zero-Shot Dense Retrieval without Relevance Labels' |
| **In-Context / Retrieval-Augmented Long-Context (kNN-LM family)** <br/><sub>kNN-LM, retrieval-augmented LM, RETRO</sub> | Augments next-token prediction at the model/decoding level by retrieving nearest-neighbor contexts from a datastore, blending retrieved continuations into the LM's predictions. *Interpolates the LM's output distribution with a distribution over retrieved nearest-neighbor tokens (kNN-LM), or cross-attends to retrieved chunks via dedicated layers (RETRO), injecting knowledge below the prompt level.* | accuracy/quality, memory, context-capacity | research | Khandelwal et al., 2020 (kNN-LM); Borgeaud et al., 2022 (RETRO) |
| **IRCoT (Interleaving Retrieval with Chain-of-Thought)** <br/><sub>IRCoT, retrieval-interleaved CoT</sub> | Interleaves retrieval with each step of chain-of-thought reasoning for multi-step questions, using the evolving reasoning to guide subsequent retrieval and vice versa. *Alternates between generating a reasoning step and retrieving documents using the latest reasoning sentence as the query, so retrieval tracks the reasoning trajectory across hops.* | accuracy/quality, reliability | research | Trivedi et al., 2023, 'Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions' |
| **Late Chunking** <br/><sub>late interaction chunking</sub> | Embeds the entire document with a long-context embedding model first, then pools token embeddings into chunk vectors, so each chunk embedding carries full-document context. *Runs the transformer over the long document once and only afterward splits the contextualized token embeddings into chunks via mean-pooling, preserving cross-chunk context without per-chunk LLM calls.* | accuracy/quality, cost/tokens | emerging | Günther et al., 2024, 'Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models' (Jina AI) |
| **Maximal Marginal Relevance retrieval** <br/><sub>MMR, diversity-aware retrieval</sub> | Selects retrieved passages that balance relevance to the query against novelty relative to already-selected passages, reducing redundancy in the context. *Greedily picks each next document to maximize a weighted trade-off between query similarity and dissimilarity to the already-chosen set, diversifying the retrieved evidence.* | accuracy/quality, cost/tokens | production-standard | Carbonell & Goldstein, 1998, 'The Use of MMR for Reordering Documents'; standard option in modern RAG retrievers |
| **Multi-Query Retrieval** <br/><sub>query fan-out, multi-query expansion, RAG-Fusion</sub> | Generates several diverse paraphrases of a single query, retrieves for each, and fuses the results to improve recall and robustness to query phrasing. *An LLM produces multiple query variants, runs retrieval per variant, and combines the candidate sets (often with RRF) before re-ranking or generation.* | accuracy/quality, reliability | production-standard | RAG-Fusion (Raudaschl, 2023); LangChain MultiQueryRetriever |
| **Query Decomposition** <br/><sub>sub-question decomposition, multi-query, least-to-most retrieval</sub> | Breaks a complex or multi-hop question into simpler sub-queries that are retrieved and answered independently, then composed into a final answer. *An LLM splits the query into sub-questions, retrieves evidence per sub-question, and aggregates the partial answers, improving coverage on compositional/multi-hop questions.* | accuracy/quality, reliability | production-standard | Common in multi-hop RAG; cf. least-to-most prompting (Zhou et al., 2022) and decomposed retrieval pipelines |
| **Query Expansion** <br/><sub>query augmentation, pseudo-relevance feedback, Query2Doc</sub> | Enriches the query with additional terms, synonyms, or LLM-generated context to broaden recall over the corpus. *Appends generated or feedback-derived terms (e.g. an LLM expansion or pseudo-relevance-feedback terms) to the query so the retriever matches more relevant passages.* | accuracy/quality | production-standard | Wang et al., 2023, 'Query2doc: Query Expansion with Large Language Models' |
| **Query Rewriting** <br/><sub>query transformation, Rewrite-Retrieve-Read</sub> | Reformulates the user's raw query into one or more retrieval-optimized queries before search, improving alignment with how relevant documents are phrased. *An LLM (often a small trainable rewriter) rewrites the query, optionally guided by reinforcement signals from downstream answer quality, then the rewritten query drives retrieval.* | accuracy/quality, reliability | production-standard | Ma et al., 2023, 'Query Rewriting for Retrieval-Augmented Large Language Models' |
| **RankGPT / LLM listwise reranking** | Using the LLM itself to rerank retrieved passages via a listwise permutation prompt rather than a separate cross-encoder. *Sliding-window listwise prompts ask the LLM to output a permutation of candidate doc ids by relevance, optionally distilled into a smaller reranker.* | accuracy/quality | emerging | Sun et al., 2023 (Is ChatGPT Good at Search? / RankGPT) |
| **Reciprocal Rank Fusion** <br/><sub>RRF, rank fusion</sub> | A parameter-light method to merge multiple ranked retrieval lists (e.g. from different retrievers or query variants) into a single robust ranking. *Scores each document by summing 1/(k + rank) across all input lists, rewarding items ranked highly by several retrievers without needing score calibration.* | accuracy/quality, reliability | production-standard | Cormack et al., 2009, 'Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods' |
| **REPLUG** <br/><sub>retrieve-then-ensemble, black-box LM retrieval</sub> | Augments a frozen black-box LLM by retrieving documents and ensembling the model's output distributions across them, optionally tuning the retriever to the LM's likelihood signal. *Prepends each retrieved document separately, computes the LM's next-token distribution per document, and marginalizes them weighted by retrieval scores; the retriever is trained to match LM perplexity gains.* | accuracy/quality | research | Shi et al., 2023, 'REPLUG: Retrieval-Augmented Black-Box Language Models' |
| **Retrieval Routing** <br/><sub>query routing, adaptive RAG routing, retriever selection</sub> | Routes each query to the most appropriate retrieval source, strategy, or no-retrieval path based on query characteristics or estimated complexity. *A classifier or LLM router predicts query type/complexity and dispatches to the matching index, tool, or pipeline (e.g. simple direct answer vs single-step vs multi-step retrieval).* | accuracy/quality, cost/tokens, latency | emerging | Adaptive-RAG, Jeong et al., 2024, 'Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity' |
| **Retrieval-Augmented Fine-Tuning** <br/><sub>RAFT, RFT, best-of-n distillation</sub> | Fine-tunes the LLM to read retrieved documents that include distractors, teaching it to cite the relevant ones and ignore irrelevant passages in the RAG setting. *Trains on question + golden documents + distractor documents with chain-of-thought answers, so the model learns domain-specific reading and robustness to imperfect retrieval.* <br/><sub>↔ also: Training & adaptation for agentic capability</sub> | accuracy/quality, reliability | research | Zhang et al., 2024, 'RAFT: Adapting Language Model to Domain Specific RAG' |
| **Retrieval-Augmented Generation** <br/><sub>RAG, naive RAG, vanilla RAG</sub> | A pattern that retrieves relevant documents from an external corpus and conditions the LLM's generation on them, so the model answers from fetched evidence rather than parametric memory alone. *Embeds the query, runs nearest-neighbor search over a vector index of chunked documents, and concatenates the top-k passages into the prompt context before generation.* | accuracy/quality, reliability, cost/tokens, context-capacity | production-standard | Lewis et al., 2020, 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' |
| **Self-Querying Retrieval (metadata-filtered retrieval)** <br/><sub>self-query retriever, metadata filtering, structured query construction</sub> | Has the LLM translate a natural-language query into both a semantic search string and a structured metadata filter, combining vector search with attribute constraints. *An LLM parses the query into a (query, filter) pair against a known metadata schema, so retrieval applies exact filters (date, author, type) alongside embedding similarity.* | accuracy/quality, reliability, latency | production-standard | LangChain Self-Query Retriever pattern, 2023 |
| **Self-RAG** <br/><sub>self-reflective RAG</sub> | An LLM trained to decide on demand whether to retrieve, then critique its own retrieved passages and generations using special reflection tokens. *The model emits 'retrieve' and 'critique' (relevance/support/usefulness) reflection tokens during decoding, enabling adaptive retrieval and self-assessment that can gate or revise output.* | accuracy/quality, reliability, cost/tokens | research | Asai et al., 2023, 'Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection' |
| **Sentence-Window / Parent-Document Retrieval** <br/><sub>small-to-big retrieval, parent-child chunking, auto-merging retrieval</sub> | Retrieves on small precise units (sentences or child chunks) but returns the larger surrounding window or parent chunk to the LLM for fuller context. *Indexes fine-grained chunks for accurate matching while linking each to a larger parent/window that is substituted in at generation time, decoupling retrieval granularity from context granularity.* | accuracy/quality, context-capacity | production-standard | LlamaIndex Sentence-Window / LangChain Parent-Document Retriever patterns, 2023-2024 |

## 8. KV cache & prefix reuse

The attention KV cache is the agent's hottest reusable asset. These methods avoid recomputing it: reuse shared prefixes, page and offload it, share it across heads/layers/agents, quantize it, or evict the least-useful spans. fak's deep treatment: [agentic caching SOTA](AGENTIC-CACHING-SOTA-2026-06-19.md) and the [addressable KV cache](../explainers/addressable-kv-cache.md) with bit-exact mid-run eviction (`max|Δ|=0`).

*25 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Cache-aware load balancing / routing** <br/><sub>KV-cache-aware routing, prefix-aware routing, sticky routing</sub> | A request router that directs requests to the serving replica most likely to already hold the relevant KV prefix, maximizing cache hit rate across a fleet. *The router tracks (or approximates) which prefixes each instance has cached and routes by longest-prefix affinity rather than pure round-robin, while balancing load.* | latency, throughput, cost/tokens | emerging | Preble (Srivatsa et al., 2024); SGLang / vLLM production-stack cache-aware routing |
| **CacheBlend (segment-level / non-prefix KV reuse)** <br/><sub>non-prefix caching, modular KV fusion</sub> | A technique that reuses precomputed KV for arbitrary text chunks even when they are not a contiguous prefix (e.g., multiple retrieved RAG passages concatenated in different orders). *It loads cached KV for each chunk and selectively recomputes only the small fraction of tokens whose attention is most affected by cross-chunk context, restoring accuracy at a fraction of full recompute cost.* | latency, cost/tokens, throughput | emerging | Yao et al., CacheBlend, 2024 |
| **Cascade / multi-tier KV inference (CascadeAttention)** <br/><sub>cascade inference, shared-prefix cascade, FlashInfer cascade</sub> | An attention execution that splits a shared prefix's KV (computed once, kept in fast memory) from per-request suffix KV, running two attention passes and merging them. *It computes attention over the large shared prefix with a memory-efficient kernel and over the small unique suffix separately, then combines via log-sum-exp merging, avoiding redundant prefix work.* | throughput, latency, memory | emerging | Ye et al., Cascade Inference / FlashInfer, 2024 |
| **ChunkAttention / prefix-aware fused attention** <br/><sub>ChunkAttention, prefix-aware KV kernel</sub> | An attention kernel that detects and batches shared KV prefixes across requests using a prefix-tree of KV chunks, then fuses the shared-prefix attention computation. *KV is broken into chunks organized in a prefix tree; the kernel runs a single batched two-phase attention over shared chunks and per-request chunks, removing duplicated memory traffic.* | throughput, memory, latency | research | Ye et al., ChunkAttention, ACL 2024 |
| **Context caching for agent / tool-loop reuse** <br/><sub>agent prefix reuse, tool-schema caching, trajectory KV reuse</sub> | Applying prefix/KV caching specifically to the agent loop, where a long stable prefix (system instructions, tool definitions, prior turns) is reused across every step of a multi-turn trajectory. *Keeping the agent's stable context as a cached prefix (provider cache or local KV) so each tool-call iteration only prefills the new appended observation rather than the entire growing transcript.* | cost/tokens, latency, throughput | emerging | Agent framework caching guidance (e.g., Anthropic/OpenAI agent docs, 2024); common practice in agent infra |
| **Cross-layer KV sharing (YOCO / CLA)** <br/><sub>YOCO, Cross-Layer Attention, layer-shared KV</sub> | Architectural schemes that share or reuse KV cache across transformer layers so the cache is stored once (or for few layers) rather than per layer. *Later layers attend to KV computed at an earlier (or single global) layer, eliminating per-layer KV storage and roughly dividing cache size by the sharing factor.* | memory, context-capacity, throughput | emerging | Sun et al., You Only Cache Once (YOCO), 2024; Brandon et al., Reducing Transformer KV Cache Size with Cross-Layer Attention (CLA), 2024 |
| **Disaggregated KV transfer / KV-aware disaggregation** <br/><sub>prefill-decode disaggregation KV transfer, NVIDIA Dynamo, Mooncake</sub> | An architecture that separates prefill and decode onto different GPU pools and ships the computed KV cache between them, with a global KV store enabling cross-instance prefix reuse. *Prefill workers compute KV and stream it over fast interconnect to decode workers, while a distributed KV cache pool lets any instance reuse another's cached prefixes.* | throughput, latency, cost/tokens | production-standard | Mooncake (Qin et al., 2024); NVIDIA Dynamo (2025) |
| **Distributed / global KV cache pool** <br/><sub>shared KV pool, global prefix store, KV cache cluster</sub> | A cluster-wide shared store of KV blocks (across GPUs and tiers) that lets any worker reuse prefixes computed by any other worker. *KV blocks are content-addressed and registered in a shared index (often RDMA/NVLink-backed), so a cache lookup spans the whole fleet rather than a single instance's local cache.* | throughput, cost/tokens, memory | emerging | Mooncake distributed KVCache (Qin et al., 2024); NVIDIA Dynamo KV manager |
| **DuoAttention (retrieval vs streaming head split)** | Classifying attention heads into 'retrieval heads' (need full KV) and 'streaming heads' (need only recent + sink), keeping the full cache only for the former. *Only retrieval heads retain the full KV cache; streaming heads use a small recent+sink window, sharply cutting long-context KV memory and latency.* | memory, latency, context-capacity | emerging | Xiao et al., 2024 (DuoAttention) |
| **FastGen (adaptive KV compression)** <br/><sub>H2O, Heavy-Hitter Oracle, Scissorhands</sub> | An adaptive, head-aware KV compression method that assigns different retention policies to different attention heads based on their observed structure. *A lightweight profiling pass classifies each head (e.g., local, special-token, punctuation, broad) and applies the cheapest policy that preserves that head's behavior, compressing the cache per-head.* <br/><sub>↔ also: Context-window management & compression</sub> | memory, throughput, context-capacity, latency | research | Ge et al., Model Tells You What to Discard: Adaptive KV Cache Compression (FastGen), ICLR 2024 |
| **Grouped-Query Attention** <br/><sub>GQA, MQA, shared KV heads</sub> | An attention variant that interpolates between multi-head and multi-query attention by sharing each K/V head across a group of query heads. *Uses G key/value heads (1 < G < num_heads) so the KV cache is smaller than MHA while retaining more quality than MQA, with a quick uptraining to convert an MHA checkpoint.* <br/><sub>↔ also: Inference / serving optimizations beneath agents</sub> | memory, throughput, context-capacity, latency, accuracy/quality | production-standard | Ainslie et al., 2023 (GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints) |
| **Heavy KV merging (CaM / KVMerger)** <br/><sub>cache merging, KV merging, CaM</sub> | Compression methods that, instead of evicting tokens outright, merge soon-to-be-evicted KV entries into retained ones to preserve their information. *Tokens marked for eviction are weighted-merged (by attention or similarity) into surviving KV slots, reducing cache size while limiting the accuracy loss of pure dropping.* | memory, accuracy/quality, context-capacity | research | Zhang et al., CaM: Cache Merging for Memory-efficient LLMs Inference, ICML 2024 |
| **Hydragen (shared-prefix attention decomposition)** <br/><sub>Hydragen, high-throughput shared-prefix attention</sub> | A high-throughput attention method for large batches that share a common prefix, decomposing attention into prefix and suffix parts to enable tensor-core-efficient batching. *It batches the shared-prefix attention across all sequences into dense matmuls and handles unique suffixes separately, then combines them, turning memory-bound work into compute-efficient work.* | throughput, latency | research | Juravsky et al., Hydragen: High-Throughput LLM Inference with Shared Prefixes, 2024 |
| **KV cache deduplication / content-addressed block sharing** <br/><sub>KV dedup, content-hash KV sharing, copy-on-write KV</sub> | Eliminating duplicate KV storage by sharing identical KV blocks (same tokens, same position) across requests via content addressing and copy-on-write. *Each KV block is hashed by its token content and position; identical blocks point to one physical copy until a write diverges (copy-on-write), collapsing redundant storage.* | memory, throughput | production-standard | vLLM block hashing / PagedAttention CoW (Kwon et al., 2023); vLLM prefix caching |
| **KV cache offloading (CPU / disk tiering)** <br/><sub>LMCache, KV offload, CPU/disk KV tier</sub> | Extending KV cache capacity beyond GPU HBM by storing warm/cold KV blocks in CPU DRAM, local disk, or remote storage and fetching them back on demand. *A tiered cache manager evicts cold KV blocks from GPU to slower tiers and reloads (or recomputes) them when a matching prefix recurs, trading bandwidth for far larger effective cache.* | memory, cost/tokens, context-capacity | production-standard | LMCache (project, 2024); NVIDIA Dynamo KV offload |
| **KV cache quantization (KIVI)** <br/><sub>KV cache quantization, KVQuant, KIVI</sub> | Compressing the stored KV cache to very low bit-width (e.g., 2-bit) to shrink memory footprint while preserving generation quality. *It quantizes keys per-channel and values per-token (asymmetric) to exploit their differing outlier distributions, keeping a small residual of recent tokens in full precision.* <br/><sub>↔ also: Context-window management & compression; Inference / serving optimizations beneath agents</sub> | memory, context-capacity, throughput | emerging | Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, 2024 |
| **MInference (dynamic sparse prefill attention)** | Accelerating long-context prefill by detecting per-head sparse attention patterns (A-shape, vertical-slash, block-sparse) and computing only those. *Offline-classified head patterns drive a sparse attention kernel at prefill time, cutting the quadratic prefill cost on 100K+ contexts with little accuracy loss.* | latency, throughput, context-capacity | emerging | Jiang et al., 2024 (MInference 1.0) |
| **Multi-head Latent Attention (MLA)** <br/><sub>MLA, latent KV compression, DeepSeek MLA</sub> | An attention variant that compresses keys and values into a low-rank latent vector cached per token, drastically reducing KV memory versus standard multi-head attention. *K/V are projected down to a shared latent representation that is cached and up-projected at attention time, so only the small latent (not full per-head K/V) is stored.* <br/><sub>↔ also: Inference / serving optimizations beneath agents</sub> | memory, throughput, context-capacity | production-standard | DeepSeek-V2 (DeepSeek-AI, 2024) |
| **PagedAttention** <br/><sub>vLLM paged KV, vLLM KV paging</sub> | A KV-cache memory manager that stores the cache in fixed-size non-contiguous blocks (pages), eliminating fragmentation and enabling near-zero-waste sharing of KV memory. *Borrowing OS virtual-memory paging, it maps logical KV positions to physical blocks via a block table, allowing copy-on-write sharing of identical blocks across sequences (e.g., shared prefixes, parallel samples).* <br/><sub>↔ also: Inference / serving optimizations beneath agents</sub> | memory, throughput | production-standard | Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM), SOSP 2023 |
| **Prompt Cache (modular position-independent KV)** <br/><sub>modular prompt caching, prompt schema reuse, exact-match cache</sub> | A scheme that precomputes and reuses KV for reusable prompt modules (snippets) defined in a schema, allowing them to appear at varying positions across prompts. *Position-independent KV is computed for each named module with discontinuous position-id assignment so cached modules can be spliced into different prompts without recomputation.* <br/><sub>↔ also: Model routing, cascades & cost optimization</sub> | latency, cost/tokens, throughput | emerging | Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference, MLSys 2024 |
| **Provider prompt caching (OpenAI / Anthropic / Gemini)** <br/><sub>KV prefix cache, prompt caching, RadixAttention</sub> | Provider-side hosted caching of a request's prefix KV state so repeated leading content (system prompt, tool schemas, few-shot examples, large documents) is billed at a steep discount and served faster on subsequent calls. *The provider stores the attention KV for an already-processed token prefix keyed by its exact bytes and, on a cache hit, skips re-prefilling that prefix, charging cached-input rates instead of full input rates.* <br/><sub>↔ also: Context-window management & compression; Tool use & function calling optimization; Model routing, cascades & cost optimization; Inference / serving optimizations beneath agents</sub> | latency, cost/tokens, throughput, memory | production-standard | Anthropic prompt caching docs (2024); OpenAI prompt caching (2024); Google Gemini context caching (2024) |
| **PyramidKV / PyramidInfer (layer-pyramidal KV budget)** <br/><sub>PyramidKV, PyramidInfer, layer-wise KV budget allocation</sub> | KV compression schemes that allocate a larger cache budget to lower layers and a smaller budget to higher layers, matching the pyramidal information-aggregation pattern across depth. *Per-layer retention budgets decrease with depth (information funnels into fewer tokens at higher layers), so each layer keeps only its needed KV positions.* | memory, context-capacity, throughput | research | Cai et al., PyramidKV, 2024; Yang et al., PyramidInfer, 2024 |
| **Quest (query-aware KV page selection)** <br/><sub>Quest, query-aware sparse KV</sub> | A query-dependent sparse-attention method that, at each decoding step, selects only the most relevant KV pages to attend over, accelerating long-context decode. *It keeps per-page min/max key summaries and uses the current query to estimate page criticality, loading and attending to only the top-k pages while leaving the rest in memory unused.* | latency, throughput, context-capacity | research | Tang et al., Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024 |
| **Retrieval-head-aware KV eviction (RazorAttention / retrieval heads)** | Using the discovery that a few 'retrieval heads' carry long-range copying to protect their KV while compressing the rest. *Identify retrieval heads (those that drive copy/induction over long range) and exempt their KV from eviction/compression, preserving long-context recall under a tight budget.* | memory, context-capacity | research | Tang et al., 2024 (RazorAttention); Wu et al., 2024 (Retrieval Heads) |
| **StreamingLLM (attention sinks)** <br/><sub>attention sinks, rolling KV with sink tokens</sub> | An inference scheme enabling effectively unbounded streaming generation by keeping a small fixed set of initial 'sink' tokens plus a sliding window of recent KV. *It retains the first few tokens (which absorb disproportionate attention as 'sinks') alongside a rolling recent-token window, preventing the perplexity blow-up that naive window truncation causes.* | memory, context-capacity, latency | production-standard | Xiao et al., Efficient Streaming Language Models with Attention Sinks, ICLR 2024 |

# Part III — Action, planning & orchestration

## 9. Tool use & function calling optimization

An agent is only as good as its tool calls. These methods make tool use cheaper and more reliable: select the right tools from a large catalog, design schemas the model can follow, call in parallel, constrain output to valid calls, cache results, and recover from errors. Cross-cut: fak's [grammar-constrained tool-call decoding](RESEARCH-grammar-constrained-tool-call-decoding-2026-06-22.md) and the `internal/vdso` tool-result fast path.

*26 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Code generation for API composition (programmatic tool use)** <br/><sub>program-of-thought tool use, ViperGPT-style, API graph as code</sub> | Generate a program that calls multiple APIs/tools and intermediate values, instead of one call at a time, for complex multi-tool tasks. *The model writes a script binding tool outputs to variables and passing them between calls, so multi-tool dependencies are resolved by execution rather than by re-prompting.* | accuracy/quality, cost/tokens, autonomy | emerging | Surís et al., ViperGPT, 2023; Gao et al., PAL, 2022 |
| **Code-as-Action (CodeAct)** <br/><sub>CodeAct, code actions, executable action agents</sub> | Have the agent express its actions as executable code (e.g., Python) rather than discrete JSON tool calls, letting one code block orchestrate many tool invocations, control flow, and data handling. *Unifies the action space into a code interpreter so the LLM composes tools with loops/conditionals/variables in a single executable action, reducing round-trips.* | accuracy/quality, latency, cost/tokens, autonomy | emerging | Wang et al., 'Executable Code Actions Elicit Better LLM Agents' (CodeAct), 2024 |
| **Demonstration / few-shot tool-call exemplars** <br/><sub>in-context tool examples, few-shot function calling</sub> | Provide worked examples of correct tool calls (and tricky edge cases) in the prompt to steer the model's tool selection and argument formatting. *In-context demonstrations bias the model toward the demonstrated call format and selection logic without weight updates.* | accuracy/quality, reliability | production-standard | Brown et al., in-context learning, 2020; function-calling best-practice guides, 2023-2024 |
| **Forced / directed tool choice** <br/><sub>tool_choice=required, function_call forcing, any/auto/none tool choice</sub> | Constrain whether the model must call a tool, may choose, must call a specific tool, or must not call any tool on a given turn. *An API parameter (auto/any/required/none/specific) biases or hard-restricts the decode toward the desired tool-calling behavior.* | reliability, accuracy/quality, latency | production-standard | OpenAI tool_choice / Anthropic tool_choice docs, 2023-2024 |
| **Gorilla / API-aware retriever finetuning** <br/><sub>Gorilla, RAT (retriever-aware training)</sub> | A finetuned LLM specialized for generating correct API/tool calls against large, evolving API catalogs, optionally coupled with a document retriever at inference. *Trains the model on (instruction, API doc, call) triples and uses retriever-aware training so the model adapts to retrieved docs and reduces hallucinated API calls.* | accuracy/quality, reliability | research | Patil et al., 'Gorilla: Large Language Model Connected with Massive APIs', 2023 |
| **Irrelevance / no-call detection** <br/><sub>abstention from tool use, relevance gating, should-not-call detection</sub> | Train or prompt the model to recognize when no tool is needed (or no available tool fits) and answer directly instead of forcing an inappropriate call. *Adds negative/irrelevance examples and a 'none' option so the policy learns to abstain, reducing spurious and unsafe tool invocations.* | accuracy/quality, reliability, cost/tokens | production-standard | BFCL irrelevance category (Yan et al., 2024); function-calling best practices, 2024 |
| **Lifting text/inline tool calls at the boundary** <br/><sub>XML/inline tool-call parsing, tool-call extraction, non-native function calling</sub> | Parse tool calls emitted as inline text (XML, JSON-in-text, or DSL) for models without native function-calling, normalizing them into structured calls at the runtime boundary. *A parser at the gateway extracts the structured call from free-form generation (e.g., qwen3_coder XML), so any model can drive the tool loop.* | reliability, accuracy/quality | production-standard | SGLang/vLLM tool-call parsers (e.g., qwen3_coder, hermes), 2024 |
| **LLM-as-Compiler / LLMCompiler parallel tool orchestration** | Planning a DAG of tool calls up front and dispatching independent calls in parallel with a task fetcher and executor. *A planner emits an inter-dependent task graph; a fetching unit resolves arguments and an executor runs independent nodes concurrently, cutting latency and token cost vs sequential ReAct.* | latency, cost/tokens, accuracy/quality | emerging | Kim et al., 2023 (An LLM Compiler for Parallel Function Calling) |
| **Memoized / deduplicated tool execution caching** <br/><sub>tool call memoization, idempotent-call cache, result dedup</sub> | Cache the outputs of deterministic/idempotent tool calls keyed by (tool, args) so repeated identical calls skip re-execution. *A keyed cache short-circuits the round-trip to the external tool when the same call signature recurs, returning the stored result.* | latency, cost/tokens, throughput | production-standard | Common agent-framework practice (LangChain/LlamaIndex tool caching), 2023-2024 |
| **Model Context Protocol (MCP) / tool standardization** <br/><sub>MCP, tool server protocol, open tool standard</sub> | An open protocol standardizing how agents discover and call external tools, resources, and prompts via interchangeable servers, decoupling tool implementations from the agent. *Defines a client-server JSON-RPC interface for listing tools/resources and invoking them, so any compliant host can use any compliant tool server.* | autonomy, reliability | production-standard | Anthropic, 'Model Context Protocol', 2024 |
| **Parallel tool calling** <br/><sub>parallel function calling, multi-tool calls, batched tool calls</sub> | Allowing the model to emit multiple independent tool calls in a single turn so they can be executed concurrently rather than sequentially. *The model returns a list of tool_use blocks in one response; the runtime dispatches the independent calls in parallel and feeds all results back together.* | latency, throughput, cost/tokens | production-standard | OpenAI parallel function calling, 2023; Anthropic tool use docs, 2024 |
| **ReAct** <br/><sub>Reason+Act, reasoning and acting, thought-action-observation loop</sub> | An agent framework that interleaves free-form reasoning ('thoughts') with tool/environment actions and observations in a single loop. *Alternating reasoning traces and actions lets the model plan, ground reasoning in external observations, and recover from errors, making it the backbone of tool-using agents.* <br/><sub>↔ also: Prompting & reasoning strategies</sub> | accuracy/quality, reliability, autonomy | production-standard | Yao et al., 2022, 'ReAct: Synergizing Reasoning and Acting in Language Models' |
| **Reinforcement learning for tool use** <br/><sub>ToolRL, RL tool-use training, execution-feedback RL</sub> | Train tool-using policies with reinforcement learning where rewards come from tool-execution outcomes or task success rather than imitation alone. *Uses outcome/format rewards from real tool execution to optimize when, which, and how the model calls tools via policy-gradient methods.* | accuracy/quality, reliability, autonomy | research | Qian et al., ToolRL, 2025; Feng et al., ReTool, 2025 |
| **Retry with backoff / circuit breaking for flaky tools** <br/><sub>tool retry policy, exponential backoff, circuit breaker</sub> | Operational policies that retry transient tool failures with backoff and trip a circuit breaker for persistently failing tools to keep the agent loop stable. *The runtime wraps tool calls with retry/backoff and failure-rate thresholds that disable or route around a degraded tool.* | reliability, latency | production-standard | Standard distributed-systems practice applied to agent tool calls, 2023-2024 |
| **Sandboxed code-action execution** <br/><sub>secure code interpreter, tool sandboxing, containerized actions</sub> | Execute code-as-action and tool side effects inside an isolated, resource-limited sandbox to contain errors and untrusted behavior. *Runs generated code/tool calls in a constrained container or VM with restricted filesystem/network and timeouts, returning captured stdout/errors as observations.* | reliability, autonomy | production-standard | Code-interpreter / E2B / sandbox patterns, 2023-2024 |
| **Speculative / predictive tool prefetching** <br/><sub>tool prefetch, speculative tool execution, anticipatory calls</sub> | Begin executing likely-needed tool calls before the model formally requests them, or speculatively run candidate calls, to hide latency. *A predictor anticipates the next tool from context and kicks off execution in the background, discarding wasted work if the prediction is wrong.* | latency, throughput | research | Emerging agent-serving optimization; analogous to speculative execution, 2024-2025 |
| **Tool masking / dynamic gating** <br/><sub>tool gating, allowed-tools constraint, state-dependent tool masking</sub> | Restrict, at each step, which tools the model is permitted to call based on agent state, permissions, or workflow phase. *The runtime supplies or logit-masks only the currently-valid tools (or rejects out-of-policy calls), shrinking the choice space and enforcing safety/sequencing.* | accuracy/quality, reliability, cost/tokens | production-standard | Agent-framework practice; OpenAI/Anthropic tool_choice controls, 2024 |
| **Tool pruning / fewer-tools curation** <br/><sub>tool deduplication, minimal toolset, tool consolidation</sub> | Reduce the number of exposed tools by merging overlapping ones, removing rarely-used ones, or designing a small high-coverage toolset. *Curates the catalog so fewer, more orthogonal tools are presented, shrinking the selection space and the schema token cost while improving disambiguation.* | accuracy/quality, cost/tokens, context-capacity | production-standard | Practitioner guidance (Anthropic/OpenAI tool-design best practices), 2024 |
| **Tool Retrieval / RAG-based tool selection** <br/><sub>tool RAG, dynamic tool retrieval, retrieval-augmented tool selection</sub> | At scale (hundreds to thousands of tools), retrieve only the top-k most relevant tool schemas for a query and inject just those into the prompt instead of the full tool catalog. *Embeds the user query and tool descriptions into a vector space (or uses a learned retriever) and selects the nearest tools so the model sees a small, relevant subset.* | accuracy/quality, cost/tokens, latency, context-capacity | production-standard | Patil et al., Gorilla, 2023; Qin et al., ToolLLM/ToolBench, 2023 |
| **Tool schema / documentation design** <br/><sub>tool description engineering, function-spec design, JSON-schema tooling</sub> | Authoring clear tool names, parameter descriptions, types, enums, examples, and constraints so the model selects and fills tools correctly. *Encodes intent and constraints directly into the function schema (descriptive names, required/optional flags, enums, in-line examples) that the model conditions on at call time.* | accuracy/quality, reliability, cost/tokens | production-standard | Anthropic & OpenAI function-calling / tool-use documentation, 2023-2024 |
| **Tool-call validation / argument schema enforcement** <br/><sub>argument validation, post-hoc schema check, guardrail validation</sub> | Validate generated tool arguments against the schema (types, ranges, required fields, enums) before execution, rejecting or repairing invalid calls. *A validator parses the emitted arguments against the declared JSON schema and either auto-repairs, re-prompts, or blocks malformed calls.* | reliability, accuracy/quality | production-standard | Pydantic/JSON-schema validation in agent frameworks (e.g., Instructor, Guardrails), 2023-2024 |
| **Tool-result compression / truncation** <br/><sub>observation compression, result summarization, tool output trimming</sub> | Compress, summarize, paginate, or truncate large tool outputs before they re-enter the context so they don't blow the context budget. *Applies extractive/abstractive summarization, head+tail windowing, pointers/handles, or schema-projection to the tool result prior to feeding it back to the model.* | cost/tokens, context-capacity, latency | production-standard | Agent-framework practice; LLMLingua-style compression (Jiang et al., 2023) |
| **Tool-use benchmarking / capability gating** <br/><sub>BFCL (Berkeley Function-Calling Leaderboard), API-Bank, ToolBench eval</sub> | Standardized benchmarks that measure tool-selection accuracy, argument correctness, parallel/sequential calling, and irrelevance detection, used to gate model/prompt choices. *Scores models on held-out tool-calling tasks (including 'should not call') so practitioners select models and prompts empirically rather than by intuition.* | accuracy/quality, reliability | production-standard | Berkeley Function-Calling Leaderboard (Yan et al., 2024); Li et al., API-Bank, 2023; tau-bench (Yao et al., 2024) |
| **Toolformer** | A self-supervised method that teaches a model to decide which APIs to call, when, and with what arguments by inserting API calls into its own training text. *Samples candidate API calls in-context, keeps only those whose results reduce LM loss on subsequent tokens, then fine-tunes on the filtered, API-annotated corpus.* <br/><sub>↔ also: Training & adaptation for agentic capability</sub> | accuracy/quality, autonomy | research | Schick et al., 2023, 'Toolformer: Language Models Can Teach Themselves to Use Tools' |
| **ToolkenGPT (tools as learned tokens)** <br/><sub>toolkits, tool namespaces, meta-tools</sub> | Representing each tool as a special 'toolken' embedding so the model can call tools by predicting a token, without per-tool fine-tuning of the whole model. *Tool embeddings are added to the vocabulary; predicting a toolken triggers the call and switches to argument-filling mode, enabling massive tool sets via plug-in embeddings.* | accuracy/quality, context-capacity, cost/tokens | research | Hao et al., 2023 (ToolkenGPT) |
| **ToolLLM / ToolBench (large-scale tool-use instruction tuning)** <br/><sub>ToolLLM, ToolBench, DFSDT</sub> | A framework and dataset for teaching LLMs to use 16k+ real-world REST APIs, including a depth-first search-based decision tree (DFSDT) for multi-step tool reasoning. *Generates multi-tool instruction data with ChatGPT and trains models to plan and call tools, using DFSDT to expand and backtrack over candidate tool-call paths.* | accuracy/quality, autonomy, reliability | research | Qin et al., 'ToolLLM', 2023 |

## 10. Planning algorithms & world models for agents

Decide *what to do* before doing it. These methods span hybrid classical planners, tree-search over actions, and plan caching — separating deliberation from execution so the agent commits fewer, better-chosen steps.

*26 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **AdaPlanner** <br/><sub>Adaptive Planner</sub> | A closed-loop LLM agent that adaptively refines its self-generated plan in response to environment feedback, supporting both in-plan and out-of-plan refinement. *Generates a program-style plan, then uses two refiners that correct mistakes mid-trajectory and inject skills/code-based feedback, mitigating hallucination via a skill-discovery mechanism.* | accuracy/quality, reliability, autonomy | research | Sun et al., 2023, 'AdaPlanner: Adaptive Planning from Feedback with Language Models' |
| **ADaPT (As-Needed Decomposition and Planning)** <br/><sub>ADaPT</sub> | Recursively decomposes a task only when the executor fails, adapting decomposition depth to task complexity and the LLM's capability. *Interleaves a planner that splits a subtask into sub-subtasks on demand with an executor, avoiding over-decomposition of easy tasks while handling hard ones.* | accuracy/quality, cost/tokens, reliability | research | Prasad et al., 2023, 'ADaPT: As-Needed Decomposition and Planning with Language Models' |
| **Code as Policies** <br/><sub>CaP</sub> | Uses LLM-written code as the policy: the model emits executable robot-control programs that invoke perception and control APIs to realize a plan. *Leverages code generation to express plans with precise logic, parameterization, and feedback handling, executing directly against an API layer.* | reliability, accuracy/quality, autonomy | research | Liang et al., 2022, 'Code as Policies: Language Model Programs for Embodied Control' |
| **Describe, Explain, Plan and Select (DEPS)** <br/><sub>DEPS</sub> | An interactive planning approach for open-world multi-task agents that describes the current plan-execution state, explains feedback on failures, replans, and selects the nearest feasible subgoal. *A selector ranks parallel candidate subgoals by estimated proximity/feasibility and the describe-explain loop corrects long-horizon plans against environment feedback.* | accuracy/quality, reliability, autonomy | research | Wang et al., 2023, 'Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents' |
| **Dynamic replanning** <br/><sub>Online replanning, Closed-loop replanning</sub> | Revising or regenerating the agent's plan during execution whenever observations diverge from expectations or a step fails, rather than committing to a fixed open-loop plan. *Monitors execution feedback and triggers partial or full plan regeneration on deviation, turning open-loop planning into closed-loop control.* | reliability, autonomy, accuracy/quality | production-standard | Inner Monologue, Huang et al., 2022; general closed-loop agent control |
| **Generative-agent reflection-and-planning memory** <br/><sub>Generative Agents, Reflect-and-plan memory stream</sub> | Believable agents that synthesize a memory stream into higher-level reflections and daily/hierarchical plans which guide and are recursively refined during behavior. *Periodically reflects over salient memories to form abstractions, then generates broad-to-detailed plans that are reactively adjusted as new observations arrive.* | autonomy, accuracy/quality, context-capacity | research | Park et al., 2023, 'Generative Agents: Interactive Simulacra of Human Behavior' |
| **Goal decomposition / task decomposition** <br/><sub>Subgoal decomposition, Task splitting</sub> | Breaking a complex high-level goal into a set or sequence of smaller, individually solvable subgoals that the agent tackles one at a time. *Reduces a hard reasoning/action problem into easier subproblems whose solutions compose, lowering per-step difficulty and error rate.* | accuracy/quality, reliability, context-capacity | production-standard | Least-to-Most Prompting, Zhou et al., 2022; HuggingGPT, Shen et al., 2023 |
| **Hierarchical Task Network planning** <br/><sub>HTN, HTN planning</sub> | Planning by recursively decomposing high-level compound tasks into ordered subtasks via predefined methods until only primitive executable actions remain. *Uses a library of task-decomposition methods to expand abstract goals into concrete action sequences, constraining search to author-validated structures.* | reliability, accuracy/quality | production-standard | Erol, Hendler & Nau, 1994 (SHOP/SHOP2 lineage); applied to LLM agents |
| **LATS (Language Agent Tree Search)** <br/><sub>LATS, language agent tree search, MCTS over agents</sub> | A framework unifying reasoning, acting, and planning by running Monte Carlo Tree Search over language-agent trajectories with environment feedback and self-reflection. *Uses MCTS selection/expansion/evaluation/backpropagation where the LLM proposes actions, a value function scores nodes, and reflections from failed branches improve future rollouts.* <br/><sub>↔ also: Multi-agent orchestration patterns</sub> | accuracy/quality, reliability, autonomy | research | Zhou et al., 2023, 'Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models' |
| **LLM+P (LLM + Classical Planner)** <br/><sub>LLMP, LLM plus Planner</sub> | A hybrid that translates a natural-language task into a formal PDDL problem, hands it to a sound classical planner, and translates the returned plan back to natural language. *Offloads the combinatorial search to a provably correct symbolic planner (e.g., Fast Downward) so the LLM only does NL-to-PDDL translation, not the planning itself.* | accuracy/quality, reliability | research | Liu et al., 2023, 'LLM+P: Empowering Large Language Models with Optimal Planning Proficiency' |
| **LLM-DP (LLM Dynamic Planner)** <br/><sub>LLM-DP</sub> | A neuro-symbolic embodied agent that combines an LLM with a symbolic planner, using the LLM to hypothesize world state and the planner to produce valid action sequences. *The LLM proposes plausible PDDL world states/goals from observations and a classical planner solves them, leveraging the LLM's commonsense plus planner soundness for fast valid plans.* | accuracy/quality, latency, reliability | research | Dagan et al., 2023, 'Dynamic Planning with a LLM' |
| **Monte Carlo Tree Search for LLM planning** <br/><sub>MCTS planning, MCTS-for-LLM</sub> | Applying the classic selection-expansion-simulation-backpropagation search loop to LLM-generated action/reasoning trees to find high-value trajectories. *Balances exploration and exploitation via UCT over LLM-proposed branches, using a learned or LLM value estimate as the rollout reward to prune low-value paths.* | accuracy/quality, reliability | research | Kocsis & Szepesvari, 2006 (UCT); applied to LLMs in e.g. Hao et al., 2023 (RAP) |
| **Monte Carlo Tree Self-refine / AlphaLLM-style self-improvement** <br/><sub>AlphaLLM, MCTSr</sub> | Couples MCTS with LLMs in a self-improving loop, using tree search to generate high-quality trajectories that train a better policy/value, then re-searching. *Tree search with critic-based value estimation produces refined solution trajectories used as training/self-correction signal, iteratively bootstrapping planning quality.* | accuracy/quality, reliability | research | Tian et al., 2024, 'Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (AlphaLLM)'; Zhang et al., 2024 (MCTSr) |
| **Plan caching / plan templates** <br/><sub>Plan reuse, Plan library, Cached plans</sub> | Storing previously generated successful plans (or parameterized plan templates) and retrieving/reusing them for similar future tasks instead of replanning from scratch. *Matches a new task to a cached plan by similarity and instantiates the template, amortizing planning cost and latency over repeated task types.* | latency, cost/tokens, reliability | emerging | Case-based planning lineage (Hammond, 1989); LLM agent plan-reuse patterns |
| **Plan-and-Solve Prompting** <br/><sub>PS, PS+</sub> | A zero-shot prompting strategy that asks the model to first devise a plan dividing the task into subtasks, then carry out the subtasks step by step. *Explicitly separates the 'understand and plan' phase from the 'execute' phase in a single prompt, reducing missing-step and calculation errors versus plain chain-of-thought.* <br/><sub>↔ also: Prompting & reasoning strategies</sub> | accuracy/quality, reliability | emerging | Wang et al., 2023, 'Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning' |
| **Planner-Executor (Plan-and-Execute)** <br/><sub>Plan-and-Solve, Plan-and-Execute, decompose-then-act</sub> | A two-role pattern where one agent produces a multi-step plan up front and a separate executor agent (or loop) carries out each step, optionally re-planning when steps fail. *Separating planning from execution lets a strong model reason globally once, then delegate cheap per-step execution, reducing redundant re-reasoning and bounding wandering.* <br/><sub>↔ also: Multi-agent orchestration patterns</sub> | accuracy/quality, cost/tokens, reliability, autonomy, latency | production-standard | Wang et al., 'Plan-and-Solve Prompting', 2023; LangChain Plan-and-Execute agents |
| **ProgPrompt (programmatic planning)** <br/><sub>ProgPrompt</sub> | Represents the plan as executable program code (functions and control flow) generated by the LLM, where each statement is a primitive action or assertion. *Prompts the LLM with Pythonic program structure so plans inherit loops/conditionals/assertions, producing structured, situated, and verifiable action sequences.* | accuracy/quality, reliability | research | Singh et al., 2022, 'ProgPrompt: Generating Situated Robot Task Plans using Large Language Models' |
| **RAP (Reasoning via Planning)** <br/><sub>Monte Carlo Tree Search reasoning, MCTS-LLM, RAP</sub> | Repurposes the LLM as both a world model and a reasoning agent, doing principled MCTS planning over a tree of reasoning states with the LLM predicting next states and rewards. *The LLM predicts the next world state given an action and self-evaluates rewards, and MCTS uses these to search a state-action reasoning tree toward high-reward solutions.* <br/><sub>↔ also: Test-time compute scaling & search over reasoning</sub> | accuracy/quality, reliability | research | Hao et al., 2023, 'Reasoning with Language Model is Planning with World Model' |
| **Reflexion-for-planning successor — AdaPlanner closed-loop refinement (explicit in-plan + out-of-plan)** | An LLM planner that refines its plan from environment feedback using both in-plan (continue) and out-of-plan (revise) corrections with code-style plans. *Skill discovery plus dual feedback channels let the agent patch the current plan locally or replan globally, raising long-horizon success with few samples.* | accuracy/quality, autonomy | research | Sun et al., 2023 (AdaPlanner) |
| **ReWOO (Reasoning WithOut Observation)** <br/><sub>ReWOO, Reasoning WithOut Observation</sub> | A plan-then-work architecture that produces the entire reasoning blueprint (a list of interdependent tool calls with variable substitution) up front, separating reasoning from observation/tool execution. *A Planner emits a full plan with placeholder variables, a Worker fills them by executing tools once, and a Solver composes the final answer, removing repeated context re-feeding per ReAct step.* <br/><sub>↔ also: Prompting & reasoning strategies</sub> | cost/tokens, latency, accuracy/quality | emerging | Xu et al., 2023, 'ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models' |
| **SayCan (grounded affordance planning)** <br/><sub>SayCan, Do As I Can, Not As I Say</sub> | Grounds LLM-proposed actions in a robot's real capabilities by combining the LLM's task-relevance score with a learned affordance value for each skill. *Selects the next skill by multiplying the LLM 'say' probability with a value-function 'can' affordance, ensuring the plan stays within feasible, executable actions.* | reliability, accuracy/quality | research | Ahn et al., 2022, 'Do As I Can, Not As I Say: Grounding Language in Robotic Affordances' |
| **ToolChain*** <br/><sub>ToolChain-star</sub> | An efficient tree-search-based planning algorithm for tool-use agents that navigates the action space with an A*-style heuristic over decision trees of API calls. *Combines tree search with task-specific cost functions (a heuristic g+h) to expand the most promising tool-call branches first, pruning expensive exhaustive exploration.* | accuracy/quality, latency, cost/tokens | research | Zhuang et al., 2023, 'ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search' |
| **Tree-of-Mixed-Thought / Thought-of-Search and learned planning programs** <br/><sub>Thought of Search, ToS</sub> | Has the LLM author the planning components themselves (successor function and goal test) as code, then runs a sound classical search algorithm with those components. *Shifts the LLM from generating plans to generating the search machinery, so a complete/sound search procedure guarantees valid plans across all instances.* | reliability, accuracy/quality, cost/tokens | research | Katz et al., 2024, 'Thought of Search: Planning with Language Models Through The Lens of Efficiency' |
| **Tree-of-Thoughts agent search successor — ToolChain* / A*-guided tool-action search (explicit A*)** | Treating multi-step tool action selection as A* search over an action tree with a learned/heuristic cost-to-go to prune the branching factor. *An admissible-ish heuristic scores partial plans so the agent expands only promising tool-action paths, cutting cost vs exhaustive tree search.* | accuracy/quality, cost/tokens | research | Zhuang et al., 2023 (ToolChain*) |
| **Tree-Planner** <br/><sub>Tree Planner</sub> | A plan-sample-then-search method that first samples a set of candidate whole plans, aggregates them into a single action tree, then grounds the tree by decision-making during execution. *Decouples expensive plan generation (sample many plans once) from cheap tree-grounded decision-making, cutting redundant LLM calls and correcting mistakes via the merged action tree.* | cost/tokens, accuracy/quality, reliability | research | Hu et al., 2023, 'Tree-Planner: Efficient Close-loop Task Planning with Large Language Models' |
| **World-model / look-ahead simulation** <br/><sub>Model-based look-ahead, Mental simulation</sub> | Using a learned or LLM-internalized model of environment dynamics to simulate the consequences of candidate actions before committing to one. *Predicts future states/rewards for hypothetical action rollouts and selects actions by their simulated outcomes, enabling planning without real-environment trial-and-error.* | accuracy/quality, reliability, cost/tokens | research | Ha & Schmidhuber, 2018 ('World Models'); LLM instantiations e.g. RAP, Hao et al., 2023 |

## 11. Multi-agent orchestration patterns

Decompose work across several agents — a planner and workers, a debate, a pipeline, a routed swarm. The win is parallelism, specialization, and adversarial checking; the cost is coordination overhead and duplicated context. fak's [scaling laws of agents](SCALING-LAWS-OF-AGENTS-2026-06-19.md) frames when fan-out pays.

*27 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Agent-as-Tool** <br/><sub>agent-as-function, sub-agent tool, callable agent</sub> | An entire agent is exposed to a calling agent as if it were a single tool/function, hiding its internal multi-step reasoning behind one invocation. *Encapsulating a sub-agent behind a tool interface isolates its token-heavy internal context from the caller, preserving the parent's context budget while reusing the sub-agent's capability.* | context-capacity, cost/tokens, reliability, autonomy | production-standard | OpenAI Agents SDK 'agents as tools'; Anthropic subagents; LangGraph subgraph-as-tool |
| **AutoGen conversable-agent framework (programmable multi-agent conversation)** | A framework pattern of conversable agents that interleave LLM, tool, and human turns under programmable termination/auto-reply rules. *Agents exchange messages with customizable reply functions and group-chat managers, composing flexible multi-agent workflows from a small primitive set.* | autonomy, accuracy/quality | production-standard | Wu et al., 2023 (AutoGen) |
| **AutoGPT-style Autonomous Loop** <br/><sub>autonomous agent loop, self-directed goal agent, BabyAGI</sub> | A single autonomous agent that, given a high-level goal, iteratively generates tasks, executes them with tools/memory, and creates follow-up tasks until the goal is met. *A self-generating task queue plus a perceive-plan-act-memory loop lets the agent pursue open-ended goals without per-step human direction.* | autonomy | emerging | AutoGPT (Significant Gravitas, 2023); BabyAGI (Nakajima, 2023) |
| **Blackboard Architecture** <br/><sub>shared-memory coordination, blackboard systems</sub> | Agents (knowledge sources) communicate indirectly by reading from and writing to a shared structured workspace (the blackboard) rather than messaging each other directly. *A shared mutable state plus a control component lets heterogeneous specialists opportunistically contribute when the current state matches their expertise, decoupling agents.* | reliability, context-capacity, autonomy | emerging | Hayes-Roth, 'A Blackboard Architecture for Control', 1985; HEARSAY-II (Erman et al., 1980) |
| **ChatDev (Communicative Software Agents)** <br/><sub>chat-powered software dev, phased multi-agent dev</sub> | A virtual software company of communicative agents that collaborate through structured chats across design, coding, testing, and documentation phases. *Chaining phase-specific dual-agent chats with a chat-chain and self-reflection keeps each phase focused and propagates validated decisions forward.* | accuracy/quality, reliability, autonomy | research | Qian et al., 'ChatDev: Communicative Agents for Software Development', 2023 |
| **ChatEval-style role-diverse multi-agent evaluation (referee-team)** | A debate-driven evaluator where multiple role-conditioned agents discuss before scoring, improving alignment with human judgments over single LLM-judge. *Heterogeneous evaluator personas exchange critiques across rounds (one-by-one / simultaneous-talk) and converge to a verdict, reducing single-judge bias.* | accuracy/quality, reliability | research | Chan et al., 2023 (ChatEval) |
| **Contract-Net / Auction Allocation** <br/><sub>contract net protocol, CNP, bidding/auction task allocation</sub> | A manager announces a task, candidate worker agents bid based on their fitness/cost, and the manager awards the contract to the best bidder. *Market-style bidding dynamically matches tasks to the most capable or cheapest available agent, optimizing allocation without static assignment.* | cost/tokens, throughput, reliability, autonomy | research | Smith, 'The Contract Net Protocol', 1980 (applied to LLM agent task markets) |
| **Cross-Model Mixture / Model Diversity Ensemble** <br/><sub>heterogeneous-model panel, multi-LLM ensemble, model committee</sub> | An orchestration that deliberately combines agents backed by different underlying models so their complementary strengths and uncorrelated errors are pooled. *Using diverse base models reduces correlated failure modes, so aggregation (vote, debate, or synthesis) yields more reliable answers than any single model.* | accuracy/quality, reliability | research | Jiang et al., 'LLM-Blender: Ensembling LLMs with Pairwise Ranking and Generative Fusion', 2023 |
| **Dynamic / Self-Organizing Agent Topology** <br/><sub>DyLAN, GPTSwarm, learnable agent graphs</sub> | Frameworks that represent the multi-agent system as a graph and automatically learn or prune which agents participate and how they connect for a given task. *Treating the agent communication topology as an optimizable graph (importance scoring or graph optimization) removes low-value agents and edges, cutting cost while preserving quality.* | cost/tokens, accuracy/quality, throughput | research | Liu et al., 'DyLAN: Dynamic LLM-Agent Network', 2023; Zhuge et al., 'GPTSwarm', 2024 |
| **Generator-Critic (Actor-Critic / Reviewer)** <br/><sub>generator-evaluator, critic agent, evaluator-optimizer</sub> | One agent produces a candidate output and a second critic agent evaluates it against criteria, returning feedback that the generator uses to revise, looping until acceptable. *Externalizing evaluation into a dedicated critic gives an explicit acceptance signal and targeted feedback, driving iterative refinement toward higher-quality output.* | accuracy/quality, reliability | production-standard | Anthropic 'Building effective agents' (evaluator-optimizer), 2024; Madaan et al., 'Self-Refine', 2023 |
| **Group Chat (Conversational Multi-Agent)** <br/><sub>group chat manager, round-robin agents, multi-agent conversation</sub> | Multiple agents converse in a shared chat thread, with a manager (or selection policy) choosing which agent speaks next until the task is resolved. *A speaker-selection policy over a shared conversation lets agents collaborate, build on each other, and self-organize without a rigid predefined pipeline.* | accuracy/quality, autonomy, reliability | production-standard | Wu et al., 'AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation', 2023 |
| **Handoff** <br/><sub>agent handoff, transfer-to-agent, delegation handoff</sub> | An agent dynamically transfers control (and conversation state) of an in-progress task to another agent better suited to continue it. *Representing handoff as a callable transfer turns control flow into a runtime decision, letting the active agent always be the most relevant specialist without a central planner.* | accuracy/quality, autonomy, latency, reliability | production-standard | OpenAI Swarm / OpenAI Agents SDK handoffs, 2024-2025 |
| **Hierarchical / Recursive Agents** <br/><sub>agent hierarchies, recursive decomposition, nested supervisors</sub> | Agents arranged in a tree where higher-level agents spawn and supervise lower-level agents, which may themselves recursively decompose subtasks. *Recursive task decomposition matches problem structure to agent structure, letting each level operate within a bounded, locally-coherent context.* | accuracy/quality, context-capacity, autonomy, reliability | production-standard | LangGraph hierarchical agent teams; AutoGen GroupChat nesting; classic HTN planning lineage |
| **LLM-as-Judge Orchestration** <br/><sub>judge agent, verifier agent, pairwise judge</sub> | A dedicated judge agent scores, ranks, or accepts/rejects the outputs of other agents against a rubric to gate or select the final answer. *A separate evaluation model applies explicit criteria to candidate outputs, providing a selection/gating signal that filters out low-quality or unsafe results.* | accuracy/quality, reliability | production-standard | Zheng et al., 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena', 2023 |
| **Map-Reduce over Agents** <br/><sub>fan-out/fan-in, agent map-reduce, parallel decomposition</sub> | A task is split into independent shards processed in parallel by worker agents (map), whose partial results are then aggregated by a reducer agent (reduce). *Embarrassingly-parallel decomposition fans work across agents simultaneously and merges results, cutting wall-clock latency and letting each shard fit a smaller context.* | latency, throughput, context-capacity, accuracy/quality | production-standard | LangGraph map-reduce (Send API); classic MapReduce (Dean & Ghemawat, 2004) applied to agents |
| **MetaGPT / SOP-Encoded Pipeline** <br/><sub>standardized operating procedures, assembly-line agents, meta-programming framework</sub> | A multi-agent software company where agents take human-style roles and follow encoded Standard Operating Procedures that produce structured intermediate artifacts. *Encoding SOPs and structured outputs into the agent workflow constrains communication to validated artifacts, reducing error propagation across the assembly line.* | accuracy/quality, reliability | research | Hong et al., 'MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework', 2023 |
| **Mixture-of-Agents (for cost/quality)** <br/><sub>mixture-of-agents, MoA, agent blending</sub> | A layered architecture where multiple (often open/cheaper) LLMs each produce a response and an aggregator synthesizes them, sometimes matching a frontier model at lower cost. *Proposer agents in each layer generate candidate answers that downstream aggregator agents iteratively refine, exploiting collaborative diversity instead of one large model.* <br/><sub>↔ also: Self-improvement & feedback loops at inference; Model routing, cascades & cost optimization</sub> | accuracy/quality, reliability, cost/tokens | research | Wang et al., 'Mixture-of-Agents Enhances Large Language Model Capabilities', 2024 |
| **Multi-agent debate** <br/><sub>LLM debate, society of minds, self-consistency via debate</sub> | Multiple LLM instances independently answer, then read each other's answers and reasoning across several rounds, converging on a refined consensus answer. *Each agent revises its response conditioned on the other agents' arguments over multiple debate rounds, surfacing and correcting errors through cross-examination before a final answer is taken.* <br/><sub>↔ also: Self-improvement & feedback loops at inference; Test-time compute scaling & search over reasoning</sub> | accuracy/quality, reliability, autonomy | emerging | Du et al., 2023, 'Improving Factuality and Reasoning in Language Models through Multiagent Debate' |
| **Orchestrator-Worker (Manager-Workers)** <br/><sub>coordinator-worker, manager-agent, supervisor pattern</sub> | A central orchestrator agent decomposes a task and dispatches subtasks to specialized worker agents, then integrates their outputs into a final result. *A single coordinating locus assigns work dynamically and synthesizes results, enabling parallelism and clean separation of concerns while keeping one accountable controller.* | accuracy/quality, throughput, autonomy, reliability | production-standard | Anthropic 'Building a multi-agent research system', 2025; Anthropic 'Building effective agents', 2024 |
| **Parallelization (Sectioning & Voting)** <br/><sub>voting ensemble, sectioning, self-consistency across agents</sub> | The same task is run across multiple agents in parallel either by splitting into independent sections or by sampling several attempts and aggregating via vote/merge. *Aggregating diverse parallel attempts (majority vote or merge) cancels independent errors and improves both robustness and latency versus serial retries.* | accuracy/quality, reliability, latency | production-standard | Anthropic 'Building effective agents' (parallelization), 2024; Wang et al., 'Self-Consistency', 2022 |
| **Prompt-Chaining (Sequential Pipeline)** <br/><sub>chain workflow, sequential agents, pipeline of agents</sub> | A fixed sequence of agents/LLM calls where each step's output feeds the next, optionally with programmatic gates between steps. *Decomposing a task into ordered subtasks lets each step be simpler and individually validated, trading a little latency for higher per-step accuracy.* | accuracy/quality, reliability | production-standard | Anthropic 'Building effective agents' (prompt chaining), 2024 |
| **Role Specialization (Persona / Expert Agents)** <br/><sub>role-play agents, expert personas, specialized roles</sub> | Distinct agents are assigned focused roles or personas (e.g., coder, tester, reviewer, researcher) with tailored prompts, tools, and context windows. *Narrowing each agent's scope concentrates relevant instructions and tools per agent, raising per-task quality and keeping each context window focused.* | accuracy/quality, context-capacity, reliability | production-standard | Qian et al., 'ChatDev: Communicative Agents for Software Development', 2023; CAMEL (Li et al., 2023) |
| **Role-Playing Cooperative Agents (Instructor-Assistant)** <br/><sub>CAMEL, inception prompting, user-assistant role play</sub> | Two agents are assigned complementary roles (e.g., a task-giving user and a task-solving assistant) and autonomously converse to complete a task without further human input. *Inception prompting fixes the roles so the agents drive each other forward, generating a self-sustaining cooperative dialogue toward task completion.* | autonomy, accuracy/quality | research | Li et al., 'CAMEL: Communicative Agents for Mind Exploration of Large Scale Language Model Society', 2023 |
| **Routing / Dispatch** <br/><sub>router agent, intent routing, classifier-router</sub> | A lightweight router classifies an incoming request and forwards it to the most appropriate specialized agent, model, or workflow. *Front-loading a cheap classification step sends each input to the cheapest sufficient handler, cutting cost and latency versus routing everything to the strongest agent.* | cost/tokens, latency, accuracy/quality, throughput | production-standard | Anthropic 'Building effective agents' (routing workflow), 2024; RouteLLM (Ong et al., 2024) |
| **Self-Collaboration / Single-Model Multi-Persona** <br/><sub>solo performance prompting, SPP, single-LLM multi-role</sub> | A single model simulates multiple cooperating personas within one context to gain multi-agent benefits without running separate model instances. *Eliciting distinct personas inside one model's context provides internal division of labor and self-critique while avoiding the cost of multiple agent processes.* | accuracy/quality, cost/tokens | research | Wang et al., 'Unleashing the Emergent Cognitive Synergy in LLMs: Solo Performance Prompting', 2023; Dong et al., 'Self-Collaboration Code Generation', 2023 |
| **Shared Long-Term Memory / Knowledge Base Coordination** <br/><sub>shared memory store, collective memory, vector-store coordination</sub> | Agents read and write a persistent shared memory (vector store, KB, or scratchpad) so knowledge produced by one agent is reusable by others across time. *Externalizing experience into a queryable shared store lets agents reuse prior findings instead of recomputing, saving tokens and improving cross-agent consistency.* | cost/tokens, memory, context-capacity, accuracy/quality | emerging | Park et al., 'Generative Agents', 2023 (memory stream); Generative Agents shared memory patterns |
| **Stigmergic Coordination (Environment-Mediated)** <br/><sub>stigmergy, trace-based coordination, shared-artifact coordination</sub> | Agents coordinate indirectly by leaving and reacting to traces in a shared environment or artifact store rather than communicating directly. *Modifications to shared state act as implicit signals that trigger other agents' actions, enabling decentralized self-organization without explicit messaging.* | throughput, autonomy, reliability | research | Theraulaz & Bonabeau, 'A Brief History of Stigmergy', 1999 (applied to LLM agent file/scratchpad coordination) |

## 12. Context engineering & agent-design patterns

Production patterns (mostly 2024–2025, from teams shipping agents at scale) for shaping context so the *rest* of the stack works better — keeping a cacheable stable prefix, masking tools instead of removing them, isolating sub-agent contexts, reciting goals to fight drift. The practitioner-facing complement to the academic methods above.

*5 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Context offloading via structured note-taking to a scratchpad tool (Anthropic 'think' tool)** | Giving the agent an explicit no-op 'think' / note tool to externalize intermediate reasoning and policy reminders mid-trajectory. *A dedicated tool call records reasoning/plan to the transcript without side effects, improving multi-step tool-use adherence and reducing dropped constraints.* | accuracy/quality, reliability | production-standard | Anthropic, 2025 (The 'think' tool) |
| **Prompt-cache-friendly prompt ordering (stable prefix / append-only design)** | Structuring prompts so static content (system, tools, few-shot) is a fixed prefix and only volatile content changes at the tail, maximizing prefix-cache hits. *Keeping the cacheable prefix byte-stable and appending new turns at the end lets the serving stack reuse KV for the prefix, cutting TTFB and input cost.* | cost/tokens, latency | production-standard | Provider prompt-caching best practices (OpenAI/Anthropic, 2024); 'KV-cache hit rate' agent design (Manus, 2025) |
| **Recitation / re-stating goals to combat goal drift (todo.md recitation)** | Periodically rewriting the task objective and remaining plan into the recent context so attention stays on the goal over long trajectories. *Re-injecting an updated todo/plan at the tail exploits recency to keep the global objective salient, reducing lost-in-the-middle goal drift on long agent runs.* | reliability, accuracy/quality | emerging | Manus context-engineering writeup, 2025 (recitation / todo.md) |
| **Sub-agent context isolation / orchestrator-with-clean-subcontexts (Claude Code / Cognition pattern)** | Spawning subagents that each receive a narrowly scoped fresh context and return only a compact result, keeping the orchestrator's window clean. *Heavy exploration/tool output is confined to a child context; only a distilled summary returns, preventing context pollution and preserving the main thread's budget.* | context-capacity, cost/tokens, accuracy/quality | production-standard | Anthropic multi-agent research system, 2025; Cognition 'Don't Build Multi-Agents' (context-engineering), 2025 |
| **Tool masking over tool removal for stable cache (logit-masked action space)** | Keeping the full tool definitions in the (cached) context and constraining the available actions via decode-time logit masking instead of editing the tool list. *Masking disallowed tools at the token level preserves the cacheable tool-definition prefix while still gating actions per state, avoiding cache invalidation.* | cost/tokens, latency, reliability | emerging | Manus context-engineering writeup, 2025 (mask, don't remove) |

# Part IV — Cost, serving & training

## 13. Model routing, cascades & cost optimization

Not every query needs the biggest model. These methods send each request to the cheapest model that can handle it — cascades that escalate on low confidence, learned routers, semantic caches that skip the model entirely, and token-budget controls.

*22 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **AutoMix (self-verification mixing)** <br/><sub>AutoMix, self-verify-and-route</sub> | An approach where a smaller model both answers and few-shot self-verifies its own answer, and a meta-verifier (POMDP-based) decides whether to route the query to a larger model. *Noisy self-verification signals feed a Markov-decision-process meta-controller that learns the cost-optimal escalation policy across model sizes.* | cost/tokens, accuracy/quality | research | Madaan et al., 'AutoMix: Automatically Mixing Language Models', 2023 |
| **Batch / off-peak request scheduling** <br/><sub>batch API, asynchronous batch, off-peak discount routing</sub> | Deferring latency-tolerant requests to a discounted batch/asynchronous tier instead of synchronous real-time inference. *Requests are queued and processed in bulk during off-peak windows at a reduced per-token price (commonly ~50% off provider batch APIs).* | cost/tokens, throughput | production-standard | OpenAI Batch API / Anthropic Message Batches, 2024 |
| **Cache-augmented generation (CAG)** <br/><sub>CAG, preloaded-context caching</sub> | Preloading a fixed knowledge corpus into the model's context and precomputing its KV cache so repeated queries against that knowledge skip both retrieval and prefill recomputation. *The corpus KV state is computed once and reused across queries, replacing per-query retrieval+prefill with a cache load.* | cost/tokens, latency, throughput | emerging | Chan et al., 'Don't Do RAG: Cache-Augmented Generation', 2024 |
| **Cascade threshold / abstention tuning** <br/><sub>deferral threshold optimization, confidence-threshold cascade, Wisdom of Committees</sub> | Optimizing the per-stage confidence/deferral thresholds (and which confidence signal to use) that govern when a cascade stops vs escalates. *Thresholds on the model's confidence (or a post-hoc calibrated score) are tuned on validation data to hit a target cost-accuracy operating point along the deferral curve.* | cost/tokens, accuracy/quality, reliability | emerging | Gupta et al., 'Language Model Cascades: Token-Level Uncertainty And Beyond', 2024; Wang et al. 'Wisdom of Committees', 2020 (cascade theory) |
| **Difficulty / complexity-based routing** <br/><sub>query difficulty routing, complexity routing, adaptive routing</sub> | Routing that classifies incoming queries by estimated difficulty or complexity and assigns simpler queries to smaller/cheaper models and harder queries to larger models. *A difficulty estimator (classifier, heuristic, or LLM self-assessment of required reasoning depth) produces a complexity score that maps to a model tier via thresholds.* | cost/tokens, accuracy/quality, latency | production-standard | General routing framework; see Šakota et al. 'Fly-Swat or Cannon? Cost-Effective Language Model Choice', 2024 and related routing surveys |
| **Embedding/predictive router (zero-shot model selection)** <br/><sub>semantic router, embedding router, ZOOTER</sub> | A lightweight router that maps the query embedding directly to the best-suited model among many, without running any candidate model first. *Query embeddings are scored against learned per-model competence regions (reward-distilled or nearest-neighbor) to pick the cheapest model expected to succeed.* | cost/tokens, latency, accuracy/quality | emerging | Lu et al., 'Routing to the Expert (ZOOTER)', 2023; semantic-router open-source library (Aurelio) |
| **FrugalGPT LLM cascade** <br/><sub>FrugalGPT, LLM cascade, cascade of LLMs</sub> | A query is sent to a sequence of progressively more expensive LLMs, and a learned scorer decides at each stage whether the cheap model's answer is good enough to return or whether to escalate to the next model. *A per-query 'generation scorer' (DistilBERT-style reliability predictor) accepts the cheapest answer above a learned threshold, stopping the cascade early so only hard queries reach the expensive model.* | cost/tokens, accuracy/quality, latency | research | Chen, Zaharia & Zou, 'FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance', 2023 |
| **Hybrid LLM routing (cost-aware quality threshold)** <br/><sub>Hybrid LLM, router with desired quality level</sub> | A router trained to send only the queries that the small model would handle as well as the large model to the small model, exposing a knob for the user's tolerable quality drop. *A BERT-based router predicts per-query quality gap between models and routes to the cheap model when the predicted drop is within the allowed budget.* | cost/tokens, accuracy/quality | research | Ding et al., 'Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing', 2024 |
| **LLM ensembling for cost/quality** <br/><sub>model ensembling, ensemble routing, LLM blending</sub> | Combining outputs from several (often smaller/cheaper) models to reach quality competitive with a single large model at lower aggregate cost. *Multiple models answer and a fusion/voting/selection step picks or merges responses, trading parallel cheap calls for one expensive call.* | accuracy/quality, cost/tokens, reliability | research | Jiang et al., 'LLM-Blender: Ensembling LLMs with Pairwise Ranking and Generative Fusion', 2023; 'Blending Is All You Need', 2024 |
| **Mixture-of-Thought / model-internal effort routing (reasoning-effort parameter)** | A provider-level control that routes a request to more or less internal reasoning compute (e.g. reasoning_effort low/medium/high) to trade cost for accuracy. *A single model exposes a thinking-budget knob; the request specifies effort, and the serving stack allocates more hidden reasoning tokens for hard queries only.* | cost/tokens, latency, accuracy/quality | production-standard | OpenAI o-series / Anthropic extended-thinking reasoning_effort (2024-2025) |
| **Model cascade / quantization-tier serving** <br/><sub>quantized-model cascade, precision cascade, mixed-precision tiering</sub> | Serving the same base model at multiple quantization/precision tiers and routing easy queries to the cheaper low-precision variant, escalating to higher precision when needed. *A confidence or difficulty gate selects the lowest-precision tier likely to be correct, reserving full-precision compute for hard inputs.* | cost/tokens, latency, memory, throughput | emerging | Cascade/serving design pattern; generalization of FrugalGPT applied to precision tiers |
| **Multi-LLM optimal assignment / portfolio routing** <br/><sub>LLM portfolio, Fly-Swat or Cannon, meta-model selection</sub> | Given a pool of many candidate LLMs/APIs, choose per query the model (or set) that maximizes expected quality under a cost budget, framed as an assignment/optimization problem. *A meta-model predicts each model's expected performance and price for the query and solves a constrained optimization (or scoring) to pick the best cost-quality candidate.* | cost/tokens, accuracy/quality | research | Šakota et al., 'Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling', 2024 |
| **Output-length control / token budgeting** <br/><sub>length control, token budgeting, max-token capping</sub> | Techniques that constrain or shorten model output (and sometimes reasoning) length to reduce generated tokens, which dominate cost and latency. *Caps (max_tokens), length-conditioned prompts/instructions, or fine-tuned brevity steer the decoder to stop sooner while preserving answer content.* | cost/tokens, latency | production-standard | Common practice; see length-controlled generation and concise-CoT studies (e.g. Nayab et al. 'Concise Thoughts', 2024) |
| **Provider/API cost routing (cheapest-provider arbitrage)** <br/><sub>provider routing, API arbitrage, load-balanced model routing</sub> | Routing a request among multiple providers or deployments hosting equivalent models to the cheapest, fastest, or most-available endpoint. *A gateway compares per-token price, latency, and availability across providers and dispatches to the best endpoint, with fallback on failure.* | cost/tokens, latency, reliability, throughput | production-standard | Gateway practice; OpenRouter / LiteLLM / portkey-style routing, 2023-2025 |
| **Reasoning-budget control / token-budget-aware reasoning** <br/><sub>thinking budget, reasoning effort control, TALE</sub> | Methods that allocate or cap the number of chain-of-thought / 'thinking' tokens a reasoning model spends per query, matched to the query's difficulty. *A predicted or instructed token budget bounds the reasoning trace (e.g. 'reasoning_effort' levels or an estimated optimal budget) so easy queries do not over-think.* | cost/tokens, latency, accuracy/quality | emerging | Han et al., 'Token-Budget-Aware LLM Reasoning (TALE)', 2024; provider 'reasoning effort' controls, 2024-2025 |
| **RouteLLM (learned LLM router)** <br/><sub>RouteLLM, learned router, binary strong/weak router</sub> | A trained router predicts, per query, whether a cheap (weak) model or an expensive (strong) model should answer, sending each query to exactly one model rather than running them in sequence. *Routers (matrix factorization, BERT classifier, similarity-weighted ranking) trained on preference data (e.g. Chatbot Arena) score query difficulty and dispatch to strong vs weak model at a tunable cost-quality threshold.* | cost/tokens, accuracy/quality, latency | production-standard | Ong et al., 'RouteLLM: Learning to Route LLMs with Preference Data', 2024 |
| **Self-consistency with adaptive sampling** <br/><sub>CoT-SC, self-consistency decoding, sample-and-vote</sub> | Sampling multiple reasoning paths and majority-voting the final answer, with adaptive variants that stop sampling early once the vote is confident to save tokens. *A confidence/consensus stopping rule monitors the running vote distribution and halts generation of further samples when the answer is statistically settled.* <br/><sub>↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference; Test-time compute scaling & search over reasoning; Reliability, structured generation & evaluation-as-optimization</sub> | accuracy/quality, reliability, cost/tokens | research | Wang et al., 'Self-Consistency Improves Chain of Thought Reasoning', 2022; Aggarwal et al., 'Adaptive-Consistency', 2023 |
| **Semantic caching (GPTCache)** <br/><sub>GPTCache, semantic cache, embedding cache</sub> | A cache keyed on the semantic meaning (embedding) of a request so that semantically similar prompts return a stored answer instead of re-invoking the LLM. *Incoming queries are embedded and a vector-similarity lookup above a threshold returns the cached response, skipping the model call entirely.* | cost/tokens, latency, throughput | production-standard | Bang, 'GPTCache: An Open-Source Semantic Cache for LLM Applications', 2023 (Zilliz) |
| **Small-model-first escalation** <br/><sub>model escalation, tiered fallback, weak-to-strong escalation</sub> | Always try the smallest/cheapest model first and escalate to a larger model only when the small model's answer fails a quality or confidence check. *A verifier or confidence gate after the small model's response triggers a retry with the next-larger model on failure, so escalation cost is paid only for hard cases.* | cost/tokens, latency, reliability | production-standard | Cascade/fallback design pattern; generalization of FrugalGPT and provider 'fallback' routing |
| **Speculative cascades** <br/><sub>speculative cascade, cascade + speculative decoding</sub> | A hybrid that fuses model cascades with speculative decoding: a small model drafts tokens and a deferral rule decides token-by-token whether the large model needs to verify/take over, rather than deferring whole queries. *Combines a cascade's quality-aware deferral with speculative decoding's parallel draft-and-verify, applying an optimal token-level deferral rule so the large model is invoked only where the small model is likely wrong.* | cost/tokens, latency, accuracy/quality | research | Narasimhan et al. (Google Research), 'Faster Cascades via Speculative Decoding' / speculative cascades, 2024-2025 |
| **Tool/retrieval gating for cost** <br/><sub>adaptive retrieval, should-I-retrieve gating, selective tool use</sub> | Deciding per query whether expensive auxiliary steps (retrieval, tool/API calls, web search) are needed at all, skipping them for queries the model can answer directly. *A learned or self-reflective gate (e.g. a retrieve/no-retrieve token or classifier) suppresses unnecessary context-fetching calls and their token cost.* | cost/tokens, latency, accuracy/quality | emerging | Asai et al., 'Self-RAG', 2023; adaptive-retrieval literature (e.g. Jeong et al. 'Adaptive-RAG', 2024) |
| **Verifier-gated cascade / answer verification deferral** <br/><sub>verification cascade, judge-gated deferral, quality-gate routing</sub> | A cascade where the decision to accept a cheap model's output or escalate is made by an explicit verifier (LLM-as-judge, reward model, or rule check) rather than the generating model's own score. *An independent verifier scores the candidate answer and routes below-threshold answers to a stronger model or a re-generation step.* | accuracy/quality, cost/tokens, reliability | emerging | Cascade literature; LLM-as-judge deferral (e.g. AutoMix, Madaan et al. 2023) |

## 14. Inference / serving optimizations beneath agents

The engine layer the agent rides on. Quantization, speculative decoding, batching, attention kernels, and parallelism set the throughput/latency floor — almost all implemented in the serving engine (vLLM/SGLang/llama.cpp/TensorRT-LLM), not the agent. fak's honest baseline: [SOTA serving optimizations](../explainers/sota-optimizations.md); quantization: [AWQ](../explainers/awq-quantization.md).

*31 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **AWQ** <br/><sub>Activation-aware Weight Quantization</sub> | A post-training weight-only quantization method that protects a small fraction of salient weights to preserve accuracy at low bit-widths. *Identifies salient weight channels from activation magnitude statistics and applies per-channel scaling so quantization error concentrates on less-important weights.* | memory, cost/tokens, throughput, accuracy/quality | production-standard | Lin et al., 2023 (AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration) |
| **Chunked prefill** <br/><sub>dynamic chunking, split-fuse, piggybacking</sub> | A scheduling technique that splits a long prompt's prefill into smaller chunks and interleaves them with ongoing decode steps. *Breaks prefill into token-budgeted chunks and batches them together with decode iterations so long prompts no longer stall token generation for other requests, balancing compute-bound prefill and memory-bound decode.* | latency, throughput | production-standard | Agrawal et al., 2023/2024 (SARATHI / Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve) |
| **Continuous / in-flight batching** <br/><sub>iteration-level scheduling, dynamic batching, in-flight batching</sub> | A serving scheduler that admits and retires requests at the granularity of individual decode iterations rather than whole-batch boundaries. *After each token-generation step it removes finished sequences and slots waiting requests into the running batch, keeping the GPU saturated instead of waiting for the slowest sequence in a static batch.* | throughput, latency, cost/tokens | production-standard | Yu et al., 2022 (Orca: A Distributed Serving System for Transformer-Based Generative Models) |
| **CUDA graphs** <br/><sub>graph capture, CUDA graph replay</sub> | A mechanism that records a sequence of GPU kernel launches as a graph and replays it with a single launch. *Captures the static decode-step kernel sequence once and replays the whole graph per step, eliminating per-kernel CPU launch overhead that otherwise dominates small-batch autoregressive decode.* | latency, throughput | production-standard | NVIDIA CUDA Graphs (CUDA Toolkit documentation) |
| **Disaggregated prefill/decode** <br/><sub>P/D disaggregation, prefill-decode disaggregation</sub> | A serving architecture that runs the prefill (prompt) phase and the decode (generation) phase on separate GPU pools. *Routes prompts to compute-optimized prefill workers and streams the resulting KV cache to memory-bandwidth-optimized decode workers, eliminating prefill/decode interference and letting each phase use its own parallelism and hardware.* | latency, throughput, cost/tokens | production-standard | Zhong et al., 2024 (DistServe); Patel et al., 2024 (Splitwise) |
| **EAGLE** <br/><sub>Extrapolation Algorithm for Greater Language-model Efficiency, EAGLE-2, EAGLE-3</sub> | A speculative decoding method that drafts at the feature (hidden-state) level rather than the token level for higher acceptance rates. *A small autoregressive head predicts the next hidden feature given prior features plus the sampled token, then decodes draft tokens from it and verifies with the target, with later versions adding dynamic draft trees.* | latency, throughput | emerging | Li et al., 2024 (EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty) |
| **Expert parallelism** <br/><sub>EP, MoE expert sharding</sub> | A parallelism strategy for Mixture-of-Experts models that distributes distinct experts across devices. *Places different experts on different GPUs and uses all-to-all communication to route tokens to their selected experts and gather results, scaling parameter count without scaling per-token compute.* | memory, throughput, cost/tokens | production-standard | Lepikhin et al., 2020 (GShard); Fedus et al., 2021 (Switch Transformer) |
| **FlashAttention** <br/><sub>FlashAttention-2, FlashAttention-3</sub> | An exact, IO-aware attention kernel that computes attention without materializing the full N×N score matrix in HBM. *Tiles the attention computation into SRAM and uses online softmax to fuse the softmax and matmuls in a single pass, minimizing slow HBM reads/writes; later versions add better work partitioning and FP8/async support.* | latency, memory, throughput | production-standard | Dao et al., 2022 (FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness) |
| **FlashDecoding / FlashDecoding++** <br/><sub>Flash-Decoding</sub> | An attention kernel specialization for the decode phase that parallelizes over the KV sequence length. *Splits the long key/value sequence into chunks processed in parallel and combines partial softmax results, exposing parallelism when batch and query length are tiny (single-token decode).* | latency, throughput | production-standard | Dao et al., 2023 (Flash-Decoding for long-context inference); Hong et al., 2023 (FlashDecoding++) |
| **FP8 quantization** <br/><sub>E4M3, E5M2, 8-bit floating point</sub> | Inference (and training) in 8-bit floating-point formats supported natively on Hopper/Ada and later GPUs. *Represents weights and/or activations and KV cache in FP8 (E4M3/E5M2) so tensor cores execute matmuls at 8-bit with per-tensor or finer scaling factors to preserve dynamic range.* | memory, throughput, latency, cost/tokens | production-standard | Micikevicius et al., 2022 (FP8 Formats for Deep Learning) |
| **GGUF / llama.cpp k-quants** <br/><sub>GGUF, GGML, Q4_K_M</sub> | A file format and family of mixed-precision block quantization schemes used by llama.cpp for CPU/edge and consumer-GPU inference. *Stores weights in small blocks with per-block scales (and super-block scales for k-quants), mixing bit-widths across tensors to balance size and quality for memory-constrained hardware.* | memory, cost/tokens | production-standard | Gerganov et al., llama.cpp project (GGUF format) |
| **GPTQ** <br/><sub>Generative Pre-trained Transformer Quantization</sub> | A post-training weight-only quantization method that compresses LLM weights to 3-4 bits with minimal accuracy loss. *Uses approximate second-order (Hessian-based) information to quantize weights layer-by-layer, greedily updating remaining weights to compensate for rounding error.* | memory, cost/tokens, throughput | production-standard | Frantar et al., 2022 (GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers) |
| **Hadamard / incoherence-processed quantization (QuIP and QuIP#)** | Post-training quantization that pre-processes weights with incoherence (random orthogonal) transforms to make them quantization-friendly at 2-3 bits. *Incoherence processing + lattice codebooks let weights compress to ~2 bits with adaptive rounding, retaining usable quality at extreme compression.* | memory, cost/tokens | research | Chee et al., 2023 (QuIP); Tseng et al., 2024 (QuIP#) |
| **Hydra / multi-token Medusa successors and SpecInfer tree speculation** | Tree-structured speculative decoding that verifies many candidate continuations in one forward pass via a token tree and tree attention. *A draft (small model or heads) proposes a token tree; the target verifies all branches at once with tree-attention, accepting the longest valid prefix.* | latency, throughput | emerging | Miao et al., 2023 (SpecInfer); Ankner et al., 2024 (Hydra) |
| **KV cache reuse across requests via offload + load (CacheGen / LMCache streaming load)** | Encoding and streaming precomputed KV caches between storage/network and the serving node so reused contexts skip recomputation even when not a live prefix. *KV tensors are compactly encoded and loaded on demand, trading bandwidth for prefill compute when the same context recurs across sessions/nodes.* | latency, throughput, cost/tokens | emerging | Liu et al., 2024 (CacheGen); LMCache project |
| **Lookahead decoding** <br/><sub>Jacobi decoding, lookahead</sub> | A draft-model-free parallel decoding method that breaks the sequential dependency of autoregressive generation. *Runs a parallel Jacobi-style iteration that generates and caches n-gram candidates in a lookahead branch and verifies them in a verification branch within the same step, trading FLOPs for fewer decoding steps.* | latency | emerging | Fu et al., 2024 (Break the Sequential Dependency of LLM Inference Using Lookahead Decoding) |
| **Medusa** | A self-speculative decoding method that adds multiple lightweight decoding heads to the model to predict several future tokens at once. *Extra trained heads predict tokens at positions t+1, t+2, ... and a tree-attention scheme verifies many candidate continuations in a single forward pass without a separate draft model.* | latency, throughput | emerging | Cai et al., 2024 (Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads) |
| **MoE routing (sparse gating / top-k)** <br/><sub>Mixture-of-Experts, sparse MoE, top-k gating</sub> | A sparse architecture/inference pattern where a gating network activates only a few experts per token. *A learned router selects top-k of N expert FFNs per token so only a small fraction of parameters compute per token, decoupling model capacity from per-token FLOPs; serving variants add capacity factors, load balancing, and expert caching.* | cost/tokens, throughput, accuracy/quality | production-standard | Shazeer et al., 2017 (Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer) |
| **Pipeline parallelism** <br/><sub>PP, inter-layer model parallelism</sub> | A model-parallel strategy that partitions the model's layers into stages placed on different GPUs. *Passes activations stage-to-stage and processes micro-batches in a pipeline so multiple stages stay busy simultaneously, fitting models too large for one device with lower inter-device communication than tensor parallelism.* | memory, throughput | production-standard | Huang et al., 2019 (GPipe); Narayanan et al., 2019 (PipeDream) |
| **Prompt Lookup Decoding / n-gram speculative decoding** | A draft-model-free speculative decoding that proposes continuation tokens by matching recent n-grams already present in the prompt/context. *When generation is input-grounded (RAG, code edits, summarization), candidate spans are copied from the context and verified in one pass, accelerating decode for free.* | latency, throughput | production-standard | Saxena, 2023 (Prompt Lookup Decoding); LLMA / Yang et al., 2023 (Inference with Reference) |
| **Quantization-aware / low-bit serving (W4A16, QLoRA-style)** <br/><sub>W4A16, marlin kernels, bitsandbytes NF4</sub> | End-to-end low-bit serving combining sub-4-bit weight formats with optimized mixed-precision GEMM kernels. *Pairs aggressive weight quantization (e.g., NF4, 2-3 bit via incoherence processing) with custom dequant-fused GEMM kernels (e.g., Marlin) so low-bit weights are unpacked on-chip during the matmul for near-full-precision quality at much lower memory/bandwidth.* | memory, throughput, latency, cost/tokens | emerging | Dettmers et al., 2023 (QLoRA / NF4); Tseng et al., 2024 (QuIP#); Frantar et al., 2024 (Marlin) |
| **QuaRot / SpinQuant (rotation-based outlier-free quantization)** | Applying learned/Hadamard rotations to weights and activations so outliers are spread out, enabling clean 4-bit weight+activation+KV quantization. *Rotation matrices are fused into the network to remove activation outliers, after which uniform low-bit quantization (W4A4/KV4) keeps accuracy.* | memory, throughput, cost/tokens | emerging | Ashkboos et al., 2024 (QuaRot); Liu et al., 2024 (SpinQuant) |
| **Ring Attention / Striped Attention / Blockwise context parallel (explicit)** | Distributing a single long sequence's attention across devices in a ring so context length scales with the number of accelerators near-losslessly. *Blockwise KV blocks are passed around a device ring overlapping compute with communication, enabling million-token contexts beyond single-device memory.* | context-capacity, memory, throughput | emerging | Liu et al., 2023 (Ring Attention with Blockwise Transformers); Brandon et al., 2023 (Striped Attention) |
| **Self-speculative decoding** <br/><sub>layer-skip drafting, Draft & Verify</sub> | Speculative decoding that uses the target model itself (a subset of its layers) as the draft, requiring no separate model. *Drafts tokens by skipping some intermediate/later layers (early exit) of the same model, then verifies them with the full model in one pass, accepting the consistent prefix.* | latency, memory | emerging | Zhang et al., 2023 (Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding); Elhoushi et al., 2024 (LayerSkip) |
| **Sequence / context parallelism** <br/><sub>SP, Ring Attention, context parallelism</sub> | A parallelism strategy that shards the sequence (token) dimension across devices to support very long contexts. *Each device holds a slice of the sequence and exchanges keys/values in a ring (or via all-gather) so attention can be computed over the full sequence without any single device holding all tokens.* | memory, context-capacity, throughput | production-standard | Liu et al., 2023 (Ring Attention with Blockwise Transformers for Near-Infinite Context); Korthikanti et al., 2022 (sequence parallelism in Megatron) |
| **Sequence packing / sample packing for training & prefill efficiency** | Concatenating multiple short sequences into one packed sequence (with attention masks blocking cross-doc attention) to eliminate padding waste. *A bin-packing of variable-length samples plus document-boundary attention masking raises hardware utilization without leaking attention across samples.* | throughput, cost/tokens | production-standard | Krell et al., 2021 (Efficient Sequence Packing); FlashAttention varlen packing |
| **SmoothQuant** | A post-training quantization technique enabling accurate W8A8 (8-bit weight and activation) quantization for LLMs. *Migrates quantization difficulty from hard-to-quantize activations to weights via a per-channel mathematically-equivalent scaling transformation, smoothing activation outliers.* | memory, throughput, latency | production-standard | Xiao et al., 2022 (SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models) |
| **Speculative decoding (draft model)** <br/><sub>speculative decoding, speculative sampling, draft model decoding</sub> | An exact-distribution acceleration where a small fast draft model proposes multiple tokens that the large target model verifies in parallel. *The draft model autoregressively generates a token chunk; the target model scores them in one forward pass and accepts the longest prefix consistent with its distribution via a rejection-sampling correction, preserving the target's output distribution.* <br/><sub>↔ also: Model routing, cascades & cost optimization</sub> | latency, throughput, cost/tokens | production-standard | Leviathan et al., 2023 / Chen et al., 2023 (Speculative Decoding / Accelerating LLM Decoding with Speculative Sampling) |
| **Structured pruning / sparsity** <br/><sub>2:4 sparsity, Wanda, SparseGPT</sub> | Removing weights (unstructured or hardware-friendly structured patterns like N:M) to reduce model size and compute. *Selects prunable weights by magnitude and/or activation-weighted importance, often in one shot post-training, and uses sparse tensor-core support (e.g., 2:4) to accelerate matmuls at reduced precision/density.* | memory, throughput, cost/tokens | emerging | Frantar & Alistarh, 2023 (SparseGPT); Sun et al., 2023 (Wanda: A Simple and Effective Pruning Approach for LLMs) |
| **Tensor parallelism** <br/><sub>TP, intra-layer model parallelism, Megatron-style</sub> | A model-parallel strategy that shards individual weight matrices (and their matmuls) across multiple GPUs within a layer. *Splits attention and MLP matmuls column/row-wise across devices with all-reduce/all-gather collectives to combine partial results, fitting large layers and reducing per-GPU compute/latency.* | memory, latency, throughput | production-standard | Shoeybi et al., 2019 (Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism) |
| **Weight offloading / heterogeneous inference** <br/><sub>CPU offload, FlexGen, ZeRO-Inference</sub> | Serving models larger than GPU memory by holding weights/KV in CPU RAM or disk and streaming them to the GPU as needed. *Schedules layer-by-layer transfer of weights/activations/KV between GPU, CPU, and NVMe with overlap and tensor compression, trading bandwidth for the ability to run otherwise-too-large models.* | memory, cost/tokens | production-standard | Sheng et al., 2023 (FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU); Aminabadi et al., 2022 (DeepSpeed-Inference) |

## 15. Training & adaptation for agentic capability

Bake agentic skill into the weights. Supervised tuning on trajectories, preference optimization (DPO and its family), RL with verifiable or execution rewards, self-training loops, and parameter-efficient adapters turn a base model into a competent tool-user and reasoner.

*34 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Adapter / Prefix / Prompt Tuning (PEFT family)** <br/><sub>adapters, prefix tuning, P-tuning</sub> | A family of parameter-efficient tuning methods that add small trainable modules or learned continuous prompts instead of updating full weights. *Inserts bottleneck adapter layers, learnable prefix/prompt vectors, or learned activation scalings while keeping the backbone frozen, so few parameters carry the task adaptation.* | memory, cost/tokens, throughput | production-standard | Houlsby et al., 2019 (Adapters); Li & Liang, 2021 (Prefix Tuning); Lester et al., 2021 (Prompt Tuning); Liu et al., 2022 ((IA)^3) |
| **Agentic RL with Environment Interaction** <br/><sub>agentic RL, multi-turn tool-use RL, RL on agent trajectories</sub> | Reinforcement learning where the LLM agent acts over multiple turns in a real or simulated environment (tools, code, web, OS) and is rewarded on task outcomes. *The policy rolls out multi-step tool-calling episodes against an environment, and trajectory-level rewards (with credit assignment over turns) update the policy via PPO/GRPO-style RL.* | accuracy/quality, autonomy, reliability | emerging | Carta et al., 2023 (GLAM); Bai et al., 2024 (DigiRL); SWE-RL / agent-RL line, 2024-2025 |
| **AgentTuning** <br/><sub>AgentInstruct (dataset)</sub> | An approach that fine-tunes LLMs into generalist agents using a mixed corpus of agent-interaction trajectories (AgentInstruct) plus general instructions to preserve broad ability. *Blends high-quality multi-task agent trajectories with general SFT data so agentic skill is gained without catastrophic loss of general capability.* | accuracy/quality, autonomy, reliability | research | Zeng et al., 2023, 'AgentTuning: Enabling Generalized Agent Abilities for LLMs' |
| **Constitutional AI (critique-and-revise)** <br/><sub>CAI, RLAIF critique step, self-critique against principles</sub> | A method where the model critiques and revises its own responses against an explicit set of written principles (a 'constitution'), used both at inference and to generate training data. *At the critique-revise stage, the model is prompted to find where a response violates a principle and rewrite it, replacing human harmlessness labels with principle-conditioned self-feedback.* <br/><sub>↔ also: Self-improvement & feedback loops at inference</sub> | accuracy/quality, reliability, cost/tokens | production-standard | Bai et al., 2022, 'Constitutional AI: Harmlessness from AI Feedback' (Anthropic) |
| **DAPO / Dr.GRPO (decoupled-clip and bias-corrected GRPO variants)** | GRPO refinements that fix length/difficulty bias and clipping issues to stabilize large-scale RL for long-CoT reasoning. *Decoupled clipping, dynamic sampling, token-level loss and removal of the length/std normalization bias improve sample efficiency and stop reward hacking in reasoning RL.* <br/><sub>↔ also: RL for agents & verifiable reasoning</sub> | accuracy/quality, autonomy | emerging | Yu et al., 2025 (DAPO); Liu et al., 2025 (Dr. GRPO) |
| **Direct Preference Optimization** <br/><sub>DPO</sub> | An offline preference-alignment method that fine-tunes directly on preferred/dispreferred response pairs without training a separate reward model or running online RL. *Reframes the RLHF objective as a closed-form classification loss over preference pairs, implicitly fitting a reward through the policy/reference log-ratio.* | accuracy/quality, reliability, cost/tokens | production-standard | Rafailov et al., 2023, 'Direct Preference Optimization' |
| **DoRA (weight-decomposed low-rank adaptation)** | A PEFT method that decomposes pretrained weights into magnitude and direction, adapting direction with LoRA and learning magnitude separately. *Separating magnitude/direction lets the low-rank update behave more like full fine-tuning, closing the LoRA-vs-FT gap at similar parameter cost.* | accuracy/quality, memory | emerging | Liu et al., 2024 (DoRA) |
| **FireAct** | Fine-tuning language model agents on ReAct-style trajectories generated by a stronger teacher across multiple tasks and prompting methods. *Distills diverse GPT-4-produced ReAct/CoT-with-tools trajectories into a smaller open model via SFT, improving robustness over few-shot prompting.* | accuracy/quality, reliability, cost/tokens | research | Chen et al., 2023, 'FireAct: Toward Language Agent Fine-tuning' |
| **Group Relative Policy Optimization** <br/><sub>GRPO</sub> | A critic-free policy-gradient algorithm that estimates advantages by normalizing rewards within a group of sampled responses to the same prompt. *Samples a group of completions per prompt, uses the group's mean/std reward as the baseline (no value network), and applies a PPO-style clipped update.* | accuracy/quality, memory, cost/tokens | emerging | Shao et al., 2024 (DeepSeekMath GRPO) |
| **Identity Preference Optimization** <br/><sub>IPO</sub> | A DPO variant that adds an explicit regularizer to prevent overfitting to deterministic/near-certain preferences. *Optimizes a squared-loss objective on the preference log-ratio gap to a target margin, bounding the implicit reward and reducing preference overfitting.* | accuracy/quality, reliability | emerging | Azar et al., 2023, 'A General Theoretical Paradigm to Understand Learning from Human Preferences' |
| **Instruction Tuning for Tool Use** <br/><sub>tool-use instruction tuning, function-calling fine-tuning</sub> | Instruction-format fine-tuning on (instruction, tool-call, tool-result, answer) examples to teach a model when and how to emit structured function/tool calls. *Trains on instruction-response pairs whose responses include well-formed API/function-call syntax so the model generalizes the calling format and tool-selection policy.* | accuracy/quality, reliability, autonomy | production-standard | Gorilla (Patil et al., 2023); ToolLLM/ToolBench (Qin et al., 2023) |
| **Iterative / Online DPO** <br/><sub>iterative DPO, online preference optimization, online DPO</sub> | An on-policy variant of preference optimization that repeatedly samples fresh model responses, labels new preferences, and re-runs DPO each round. *Each iteration regenerates on-policy pairs from the latest policy and an updated reward/judge, narrowing distribution shift versus a single offline DPO pass.* | accuracy/quality, reliability | emerging | Xu et al., 2023 / Guo et al., 2024 (online/iterative DPO line) |
| **Kahneman-Tversky Optimization** <br/><sub>KTO</sub> | A preference-alignment method that learns from single examples labeled merely desirable or undesirable, without needing paired comparisons. *Applies a prospect-theory-inspired utility loss to per-example binary signals relative to a reference, removing the need for matched pairs.* | accuracy/quality, reliability, cost/tokens | emerging | Ethayarajh et al., 2024, 'KTO: Model Alignment as Prospect Theoretic Optimization' |
| **Knowledge Distillation** <br/><sub>teacher-student distillation, KD</sub> | Transfers capability from a larger teacher model to a smaller student by training the student to mimic the teacher's outputs or distributions. *The student minimizes divergence to teacher soft logits (or imitates teacher-generated outputs), compressing teacher behavior into a cheaper model.* | cost/tokens, latency, memory, throughput | production-standard | Hinton et al., 2015, 'Distilling the Knowledge in a Neural Network' |
| **Low-Rank Adaptation** <br/><sub>LoRA, PEFT</sub> | A parameter-efficient fine-tuning method that freezes base weights and learns small low-rank update matrices injected into selected layers. *Represents weight deltas as a product of two low-rank matrices trained while the base model stays frozen, drastically cutting trainable parameters and optimizer memory.* | memory, cost/tokens, throughput | production-standard | Hu et al., 2021, 'LoRA: Low-Rank Adaptation of Large Language Models' |
| **Odds Ratio Preference Optimization** <br/><sub>ORPO</sub> | A monolithic method that combines instruction SFT and preference alignment in a single training stage with no reference model. *Adds an odds-ratio preference penalty to the SFT loss, jointly raising chosen-response likelihood while penalizing the rejected response.* | accuracy/quality, reliability, cost/tokens | emerging | Hong et al., 2024, 'ORPO: Monolithic Preference Optimization without Reference Model' |
| **Outcome-Reward Reinforcement Learning for Reasoning** <br/><sub>RLVR, reinforcement learning with verifiable rewards, RL on reasoning</sub> | RL that fine-tunes a model using automatically verifiable outcome rewards (e.g., correct answer, passing tests) to elicit long-form reasoning. *A programmatic verifier returns sparse 0/1 outcome reward over sampled solutions, and policy-gradient updates increase the probability of solution traces that verify.* | accuracy/quality, reliability, autonomy | emerging | DeepSeek-AI, 2025 (DeepSeek-R1); Lambert et al., 2024 (Tulu 3 RLVR) |
| **Process Supervision / Process Reward Models** <br/><sub>PRM, process-supervised verifier, step-level reward model</sub> | Trains reward models to score the correctness of each intermediate reasoning step rather than only the final outcome. *Human or automatically labeled step-level correctness signals supervise a PRM, which then guides search, reranking, or RL toward valid reasoning chains.* <br/><sub>↔ also: Test-time compute scaling & search over reasoning</sub> | accuracy/quality, reliability | emerging | Lightman et al., 2023, 'Let's Verify Step by Step'; Uesato et al., 2022 |
| **Quantized Low-Rank Adaptation** <br/><sub>QLoRA</sub> | Memory-efficient fine-tuning that backpropagates LoRA adapters through a frozen 4-bit quantized base model. *Stores the base in 4-bit NF4 with double quantization and paged optimizers, training only LoRA adapters so large models fine-tune on a single GPU.* | memory, cost/tokens | production-standard | Dettmers et al., 2023, 'QLoRA: Efficient Finetuning of Quantized LLMs' |
| **Rank Responses to Align Human Feedback** <br/><sub>RRHF</sub> | A lightweight alignment method that aligns model scores with a ranking of multiple sampled responses using a ranking loss. *Scores candidate responses by length-normalized log-probability and applies a pairwise ranking loss so preferred responses outrank dispreferred ones, plus an SFT term.* | accuracy/quality, cost/tokens | research | Yuan et al., 2023, 'RRHF: Rank Responses to Align Language Models with Human Feedback' |
| **Reasoning Distillation** <br/><sub>CoT distillation, distilling reasoning chains, chain-of-thought distillation</sub> | Distills a strong reasoning/agent model into a smaller one by training on the teacher's long chain-of-thought and tool-use traces. *SFT the student on teacher-generated step-by-step reasoning (and tool trajectories), transferring the reasoning policy without running RL on the student.* | accuracy/quality, cost/tokens, latency | emerging | DeepSeek-AI, 2025 (R1 distilled models); Hsieh et al., 2023 (Distilling Step-by-Step) |
| **Reflection / Verbal-Feedback Self-Improvement Tuning** <br/><sub>Reflexion-style tuning, self-correction training, SCoRe</sub> | Methods that train agents to improve via self-generated natural-language critiques and corrections of their own prior attempts. *Collects (attempt, reflection, improved attempt) episodes and fine-tunes (often with RL) so the model learns to self-critique and revise toward correct outcomes.* | accuracy/quality, reliability, autonomy | emerging | Shinn et al., 2023 (Reflexion); Kumar et al., 2024 (SCoRe) |
| **Reinforced Self-Training** <br/><sub>ReST, ReST-EM, ReST^EM</sub> | An offline batched RL/self-training scheme that alternates generating a dataset of model samples (Grow) and fine-tuning on the reward-filtered subset (Improve). *Iteratively samples outputs, scores them with a reward/verifier, filters to high-reward samples, and re-fines on them (expectation-maximization view), repeating.* | accuracy/quality, reliability, cost/tokens | research | Gulcehre et al., 2023 (ReST); Singh et al., 2023 (ReST-EM) |
| **Reinforcement Learning from Human Feedback** <br/><sub>RLHF, PPO-RLHF</sub> | Aligns a model to human preferences by training a reward model from human comparisons and optimizing the policy against it with RL. *A learned reward model scores outputs and a policy-gradient algorithm (typically PPO) updates the LLM to maximize reward while a KL penalty keeps it near the reference policy.* | accuracy/quality, reliability | production-standard | Ouyang et al., 2022 (InstructGPT); Christiano et al., 2017 |
| **ReST-MCTS* / self-training with tree-search-generated rationales** | Bootstrapping reasoning data by running MCTS with a process reward to mine high-quality solution traces, then fine-tuning the policy (and PRM) on them. *Tree search produces per-step value estimates and verified traces without human step labels; these supervise both the policy and the process reward model iteratively.* | accuracy/quality, autonomy | research | Zhang et al., 2024 (ReST-MCTS*) |
| **Reward-Ranked / Conditioned Fine-Tuning** <br/><sub>quark, reward-conditioned SFT, control-token tuning</sub> | Fine-tuning that conditions generation on reward/quality control tokens so the model can be steered toward high-reward behavior at inference. *Labels training samples by reward quantile with control tokens (or unlearns low-reward behavior), letting the model learn reward-conditioned policies without online RL.* | accuracy/quality, reliability | research | Lu et al., 2022 (Quark); Korbak et al., 2023 (conditional training) |
| **RLEF / execution-feedback RL for code & tool agents** | Reinforcement learning where the reward comes from executing the agent's code/tool calls (tests pass, environment success) over multi-turn trajectories. *Grounding rewards in real execution outcomes across turns teaches the model to iterate on failures, improving code/agent success beyond single-shot supervised tuning.* <br/><sub>↔ also: RL for agents & verifiable reasoning</sub> | accuracy/quality, autonomy | emerging | Gehring et al., 2024 (RLEF: Grounding Code LLMs in Execution Feedback) |
| **Self-Instruct** <br/><sub>instruction self-generation</sub> | A bootstrapping method that uses a model to generate its own diverse instruction-following examples for instruction tuning with minimal seed data. *Prompts the model to expand a small seed set into new instructions and responses, filters for quality/diversity, and fine-tunes on the synthesized dataset.* | accuracy/quality, cost/tokens | production-standard | Wang et al., 2022, 'Self-Instruct: Aligning Language Models with Self-Generated Instructions' |
| **Self-Play preference / SPIN and SPPO self-play fine-tuning** | Iterative self-play where the current model generates responses treated as 'losers' against prior human/own data, optimizing toward a Nash equilibrium policy. *Each round the model plays against its previous self under a preference objective, distilling more capability from a fixed dataset without new human labels.* | accuracy/quality, autonomy | research | Chen et al., 2024 (SPIN); Wu et al., 2024 (Self-Play Preference Optimization / SPPO) |
| **Self-Rewarding / LLM-as-Judge Self-Training** <br/><sub>self-rewarding LLM, self-rewarding language models, iterative self-judging</sub> | An iterative alignment loop where the same model both generates candidate responses and judges them to create its own preference data. *The model acts as its own reward judge to label preference pairs each round, then trains via DPO/preference optimization on the self-generated labels, iterating.* <br/><sub>↔ also: Self-improvement & feedback loops at inference</sub> | accuracy/quality, autonomy, reliability | research | Yuan et al., 2024, 'Self-Rewarding Language Models' |
| **Self-Taught Reasoner (inference-time bootstrapping)** <br/><sub>STaR, rationale bootstrapping, rejection-sampling fine-tuning loop</sub> | A loop where the model generates rationales, keeps those that lead to correct answers (rationalizing the rest from the answer), and learns from them; the generate-filter-by-correctness loop is the feedback core. *Correctness of the final answer filters self-generated reasoning traces, and rationalization repairs wrong ones, creating a self-improving feedback loop over reasoning.* <br/><sub>↔ also: Self-improvement & feedback loops at inference; Test-time compute scaling & search over reasoning</sub> | accuracy/quality, autonomy | research | Zelikman et al., 2022, 'STaR: Bootstrapping Reasoning with Reasoning' |
| **SimPO (reference-free simple preference optimization)** | A preference-optimization objective using length-normalized average log-probability as the implicit reward with a target margin, removing the reference model. *Drops DPO's reference model and adds a reward margin on length-normalized sequence likelihood, simplifying training and improving alignment efficiency.* | accuracy/quality, cost/tokens | emerging | Meng et al., 2024 (SimPO) |
| **Spectrum / GaLore (memory-efficient full-rank training)** | Training/fine-tuning at full rank while keeping optimizer memory low by projecting gradients into a low-rank subspace (GaLore) or training only high-SNR layers (Spectrum). *GaLore periodically updates a low-rank gradient projection so Adam states stay small; Spectrum freezes low-information layers, both cutting memory vs full FT.* | memory, accuracy/quality | emerging | Zhao et al., 2024 (GaLore); Hartford et al., 2024 (Spectrum) |
| **Supervised Fine-Tuning on Agent Trajectories** <br/><sub>SFT, behavioral cloning, trajectory SFT</sub> | Fine-tuning a base or instruct model on curated multi-step agent trajectories (observation-thought-action-tool-result sequences) so it imitates competent tool-using, multi-turn behavior. *Minimizes next-token cross-entropy over recorded successful trajectories, copying expert/teacher action sequences into the policy.* | accuracy/quality, reliability, autonomy | production-standard | Standard practice; see AgentTuning (Zeng et al., 2023) and FireAct (Chen et al., 2023) |

# Part V — Reliability & evaluation

## 16. Reliability, structured generation & evaluation-as-optimization

Making agents *dependable* is itself an optimization target. Constrained/structured decoding guarantees parseable output; guardrails and validation catch failures; automatic prompt optimizers (DSPy/OPRO/GEPA) tune the system against an eval; observability and offline replay turn production into a feedback signal. fak's [policy-in-the-kernel](../explainers/policy-in-the-kernel.md) is the default-deny floor; trajectory replay lives in `internal/turnbench`.

*27 methods.*

| Method | What it is & how it optimizes | Optimizes | Maturity | Representative reference |
|---|---|---|---|---|
| **Agent observability & tracing** <br/><sub>LLM observability, span tracing, OpenTelemetry GenAI</sub> | Instrumentation that captures structured traces (prompts, tool calls, latencies, tokens, costs, scores) of every step in an LLM/agent run for debugging and evaluation. *Emits nested spans per LLM/tool/retriever call with inputs, outputs, and metadata to a tracing backend, enabling replay, regression eval, and cost/latency attribution.* | reliability, cost/tokens, latency | production-standard | LangSmith; Langfuse; Arize Phoenix; OpenTelemetry GenAI semantic conventions, 2023-2025 |
| **Automatic Prompt Engineer** <br/><sub>APE</sub> | A method that automatically generates and selects instruction prompts by treating prompt engineering as a black-box optimization problem. *An LLM proposes candidate instructions from input-output demonstrations, scores each by execution accuracy on held-out data, and iteratively resamples around the best candidates.* | accuracy/quality | research | Zhou et al., 'Large Language Models Are Human-Level Prompt Engineers', 2022 |
| **Chain-of-Verification** <br/><sub>CoVe</sub> | A method to reduce hallucination where the model drafts an answer, plans verification questions, answers them independently, and revises the answer accordingly. *Independently answering self-generated verification questions (without conditioning on the original answer) catches inconsistencies that are then folded into a corrected final response.* <br/><sub>↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference</sub> | accuracy/quality, reliability | emerging | Dhuliawala et al., 2023, 'Chain-of-Verification Reduces Hallucination in Large Language Models' |
| **Constrained / grammar-guided decoding** <br/><sub>structured decoding, JSON-schema-constrained decoding, grammar-constrained decoding</sub> | A decoding-time technique that restricts the model's next-token choices to only those allowed by a formal grammar or finite-state machine, guaranteeing syntactically valid output (e.g. valid JSON, valid SQL, a regex-matching string). *At each step it masks the logits of any token that would violate the grammar/FSM, so sampling can only produce a parse-valid continuation.* <br/><sub>↔ also: Tool use & function calling optimization</sub> | reliability, accuracy/quality, latency | production-standard | Willard & Louf, 'Efficient Guided Generation for LLMs' (Outlines), 2023; Geng et al. (GCD), 2023 |
| **DSPy** <br/><sub>declarative self-improving pipelines, programming-not-prompting, DSP</sub> | A framework that expresses LLM pipelines as typed modules with signatures and uses optimizers (teleprompters) to compile prompts/weights automatically against a metric. *Treats the pipeline as a program and runs optimizers (e.g. bootstrap few-shot, MIPRO) that search instructions and demonstrations to maximize a developer-supplied metric on a trainset.* | accuracy/quality, reliability | production-standard | Khattab et al., 'DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines', 2023 |
| **Eval-harness-driven optimization** <br/><sub>benchmark-driven development, SWE-bench / WebArena / GAIA / tau-bench harnesses</sub> | Using standardized agent benchmarks with executable verifiers as the objective that an agent system is iteratively tuned against. *Runs the agent inside a reproducible harness whose pass/fail verifier (unit tests, task completion checks) provides an objective score that guides scaffold, prompt, and tool changes.* | accuracy/quality, reliability, autonomy | production-standard | Jimenez et al., 'SWE-bench', 2023; Zhou et al., 'WebArena', 2023; Mialon et al., 'GAIA', 2023; Yao et al., 'tau-bench', 2024 |
| **EvoPrompt (evolutionary prompt optimization)** | Discrete prompt optimization that evolves a population of prompts via LLM-driven mutation/crossover guided by dev-set scores. *Genetic operators implemented as LLM rewrites plus selection on a held-out metric search the prompt space without gradients, a peer of PromptBreeder/GEPA.* | accuracy/quality | research | Guo et al., 2023 (EvoPrompt / Connecting LLMs with Evolutionary Algorithms) |
| **GEPA** <br/><sub>Genetic-Pareto prompt optimization, reflective prompt evolution</sub> | A prompt optimizer that uses reflective natural-language mutation plus Pareto-based selection to evolve prompts sample-efficiently. *Reflects on full execution traces to propose targeted prompt edits and maintains a Pareto frontier of candidates across instances, often matching RL with far fewer rollouts.* | accuracy/quality, cost/tokens | emerging | Agrawal et al., 'GEPA: Reflective Prompt Evolution Can Outperform RL', 2025 |
| **Guardrails / I-O safety filtering** <br/><sub>input-output guardrails, Llama Guard, NeMo Guardrails</sub> | Policy-enforcing classifiers and programmable rails placed around an LLM/agent to screen inputs and outputs for unsafe, off-topic, or policy-violating content and to enforce conversational flow. *Dedicated guard models or rule/colang flows classify each turn and block, rewrite, or redirect responses that violate the configured policy before they reach the user or downstream tools.* | reliability | production-standard | Inan et al., 'Llama Guard', 2023; NVIDIA NeMo Guardrails (Rebedea et al., 2023) |
| **JSON mode / structured outputs API** <br/><sub>JSON mode, function-calling schema enforcement, response_format json_schema</sub> | A provider-side feature (OpenAI, Anthropic, etc.) that guarantees the model's response conforms to a supplied JSON Schema or tool/function signature. *The provider applies constrained decoding and/or schema validation against the supplied schema during generation so the returned object always parses.* | reliability, accuracy/quality | production-standard | OpenAI Structured Outputs (2024); Anthropic tool use / Claude API |
| **LLM-as-a-judge** <br/><sub>LLM evaluator, model-graded evaluation, AI feedback evaluator</sub> | Using a strong LLM as an automatic evaluator that scores or pairwise-ranks candidate outputs against a rubric or reference. *Prompts an evaluator model with the task, rubric, and candidate(s) to emit a graded verdict, enabling cheap scalable evaluation and reward signals (with known position/verbosity biases to mitigate).* <br/><sub>↔ also: Self-improvement & feedback loops at inference</sub> | accuracy/quality, reliability | production-standard | Zheng et al., 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena', 2023 |
| **Lookahead / coherence-checking decoding for structured output (jsonformer / token-masking schema decoding)** | Filling only the value slots of a known JSON/structured schema by generating data tokens and emitting the fixed scaffolding deterministically. *The schema's literal structure is emitted programmatically and the model is only asked to produce typed values under a token mask, guaranteeing valid structure with fewer tokens.* | reliability, cost/tokens, latency | production-standard | Jsonformer (Singh, 2023); token-masking structured decoding |
| **MIPRO** <br/><sub>MIPROv2, Multi-prompt Instruction Proposal Optimizer</sub> | A DSPy optimizer that jointly optimizes instructions and few-shot demonstrations for multi-stage LLM programs. *Bayesian/surrogate search proposes candidate instructions (grounded in data and program structure) and demo sets, evaluating combinations against the metric to pick the best configuration.* | accuracy/quality | emerging | Opsahl-Ong et al., 'Optimizing Instructions and Demonstrations for Multi-Stage LM Programs', 2024 |
| **OPRO** <br/><sub>Optimization by PROmpting</sub> | Uses an LLM as a general-purpose optimizer where the optimization trajectory (prior prompts and their scores) is given in natural language and the model proposes the next candidate. *Iteratively feeds the LLM a meta-prompt containing previously tried solutions sorted by score, asking it to generate a new, higher-scoring solution (e.g. a better instruction).* | accuracy/quality | research | Yang et al., 'Large Language Models as Optimizers' (OPRO), 2023 |
| **Outlines** <br/><sub>regex/JSON-schema guided generation</sub> | An open-source library that compiles a regex, JSON Schema, or context-free grammar into an FSM index over the tokenizer to drive structured generation. *Precomputes, per FSM state, the set of allowed next tokens so logit masking is an O(1) lookup, making constrained decoding nearly free at serving time.* | reliability, latency, accuracy/quality | production-standard | Willard & Louf, 2023 (dottxt-ai/outlines) |
| **Output validation & auto-repair** <br/><sub>self-healing parsing, reask / retry-on-validation-failure, Guardrails AI</sub> | A wrapper layer that validates model output against a schema/type/assertions and, on failure, automatically re-prompts the model with the validation error to obtain a corrected output. *Runs validators (Pydantic types, regex, custom checks) on the output and feeds any failures back as a corrective 'reask' until validation passes or a retry budget is exhausted.* | reliability, accuracy/quality | production-standard | Guardrails AI (Rajpal et al.); Instructor (Liu); LangChain OutputFixingParser |
| **PromptBreeder** <br/><sub>self-referential prompt evolution</sub> | An evolutionary, self-referential method that evolves a population of task-prompts and the mutation-prompts that modify them. *Runs a genetic algorithm where an LLM mutates task-prompts using mutation-prompts that are themselves evolved, selecting on fitness measured against a task metric.* | accuracy/quality | research | Fernando et al., 'Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution', 2023 |
| **Regression / canary eval gating in CI** <br/><sub>eval-driven CI, prompt regression tests, golden-set gating</sub> | Running a curated evaluation suite automatically on every prompt/model/agent change and blocking deploys that regress the metric. *Executes a deterministic eval set (with assertions or LLM-judge scores) in continuous integration and fails the pipeline if scores drop below a baseline threshold.* | reliability, accuracy/quality | production-standard | promptfoo; OpenAI Evals; industry eval-driven-development practice, 2023-2025 |
| **Selective prediction / abstention** <br/><sub>reject option, I-don't-know routing, selective answering</sub> | Letting the model decline to answer (or escalate) when its confidence is below a threshold rather than emitting a likely-wrong answer. *Thresholds an uncertainty/confidence score to choose between answering and abstaining/deferring, trading coverage for higher accuracy on answered cases.* | reliability, accuracy/quality | emerging | Ren et al., selective prediction for LLMs, 2023; Cole et al., 'Selectively Answering Ambiguous Questions', 2023 |
| **Self-verification / generate-then-verify** <br/><sub>verifier models, self-checking, answer verification</sub> | Pairing a generator with a separate verification step (model or executable check) that accepts, ranks, or rejects candidate outputs. *Generates candidates then runs a learned or rule-based verifier (e.g. backward reasoning, test execution) to select the most likely-correct answer or trigger a retry.* | accuracy/quality, reliability | emerging | Weng et al., 'Large Language Models are Better Reasoners with Self-Verification', 2022; Cobbe et al. verifiers, 2021 |
| **SelfCheckGPT (sampling-based hallucination detection)** | A zero-resource hallucination detector that samples multiple responses and flags statements that are not consistent across the samples. *Consistency of a claim across stochastic samples (via NLI/QA/n-gram scoring) estimates factuality without external knowledge or token-probabilities.* | reliability, accuracy/quality | emerging | Manakul et al., 2023 (SelfCheckGPT) |
| **Semantic entropy / lexical-invariant uncertainty for abstention** | Estimating hallucination by clustering sampled answers into meaning-equivalence classes and measuring entropy over the clusters rather than tokens. *Bidirectional-entailment clustering collapses paraphrases so entropy reflects semantic uncertainty, giving a calibrated confound-free hallucination signal.* | reliability, accuracy/quality | emerging | Kuhn et al., 2023; Farquhar et al., 2024 (Semantic Entropy, Nature) |
| **Speculative / constrained tool-call repair at the boundary** <br/><sub>function-call validation, tool-argument schema enforcement, arg auto-repair</sub> | Validating an agent's emitted tool/function call against the tool's declared schema and repairing or rejecting malformed calls before execution. *Parses the tool call, checks argument names/types against the tool signature, and either constrains generation to the schema or re-asks the model to fix invalid arguments.* | reliability, autonomy | production-standard | OpenAI/Anthropic tool-use schema enforcement; Instructor/Guardrails tool validation, 2023-2025 |
| **TextGrad** <br/><sub>textual gradients, automatic differentiation via text</sub> | A framework that optimizes components of compound LLM systems using natural-language 'gradients' (textual feedback) backpropagated through the system. *An LLM produces critique-as-gradient for each variable (prompt, solution, code) and a textual analog of gradient descent applies those critiques to improve the variable.* | accuracy/quality | emerging | Yuksekgonul et al., 'TextGrad: Automatic Differentiation via Text', 2024 |
| **Trajectory replay / offline policy evaluation** <br/><sub>off-policy evaluation, OPE, counterfactual evaluation</sub> | Scoring or comparing candidate agent policies against previously recorded interaction traces instead of re-executing live, to estimate which policy is better cheaply. *Replays a logged trajectory deterministically and re-derives each candidate policy's decisions/score against the recorded environment responses, turning a product of live runs into a sum over one recorded run.* | cost/tokens, reliability, accuracy/quality | emerging | Precup et al., off-policy evaluation literature; applied LLM-agent trajectory-replay practice, 2024-2025 |
| **Uncertainty estimation / confidence calibration** <br/><sub>confidence elicitation, semantic entropy, calibration</sub> | Methods that estimate how likely an LLM's output is to be correct, used to flag low-confidence answers. *Derives a confidence signal from token logprobs, verbalized confidence, sampled-answer agreement, or semantic-entropy over clustered samples, and calibrates it against observed accuracy.* | reliability | emerging | Kuhn et al., 'Semantic Uncertainty', 2023; Kadavath et al., 'Language Models (Mostly) Know What They Know', 2022 |
| **XGrammar** <br/><sub>context-free-grammar constrained decoding</sub> | A high-performance constrained-decoding engine for general context-free grammars integrated into serving stacks (vLLM, SGLang, MLC). *Splits tokens into context-independent vs context-dependent, precomputes an adaptive token mask cache, and overlaps mask generation with GPU compute to make CFG-constrained decoding near-zero-overhead.* | reliability, latency, throughput | production-standard | Dong et al., 'XGrammar: Flexible and Efficient Structured Generation', 2024 |

---

## Alphabetical master index

All 370 methods, A→Z, with their home family — for lookup and grep.

| Method | Family | Maturity |
|---|---|---|
| A-MEM (agentic memory / Zettelkasten-style) | 5. Context-window management & compression | research |
| Activation Beacon (sliding context condensation) | 5. Context-window management & compression | research |
| AdaPlanner | 10. Planning algorithms & world models for agents | research |
| ADaPT (As-Needed Decomposition and Planning) | 10. Planning algorithms & world models for agents | research |
| Adapter / Prefix / Prompt Tuning (PEFT family) | 15. Training & adaptation for agentic capability | production-standard |
| Adaptive / early-stopping sampling | 2. Test-time compute scaling & search over reasoning | emerging |
| Adaptive-RAG (query-complexity-routed retrieval depth) | 7. Retrieval & knowledge augmentation | emerging |
| Agent observability & tracing | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Agent-as-Tool | 11. Multi-agent orchestration patterns | production-standard |
| Agentic RAG | 7. Retrieval & knowledge augmentation | emerging |
| Agentic RL with Environment Interaction | 15. Training & adaptation for agentic capability | emerging |
| AgentTuning | 15. Training & adaptation for agentic capability | research |
| Algorithm of Thoughts | 1. Prompting & reasoning strategies | research |
| Analogical prompting | 1. Prompting & reasoning strategies | emerging |
| Associative graph-traversal recall (HippoRAG-style) | 6. Agent memory architectures | emerging |
| Astute RAG / knowledge-conflict resolution | 7. Retrieval & knowledge augmentation | emerging |
| Attention sinks / StreamingLLM | 5. Context-window management & compression | production-standard |
| AutoGen conversable-agent framework (programmable multi-agent conversation) | 11. Multi-agent orchestration patterns | production-standard |
| AutoGPT-style Autonomous Loop | 11. Multi-agent orchestration patterns | emerging |
| Automatic Chain-of-Thought | 1. Prompting & reasoning strategies | research |
| Automatic Prompt Engineer | 16. Reliability, structured generation & evaluation-as-optimization | research |
| AutoMix (self-verification mixing) | 13. Model routing, cascades & cost optimization | research |
| AWQ | 14. Inference / serving optimizations beneath agents | production-standard |
| Batch / off-peak request scheduling | 13. Model routing, cascades & cost optimization | production-standard |
| Best-of-N sampling | 2. Test-time compute scaling & search over reasoning | production-standard |
| Best-of-N with reward/verifier model | 3. Self-improvement & feedback loops at inference | production-standard |
| Blackboard Architecture | 11. Multi-agent orchestration patterns | emerging |
| Budget forcing / long-thinking (s1-style) | 2. Test-time compute scaling & search over reasoning | emerging |
| Cache-augmented generation (CAG) | 13. Model routing, cascades & cost optimization | emerging |
| Cache-aware load balancing / routing | 8. KV cache & prefix reuse | emerging |
| CacheBlend (segment-level / non-prefix KV reuse) | 8. KV cache & prefix reuse | emerging |
| Cascade / multi-tier KV inference (CascadeAttention) | 8. KV cache & prefix reuse | emerging |
| Cascade threshold / abstention tuning | 13. Model routing, cascades & cost optimization | emerging |
| Chain-of-Note (CoN) | 7. Retrieval & knowledge augmentation | research |
| Chain-of-Thought prompting | 1. Prompting & reasoning strategies | production-standard |
| Chain-of-Thought without prompting (CoT-decoding) | 1. Prompting & reasoning strategies | research |
| Chain-of-Verification | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| ChatDev (Communicative Software Agents) | 11. Multi-agent orchestration patterns | research |
| ChatEval-style role-diverse multi-agent evaluation (referee-team) | 11. Multi-agent orchestration patterns | research |
| ChunkAttention / prefix-aware fused attention | 8. KV cache & prefix reuse | research |
| Chunked prefill | 14. Inference / serving optimizations beneath agents | production-standard |
| Code as Policies | 10. Planning algorithms & world models for agents | research |
| Code generation for API composition (programmatic tool use) | 9. Tool use & function calling optimization | emerging |
| Code-as-Action (CodeAct) | 9. Tool use & function calling optimization | emerging |
| Cognitive memory typing (working / episodic / semantic / procedural) | 6. Agent memory architectures | emerging |
| ColBERT Late-Interaction Retrieval | 7. Retrieval & knowledge augmentation | emerging |
| Complexity-based prompting | 1. Prompting & reasoning strategies | research |
| Compute-optimal test-time scaling | 2. Test-time compute scaling & search over reasoning | emerging |
| Constitutional AI (critique-and-revise) | 15. Training & adaptation for agentic capability | production-standard |
| Constrained / grammar-guided decoding | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Context caching for agent / tool-loop reuse | 8. KV cache & prefix reuse | emerging |
| Context offloading via structured note-taking to a scratchpad tool (Anthropic 'think' tool) | 12. Context engineering & agent-design patterns | production-standard |
| Context-aware / Context-Aware Decoding (CAD, PMI context amplification) | 4. Decoding-time control | research |
| Contextual Compression / Retrieved-Context Pruning | 5. Context-window management & compression | emerging |
| Contextual Retrieval | 7. Retrieval & knowledge augmentation | emerging |
| Continuous / in-flight batching | 14. Inference / serving optimizations beneath agents | production-standard |
| Contract-Net / Auction Allocation | 11. Multi-agent orchestration patterns | research |
| Contrastive Decoding (expert vs amateur) | 4. Decoding-time control | research |
| Conversation summarization / compaction | 5. Context-window management & compression | production-standard |
| Conversational summary-buffer memory | 6. Agent memory architectures | production-standard |
| Corrective RAG | 7. Retrieval & knowledge augmentation | research |
| CRITIC (tool-augmented self-correction) | 3. Self-improvement & feedback loops at inference | emerging |
| Cross-Encoder Re-ranking | 7. Retrieval & knowledge augmentation | production-standard |
| Cross-layer KV sharing (YOCO / CLA) | 8. KV cache & prefix reuse | emerging |
| Cross-Model Mixture / Model Diversity Ensemble | 11. Multi-agent orchestration patterns | research |
| CUDA graphs | 14. Inference / serving optimizations beneath agents | production-standard |
| DAPO / Dr.GRPO (decoupled-clip and bias-corrected GRPO variants) | 15. Training & adaptation for agentic capability | emerging |
| Deduplication / redundancy elimination in context | 5. Context-window management & compression | emerging |
| Demonstration / few-shot tool-call exemplars | 9. Tool use & function calling optimization | production-standard |
| Dense Passage Retrieval | 7. Retrieval & knowledge augmentation | production-standard |
| Describe, Explain, Plan and Select (DEPS) | 10. Planning algorithms & world models for agents | research |
| Difficulty / complexity-based routing | 13. Model routing, cascades & cost optimization | production-standard |
| Direct Preference Optimization | 15. Training & adaptation for agentic capability | production-standard |
| Disaggregated KV transfer / KV-aware disaggregation | 8. KV cache & prefix reuse | production-standard |
| Disaggregated prefill/decode | 14. Inference / serving optimizations beneath agents | production-standard |
| Distributed / global KV cache pool | 8. KV cache & prefix reuse | emerging |
| DMC / Dynamic Memory Compression (learned KV merging at decode) | 5. Context-window management & compression | research |
| DoLa (Decoding by Contrasting Layers) | 4. Decoding-time control | research |
| DoRA (weight-decomposed low-rank adaptation) | 15. Training & adaptation for agentic capability | emerging |
| DSPy | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| DuoAttention (retrieval vs streaming head split) | 8. KV cache & prefix reuse | emerging |
| Dynamic / Self-Organizing Agent Topology | 11. Multi-agent orchestration patterns | research |
| Dynamic replanning | 10. Planning algorithms & world models for agents | production-standard |
| EAGLE | 14. Inference / serving optimizations beneath agents | emerging |
| Editable core / persona memory blocks | 6. Agent memory architectures | production-standard |
| Embedding/predictive router (zero-shot model selection) | 13. Model routing, cascades & cost optimization | emerging |
| Emotion Prompting (EmotionPrompt) | 1. Prompting & reasoning strategies | research |
| Entity / profile memory | 6. Agent memory architectures | production-standard |
| Eval-harness-driven optimization | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| EvoPrompt (evolutionary prompt optimization) | 16. Reliability, structured generation & evaluation-as-optimization | research |
| Experiential memory / case-based experience reuse | 6. Agent memory architectures | emerging |
| Expert parallelism | 14. Inference / serving optimizations beneath agents | production-standard |
| Faithful Chain-of-Thought | 1. Prompting & reasoning strategies | research |
| FastGen (adaptive KV compression) | 8. KV cache & prefix reuse | research |
| Few-shot prompting | 1. Prompting & reasoning strategies | production-standard |
| FireAct | 15. Training & adaptation for agentic capability | research |
| FLARE (Forward-Looking Active Retrieval) | 7. Retrieval & knowledge augmentation | research |
| FlashAttention | 14. Inference / serving optimizations beneath agents | production-standard |
| FlashDecoding / FlashDecoding++ | 14. Inference / serving optimizations beneath agents | production-standard |
| Forced / directed tool choice | 9. Tool use & function calling optimization | production-standard |
| Forest-of-Thoughts / ensemble-of-trees | 2. Test-time compute scaling & search over reasoning | research |
| FP8 quantization | 14. Inference / serving optimizations beneath agents | production-standard |
| FrugalGPT LLM cascade | 13. Model routing, cascades & cost optimization | research |
| Generate-then-Read / GenRead (LLM-as-retriever) | 7. Retrieval & knowledge augmentation | research |
| Generative / parametric memory (memory tokens & soft prompts) | 5. Context-window management & compression | research |
| Generative / RAG-free memory via Cache-Augmented Generation distinction — RecurrentGPT (language-based recurrence) | 6. Agent memory architectures | research |
| Generative-agent reflection-and-planning memory | 10. Planning algorithms & world models for agents | research |
| Generator-Critic (Actor-Critic / Reviewer) | 11. Multi-agent orchestration patterns | production-standard |
| GEPA | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| GGUF / llama.cpp k-quants | 14. Inference / serving optimizations beneath agents | production-standard |
| Goal decomposition / task decomposition | 10. Planning algorithms & world models for agents | production-standard |
| Gorilla / API-aware retriever finetuning | 9. Tool use & function calling optimization | research |
| GPTQ | 14. Inference / serving optimizations beneath agents | production-standard |
| Graph of Thoughts | 2. Test-time compute scaling & search over reasoning | research |
| Graph-based agent memory | 6. Agent memory architectures | production-standard |
| GraphRAG | 7. Retrieval & knowledge augmentation | emerging |
| Group Chat (Conversational Multi-Agent) | 11. Multi-agent orchestration patterns | production-standard |
| Group Relative Policy Optimization | 15. Training & adaptation for agentic capability | emerging |
| Grouped-Query Attention | 8. KV cache & prefix reuse | production-standard |
| Guardrails / I-O safety filtering | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Guidance / constrained generation interleaving (template-guided + acceleration) | 4. Decoding-time control | production-standard |
| Hadamard / incoherence-processed quantization (QuIP and QuIP#) | 14. Inference / serving optimizations beneath agents | research |
| Handoff | 11. Multi-agent orchestration patterns | production-standard |
| Heavy KV merging (CaM / KVMerger) | 8. KV cache & prefix reuse | research |
| Hierarchical / Recursive Agents | 11. Multi-agent orchestration patterns | production-standard |
| Hierarchical / recursive summarization | 5. Context-window management & compression | production-standard |
| Hierarchical Task Network planning | 10. Planning algorithms & world models for agents | production-standard |
| Hybrid LLM routing (cost-aware quality threshold) | 13. Model routing, cascades & cost optimization | research |
| Hybrid Retrieval (sparse + dense) | 7. Retrieval & knowledge augmentation | production-standard |
| Hydra / multi-token Medusa successors and SpecInfer tree speculation | 14. Inference / serving optimizations beneath agents | emerging |
| Hydragen (shared-prefix attention decomposition) | 8. KV cache & prefix reuse | research |
| Hypothetical Document Embeddings | 7. Retrieval & knowledge augmentation | emerging |
| Identity Preference Optimization | 15. Training & adaptation for agentic capability | emerging |
| Importance scoring / memory prioritization | 6. Agent memory architectures | emerging |
| In-Context / Retrieval-Augmented Long-Context (kNN-LM family) | 7. Retrieval & knowledge augmentation | research |
| In-context autoencoding / 500x context compression (ICAE — explicit), and prompt-distillation | 5. Context-window management & compression | research |
| Instruction Tuning for Tool Use | 15. Training & adaptation for agentic capability | production-standard |
| IRCoT (Interleaving Retrieval with Chain-of-Thought) | 7. Retrieval & knowledge augmentation | research |
| Irrelevance / no-call detection | 9. Tool use & function calling optimization | production-standard |
| Iterative / Online DPO | 15. Training & adaptation for agentic capability | emerging |
| Iterative refinement / generate-evaluate-refine loop | 3. Self-improvement & feedback loops at inference | production-standard |
| JSON mode / structured outputs API | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Kahneman-Tversky Optimization | 15. Training & adaptation for agentic capability | emerging |
| Knowledge Distillation | 15. Training & adaptation for agentic capability | production-standard |
| KV cache deduplication / content-addressed block sharing | 8. KV cache & prefix reuse | production-standard |
| KV cache offloading (CPU / disk tiering) | 8. KV cache & prefix reuse | production-standard |
| KV cache quantization (KIVI) | 8. KV cache & prefix reuse | emerging |
| KV cache reuse across requests via offload + load (CacheGen / LMCache streaming load) | 14. Inference / serving optimizations beneath agents | emerging |
| Late Chunking | 7. Retrieval & knowledge augmentation | emerging |
| LATS (Language Agent Tree Search) | 10. Planning algorithms & world models for agents | research |
| Least-to-Most / Decomposed Prompting | 2. Test-time compute scaling & search over reasoning | research |
| Lifting text/inline tool calls at the boundary | 9. Tool use & function calling optimization | production-standard |
| LLM ensembling for cost/quality | 13. Model routing, cascades & cost optimization | research |
| LLM+P (LLM + Classical Planner) | 10. Planning algorithms & world models for agents | research |
| LLM-as-a-judge | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| LLM-as-Compiler / LLMCompiler parallel tool orchestration | 9. Tool use & function calling optimization | emerging |
| LLM-as-judge debate / multi-agent evaluation | 3. Self-improvement & feedback loops at inference | emerging |
| LLM-as-Judge Orchestration | 11. Multi-agent orchestration patterns | production-standard |
| LLM-DP (LLM Dynamic Planner) | 10. Planning algorithms & world models for agents | research |
| LLMLingua | 5. Context-window management & compression | emerging |
| LLMLingua-2 | 5. Context-window management & compression | emerging |
| Long chain-of-thought RL reasoning (o1/R1-style) | 2. Test-time compute scaling & search over reasoning | production-standard |
| LongLLMLingua | 5. Context-window management & compression | emerging |
| Lookahead / coherence-checking decoding for structured output (jsonformer / token-masking schema decoding) | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Lookahead / rollout-guided decoding | 2. Test-time compute scaling & search over reasoning | research |
| Lookahead decoding | 14. Inference / serving optimizations beneath agents | emerging |
| Lost-in-the-middle mitigation via reordering | 5. Context-window management & compression | production-standard |
| Low-Rank Adaptation | 15. Training & adaptation for agentic capability | production-standard |
| Maieutic prompting | 1. Prompting & reasoning strategies | research |
| Map-Reduce over Agents | 11. Multi-agent orchestration patterns | production-standard |
| Maximal Marginal Relevance retrieval | 7. Retrieval & knowledge augmentation | production-standard |
| Medusa | 14. Inference / serving optimizations beneath agents | emerging |
| Mem0 (extraction-and-update memory layer) | 6. Agent memory architectures | production-standard |
| MemGPT (paged/virtual context memory) | 5. Context-window management & compression | production-standard |
| Memoized / deduplicated tool execution caching | 9. Tool use & function calling optimization | production-standard |
| Memory consolidation | 6. Agent memory architectures | emerging |
| Memory forgetting / decay | 6. Agent memory architectures | emerging |
| Memory retrieval re-ranking / relevance gating | 6. Agent memory architectures | emerging |
| Memory writeback policy | 6. Agent memory architectures | emerging |
| MemoryBank with Ebbinghaus forgetting curve | 6. Agent memory architectures | research |
| MetaGPT / SOP-Encoded Pipeline | 11. Multi-agent orchestration patterns | research |
| Min-p sampling | 4. Decoding-time control | emerging |
| MInference (dynamic sparse prefill attention) | 8. KV cache & prefix reuse | emerging |
| MIPRO | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Mixture-of-Agents (for cost/quality) | 11. Multi-agent orchestration patterns | research |
| Mixture-of-Thought / model-internal effort routing (reasoning-effort parameter) | 13. Model routing, cascades & cost optimization | production-standard |
| Model cascade / quantization-tier serving | 13. Model routing, cascades & cost optimization | emerging |
| Model Context Protocol (MCP) / tool standardization | 9. Tool use & function calling optimization | production-standard |
| MoE routing (sparse gating / top-k) | 14. Inference / serving optimizations beneath agents | production-standard |
| Monte Carlo Tree Search for LLM planning | 10. Planning algorithms & world models for agents | research |
| Monte Carlo Tree Self-refine / AlphaLLM-style self-improvement | 10. Planning algorithms & world models for agents | research |
| Multi-agent debate | 11. Multi-agent orchestration patterns | emerging |
| Multi-head Latent Attention (MLA) | 8. KV cache & prefix reuse | production-standard |
| Multi-LLM optimal assignment / portfolio routing | 13. Model routing, cascades & cost optimization | research |
| Multi-Query Retrieval | 7. Retrieval & knowledge augmentation | production-standard |
| Odds Ratio Preference Optimization | 15. Training & adaptation for agentic capability | emerging |
| OPRO | 16. Reliability, structured generation & evaluation-as-optimization | research |
| Orchestrator-Worker (Manager-Workers) | 11. Multi-agent orchestration patterns | production-standard |
| Outcome Reward Model reranking | 2. Test-time compute scaling & search over reasoning | production-standard |
| Outcome-Reward Reinforcement Learning for Reasoning | 15. Training & adaptation for agentic capability | emerging |
| Outlines | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Output validation & auto-repair | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Output-length control / token budgeting | 13. Model routing, cascades & cost optimization | production-standard |
| PagedAttention | 8. KV cache & prefix reuse | production-standard |
| Parallel tool calling | 9. Tool use & function calling optimization | production-standard |
| Parallelization (Sectioning & Voting) | 11. Multi-agent orchestration patterns | production-standard |
| Pipeline parallelism | 14. Inference / serving optimizations beneath agents | production-standard |
| Plan caching / plan templates | 10. Planning algorithms & world models for agents | emerging |
| Plan-and-Solve Prompting | 10. Planning algorithms & world models for agents | emerging |
| Planner-Executor (Plan-and-Execute) | 10. Planning algorithms & world models for agents | production-standard |
| Procedural memory / learned skills writeback | 6. Agent memory architectures | emerging |
| Process Reward Model guided refinement | 3. Self-improvement & feedback loops at inference | emerging |
| Process Supervision / Process Reward Models | 15. Training & adaptation for agentic capability | emerging |
| ProgPrompt (programmatic planning) | 10. Planning algorithms & world models for agents | research |
| Program-Aided Language models | 1. Prompting & reasoning strategies | emerging |
| Program-of-Thoughts | 1. Prompting & reasoning strategies | emerging |
| Prompt Cache (modular position-independent KV) | 8. KV cache & prefix reuse | emerging |
| Prompt Lookup Decoding / n-gram speculative decoding | 14. Inference / serving optimizations beneath agents | production-standard |
| Prompt-cache-friendly prompt ordering (stable prefix / append-only design) | 12. Context engineering & agent-design patterns | production-standard |
| Prompt-Chaining (Sequential Pipeline) | 11. Multi-agent orchestration patterns | production-standard |
| PromptBreeder | 16. Reliability, structured generation & evaluation-as-optimization | research |
| Provider prompt caching (OpenAI / Anthropic / Gemini) | 8. KV cache & prefix reuse | production-standard |
| Provider/API cost routing (cheapest-provider arbitrage) | 13. Model routing, cascades & cost optimization | production-standard |
| PyramidKV / PyramidInfer (layer-pyramidal KV budget) | 8. KV cache & prefix reuse | research |
| Quantization-aware / low-bit serving (W4A16, QLoRA-style) | 14. Inference / serving optimizations beneath agents | emerging |
| Quantized Low-Rank Adaptation | 15. Training & adaptation for agentic capability | production-standard |
| QuaRot / SpinQuant (rotation-based outlier-free quantization) | 14. Inference / serving optimizations beneath agents | emerging |
| Query Decomposition | 7. Retrieval & knowledge augmentation | production-standard |
| Query Expansion | 7. Retrieval & knowledge augmentation | production-standard |
| Query Rewriting | 7. Retrieval & knowledge augmentation | production-standard |
| Quest (query-aware KV page selection) | 8. KV cache & prefix reuse | research |
| Quiet-STaR | 2. Test-time compute scaling & search over reasoning | research |
| Rank Responses to Align Human Feedback | 15. Training & adaptation for agentic capability | research |
| RankGPT / LLM listwise reranking | 7. Retrieval & knowledge augmentation | emerging |
| RAP (Reasoning via Planning) | 10. Planning algorithms & world models for agents | research |
| RAPTOR | 5. Context-window management & compression | emerging |
| ReAct | 9. Tool use & function calling optimization | production-standard |
| Reasoning Distillation | 15. Training & adaptation for agentic capability | emerging |
| Reasoning-budget control / token-budget-aware reasoning | 13. Model routing, cascades & cost optimization | emerging |
| Reciprocal Rank Fusion | 7. Retrieval & knowledge augmentation | production-standard |
| Recitation / re-stating goals to combat goal drift (todo.md recitation) | 12. Context engineering & agent-design patterns | emerging |
| Recurrent / Memorizing-Transformer memory | 6. Agent memory architectures | research |
| Recurrent / segment-level memory transformers | 5. Context-window management & compression | research |
| Recursive Criticism and Improvement | 3. Self-improvement & feedback loops at inference | emerging |
| Reflection / Verbal-Feedback Self-Improvement Tuning | 15. Training & adaptation for agentic capability | emerging |
| Reflection / verifier-in-the-loop agent reflection | 3. Self-improvement & feedback loops at inference | production-standard |
| Reflective / metacognitive prompting | 3. Self-improvement & feedback loops at inference | research |
| Reflexion (verbal self-reflection memory) | 3. Self-improvement & feedback loops at inference | emerging |
| Reflexion with self-generated unit tests | 3. Self-improvement & feedback loops at inference | emerging |
| Reflexion-for-planning successor — AdaPlanner closed-loop refinement (explicit in-plan + out-of-plan) | 10. Planning algorithms & world models for agents | research |
| Regression / canary eval gating in CI | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Reinforced Self-Training | 15. Training & adaptation for agentic capability | research |
| Reinforcement learning for tool use | 9. Tool use & function calling optimization | research |
| Reinforcement Learning from Human Feedback | 15. Training & adaptation for agentic capability | production-standard |
| Repeated sampling / coverage scaling | 2. Test-time compute scaling & search over reasoning | emerging |
| Rephrase and Respond | 1. Prompting & reasoning strategies | research |
| REPLUG | 7. Retrieval & knowledge augmentation | research |
| ReST-MCTS* / self-training with tree-search-generated rationales | 15. Training & adaptation for agentic capability | research |
| Retrieval Routing | 7. Retrieval & knowledge augmentation | emerging |
| Retrieval-Augmented Fine-Tuning | 7. Retrieval & knowledge augmentation | research |
| Retrieval-Augmented Generation | 7. Retrieval & knowledge augmentation | production-standard |
| Retrieval-augmented memory (retrieval-over-memory / RAG) | 6. Agent memory architectures | production-standard |
| Retrieval-based context selection | 5. Context-window management & compression | production-standard |
| Retrieval-head-aware KV eviction (RazorAttention / retrieval heads) | 8. KV cache & prefix reuse | research |
| Retrieval-scored memory (recency-importance-relevance) | 6. Agent memory architectures | emerging |
| Retry with backoff / circuit breaking for flaky tools | 9. Tool use & function calling optimization | production-standard |
| Reward-Ranked / Conditioned Fine-Tuning | 15. Training & adaptation for agentic capability | research |
| ReWOO (Reasoning WithOut Observation) | 10. Planning algorithms & world models for agents | emerging |
| Ring Attention / Striped Attention / Blockwise context parallel (explicit) | 14. Inference / serving optimizations beneath agents | emerging |
| RLEF / execution-feedback RL for code & tool agents | 15. Training & adaptation for agentic capability | emerging |
| Role Specialization (Persona / Expert Agents) | 11. Multi-agent orchestration patterns | production-standard |
| Role-Playing Cooperative Agents (Instructor-Assistant) | 11. Multi-agent orchestration patterns | research |
| RouteLLM (learned LLM router) | 13. Model routing, cascades & cost optimization | production-standard |
| Routing / Dispatch | 11. Multi-agent orchestration patterns | production-standard |
| Sandboxed code-action execution | 9. Tool use & function calling optimization | production-standard |
| SayCan (grounded affordance planning) | 10. Planning algorithms & world models for agents | research |
| Scratchpad reasoning | 5. Context-window management & compression | research |
| Selective Context | 5. Context-window management & compression | emerging |
| Selective prediction / abstention | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Self-Ask | 1. Prompting & reasoning strategies | emerging |
| Self-Collaboration / Single-Model Multi-Persona | 11. Multi-agent orchestration patterns | research |
| Self-Consistency as feedback / verifier-guided refinement | 3. Self-improvement & feedback loops at inference | research |
| Self-consistency confidence (logit/entropy-weighted) aggregation | 2. Test-time compute scaling & search over reasoning | research |
| Self-Consistency early termination by entropy / agreement (ESC, Adaptive-Consistency adaptive sampling) | 2. Test-time compute scaling & search over reasoning | emerging |
| Self-consistency over tool-augmented / program reasoning | 2. Test-time compute scaling & search over reasoning | emerging |
| Self-consistency with adaptive sampling | 13. Model routing, cascades & cost optimization | research |
| Self-Contrast (multi-perspective divergent self-checking) | 3. Self-improvement & feedback loops at inference | research |
| Self-Correction with learned corrector (Self-Correct) | 3. Self-improvement & feedback loops at inference | research |
| Self-critique / self-correction | 3. Self-improvement & feedback loops at inference | production-standard |
| Self-Debugging | 3. Self-improvement & feedback loops at inference | emerging |
| Self-Discover | 3. Self-improvement & feedback loops at inference | research |
| Self-editing / learning-to-memorize agents | 6. Agent memory architectures | research |
| Self-evaluation / self-assessment | 3. Self-improvement & feedback loops at inference | emerging |
| Self-Instruct | 15. Training & adaptation for agentic capability | production-standard |
| Self-Play preference / SPIN and SPPO self-play fine-tuning | 15. Training & adaptation for agentic capability | research |
| Self-Querying Retrieval (metadata-filtered retrieval) | 7. Retrieval & knowledge augmentation | production-standard |
| Self-RAG | 7. Retrieval & knowledge augmentation | research |
| Self-Refine | 3. Self-improvement & feedback loops at inference | production-standard |
| Self-Rewarding / LLM-as-Judge Self-Training | 15. Training & adaptation for agentic capability | research |
| Self-speculative decoding | 14. Inference / serving optimizations beneath agents | emerging |
| Self-Taught Reasoner (inference-time bootstrapping) | 15. Training & adaptation for agentic capability | research |
| Self-verification | 3. Self-improvement & feedback loops at inference | emerging |
| Self-verification / generate-then-verify | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Self-Verification / self-evaluation selection | 2. Test-time compute scaling & search over reasoning | emerging |
| SelfCheckGPT (sampling-based hallucination detection) | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Semantic caching (GPTCache) | 13. Model routing, cascades & cost optimization | production-standard |
| Semantic chunking | 5. Context-window management & compression | production-standard |
| Semantic entropy / lexical-invariant uncertainty for abstention | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Sentence-Window / Parent-Document Retrieval | 7. Retrieval & knowledge augmentation | production-standard |
| Sequence / context parallelism | 14. Inference / serving optimizations beneath agents | production-standard |
| Sequence packing / sample packing for training & prefill efficiency | 14. Inference / serving optimizations beneath agents | production-standard |
| Shared / collective multi-agent memory | 6. Agent memory architectures | research |
| Shared Long-Term Memory / Knowledge Base Coordination | 11. Multi-agent orchestration patterns | emerging |
| SimPO (reference-free simple preference optimization) | 15. Training & adaptation for agentic capability | emerging |
| Skeleton-of-Thought | 2. Test-time compute scaling & search over reasoning | research |
| Sliding-window / segment context processing | 5. Context-window management & compression | production-standard |
| Small-model-first escalation | 13. Model routing, cascades & cost optimization | production-standard |
| SmoothQuant | 14. Inference / serving optimizations beneath agents | production-standard |
| Spectrum / GaLore (memory-efficient full-rank training) | 15. Training & adaptation for agentic capability | emerging |
| Speculative / constrained tool-call repair at the boundary | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Speculative / predictive tool prefetching | 9. Tool use & function calling optimization | research |
| Speculative cascades | 13. Model routing, cascades & cost optimization | research |
| Speculative decoding (draft model) | 14. Inference / serving optimizations beneath agents | production-standard |
| Speculative reasoning / draft-then-verify reasoning skeletons (Skeleton-of-Thought parallel decode) | 2. Test-time compute scaling & search over reasoning | research |
| Step-Back prompting | 1. Prompting & reasoning strategies | emerging |
| Step-level Beam Search over reasoning | 2. Test-time compute scaling & search over reasoning | emerging |
| Stigmergic Coordination (Environment-Mediated) | 11. Multi-agent orchestration patterns | research |
| StreamingLLM (attention sinks) | 8. KV cache & prefix reuse | production-standard |
| Structured / running state notes | 5. Context-window management & compression | production-standard |
| Structured pruning / sparsity | 14. Inference / serving optimizations beneath agents | emerging |
| Sub-agent / context isolation (sub-context offload) | 5. Context-window management & compression | production-standard |
| Sub-agent context isolation / orchestrator-with-clean-subcontexts (Claude Code / Cognition pattern) | 12. Context engineering & agent-design patterns | production-standard |
| Supervised Fine-Tuning on Agent Trajectories | 15. Training & adaptation for agentic capability | production-standard |
| System 2 Attention (S2A) | 1. Prompting & reasoning strategies | research |
| Tab-CoT / Structured (tabular) reasoning | 1. Prompting & reasoning strategies | research |
| Take a Deep Breath / optimized meta-prompts (OPRO-discovered instructions) | 1. Prompting & reasoning strategies | research |
| Tensor parallelism | 14. Inference / serving optimizations beneath agents | production-standard |
| TextGrad | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Thread of Thought (ThoT) | 1. Prompting & reasoning strategies | research |
| Token / context pruning by saliency | 5. Context-window management & compression | research |
| Tool masking / dynamic gating | 9. Tool use & function calling optimization | production-standard |
| Tool masking over tool removal for stable cache (logit-masked action space) | 12. Context engineering & agent-design patterns | emerging |
| Tool pruning / fewer-tools curation | 9. Tool use & function calling optimization | production-standard |
| Tool Retrieval / RAG-based tool selection | 9. Tool use & function calling optimization | production-standard |
| Tool schema / documentation design | 9. Tool use & function calling optimization | production-standard |
| Tool-call validation / argument schema enforcement | 9. Tool use & function calling optimization | production-standard |
| Tool-result / observation truncation and windowing | 5. Context-window management & compression | production-standard |
| Tool-result compression / truncation | 9. Tool use & function calling optimization | production-standard |
| Tool-use benchmarking / capability gating | 9. Tool use & function calling optimization | production-standard |
| Tool/retrieval gating for cost | 13. Model routing, cascades & cost optimization | emerging |
| ToolChain* | 10. Planning algorithms & world models for agents | research |
| Toolformer | 9. Tool use & function calling optimization | research |
| ToolkenGPT (tools as learned tokens) | 9. Tool use & function calling optimization | research |
| ToolLLM / ToolBench (large-scale tool-use instruction tuning) | 9. Tool use & function calling optimization | research |
| Trajectory replay / offline policy evaluation | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Tree of Thoughts | 2. Test-time compute scaling & search over reasoning | research |
| Tree-of-Mixed-Thought / Thought-of-Search and learned planning programs | 10. Planning algorithms & world models for agents | research |
| Tree-of-Thoughts agent search successor — ToolChain* / A*-guided tool-action search (explicit A*) | 10. Planning algorithms & world models for agents | research |
| Tree-Planner | 10. Planning algorithms & world models for agents | research |
| Typed multi-view memory | 6. Agent memory architectures | research |
| Uncertainty estimation / confidence calibration | 16. Reliability, structured generation & evaluation-as-optimization | emerging |
| Universal Self-Consistency | 2. Test-time compute scaling & search over reasoning | emerging |
| Verifier-gated cascade / answer verification deferral | 13. Model routing, cascades & cost optimization | emerging |
| Verifier-guided / reward-guided decoding | 2. Test-time compute scaling & search over reasoning | emerging |
| Voyager skill library | 6. Agent memory architectures | research |
| Weight offloading / heterogeneous inference | 14. Inference / serving optimizations beneath agents | production-standard |
| Weighted Self-Consistency | 2. Test-time compute scaling & search over reasoning | emerging |
| World-model / look-ahead simulation | 10. Planning algorithms & world models for agents | research |
| XGrammar | 16. Reliability, structured generation & evaluation-as-optimization | production-standard |
| Zero-shot Chain-of-Thought | 1. Prompting & reasoning strategies | production-standard |

---

## Scope & honesty

- **This is a map, not a benchmark.** No number here is a fak measurement; the families fak has
  measured link out to their own witnessed result docs.
- **References are anchors, not a bibliography.** They are canonical "Author et al., YEAR" pointers
  to aid search; confirm any precise identifier at the source before relying on it.
- **The taxonomy has fuzzy edges.** Many methods could live in two families (Self-RAG is retrieval
  *and* self-improvement; speculative decoding is serving *and* latency; ToT is reasoning *and*
  test-time search). Each is filed under its primary mechanism and cross-noted (↔) where it
  matters; the deduper folded 453 raw family entries into 370 distinct methods.
- **Coverage, not completeness.** "Every publicly known method" is the goal, but the field moves
  weekly. Treat this as a living index — the same fan-out + completeness-critic loop that built it
  is the loop to re-run as the field adds methods and families.
- **Placement and maturity are judgment calls.** A method tagged `production-standard` is in wide
  use; `emerging` is adopted but not yet default; `research` is published but not broadly deployed.
  Reasonable people will move some rows a tier.

## Sources & related reading (repo cross-links)

- Serving baseline: [SOTA serving optimizations](../explainers/sota-optimizations.md)
- Agentic caching layers: [Agentic caching SOTA](AGENTIC-CACHING-SOTA-2026-06-19.md)
- Scaling laws: [Scaling laws of agents](SCALING-LAWS-OF-AGENTS-2026-06-19.md)
- Ultra-long context: [Levels, levers, naming](RESEARCH-ultra-long-context-levels-and-naming-2026-06-22.md)
- Context economics: [O(1) context window economics](../explainers/o1-context-window-economics.md)
- Context planner: [The O(1) current turn](O1-TURN-CONTEXT-PLANNER-2026-06-23.md)
- Context window baseline: [ctxwin baseline](CTXWIN-CONTEXT-WINDOW-BASELINE-2026-06-23.md)
- KV cache for agents: [KV cache as agentic context grows](../explainers/kv-cache-agentic-context.md)
- Addressable KV cache: [Addressable KV cache](../explainers/addressable-kv-cache.md)
- Grammar-constrained decoding: [Grammar-constrained tool-call decoding](RESEARCH-grammar-constrained-tool-call-decoding-2026-06-22.md)
- Quantization: [AWQ quantization](../explainers/awq-quantization.md)
- Memory layers: [The four layers of agent memory](../MEMORY-LAYERS-EXPLAINER.md) · [Context is not memory](../CONTEXT-IS-NOT-MEMORY.md)

*Last updated: 2026-06-23.*
