Self Evolving Agents Tutorial En

§0 TL;DR Cheat Sheet

1. Core problem: have an agent continuously improve its capability on long-horizon tasks, without relying on repeated human annotation. Formalized as the convergence / stability / asymptotic effectiveness of an update operator $\mathcal{T}$ such that $\pi_t \to \pi_{t+1}$.
2. Three paradigms: ① Experience-Driven (human-made tasks + reward, e.g. AgentTuning, Voyager); ② Adversarial Self-Play (Challenger-Solver, e.g. Absolute Zero, Ctx2Skill); ③ Meta-Learning / Reward-Free (task-free, reward-free exploration + outcome-based reward, e.g. Native Evolution).
3. Capability container: natural-language skill / world knowledge K written in markdown — this is the most important paradigm shift of 2024-2026, bypassing parameter updates, everything is inference-time system_prompt += K.
4. Ctx2Skill 5-role self-play (arXiv 2604.27660): Challenger / Reasoner / Judge / Proposer / Generator, frozen LM but skill set evolves. Cross-Time Replay picks $\arg\max_i \rho^h_i \cdot \rho^e_i$ to prevent adversarial collapse.
5. Native Evolution two-phase (arXiv 2604.18131): Evolution phase explores task-free and reward-free → distills markdown K; Execution phase uses K as system prompt. Training signal $R_\text{evolve}(\mathcal{K}) = \text{Success}(\mathcal{T}_E\mid\mathcal{K}) - \text{Success}(\mathcal{T}_E\mid\varnothing)$.
6. A²RD trio (arXiv 2605.06924): MVMem (textual states + frames + videos + dependency DAG) + Adaptive Segment Gen + HITS (frame-level + video-level self-check). Directly transferable as a memory + audit template for any long-horizon agent.
7. Theoretical upper bound: [arXiv:2601.05280] shows that closed-loop density matching degenerates in the absence of an exogenous grounding signal (not that all reward-free training must collapse; this is a specific conclusion for that setting); [arXiv:2507.00075] models the solver-verifier gap and empirically fits it to capability dynamics.
8. Common failures: adversarial collapse (Challenger gets extreme), memory drift (internal contradictions accumulating in K), reward hacking (self-rewarding drift), bias amplification (agent retrained on its own output), capability ceiling (self-improvement degrades when exogenous grounding is missing).

§1 Self-Evolving Agent Intuition

"Self-evolving" is not magic. An LLM agent consists of four parts:

• policy $\pi$: a parameterized or pure in-context decision function
• memory / knowledge $\mathcal{K}$: externalized long-term state, typically in markdown
• skills $\mathcal{S}$: reusable procedural knowledge, each skill being a markdown document
• environment $E$: the interactable world (web / code / paper / sandbox / OS)

"Self-evolution" is defining an update operator $\mathcal{T}$:

$$\big(\pi_t, \mathcal{K}_t, \mathcal{S}_t\big) \xrightarrow{\mathcal{T}(E, \text{trajectories})} \big(\pi_{t+1}, \mathcal{K}_{t+1}, \mathcal{S}_{t+1}\big)$$

By the object updated by $\mathcal{T}$, 2024-2026 work roughly divides into four "layers":

Two intuitions commonly missed in interviews:

• ❌ "self-evolving = automatically training new model weights" — wrong, the vast majority of work does not update parameters (Voyager, Ctx2Skill, Generative Agents, A²RD are all training-free / inference-time).
• ❌ "self-evolving = the agent does whatever it wants" — wrong, the 2026 mainstream is strictly grounded self-improvement: either a code executor (Absolute Zero), a math checker (STaR), a rubric judge (Ctx2Skill), or an outcome utility (Native Evolution).

§2 Formalizing the Three Self-Evolution Paradigms

The Native Evolution paper [arXiv:2604.18131] gives a very clear classification — we add the mathematical formulation.

2.1　Experience-Driven Evolution

Setting: humans provide a task set $\mathcal{T}$, a reward function $R: \mathcal{O} \times \mathcal{A} \to \mathbb{R}$, and a workflow. The agent runs trajectories $\tau$, weighted by $R(\tau)$ to update.

Update operator:

$$\theta_{t+1} = \theta_t + \eta \,\mathbb{E}_{\tau \sim \pi_{\theta_t}}\!\left[\nabla_\theta \log \pi_\theta(\tau)\, R(\tau)\right]$$

This is standard policy gradient — AgentTuning, ToolLLM, early Voyager variants fall in this category.

Pros: high supervision density, fast convergence. Cons: huge labor cost (each new environment needs new reward design).

2.2　Adversarial Self-Play Evolution

Setting: two agents (Challenger + Solver) evolve jointly, with no external task source — tasks are produced by Challenger, solved by Solver, with verifier feedback.

Update operator (in Absolute Zero / R-Zero formalization):

$$\theta^{\text{ch}}_{t+1}, \theta^{\text{sol}}_{t+1} = \arg\min_{\theta^{\text{ch}}, \theta^{\text{sol}}} \;\mathbb{E}_{t\sim \pi^\text{ch}_t}\big[\ell^\text{ch}(t)\big] + \lambda\, \mathbb{E}_{(t,a)\sim \pi^\text{ch}_t,\pi^\text{sol}_t}\big[\ell^\text{sol}(t,a)\big]$$

Concrete "learnability reward" (Absolute Zero, arXiv 2505.03335):

$$R^\text{learn}(t) = \pi^\text{sol}_t(\text{correct}\mid t)\cdot \big(1 - \pi^\text{sol}_t(\text{correct}\mid t)\big)$$

Maximizing it favors 50% difficulty — neither too easy nor too hard. This is the core of curriculum-as-reward.

Pros: no need for human-made task sets; Cons: still need verifier (code executor / math checker), and prone to adversarial collapse (Challenger produces extreme tasks, Solver learns trivial defense).

2.3　Meta-Learning / Reward-Free Evolution

Setting (Native Evolution): training stage provides outcome-based reward (not step-level); inference stage has no task, no reward — the agent autonomously explores → distills markdown world knowledge $\mathcal{K}$ → uses $\mathcal{K}$ as system prompt in downstream tasks.

Reward design (Native Evolution core formula):

$$\boxed{\;R_\text{evolve}(\mathcal{K}) \;=\; \text{Success}(\mathcal{T}_E \mid \mathcal{K}) \;-\; \text{Success}(\mathcal{T}_E \mid \varnothing)\;}$$

where $\mathcal{T}_E$ is the set of downstream tasks in environment $E$ (observable at training time). Reward measures the downstream utility gain of K, without needing step-level supervision.

Pros: completely task-free / reward-free at inference; Cons: high training cost (rejection sampling RFT × 2 iterations), and $\mathcal{T}_E$ still needs labeled data at training time.

2.4　Three-paradigm comparison (memorize)

§3 Markdownification of Skills / Knowledge (the most important engineering shift)

The most important paradigm shift in 2024 is: long-term memory and capability extension are not via weight updates, but via external markdown documents.

3.1　Convergence of Anthropic Skills + Native Evolution

Anthropic publicly released the skills/ paradigm in 2025 (each skill is a standalone markdown file, the agent loads on demand into the system prompt). Native Evolution explicitly cites Anthropic skills in the paper as a reference implementation for K (paper §3, footnote 1 pointing to github.com/anthropics/skills/tree/main/skills).

→ Convergence conclusion: system prompt is the new model weights, markdown documents are the new fine-tuning data.

3.2　Typical Schema of Skill / K Files

# skill_name
## Trigger / When-to-use
<which task should use this skill>

## Steps
1. ...
2. ...

## Resources / References
- file paths / URLs

## Failure modes
- Known pitfalls + fixes

Native Evolution's K also explicitly stores:

• Visual arcs (visual evolution of entities/environments)
• Spatial relations (subject-relation-object triplets)
• Camera states / Site map (environment topology)
• Token budget allocation (how many tokens each sub-page gets)

3.3　Why markdown rather than vector embedding?

• Interpretable: humans can audit, revise, merge
• Composable: skill A + skill B directly prepend
• Routable: LLM can read the trigger section and decide which to load
• Evolvable: natural language diff is more stable than vector diff

Cost: retrieval precision is worse than vector RAG; the fix is hybrid (vector for candidate selection → markdown for close reading).

§4 Voyager / Reflexion / STaR: the foundational trio

Before going into 2024-2026 frontier work, you must chew through the three foundational works — interviewers will almost certainly ask about baselines.

4.1　Voyager (Wang 2023 NeurIPS, NVIDIA + Caltech)

The first true end-to-end "automatic curriculum + skill library" agent: running GPT-4 in Minecraft, letting it propose its own tasks, write JS code (each piece of code is a skill), self-verify, and store successful skills in the library.

The core trio:

• Automatic Curriculum: GPT-4 proposes the next task based on the current inventory state
• Skill Library: each new skill is a JS function, indexed by embedding and retrieved by description
• Iterative Prompting + Self-Verification: execute → env feedback → critic agent → revise, until passes

4.2　Reflexion (Shinn 2023 NeurIPS, Northeastern)

Solidifies the concept of "verbal RL": after each failure, the agent writes a reflection in natural language about its own trajectory, stores it in episodic memory, and prepends it to the next prompt.

Formalization (pseudo-Bellman):

$$M_{t+1} = M_t \cup \big\{\text{reflect}(\tau_t, r_t)\big\}$$

where $\text{reflect}$ is a text generation of "what went wrong + how to fix" by the LLM itself.

Why it works (theoretically): reflection compresses sparse reward signal into structured text, bypassing gradient updates; equivalent to a kind of non-parametric policy improvement in the in-context domain. But lacks convergence guarantee.

4.3　STaR (Self-Taught Reasoner, Zelikman 2022 NeurIPS, Stanford)

Let the LM generate rationale → if wrong, rationalize using the true answer → SFT on correct (q, rationale, a) tuples. This is the true starting point of self-improvement on reasoning.

Pseudocode:

for iter in 1..N:
    for each (q, a_gt) in D:
        r, a_pred = LM(q)
        if a_pred == a_gt:
            collect (q, r, a_gt)
        else:
            r' = LM(q, hint=a_gt)        # rationalize
            if r' produces a_gt:
                collect (q, r', a_gt)
    SFT(LM, collected)

STaR's key flaw (also [arXiv:2601.05280]'s core argument against self-improvement): rationalization is reverse-engineering the answer, not necessarily reflecting the true reasoning process, leading to distribution drift.

§5 Ctx2Skill: 5-Role Self-Play Loop (Focus 1)

This section nearly mirrors arXiv 2604.27660's §3 word-for-word, since interviewers may quote the paper directly.

5.1　Problem formulation

Given a context $C$ (possibly 100k+ tokens of manual / paper / repo / dataset), a task set $\mathcal{T} = \{t_j\}$, each task has a binary rubric set $\mathcal{R}_j = \{r_{j,k}\}$. Solving indicator:

$$y_j(\pi; C) = \prod_k \mathbb{I}\big[r_{j,k}(a_j) = \text{pass}\big], \quad a_j \sim \pi(\,\cdot\mid C, t_j)$$

Goal: construct a markdown skill set $\mathcal{S}^R$ such that:

$$a_j \sim \pi(\,\cdot\mid \mathcal{S}^R, C, t_j) \quad \text{maximizes}\ \mathbb{E}_j y_j$$

and without updating $\pi$'s parameters — only updating $\mathcal{S}^R$.

5.2　Five frozen LM roles

Note that two sides evolve independently:

• Reasoner side: failed cases → Reasoner Proposer diagnoses "what contextual knowledge is missing" → Reasoner Generator writes new $\mathcal{S}^R_i$
• Challenger side: too-easy-solved cases → Challenger Proposer diagnoses "why is the challenger producing tasks too weak" → Challenger Generator strengthens $\mathcal{S}^C_i$

→ These two sides never exchange skill sets — maintaining strict adversarial pressure.

5.3　Cross-Time Replay mechanism (core anti-collapse)

The more iterations, the more extreme the Challenger gets, and the more the Reasoner over-specializes to extreme tasks. Returning $\mathcal{S}^R_N$ directly is bad.

Replay procedure:

1. During training, maintain two probe sets:

   • Hard set $\mathcal{Q}^h$: each iteration, pick the failed task with the lowest rubric pass rate
   • Easy set $\mathcal{Q}^e$: each iteration, pick the solved task with the fewest rubrics passed ("just barely solved")
2. After training, for each candidate $\mathcal{S}^R_i$ ($i=1\ldots N$), run the Reasoner $\pi^R$ on both probe sets:

$$\rho^h(i) = \frac{\sum_{q\in \mathcal{Q}^h} y_q(\pi^R; C, \mathcal{S}^R_i) + 1}{|\mathcal{Q}^h| + 1}, \quad \rho^e(i) = \frac{\sum_{q\in \mathcal{Q}^e} y_q(\pi^R; C, \mathcal{S}^R_i) + 1}{|\mathcal{Q}^e| + 1}$$

(Laplace smoothing prevents empty probe sets)

1. Select:

$$\boxed{\;\mathcal{S}^R_\star = \mathcal{S}^R_{i^\star}, \quad i^\star = \arg\max_i \big(\rho^h(i) \cdot \rho^e(i)\big)\;}$$

Why product, not sum: product penalizes catastrophic forgetting (if some version has $\rho^e \to 0$, the overall score → 0), forcing selection of versions that are not bad on either side. Ctx2Skill ablation shows using sum drops final accuracy by ~1.5%.

5.4　Ctx2Skill 5-role + Replay code skeleton

def ctx2skill_loop(context: str, llm, num_iters: int = 5, M: int = 5):
    """
    Ctx2Skill: 5 frozen LM roles + Cross-Time Replay.
    Returns the optimal Reasoner skill set selected by cross-time replay.
    All LM calls use the same frozen backbone; only the skill set changes.
    """
    S_R = ""                    # Reasoner skill markdown (initially empty)
    S_C = ""                    # Challenger skill markdown
    candidates = []             # Historical S_R candidates (cross-time)
    Q_hard, Q_easy = [], []     # Two probe sets

    for i in range(1, num_iters + 1):
        # ── (1) Challenger produces a batch ──
        batch = llm(role="challenger", prompt=challenger_prompt(context, S_C), n=M)
        # batch = [(t_m, rubrics_m), ...]

        failed, solved = [], []
        for t_m, rubrics_m in batch:
            # ── (2) Reasoner solves ──
            a_m = llm(role="reasoner", prompt=reasoner_prompt(context, S_R, t_m))
            # ── (3) Judge per-rubric ──
            per_rubric = [llm(role="judge", prompt=judge_prompt(a_m, r))
                          for r in rubrics_m]
            y_m = all(per_rubric)
            pass_rate = sum(per_rubric) / len(per_rubric)
            (failed if not y_m else solved).append(
                (t_m, rubrics_m, a_m, pass_rate)
            )

        # ── Maintain probe sets (preparation for Laplace smoothing) ──
        if failed:
            hardest = min(failed, key=lambda x: x[3])
            Q_hard.append((hardest[0], hardest[1]))
        if solved:
            # The "lowest pass_rate among solved" (i.e. "barely solved" — all rubrics pass but many prompts just barely pass)
            # Note: entries in `solved` all satisfy all(per_rubric), so pass_rate=1.0;
            # in production, "barely solved" should use per-rubric soft scores (e.g. LLM-judge giving [0,1] rather than 0/1),
            # here teaching version uses the task closest to the solving boundary (e.g. the task in batch with most reasoner retries)
            easiest_among_solved = solved[-1]  # Teaching simplification: take the last solved task
            Q_easy.append((easiest_among_solved[0], easiest_among_solved[1]))

        # ── (4) Two-sided Proposer diagnoses ──
        diag_R = llm(role="reasoner_proposer",
                     prompt=proposer_prompt(failed, S_R))
        diag_C = llm(role="challenger_proposer",
                     prompt=proposer_prompt(solved, S_C))

        # ── (5) Two-sided Generator writes skills ──
        S_R = llm(role="reasoner_generator",
                  prompt=generator_prompt(diag_R, S_R))
        S_C = llm(role="challenger_generator",
                  prompt=generator_prompt(diag_C, S_C))

        candidates.append(S_R)

    # ── Cross-Time Replay ──
    best_idx, best_score = 0, -1.0
    for i, cand in enumerate(candidates):
        rho_h = laplace_smoothed_rate(Q_hard, cand, llm, context)
        rho_e = laplace_smoothed_rate(Q_easy, cand, llm, context)
        score = rho_h * rho_e
        if score > best_score:
            best_score, best_idx = score, i

    return candidates[best_idx]


def laplace_smoothed_rate(probe, skill_set, llm, context):
    """ Laplace-smoothed pass rate: (sum_q y_q + 1) / (|probe| + 1).
    
    Args:
        probe:     list[(task, rubrics)]
        skill_set: candidate S_R^i markdown skill to evaluate
        llm:       frozen LM
        context:   original context (same source as ctx2skill_loop parameter; must be passed explicitly
                   to prevent closure misuse)
    """
    num_pass = 0
    for t_q, rubrics_q in probe:
        a = llm(role="reasoner",
                prompt=reasoner_prompt(context, skill_set, t_q))
        if all(llm(role="judge", prompt=judge_prompt(a, r))
               for r in rubrics_q):
            num_pass += 1
    return (num_pass + 1) / (len(probe) + 1)

5.5　Ctx2Skill experimental results (must memorize)

On CL-bench, without any parameter updates:

→ GPT-4.1 + Ctx2Skill (16.5%) surpasses Gemini 3 Pro without skills (15.8%) — confirming "high-quality skills can compensate for model gap."

5.6　Ctx2Skill ablation (interview bonus)

The drop from removing Challenger evolving is the largest — proving that "sustained adversarial pressure" is the true driver of Reasoner progress.

§6 Native Evolution: Reward-Free Meta-Learning (Focus 2)

Fully corresponds to arXiv 2604.18131. Tencent + HKUST(GZ), 2026-04-20.

6.1　Core architecture: two-phase decoupling

  ┌─────────────────────────────────┐       ┌──────────────────────────────┐
  │      Native Evolution Phase     │       │   Knowledge-Enhanced Execution│
  │      (task-free + reward-free   │       │   (uses K as system prompt at │
  │       at inference)             │       │    inference)                │
  │                                 │       │                              │
  │   π_θ(K | E)                    │  ──→  │   π_task(a_t | o_t, K, Task) │
  │   "exploring + summarizing"     │       │                              │
  │                                 │       │                              │
  └──────────────┬──────────────────┘       └──────────────────────────────┘
                 │
                 │ (during training, use outcome-based reward to supervise evolve)
                 ▼
  R_evolve(K) = Success(T_E | K) − Success(T_E | ∅)

Key design choice: evolution and execution use the same LLM (unlike RLHF which separates SFT-policy / RM); they only have different system prompts + training that goes through SFT + RFT to learn "evolution mode."

6.2　Outcome-Based Reward Design

$$\boxed{\;R_\text{evolve}(\mathcal{K}) = \underbrace{\text{Success}(\mathcal{T}_E\mid \mathcal{K})}_{\text{downstream success rate with K}} - \underbrace{\text{Success}(\mathcal{T}_E\mid \varnothing)}_{\text{no-K baseline}}\;}$$

where $\text{Success}(\mathcal{T}_E\mid \mathcal{K}) = \frac{1}{M}\sum_{j=1}^M \mathbb{I}\big[f(Q_j, \mathcal{K}) = A_j\big]$.

Why outcome-based rather than step-level?

Native Evolution chooses outcome-based for another special reason: $\mathcal{K}$ is a long markdown stretch (374.8 steps × 3322.4 tokens/step); step-level reward is nearly meaningless on such a long horizon.

6.3　Two-phase training: SFT → RFT

Stage 1 (SFT):

• Use teacher model $\pi_T$ (Gemini-2.5-Pro) to generate 3 candidates $\{\mathcal{K}_i\}_{i=1}^3$
• Compute $R_\text{evolve}(\mathcal{K}_i)$, pick the best $\mathcal{K}^\star$
• Use $T^\star = \{Q, o_1^\star, a_1^\star, \ldots, o_k^\star, a_k^\star\}$ trajectory to SFT base model $\pi_{\theta_0}$
• Training data: 600 deep search questions × 20 websites

Stage 2 (RFT, Rejection Sampling Fine-Tuning):

• Use $\pi_{\theta_1}$ itself to generate $C$ K candidates
• Pick the highest scoring by $R_\text{evolve}$
• Continue fine-tuning with the highest scoring trajectory
• Run 2 iterations

6.4　Native Evolution training + inference code skeleton

def native_evolution_pipeline(base_model, teacher_model, env_pool,
                              downstream_tasks_per_env, num_iter=2,
                              C_sft: int = 3, C_rft: int = 8):
    """
    Native Evolution: SFT + RFT (2 iter) → learn reward-free self-evolution.
    
    Args:
        C_sft: number of teacher-generated K candidates in SFT stage (paper: 3)
        C_rft: number of pi-self-generated candidates in RFT stage (paper: 8)
    """
    # ── Stage 1: SFT ──
    sft_data = []
    for E in env_pool:
        T_E = downstream_tasks_per_env[E]            # labeled downstream
        # baseline: without K
        s0 = success_rate(base_model, T_E, K=None)

        # teacher generates C_sft candidate Ks
        candidates = [explore_and_summarize(teacher_model, E)
                      for _ in range(C_sft)]
        # Evaluate reward = Success(T_E | K) − Success(T_E | ∅)
        rewards = [success_rate(base_model, T_E, K=K) - s0
                   for K in candidates]
        K_star = candidates[argmax(rewards)]
        traj_star = extract_trajectory(teacher_model, E, K_star)
        sft_data.append(traj_star)                   # ~374 steps each

    pi_1 = sft(base_model, sft_data)                 # warm-up

    # ── Stage 2: RFT × num_iter ──
    pi = pi_1
    for it in range(num_iter):
        rft_data = []
        for E in env_pool:
            T_E = downstream_tasks_per_env[E]
            s0 = success_rate(pi, T_E, K=None)
            # pi itself generates C_rft candidates
            candidates = [explore_and_summarize(pi, E) for _ in range(C_rft)]
            rewards = [success_rate(pi, T_E, K=K) - s0
                       for K in candidates]
            best = candidates[argmax(rewards)]
            rft_data.append(extract_trajectory(pi, E, best))
        pi = sft(pi, rft_data)                       # next iter

    return pi   # π_θ*: has learned native evolution


def native_evolution_inference(pi_star, new_env, task):
    """
    At inference: no task, no reward → explore → distill K → solve task with K.
    """
    K = explore_and_summarize(pi_star, new_env)      # task-free!
    answer = pi_star(task, system_prompt=K)          # K-augmented
    return answer

6.5　Native Evolution experimental results

WebVoyager + WebWalker, 14B Qwen3 / 36B Seed-OSS:

Most striking: 14B Qwen3 + transferred K from 36B → 35.6% conference accuracy; unassisted Gemini-2.5-Flash is only 31.3% — proving high-quality K can surpass pure parameter scaling.

6.6　Native Evolution vs Ctx2Skill comparison

→ They are complementary: Native Evolution lets the backbone learn how to explore; Ctx2Skill lets a frozen backbone distill context into reusable skills. They can be stacked.

§7 A²RD and Long-Horizon Memory Architecture (Focus 3)

arXiv 2605.06924, Google Cloud AI + NUS, 2026-05-07. While the paper is about video, the memory schema transfers directly to all long-horizon agents.

7.1　Retrieve → Synthesize → Refine → Update closed loop

   ┌──────────────────────────────────────────────────────────────┐
   │   for segment i = 1..N:                                       │
   │     1. Retrieve relevant context from MVMem (T_j, F_j, V_j)   │
   │     2. Decide mode: extrapolation vs interpolation            │
   │     3. Synthesize boundary frames F_i^begin, F_i^end          │
   │     4. HITS (frame-level): verify + revise frames             │
   │     5. Synthesize video segment V_i = TI2V(P_i, F_i, F^rel)   │
   │     6. HITS (video-level): verify + revise via MAPO           │
   │     7. Update MVMem with (F_i, V_i, T_i, T_{i+1}^F)           │
   └──────────────────────────────────────────────────────────────┘

7.2　MVMem schema (textual states + frames + videos)

$$\mathcal{M} := \{\mathcal{M}_1, \ldots, \mathcal{M}_N\} \cup \mathcal{R} \cup \mathcal{D}$$

Each segment $\mathcal{M}_j = \{T_j, \mathcal{F}_j, V_j\}$:

• Textual States $T_j$: contain Visual Arcs (entity identity / motion) + Spatial Relations (subject-relation-object triplets, grounding geometric layout) + Camera trajectories
• Frames $\mathcal{F}_j = \{F_j^\text{begin}, F_j^\text{end}\}$: keyframes
• Videos $V_j$: full segment

Plus:

• $\mathcal{R}$: global reference frames (background, entity references)
• $\mathcal{D}$: prompt database (also stores failed prompts)

7.3　Dependency DAG (key trick)

References have dependencies: entities depend on environment, camera depends on entity positions. A²RD builds a DAG:

$$\mathcal{G} := \text{MLLM}_\text{dep}(\mathcal{P}_\mathcal{R})$$

Then topological sort decides synthesis order. Transfers directly to ARIS-style agents: in research projects claim ← experiment ← code ← idea, typed memory is also a DAG.

7.4　HITS: Hierarchical Test-Time Self-Improvement

Two levels:

• Frame-level HITS: for each $F^\text{begin}, F^\text{end}$, use VLM to verify "matches textual state," if not → MAPO (Multi-Aspect Prompt Optimization) revises the prompt → regenerate
• Video-level HITS: for the full segment $V_i$, use VLM to verify "narrative continuity," if not → revise video prompt → regenerate

→ Self-check at two scales: inner-segment + inter-segment — stronger anti-drift than single-layer self-improvement.

7.5　Transfer to general long-horizon agents (typical cheat-sheet)

class TypedMemory:
    """A²RD MVMem idea ⇒ general long-horizon agent memory."""
    def __init__(self):
        self.segments = []          # list of {state, artifacts, deps}
        self.global_refs = {}       # Global entities (e.g. paper-level claim)
        self.dep_graph = {}         # DAG: which artifact depends on which
        self.failure_db = []        # Failure trace database

    def retrieve(self, current_segment_ctx, k=3):
        """Retrieve narratively-relevant context (top k previous segments)."""
        cands = []
        for j, M_j in enumerate(self.segments):
            score = relevance(M_j["state"], current_segment_ctx)
            cands.append((score, j))
        topk = sorted(cands, reverse=True)[:k]
        return [self.segments[j] for _, j in topk]

    def update(self, segment, deps):
        self.segments.append(segment)
        seg_id = len(self.segments) - 1
        self.dep_graph[seg_id] = deps      # parent ids

    def topo_synthesis_order(self, num_segments: int) -> list[int]:
        """A²RD's dependency DAG → decide generation order.
        
        Args:
            num_segments: total number of segments to generate; automatically adds nodes
                          not in dep_graph as roots.
        Returns:
            A valid topological order (list of segment indices).
        """
        # Automatically adds all 0..num_segments-1 to the graph (those without dependencies treated as roots)
        graph = {i: self.dep_graph.get(i, []) for i in range(num_segments)}
        return topological_sort(graph)


def long_horizon_agent_with_hits(memory, segments_to_generate, llm, verifier):
    """A²RD-style R→S→R→U closed loop.
    
    Note: segments_to_generate is a list of context descriptions for the segments to generate;
    generation order is determined by memory.dep_graph (defaults to sequential if empty).
    """
    order = memory.topo_synthesis_order(num_segments=len(segments_to_generate))
    for i in order:
        # Retrieve
        ctx = memory.retrieve(segments_to_generate[i])
        # Synthesize
        artifact = llm.generate(segments_to_generate[i], context=ctx)
        # Frame-level HITS (internal consistency of artifact)
        for _ in range(MAX_REFINES):
            if verifier.frame_check(artifact): break
            artifact = llm.refine(artifact, verifier.feedback)
        # Video-level HITS (consistency of artifact with history)
        for _ in range(MAX_REFINES):
            if verifier.video_check(artifact, ctx): break
            artifact = llm.refine(artifact, verifier.feedback)
        # Update
        memory.update(artifact, deps=ctx)

§8 Theoretical Upper Bound of Self-Improvement (L3 level)

Two 2025-2026 must-read theoretical papers — this is the L3 part that top labs may ask in interviews.

8.1　On the Limits of Self-Improving in LLMs (arXiv 2601.05280)

Full title: "On the Limits of Self-Improving in LLMs: The Singularity Is Not Near Without Symbolic Model Synthesis"

Setup: model self-training as a dynamical system on probability distributions:

$$p_{t+1} = \mathcal{T}_\text{closed}(p_t) = \mathbb{E}_{x \sim p_t}\big[\delta_{x'}\big],\quad x' = \pi_t(x)$$

i.e. $p_{t+1}$ is the distribution obtained by retraining the model on its own samples.

Main theorem (narrative version): under closed-loop density matching (no exogenous grounding signal), if $\pi_t$ has no access to ground truth, then $\{p_t\}$ generally does not converge to the target $p^\star$, and degenerates in mode collapse / drift.

Core mechanism:

$$D_\text{KL}(p^\star \,\|\, p_{t+1}) \;\ge\; D_\text{KL}(p^\star \,\|\, p_t) - \Delta_\text{grounding}$$

where $\Delta_\text{grounding}$ is the KL reduction brought by the grounding signal. With no grounding ($\Delta = 0$), the KL does not decrease but actually rises.

Positive implication: self-improvement needs exogenous grounding — code executor / math checker / human label / rubric judge — this is why Absolute Zero must hook up a code executor, STaR must use ground-truth answers for rationalization, Ctx2Skill must use a Judge to verify rubrics.

8.2　Solver-Verifier Gap (arXiv 2507.00075)

Setup: model capability evolution as coupled dynamics of two variables $\theta^\text{sol}, \theta^\text{ver}$:

$$\begin{cases} \dot\theta^\text{sol} = \eta_s\, g_s(\theta^\text{sol}, \theta^\text{ver}) \\ \dot\theta^\text{ver} = \eta_v\, g_v(\theta^\text{sol}, \theta^\text{ver}) \end{cases}$$

Empirical observation: capability $C(\theta)$ under self-improvement follows (a fitted) exponential law:

$$C(\theta_t) \approx C_\infty - (C_\infty - C_0)\, e^{-\kappa t}$$

and $\kappa$ is positively correlated with the solver-verifier gap $\Delta := C^\text{ver} - C^\text{sol}$ (larger gap → faster improvement), but too large a gap also saturates (verifier gives feedback that the solver cannot learn).

Engineering guidance:

• Cross-model reviewers (e.g. ARIS using Codex 5.5 to review Claude) naturally create a verifier-solver gap → accelerate self-improvement
• Same-model self-review has almost no gap → slow or ineffective convergence

8.3　Practical implications of the two papers

→ Combining the two: reward-free at inference + grounded at training is the fundamental reason work like Native Evolution can work; ARIS-style cross-model audit is the system-level engineering choice to accelerate self-improvement.

§9 Memory-Driven Self-Evolution

9.1　Generative Agents (Park 2023 UIST, Stanford)

The most classic long-horizon simulation: observation stream → memory store → reflection (LLM writes insights itself) → planning.

Three layers of memory:

• Observation memory: raw timestamp + literal description
• Reflection memory: "insight" generated by retrieval-augmented LLM
• Planning memory: long-term goal

Retrieval score:

$$\text{score}(m) = \alpha_\text{recency}\, r(m) + \alpha_\text{importance}\, i(m) + \alpha_\text{relevance}\, s(m, q)$$

where $r(m) = \gamma^{\Delta t}$ (exponential decay), $i(m) \in [1,10]$ (LLM self-rated), $s(m,q)$ cosine similarity.

9.2　MemGPT (Packer 2023, Berkeley)

OS-style hierarchical memory:

• Main context (LLM token budget) = "RAM"
• External archival (disk) = "HDD"
• LLM learns paging: memgpt_function_call(load, save, summarize)

Core trick: let the LLM observe its own token usage within its context, actively deciding to swap pages — this brings the OS abstraction into the LLM agent.

9.3　Relation of this layer to Ctx2Skill / Native Evolution

→ The 2024-2026 evolutionary direction of memory-driven work: from episodic (GA) → hierarchical (MemGPT) → typed + DAG (MVMem).

§10 Skill / K Retrieval and Ranking (engineering practice)

In actual deployment, with skill libraries of dozens to hundreds, you must load on demand — otherwise tokens explode.

10.1　Hybrid retrieval pipeline

def hybrid_skill_retrieval(task: str, skills: list, k=3):
    """
    Stage A: Coarse filter (vector embedding, fast)
    Stage B: Fine ranking (LLM scoring on description, accurate)
    Stage C: Exact match on trigger section (deterministic)
    """
    # ── Stage A: BM25 + dense embedding hybrid ──
    bm25_scores = bm25_search(task, [s.description for s in skills], topn=20)
    dense_scores = dense_search(task, [s.embedding for s in skills], topn=20)
    candidates = top_k(merge(bm25_scores, dense_scores), n=10)

    # ── Stage B: LLM rerank ──
    reranked = []
    for skill in candidates:
        prompt = f"task={task}\nskill trigger={skill.trigger}\n" \
                 f"Q: relevant? (yes / no / partial)"
        verdict = llm(prompt)
        score = {"yes": 1.0, "partial": 0.5, "no": 0.0}[verdict]
        reranked.append((score, skill))

    # ── Stage C: Strong keyword match ──
    keyword_hits = [s for s in skills
                    if any(kw in task.lower() for kw in s.exact_triggers)]

    # Merge and deduplicate → take top k
    final = top_k(reranked + [(2.0, s) for s in keyword_hits], k=k)
    return [s for _, s in final]

10.2　Skill ranking formula

Weighted fusion of 3 signals:

$$\text{score}(s, q) = \alpha_\text{sim}\, \cos(\mathbf{e}_s, \mathbf{e}_q) + \alpha_\text{prior}\, \log(1 + n_\text{used}(s)) + \alpha_\text{recent}\, \gamma^{\Delta t}$$

where $n_\text{used}$ is historical call count (more frequent → more reliable), $\gamma^{\Delta t}$ is recency decay.

10.3　Skill update (prevent staleness)

Each skill maintains:

• success_count, fail_count
• last_updated
• version

Trigger conditions for update:

• fail_count / total > τ_fail (high failure rate) → revise
• last_updated > T_stale (stale) → re-explore
• Environment change detection → trigger Native-Evolution-style redistillation

§11 Inference-Time Orchestration (like ARIS) vs Training-Time Meta-Learning (like Native Evolution)

L3 must-ask at top labs: master the fundamental mathematical difference between inference-time orchestration and training-time meta-learning.

11.1　Mathematical formulation comparison

11.2　Respective limitations

Inference-time orchestration:

• ❌ Cannot internalize skills into weights → tokens need loading every time
• ❌ Context length limit becomes a bottleneck
• ❌ Hard to learn low-level subtoken patterns
• ✅ Immediately interpretable, auditable
• ✅ No GPU needed

Training-time meta-learning:

• ❌ Expensive training (rejection sampling × 2 iter, 600 tasks × 20 envs)
• ❌ Once trained, evolution capability is fixed
• ❌ Cross-environment generalization remains an open problem
• ✅ One training session, long-term benefit
• ✅ Behavior can be internalized into fast inference path

11.3　Hybrid form in real systems

Mainstream production agents are often both layers:

• Bottom layer: fine-tuned for instruction follow + tool use (one-time training)
• Top layer: inference-time orchestration of skills / memory (continuous update)

This is also the actual position of ARIS-type systems — the top layer is inference-time orchestration, with the bottom layer relying on already-trained-to-follow-skill backbones like GPT-4.5 / Claude Opus.

§12 Failure Modes and Defenses (memorize)

12.1　Adversarial Collapse

Symptom: the Challenger becomes increasingly extreme, the Reasoner's learned skills only work on extreme cases, degrading on normal cases.

Ctx2Skill solution: Cross-Time Replay picks $\arg\max \rho^h \cdot \rho^e$; the product form forces retention of easy task performance.

General solutions:

• Maintain a historical probe set (not just looking at current iteration)
• Pick the most balanced rather than last-iteration skill
• Use KL penalty to prevent abrupt skill set changes: $\mathcal{L}_\text{KL} = \beta \,D_\text{KL}(\mathcal{S}_i \| \mathcal{S}_{i-1})$ (textual KL can be approximated by BLEU/edit distance)

12.2　Memory Drift

Symptom: long-horizon agent accumulates contradictory / outdated information in K or memory, getting worse with use.

A²RD solution:

• HITS cross-checks each segment against retrieved context
• Typed memory schema enforces schema validation
• failure_db separately stores "known unreliable" traces

General solutions:

• Periodically run self-consistency check: have the LLM read its own K to find internal contradictions
• Add confidence decay to each entry
• Trigger "K redistillation" mechanism (Native-Evolution-style)

12.3　Reward Hacking

Symptom: in self-rewarding training (Yuan et al. 2024 Self-Rewarding LM), the model learns to game its own reward function.

Defenses:

• Cross-model reward: have a different model family act as verifier
• Outcome-grounded reward: don't let the LLM score itself; use code executor / math checker / labeled downstream task
• Reward dropout / regularization: random subset of reward components

12.4　Bias Amplification (Echo Chamber)

Symptom: STaR-style rationalization trains the model on its own generated rationales, amplifying mode collapse.

[arXiv:2601.05280]'s KL bound directly corresponds to this case — without exogenous grounding, KL does not decrease but rises.

Defenses:

• Always retain a portion of ground-truth supervision
• Use real human feedback as anchor
• Diverse seed prompts to force multimodality

12.5　Sandbox Contamination

Symptom: the agent generates its own test cases → trains on these cases → evaluation looks high but it's actually train-test overlap.

Defenses:

• Strict holdout set, agent has zero access
• Benchmark maintained by third party
• Multi-round canonical eval (e.g. Ctx2Skill uses CL-bench)

12.6　Capability Ceiling

Symptom: after N rounds of self-improvement the curve saturates, no amount of additional compute helps.

Solver-Verifier Gap [arXiv:2507.00075] explanation: when gap $\Delta \to 0$, $\kappa \to 0$, improvement rate → 0.

Breakthrough methods:

• Introduce a stronger verifier (e.g. replace with a stronger model as judge)
• Increase task difficulty (Ctx2Skill's Challenger evolving)
• Symbolic model synthesis (the last-resort lifesaver proposed in [arXiv:2601.05280]): have the LLM maintain a simultaneous symbolic / programmatic model to prevent drift

12.7　Hallucination Compounding (independent reviewers can co-hallucinate)

Symptom: cross-model reviewers all agree on a wrong conclusion (e.g. Claude writes + GPT reviews, both miss the same bug).

Defenses (also mentioned in briefing's codex round 2):

• Reviewers must connect to raw evidence / executable checks (unit test, claim audit against raw experiment output)
• Cannot rely only on LLM textual judgment

§13 25 Frequently-Asked Interview Questions (L1 + L2 + L3)

L1 Must-Know (Q1-Q10)

L2 Advanced (Q11-Q20)

L3 Top Lab (Q21-Q25)

§A Appendix: Complete from-scratch code skeleton

A.1　Complete Skill library implementation

import json, time, math
from dataclasses import dataclass, field, asdict
from typing import Callable, Optional


@dataclass
class Skill:
    """Markdown skill with metadata for retrieval + lifecycle."""
    name: str
    trigger: str            # when-to-use section
    body: str               # The actual markdown injected into system prompt
    exact_triggers: list = field(default_factory=list)
    success_count: int = 0
    fail_count: int = 0
    last_updated: float = field(default_factory=time.time)
    version: int = 1
    embedding: list = field(default_factory=list)


class SkillLibrary:
    """Skill persistence, retrieval, lifecycle management."""

    def __init__(self):
        self.skills: dict[str, Skill] = {}
        self.callbacks_on_update: list[Callable] = []

    def add(self, s: Skill) -> None:
        self.skills[s.name] = s

    def retrieve(self, query: str, embed_fn, k: int = 3) -> list[Skill]:
        """Hybrid retrieval: keyword + dense + recency."""
        q_emb = embed_fn(query)
        scored = []
        now = time.time()
        for name, s in self.skills.items():
            sim = self._cos(s.embedding, q_emb) if s.embedding else 0.0
            prior = math.log(1 + s.success_count)
            recency = math.pow(0.999, max(0, (now - s.last_updated) / 3600))
            kw_hit = 1.0 if any(t.lower() in query.lower()
                                for t in s.exact_triggers) else 0.0
            score = 0.5 * sim + 0.2 * prior + 0.2 * recency + 0.1 * kw_hit
            scored.append((score, s))
        scored.sort(reverse=True, key=lambda x: x[0])
        return [s for _, s in scored[:k]]

    @staticmethod
    def _cos(a: list, b: list) -> float:
        if not a or not b: return 0.0
        dot = sum(x * y for x, y in zip(a, b))
        na = math.sqrt(sum(x * x for x in a))
        nb = math.sqrt(sum(y * y for y in b))
        return dot / (na * nb + 1e-9) if na > 0 and nb > 0 else 0.0

    def update_outcome(self, skill_name: str, success: bool) -> None:
        s = self.skills[skill_name]
        if success: s.success_count += 1
        else:       s.fail_count    += 1

    def should_revise(self, skill_name: str,
                      tau_fail: float = 0.5,
                      t_stale: float = 7 * 24 * 3600) -> bool:
        s = self.skills[skill_name]
        total = s.success_count + s.fail_count
        if total > 0 and s.fail_count / total > tau_fail:
            return True
        if time.time() - s.last_updated > t_stale:
            return True
        return False

    def revise(self, skill_name: str, new_body: str) -> None:
        s = self.skills[skill_name]
        s.body = new_body
        s.last_updated = time.time()
        s.version += 1
        s.success_count = 0
        s.fail_count = 0
        for cb in self.callbacks_on_update:
            cb(s)

    def serialize(self) -> str:
        return json.dumps({n: asdict(s) for n, s in self.skills.items()})

    def assemble_prompt(self, skills: list[Skill]) -> str:
        return "\n\n".join([f"# {s.name}\n{s.body}" for s in skills])

A.2　Complete Reflexion memory implementation

@dataclass
class Reflection:
    trajectory_summary: str
    failure_root_cause: str
    fix_strategy: str
    timestamp: float


class ReflexionMemory:
    """verbal-RL style memory."""

    def __init__(self, max_entries: int = 50):
        self.entries: list[Reflection] = []
        self.max_entries = max_entries

    def add(self, traj: str, llm: Callable) -> None:
        """Let the LLM generate the reflection itself."""
        prompt = (
            f"Trajectory: {traj}\n\n"
            f"Task FAILED. Write a short reflection in JSON with keys: "
            f"trajectory_summary, failure_root_cause, fix_strategy."
        )
        raw = llm(prompt)
        try:
            obj = json.loads(raw)
        except Exception:
            obj = {"trajectory_summary": traj[:400],
                   "failure_root_cause": "parse_failed",
                   "fix_strategy": raw[:400]}
        self.entries.append(Reflection(
            trajectory_summary=obj["trajectory_summary"],
            failure_root_cause=obj["failure_root_cause"],
            fix_strategy=obj["fix_strategy"],
            timestamp=time.time(),
        ))
        # Keep the most recent max_entries entries
        if len(self.entries) > self.max_entries:
            self.entries = self.entries[-self.max_entries:]

    def render(self) -> str:
        """Render as prefix prompt."""
        return "\n".join([
            f"[Reflection {i}] cause: {r.failure_root_cause}\n"
            f"           fix: {r.fix_strategy}"
            for i, r in enumerate(self.entries[-5:])
        ])

A.3　Sanity-check output (illustrative)

[a] SkillLibrary.add + retrieve            ✓ topk = ['voyager_craft', 'minecraft_kill']
[b] update_outcome accumulates success_count ✓ s.success_count = 3
[c] should_revise trigger condition (fail rate) ✓ tau_fail=0.5 → True
[d] revise increments version by 1          ✓ s.version: 1 → 2
[e] Reflexion.add parses LLM JSON           ✓ len(entries) = 1
[f] Reflexion render takes the last 5       ✓ render len = 154 chars
[g] keyword_hit weight in hybrid retrieval  ✓ keyword > dense when exact match
[h] cross-time replay arg max(rho_h * rho_e)✓ best_idx = 2 (out of 5)
[i] Native Evolution outcome reward computation ✓ R_evolve = 0.18 (≥ 0)

Code has passed independent reviewer static checks, logic constrained by dataclass / type annotations.