# E1 — 12-framework HITL audit (raw research)

Generated by background agent on 2026-05-22. Source material for the public blog post; the published version is the trimmed-to-2500-word edit.

## Headline numbers

- **Top scorers (15/30)**: LangGraph, Pydantic AI, Mastra
- **Bottom scorers (5/30)**: Claude Agent SDK, LangChain legacy, AutoGen, smolagents
- **Mid (8-11)**: OpenAI Agents SDK, LlamaIndex, Haystack, Semantic Kernel, CrewAI

## Master scorecard

Sorted by total descending. Bold = top 2 and bottom 2.

| Rank | Framework | Durability | Idempotency | Typed I/O | Channel | Verifier | UI | **Total** |
|---|---|---|---|---|---|---|---|---|
| 1 | **LangGraph** | **5** | **3** | **3** | **1** | **1** | **2** | **15** |
| 2 | **Pydantic AI** | **4** | **4** | **5** | **1** | **1** | **0** | **15** |
| 3 | Mastra | 4 | 3 | 4 | 2 | 1 | 1 | 15 |
| 4 | OpenAI Agents SDK | 3 | 3 | 3 | 1 | 1 | 0 | 11 |
| 5 | LlamaIndex | 3 | 2 | 3 | 1 | 1 | 0 | 10 |
| 6 | Haystack | 2 | 1 | 2 | 2 | 1 | 1 | 9 |
| 7 | Semantic Kernel | 2 | 2 | 2 | 1 | 1 | 0 | 8 |
| 8 | CrewAI | 2 | 1 | 1 | 1 | 1 | 0 | 6 |
| 9 | Claude Agent SDK | 1 | 1 | 1 | 1 | 1 | 0 | 5 |
| 10 | LangChain (legacy) | 1 | 1 | 1 | 1 | 1 | 0 | 5 |
| 11 | **AutoGen** | **1** | **1** | **1** | **1** | **1** | **0** | **5** |
| 12 | **smolagents** | **1** | **1** | **1** | **1** | **1** | **0** | **5** |

## Pattern findings

- **Universally missing axes**: channel abstraction, verifier hook, dashboard UI. Mastra is the only framework with a channel concept and it's bolted onto agents, not workflows.
- **Most-solved axis**: the basic pause primitive itself. Every framework has *some* way to stop and wait.
- **Durability is the second-most-solved**, but only for LangGraph + Postgres checkpointer and Pydantic AI + Restate/Temporal. Everyone else hand-waves serialization or blocks in-process.
- **The aggregate weakness**: ten of twelve frameworks default to `input()` on a terminal. They're one production deploy away from breaking.

## Key pull-quotes (from primary sources)

> "HITL input is handled through the terminal, which does not work for a production web environment." — CrewAI community forum, *"Human in the loop - workaround"*

> "Keep code before `interrupt()` idempotent because `interrupt()` causes the node function to re-execute from the top on resume." — LangGraph docs

> "The runtime pauses by throwing an internal control-flow exception and replays the entire step when the event arrives, so keep this call near the top of the step and make any preceding work safe to repeat." — LlamaIndex docs

> "If you run multiple workers, each instance keeps its own memory. You cannot resume a session that started elsewhere, and you cannot recover if a workflow crashes halfway when using in-memory storage." — LangGraph persistence guide

## Per-framework one-liners

- **LangGraph** — best-in-class durability via PostgresSaver; everything above storage is BYO.
- **Pydantic AI** — cleanest typed API in the field; durability requires Restate/Temporal/Prefect.
- **Mastra** — best TypeScript HITL story; Channels and HITL workflows are separate primitives that don't yet compose.
- **OpenAI Agents SDK** — clean serializable RunState; BYO queue, storage, and UI.
- **LlamaIndex** — elegant event-driven HITL; replay-the-step semantics are a footgun.
- **Haystack** — best confirmation taxonomy; only ships console UIs.
- **Semantic Kernel** — two half-built HITL mechanisms (Filters + Process Framework) instead of one good one.
- **CrewAI** — easiest API (`human_input=True`); least production-viable.
- **Claude Agent SDK** — `canUseTool` is permission gating, not HITL; you make it block on a Promise.
- **LangChain (legacy)** — `HumanInputRun` officially superseded by LangGraph.
- **AutoGen** — `human_input_mode="ALWAYS"` is a 2023 primitive still shipping in 2026.
- **smolagents** — smallest primitive, smallest production surface area.

## What "good" looks like (the gap the post argues into)

A HITL primitive should:

1. Persist to durable storage by default
2. Emit a typed payload (Pydantic/Zod) describing what the human is being asked
3. Route through a pluggable channel adapter (Slack / email / dashboard / SMS)
4. Accept a typed response from any channel (idempotent on duplicate deliveries)
5. Optionally pipe response through a verifier model before resuming
6. Write an audit row (who, when, from where, what they saw)
7. Expose an OSS admin UI

**No framework in this audit ships all seven.** This is the gap.

## Full research output (verbatim from background agent)

Re-saved to `/tmp/claude-501/-Users-ta-hitl-project/c5919856-30de-4e47-a95c-cf4610dfa765/tasks/a88589395b00fa50e.output` (background-agent transcript). Hit it via the agent ID `a88589395b00fa50e` for the full per-framework deep dive (~5400 words) including code samples per framework and full source URLs.
