# contextweaver — Agent Guide

> **Authority:** This file is the single source of truth for agent-facing guidance.
> Tool-specific files (`.claude/CLAUDE.md`, `.github/copilot-instructions.md`) contain
> only overrides and guardrails — they defer to this file for all shared rules.

## Purpose

contextweaver is a Python library for dynamic context management for tool-using AI agents.
It provides two integrated engines:

1. **Context Engine** — phase-specific budgeted context compilation with a context firewall
2. **Routing Engine** — bounded-choice navigation over large tool catalogs via DAG + beam search

**Non-goals:** contextweaver is not an LLM inference layer and not a tool execution runtime.
It prepares context and routes tools but never calls models or executes tools.

## Module Map

| Path | Responsibility |
|---|---|
| `types.py` | Core dataclasses and enums: `SelectableItem`, `ContextItem`, `Phase`, `ItemKind`, `Sensitivity` |
| `envelope.py` | Result types: `ResultEnvelope`, `BuildStats`, `DroppedItem`, `ContextPack`, `ChoiceCard`, `HydrationResult`, `RoutingDecision` |
| `diagnostics.py` | Versioned, payload-safe gateway events and sinks (`DiagnosticEvent`, `DiagnosticSink`, JSONL/in-memory sinks) plus deterministic aggregate reports (issues #370/#378). |
| `inspection.py` | Pure JSON/Markdown report construction for offline context, routing, and artifact inspection without raw payload content (issue #398). |
| `config.py` | Configuration: `ContextBudget`, `ContextPolicy`, `ScoringConfig` |
| `profiles.py` | Routing and profile config: `Mode`, `RoutingConfig`, `ProfileConfig`, named presets |
| `protocols.py` | Protocol interfaces: `TokenEstimator`, `EventHook`, `Summarizer`, `Extractor`, `RedactionHook`, `SensitivityClassifier` (ingestion-time labelling, issue #542), `MemorySource`, `Labeler`, `Retriever`, `Reranker`, `ClusteringEngine`, `RoutingScoreProvider` (store protocols re-exported from `store/protocols.py`). Bundled estimators: `HeuristicEstimator` (default, script-aware, dependency-free — counts CJK/Kana/Hangul/emoji ≈1 token/char, issue #525), `CharDivFourEstimator` (raw `len // 4` primitive), `TiktokenEstimator` (exact, falls back to `HeuristicEstimator` offline). Each carries a stable `name` for `BuildStats.token_estimator`. |
| `store/protocols.py` | Store-layer protocols: `EventLog`, `ArtifactStore`, `EpisodicStore`, `FactStore` |
| `exceptions.py` | Custom exception hierarchy (all errors inherit `ContextWeaverError`) |
| `_utils.py` | Text similarity primitives: `tokenize()`, `jaccard()`, `TfIdfScorer` |
| `secrets.py` | Pure, deterministic secret detection/scrubbing primitives: `scrub_secrets()`, `scrub_secrets_in_list()`, `contains_secret()`, `SecretPattern` (issue #428). Shared by the firewall secret-scrub, the `SecretRedactor` hook, the sensitivity classifier, and ChoiceCard scrubbing. No I/O; never weakens a surface (only removes characters). |
| `_version.py` | Single-source version derived from `importlib.metadata`; fallback `"0.0.0+local"` |
| `_demos.py` | Demo logic for the CLI `demo` subcommand (exempt from `print()` rule) |
| `serde.py` | Serialisation helpers for `to_dict` / `from_dict` |
| `tokens.py` | Built-in token counter (`count()`, `get_token_counter()`, `heuristic_counter()`, `TokenCounter` alias) plus the provider-estimator registry (`register_estimator()`, `registered_estimators()`, `estimator_name()`, issue #493). The **single source of truth** for token counts — firewall, sensitivity-redaction placeholders, card budgeting (`routing/cards.count_tokens`), and `FirewallStats`/`BuildStats` numbers all route through it (issues #405/#493/#530); no stray `len // 4` literals elsewhere. Owns the `tiktoken` dependency; offline it falls back to the script-aware `HeuristicEstimator`. |
| `store/` | In-memory data stores: `EventLog`, `ArtifactStore`, `EpisodicStore`, `FactStore`, `StoreBundle` |
| `store/_sqlite_base.py` | Shared SQLite connection + migration scaffolding (WAL, `foreign_keys=ON`, `_contextweaver_schema_version` table). Reused by every SQLite-backed store (issue #174). |
| `store/sqlite_event_log.py` | `SqliteEventLog` — first persistent `EventLog` backend; single-process, sync, append-only, schema-versioned (issue #223). |
| `store/json_file_artifacts.py` | `JsonFileArtifactStore` — filesystem `ArtifactStore` backend; `{enc(handle)}.data` + `{enc(handle)}.json` per artifact, re-instantiable against an existing directory (issue #42). Hardened (issue #497): **atomic** writes (temp file + `os.replace`), an in-memory handle→ref index built once on init so `list_refs` never rescans the directory, and optional `max_bytes` / `max_artifacts` quotas raising `ArtifactStoreQuotaError`. Persists `content_hash` and percent-encodes handles into filenames so the firewall's `artifact:result:…` handles are Windows-safe (issue #466). |
| `store/_json_file_io.py` | Private filesystem helpers for `json_file_artifacts.py` (keeps it ≤300 lines): on-disk suffix constants, `validate_handle` (path-traversal defense), `encode_handle` (percent-encoding), and the `atomic_write` primitive (issues #466/#497). Not public API. |
| `store/testing.py` | Store-protocol conformance kit (issue #520): framework-agnostic `check_event_log_conformance` / `check_artifact_store_conformance` / `check_episodic_store_conformance` / `check_fact_store_conformance`, each taking a factory for an empty backend and asserting the round-trip / ordering / not-found contract. No test-framework import; ships in the core wheel. `tests/test_store_conformance.py` runs every bundled backend through it. |
| `summarize/` | `SummarizationRule`, `RuleEngine`, `extract_facts()` |
| `summarize/structured.py` | Lossless JSON field projection for the firewall: `parse_path` / `project` + `StructuredFirewall(keep=[...])`. Deterministic, no LLM — keeps an allow-list of JSON paths inline and offloads the rest (issue #406). |
| `context/` | Full context pipeline, sensitivity enforcement, view registry, `ContextManager` |
| `context/firewall_api.py` | Single-call firewall facade: `compact_tool_result` / `firewalled_tool_result` → `CompactResult`. Composes structured/text strategies, schema-preserving pass-through (reserved `_cw` sidecar — a caller payload already using `_cw` raises `ConfigError` unless `overwrite_sidecar=True`, #467), the built-in token counter, and fail-closed `deterministic` mode (issues #399, #402, #403, #404, #405, #406, #467). |
| `context/manager.py` | `ContextManager` — thin orchestrator (`__init__`, properties, `drilldown`, mixin composition). Public method stubs live in flat partial-class mixins; pipeline logic lives in the delegate modules below (issue #101). |
| `context/_manager_base.py` | `_ManagerState` — private-attribute + `_build` contract the manager mixins inherit and the delegate pipeline modules type their `manager` parameter against (`ContextManager` inherits it via the mixins). Not public API (issue #101). |
| `context/_manager_ingest.py` / `_manager_build.py` / `_manager_routing.py` | `_IngestMixin` / `_BuildMixin` / `_RoutingMixin` — partial-class mixins holding `ContextManager`'s ingestion, build, and route/call-prompt method surface as thin delegations; keep `manager.py` ≤300 lines (issue #101). Not public API. |
| `context/ingest.py` | Tool-result ingestion helpers (extracted from `manager.py` to honor the <=300 line guideline). Includes `ingest_envelope` — the canonical Frame-shaped seam (weaver-spec I-05) that ingests an already-firewalled `ResultEnvelope` without re-deriving firewalling; raw-output `ingest_tool_result` / `ingest_mcp_result` are non-canonical for spec compliance (issue #352). |
| `context/memory_types.py` | `MemoryEntry` dataclass + `PHASE_SCOPE_PREFERENCES` constants for phase-aware memory ingestion (issue #293). |
| `context/memory_fixture.py` | `JsonFixtureMemorySource` — deterministic stdlib fixture adapter implementing the `MemorySource` Protocol from `protocols.py` (issue #293). |
| `context/memory_source.py` | `memory_entries_to_context_items` / `select_memory_for_phase` helpers that materialise memory entries into budgeted `memory_fact` candidates (issue #293). |
| `context/handoff_types.py` | `HandoffEntry` + `SessionHandoffPack` dataclasses and canonical handoff category constants (issue #294). |
| `context/handoff.py` | `build_session_handoff_pack` / `render_handoff_pack` — deterministic, budget-aware, sensitivity- and firewall-respecting session continuity snapshot (issue #294). |
| `context/explanation.py` | `ContextBuildExplanation` + `CandidateExplanation` opt-in debug surface returned by `ContextManager.build(..., explain=True)` (issue #291). Sister to `routing/explanation.py` on the routing side. |
| `context/classify.py` | Opt-in deterministic ingestion-time sensitivity classification (issue #542): `HeuristicSensitivityClassifier` (implements the `SensitivityClassifier` protocol) + `detect_sensitivity()`. Runs at the start of the pipeline's sensitivity stage and over fact/episode header content; may only **raise** a label, never lower it. Reuses `secrets.contains_secret` plus PII markers. |
| `context/secret_redaction.py` | Opt-in `SecretRedactor` `RedactionHook` (issue #428): substring-scrubs secret shapes from an item's text via `secrets.scrub_secrets`. Registered under the name `"secret"` for `ContextPolicy.redaction_hooks`; complements (does not replace) `MaskRedactionHook`. |
| `routing/` | `Catalog`, `ChoiceGraph`, `TreeBuilder`, `Router` (beam search), card renderer |
| `routing/filters.py` | Pre-scoring helpers: `filter_items()`, `augment_query()`, `suggest_clarifying_question()` (issues #14, #22, #112, #116) |
| `routing/manifest.py` | `GraphManifest` + `compute_catalog_hash()` for graph metadata and cache invalidation (issue #48, #15) |
| `routing/normalizer.py` | `CatalogNormalizer` + `NormalizationReport` for catalog metadata hygiene (issue #44) |
| `routing/catalog.py` (validation) | `validate_references` / `Catalog.validate_references` → `CatalogValidationReport` of dangling `depends_on`/`requires` refs; loaders take `on_invalid` (`"warn"`/`"raise"`/`"ignore"`) and raise `CatalogValidationError` in raise mode (issue #519) |
| `routing/registry.py` | `EngineRegistry` and bundled `TfIdfRetriever` / `NoOpReranker` / `JaccardClusteringEngine` defaults (issue #47) |
| `routing/trace.py` | `RouteTrace` + `TraceStep` structured routing audit (issue #51) |
| `routing/explanation.py` | `RouteResult.explanation()` Markdown / dict rendering (issue #226) |
| `routing/pipeline.py` | `RoutingPipeline` composer — explicit retrieve → rerank → navigate → pack stages (issue #56) |
| `routing/navigator.py` | `BeamSearchNavigator` (lifted from `router.py`) + `rank_collected` — the score-sort/active-filter helper is re-exported from `routing/__init__.py` for custom Navigator implementations (issues #56, #288) |
| `routing/packer.py` | `DefaultCardPacker` wrapping `make_choice_cards` for the pipeline pack stage (issue #56) |
| `routing/history.py` | `RouteHistory` dataclass + `adjust_scores` (history-aware re-routing, issue #27) |
| `routing/feedback.py` | Optional feedback-aware routing scores (issue #318): `ExecutionFeedback` (contextweaver-native, **not** a weaver-spec type), `DeterministicScoreProvider` (default no-op), `FeedbackAwareScoreProvider`, `aggregate_feedback`. Plugs into `Router(score_provider=...)`; default `None` keeps routing deterministic. |
| `extras/embeddings.py` | `SentenceTransformerBackend` + `HybridEmbeddingRetriever` + `HashingEmbeddingBackend` (re-exported) behind the `[embeddings]` extra (issue #8) |
| `extras/embeddings_hashing.py` | `HashingEmbeddingBackend` — stdlib-only deterministic `EmbeddingBackend` using blake2b hashing trick; no extras required (issue #266) |
| `_schema_gen.py` | Dataclass → JSON Schema (Draft 2020-12) generator + `make schemas-check` engine (issue #225) |
| `routing/tool_id.py` | Canonical `tool_id` grammar (`parse_tool_id` / `format_tool_id` / `compute_hash8`) per `docs/gateway_spec.md` §1 |
| `routing/path.py` | `tool_browse` path-navigation grammar (`parse_path` / `resolve_path`) per `docs/gateway_spec.md` §3 |
| `routing/hydration.py` | Public schema-hydration helpers — `SchemaSource` (from raw dict / JSON file / MCP tools-list), `hydrate_with_schema`, `lazy_schema_resolver`. Reference architectures use these to resolve a tool's full input schema from a sidecar source rather than hand-rolling a `_FULL_SCHEMAS` dict. Inline `args_schema` on the catalog item wins; sidecar only fills empties. Issue #261. |
| `adapters/` | MCP, FastMCP, A2A, weaver-spec, CrewAI, Pydantic AI, smolagents, Agno protocol adapters + MCP proxy / gateway runtime + provider-message ingestion helpers for OpenAI / Anthropic / Gemini chat histories (issues #13, #28, #29, #34, #193, #194, #219, #222, #272, #274, #275) |
| `adapters/chainweaver.py` | ChainWeaver flow-export → `SelectableItem(kind="flow")` import (`chainweaver_flow_to_selectable`, `chainweaver_flows_to_catalog`, `load_chainweaver_export`, issue #334). Pure data; no ChainWeaver dependency. Preserves name/description/input+output schemas; stamps `metadata["runtime"]="chainweaver"` + flow id/version. |
| `adapters/crewai.py` | CrewAI `BaseTool` (or equivalent plain-dict shape) ↔ `SelectableItem` (`crewai_tool_to_selectable`, `crewai_tools_to_catalog`, `infer_crewai_namespace`, `load_crewai_catalog`, issue #193) |
| `adapters/pydantic_ai.py` | Pydantic AI `Tool` ↔ `SelectableItem` and `ModelMessage` ↔ `ContextItem` lossless round-trip (`pydantic_ai_tool_to_selectable`, `pydantic_ai_tools_to_catalog`, `load_pydantic_ai_catalog`, `from_/to_pydantic_ai_messages`, issue #272) — heavy decode/encode helpers live in `adapters/_pydantic_ai_messages.py` |
| `adapters/smolagents.py` | Hugging Face smolagents `Tool` ↔ `SelectableItem` and `MultiStepAgent.memory.steps` → `ContextItem`s (`smolagents_tool_to_selectable`, `smolagents_tools_to_catalog`, `load_smolagents_catalog`, `from_smolagents_agent`, issue #274) |
| `adapters/agno.py` | Agno (formerly Phidata) `Function` / `Toolkit` ↔ `SelectableItem` and `AgentSession` → `ContextItem`s (`agno_tool_to_selectable`, `agno_tools_to_catalog`, `load_agno_catalog`, `from_agno_session`, issue #275) |
| `adapters/proxy_runtime.py` | `ProxyRuntime` shared core + `ExposureMode` enum + `UpstreamCall` Protocol (issue #29) |
| `adapters/gateway_diagnostics.py` / `gateway_catalog_diagnostics.py` | Sanitized `ProxyRuntime` instrumentation plus exact gateway/proxy static-schema exposure calculations: catalog, browse/hydrate/execute/view events, savings, artifact-view usage, and latency (issues #370/#378). |
| `adapters/mcp_gateway.py` | Two-tool gateway dispatch (`tool_browse` + `tool_execute` + `tool_view`, issues #28 / #34) |
| `adapters/mcp_proxy.py` | Transparent proxy dispatch (stripped `tools/list` + `tool_hydrate` + `tool_execute`, issue #13) |
| `adapters/mcp_upstream.py` | Concrete `UpstreamCall` adapters (`StubUpstream`, `McpClientUpstream`, `MultiplexUpstream`) |
| `adapters/mcp_gateway_server.py` | Bind `mcp_gateway` onto `mcp.server.Server` over stdio (issue #28) |
| `adapters/mcp_proxy_server.py` | Bind `mcp_proxy` onto `mcp.server.Server` over stdio (issue #13) |
| `adapters/gateway_error.py` | Structured `GatewayError` (codes + §3.4 wire shape) + `retryable` hint. Upstream-error taxonomy: `classify_upstream_exception` maps timeouts/connection/auth/permission/rate failures to `UPSTREAM_TIMEOUT`/`UPSTREAM_UNAVAILABLE`/`AUTH_FAILED`/`PERMISSION_DENIED`/`RATE_LIMITED` (fallback `UPSTREAM_ERROR`); `redact_upstream_detail` strips control chars + caps length on model-visible detail (issue #485). |
| `adapters/gateway_validation.py` | Untrusted-schema hardening for the gateway ingest path (issues #464/#484): `SchemaLimits`/`SchemaFinding`/`SkippedTool`/`CatalogRefreshReport`, `check_schema_health` (meta-validation + iterative size/depth/property bounds), `build_validator` (cached per `tool_id`). Pure, deterministic; iterative traversal avoids stack exhaustion on hostile schemas. |
| `adapters/gateway_args.py` | Opt-in deterministic tool-call argument repair (issue #488): `normalize_args` (stringified-object parse + schema-demanded `str→int/number/boolean/null` coercion) + `Repair`. Gated behind `ProxyRuntime(tolerant_args=True)`; never renames keys, drops keys, or fuzzy-matches. |
| `adapters/openai_messages.py` | OpenAI Chat Completions `messages` ↔ `ContextItem` round-trip (`from_/to_openai_messages`, issue #219) |
| `adapters/anthropic_messages.py` | Anthropic Messages API `messages` ↔ `ContextItem` round-trip (`from_/to_anthropic_messages`, issue #222) |
| `adapters/gemini_contents.py` | Google Gemini `contents[]` ↔ `ContextItem` round-trip (`from_/to_gemini_contents`, issue #222) |
| `extras/otel.py` | OpenTelemetry GenAI integration (`OTelEventHook` — `invoke_agent` / `execute_tool` spans + GenAI SemConv attributes, gated behind the `[otel]` extra, issue #224). |
| `extras/llm_summarizer.py` | Optional `LlmSummarizer` / `LlmExtractor` — LLM-backed `Summarizer` / `Extractor` plugins for the firewall. Take a user-supplied `call_fn` (no LLM SDK dep, no extra) and degrade to the rule-based path on any failure (issue #26). |
| `extras/memory/` | External-memory backend adapters that implement `EpisodicStore` / `FactStore` against an existing long-lived memory deployment without widening the Protocols (issue #195). |
| `extras/memory/mem0.py` | `Mem0EpisodicStore` + `Mem0FactStore` — wrap a `mem0.Memory` instance scoped by `user_id`; writes go through `Memory.add(infer=False)` and items are stamped with `cw_episode_id` / `cw_fact_id` metadata for canonical-ID resolution. Gated behind the `[mem0]` extra (issue #195). |
| `extras/memory/zep.py` | `ZepEpisodicStore` + `ZepFactStore` — wrap a `zep_cloud.Zep` client scoped by `user_id`; persist items as JSON graph episodes (`graph.add(type="json")`) stamped with `cw_*` IDs, resolving back via `graph.episode.get_by_user_id`. Episodic `search` is client-side (Zep graph search is edge/node-shaped). Gated behind the `[zep]` extra (issue #195). |
| `extras/memory/_zep_common.py` | Internal helpers backing `zep.py` (keeps it ≤300 lines): shared `cw_*` constants, the `ZepBackendError` exception, the JSON/scan helpers (`_episode_records` / `_episode_uuid` / `_episode_payload`), the defensive payload-coercion helpers (`_coerce_str_tags` / `_coerce_metadata`), and the `_ZepStoreBase` scope/scan/write base. Carries the same `[zep]`-extra import guard (issue #195). |
| `extras/memory/langmem.py` | `LangMemEpisodicStore` + `LangMemFactStore` — wrap any LangGraph `BaseStore` scoped by a `namespace` tuple; canonical ID is the store key, value is the dataclass `to_dict()` payload (direct, lossless KV). `search` delegates to `BaseStore.search`. Gated behind the `[langmem]` extra (issue #195). |
| `eval/` | Evaluation harness (issue #12): `EvalCase` / `EvalDataset` (gold datasets), `evaluate_routing` → `RoutingEvalReport` (top-k recall, MRR, confidence gap, beam steps), `evaluate_context` → `ContextEvalReport` (budget utilisation + token savings vs naive concat). Pure-stdlib, deterministic; backs the `eval` CLI subcommand. |
| `eval/metrics.py` | Canonical rank-based routing metrics — `recall_at_k` (classic fractional recall@k), `precision_at_k`, `reciprocal_rank` (issue #354). Single source of truth imported by both `eval/routing.py` and `benchmarks/benchmark.py` so the harness and the benchmark script can no longer define the same names with different semantics. |
| `__main__.py` | CLI: 11 subcommands (`demo`, `build`, `route`, `print-tree`, `init`, `ingest`, `replay`, `stats`, `inspect`, `budget-check`, `eval`) plus the `mcp` and `catalog` Typer sub-apps. `inspect` renders payload-safe context/routing/artifact JSON or Markdown (issue #398); `catalog lint` surfaces `NormalizationReport` + reference findings with `--json` and CI exit codes (issue #538). |
| `_mcp_cli.py` | Backs the `mcp` Typer sub-app. Hosts `mcp serve`, `mcp inspect`, and `mcp stats`; accepts native contextweaver, raw MCP `tools/list`, and `{tools:[...]}` catalog shapes. `mcp serve --diagnostics FILE` appends sanitized JSONL and `--quiet` suppresses lifecycle stderr; both are config-file keys. The packaged serve path remains a static catalog + stub upstream. |
| `data/` | Packaged data files shipped inside the wheel via `[tool.setuptools.package-data]`. Exposes `gateway_catalog_path()` (resolves `mcp_gateway_catalog.yaml` to a concrete `Path` for both editable installs and zipped wheels — falls back to a persistent cache under `tempfile.gettempdir()/contextweaver/` for zipimport). Issue #264. |
| `examples/recipes/` | MCP-client integration recipes: installed-CLI configs for Claude Desktop, Claude Code, GitHub Copilot, and Cursor plus `gateway_config.yaml`; `serve_gateway.py` remains a legacy/custom-runtime launcher (issues #278, #279, #346, #371, #429, #437). |

## Pipelines (summary)

**Context Engine** — 8 stages:

1. `generate_candidates` → 2. `dependency_closure` → 3. `sensitivity_filter` →
4. `apply_firewall` → 5. `score_candidates` → 6. `deduplicate_candidates` →
7. `select_and_pack` → 8. `render_context`

**Routing Engine** — 4 stages:

1. `Catalog` → 2. `TreeBuilder` → 3. `Router` (beam search) → 4. `ChoiceCards`

The `Router` itself composes a four-stage `RoutingPipeline` internally
(retrieve → rerank → navigate → pack, issue #56).  Each stage is
swappable via the `EngineRegistry` or by passing a custom
`RoutingPipeline` to `Router(pipeline=...)`.  History-aware re-routing
(`Router.route(history=...)`, issue #27) and the optional embedding
retriever (`Router(embedding_backend=...)`, issue #8) plug into this
same pipeline contract.

For full pipeline descriptions and design rationale, see [docs/agent-context/architecture.md](docs/agent-context/architecture.md).

## Key Types

| Type | Purpose |
|---|---|
| `SelectableItem` | Unified tool/agent/skill/flow/internal item (`kind="flow"` = external multi-step capability, e.g. a ChainWeaver flow). Alias: `ToolCard` (use `SelectableItem` in code). |
| `ContextItem` | Event log entry with `parent_id` for dependency closure |
| `ResultEnvelope` | Processed tool output: summary + facts + artifacts + views |
| `ContextPack` | Rendered prompt + stats from a context build |
| `BuildStats` | What was kept, dropped, and why. `total_candidates` is pre-sensitivity; `dropped_items` attributes every exclusion; completed builds satisfy included + dropped = total. Carries `firewall_events` + `firewall_summary()` (issues #402/#414/#459) and `token_estimator` — the estimator-path identifier that produced the build's token numbers (issue #493). |
| `FirewallStats` | Per-firewall diagnostics: `triggered`, `strategy`, original/summary chars+tokens, `artifact_ref`, `summarized_by_llm` (issues #402 / #404) |
| `CompactResult` | Output of the single-call `compact_tool_result` facade: `firewalled`, `payload`, `summary`, `facts`, `artifact_ref`, `stats` (issue #399) |
| `StructuredFirewall` | Non-summarising firewall strategy — keep an allow-list of JSON paths inline, offload the rest (issue #406) |
| `ChoiceCard` | LLM-friendly compact card (never includes full schemas) |
| `RoutingDecision` | Routing output shaped for weaver-spec interop (id, choice_cards, timestamp, selection). `choice_cards` is a flat list of CW 1:1 cards; for schema-valid spec JSON, go through `adapters.weaver_contracts.to_weaver_routing_decision()`. Build with `RouteResult.to_routing_decision(...)`. |
| `ChoiceGraph` | Bounded DAG for routing, serializable, validated on load |
| `GraphManifest` | Build-time metadata attached to every routing graph (hash, seed, engine versions, timestamp) |
| `RouteTrace` | Always-populated structured audit of a routing call; per-step expansions opt-in via `debug=True` |
| `EngineRegistry` | Pluggable registry for `Retriever`, `Reranker`, `ClusteringEngine` slots |
| `Mode` | Determinism mode (`strict` / `seeded` / `adaptive` placeholder) on `ProfileConfig` |
| `MaskRedactionHook` | Built-in redaction hook for sensitivity enforcement |
| `HydrationResult` | Result of hydrating a tool call with context |
| `ViewRegistry` | Maps content-type patterns to view generators for progressive disclosure |
| `ProxyRuntime` | Shared core for MCP proxy (#13) and gateway (#28) modes — owns upstream catalog, per-session `ContextManager`, browse / execute / view dispatch; persisted text results are returned as envelope artifact refs for `tool_view`. Hardens the untrusted-input boundary (issues #464/#484/#485/#488): `on_invalid` (skip/raise) + `schema_limits` + `last_refresh_report` at ingest, cached per-`tool_id` validators, classified+redacted upstream errors, and opt-in `tolerant_args`. |
| `ExposureMode` | `TRANSPARENT` (#13) vs `GATEWAY` (#28) for `ProxyRuntime` |
| `UpstreamCall` | Transport-agnostic Protocol over upstream MCP fan-out (used by `ProxyRuntime`) |
| `GatewayError` | Structured error payload (§3.4) returned from every gateway/proxy meta-tool. Carries a `retryable` hint and a classified upstream-error taxonomy (issue #485) plus the `SCHEMA_INVALID` ingest code (issue #484) |
| `ToolIdParts` | Destructured canonical `tool_id` (namespace / name / version / hash8) |

**Vocabulary notes:**
- `SelectableItem` is the canonical name. `ToolCard` is a user-facing alias — use `SelectableItem` in code and docs.
- "Context" is overloaded — can mean `ContextItem`, `ContextPack`, the pipeline, or the LLM context window. Disambiguate when unclear. See [docs/concepts.md](docs/concepts.md).
- "Firewall" here means context firewall (prevents large outputs from consuming the token budget), not a security firewall.

## Commands

```bash
make fmt      # ruff format src/ tests/ examples/ scripts/
make lint     # ruff check src/ tests/ examples/ scripts/
make type     # mypy src/
make test     # python -m pytest --cov=contextweaver --cov-report=term-missing -q
make example  # run all example scripts (includes architectures via the umbrella target)
make architectures  # run reference architecture scripts under examples/architectures/
make demo     # python -m contextweaver demo
make ci       # fmt + lint + type + test + schemas-check + example + demo
make docs     # mkdocs build --clean (docs site)
make docs-serve  # mkdocs serve (live preview)
make benchmark        # run benchmark harness (non-gating; writes benchmarks/results/latest.json)
make benchmark-matrix # benchmark + per-backend × per-size matrix (#208) and per-namespace breakdown (#209)
make gateway-scorecard-check  # verify gateway scorecard matches its committed JSON (gating CI; #391)
make record-demos-check       # verify committed demo casts match current output (gating CI; #390)
make smoke-eval       # non-gating CI smoke-evaluation over fixed fixtures (#331/#392); deterministic, credential-free
make scorecard        # render benchmarks/scorecard.md from benchmarks/results/latest.json
make scorecard-check  # verify scorecard.md is up to date (exits non-zero on drift)
make schemas         # regenerate schemas/ + docs/schemas/v0/ (issue #225)
make schemas-check    # verify published schemas match dataclasses (gating, in `make ci`)
make sweep-scoring    # weight sweep for ScoringConfig (#214); writes benchmarks/sweep_scoring.md
make context-rot       # render the context-rot demo: benchmarks/results/context_rot.json + docs/assets/context_rot.svg (#349)
make context-rot-check # verify context_rot.svg matches its committed JSON (gating in CI; exits non-zero on drift)
make readme-version-check  # verify README version references match pyproject.toml (gating in CI; #347)
make llms        # regenerate llms.txt and llms-full.txt from canonical docs
make llms-check  # verify llms.txt and llms-full.txt are up to date (gating in CI; #389)
make weaver-conformance  # round-trip + JSON-Schema validate the weaver-spec adapter (CI gating, fetches schemas)
```

Run `pre-commit install` once after cloning to activate git hooks
(ruff format + check + file hygiene on every commit).

For command-selection rules and sequencing, see [docs/agent-context/workflows.md](docs/agent-context/workflows.md).

## Hard Rules

These are auto-reject in review. No exceptions.

1. **No `print()` in library code.** Use hooks or logging. `__main__.py` and `_demos.py` (CLI) are exempt.
2. **No business logic in `__init__.py`.** Only re-exports allowed.

## Strong Patterns

These are strongly recommended. Engineering judgment applies — deviate with good reason.

- **Text similarity in `_utils.py` only** — `tokenize()`, `jaccard()`, `TfIdfScorer` are the single source of truth. Do not duplicate.
- **`from __future__ import annotations`** in every source file.
- **All exceptions from `contextweaver.exceptions`** — use the custom hierarchy, not bare `ValueError`/`RuntimeError`.
- **`to_dict()` / `from_dict()` on all dataclasses** — complements `serde.py`; they are not redundant. See [invariants](docs/agent-context/invariants.md#serialization-design).
- **Deterministic by default** — tie-break by ID, sorted keys. No randomness in core pipelines.
- **No wildcard imports** — never use `from contextweaver import *`.
- **Event log is append-only** — mutate only via `InMemoryEventLog.append()`.

## Coding Style

- **Python ≥ 3.10** — use `X | Y` union syntax, `match` statements where appropriate.
- **Type hints** on all public functions and methods.
- **Google-style docstrings** on all public classes and functions.
- **100-character line length** (enforced by ruff).
- **≤ 300 lines per module** — exempt: `types.py`, `envelope.py`, `__main__.py`,
  `_mcp_cli.py` (experimental Typer sub-app; size is dominated by Typer
  parameter declarations + docstrings and is expected to shrink once
  `mcp serve` graduates from `[experimental]` to stable), and `_demos.py`
  (CLI demo-output module — print-heavy walkthrough scripts backing the
  `demo` subcommand, same rationale as `__main__.py`).
- **Core runtime dependencies.** The core install pulls `tiktoken`, `PyYAML`, `rank-bm25`, plus `mcp` and `jsonschema` (added when the proxy / gateway runtimes landed — both are load-bearing for `docs/gateway_spec.md` §4.4 schema validation and the MCP transport binding).  Adding *another* core dependency requires explicit justification: broad ecosystem use, small wheel, and a default the library would otherwise have to approximate.  Heavy or runtime-specific packages (CLI, OpenTelemetry, fuzzy retrieval, ANN, NetworkX, FastMCP, LangChain) live under `[project.optional-dependencies]` and are loaded via guarded imports.
- **Dependency-constraint policy** (issue #356). Specifiers are **lower-bound-only** (`>=`), set to the lowest version *actually known to work* — no `==` pins, no speculative upper caps. The only caps kept carry an inline rationale (pre-1.0 `weaver_contracts<1`; docs-extra major pins). Two CI jobs enforce this: a **gating floor-deps job** (`uv pip install --resolution lowest-direct`, Python 3.10, in `ci.yml`) proves the `>=` floors, and a **non-gating weekly** `deps-latest-weekly.yml` (latest + pre-releases) is the no-upper-cap safety net. Raising a floor is a real change — verify with the floor-deps job. Python support is **3.10–3.13**, every cell gating in the CI matrix (3.14 is pending — the heavy dev/adapter stack still caps at `Requires-Python <3.14`).

## Testing

- Tests in `tests/test_<module>.py` — one file per module.
- `pytest.mark.asyncio` for async tests (`asyncio_mode = "auto"` is set globally).
- Do not mock internal modules — use real in-memory implementations.

## Path Conventions

**`store/`** — Protocols are backend-agnostic (must not import backend-specific libraries). Concrete implementations may import backend libs. Must implement the protocol from `protocols.py`. Data is append-only / immutable-after-write.

**`adapters/`** — Pure stateless converters. External format parsing must not leak into core. May import optional external libraries at the adapter boundary only.

**`context/`** — Async-first. All new code should be async with `_sync` wrappers.

**`routing/`** — Sync-only. Pure computation (DAG traversal, beam search). Do not make async.

**Sensitivity (`context/sensitivity.py`)** — Security-grade code. Extra review scrutiny required. Never weaken defaults. Treat changes like security-sensitive code. Redaction is effective end-to-end: `MaskRedactionHook` drops the item's `artifact_ref` and stamps `metadata["redacted"]=True` so the rendered prompt never advertises a handle that `drilldown` could dereference back to the original (issue #451). Enforcement is also applied on the prompt **header** (facts + episode summaries are routed through the floor and the `memory_fact` phase policy, issue #450) and an opt-in `SensitivityClassifier` may raise labels before enforcement (issue #542).

## Things That Must Not Be "Simplified"

1. **Protocol-based store design** — the protocol layer exists for backend extensibility. Do not collapse protocols into concrete classes.
2. **`dependency_closure` pipeline stage** — if a selected item has `parent_id`, the parent must be included. Removing it produces incoherent context (tool results without their tool calls).
3. **`serde.py` + per-class `to_dict`/`from_dict`** — complementary, not redundant. `serde.py` provides shared primitives; per-class methods handle class-specific serialization. Do not consolidate.
4. **`ContextManager` mixin composition** — its public method surface lives in flat partial-class mixins (`_IngestMixin` / `_BuildMixin` / `_RoutingMixin`) sharing the `_ManagerState` base. Do not "simplify" this into delegating composition (`manager.ingest.x()`): that would change the public method surface, which issue #101 forbids ("ContextManager still exposes all current methods"). Mixins were the deliberate trade-off to keep `manager.py` ≤300 lines without breaking the public API.

See [docs/agent-context/invariants.md](docs/agent-context/invariants.md) for the full invariants list and rationale.

## Debugging Tips

1. `make lint` — check for style and import errors.
2. `make type` — check for type errors.
3. `make test` — run the test suite.
4. Check `BuildStats` fields to understand what the context engine dropped and why.
5. Use `ContextManager.artifact_store.list_refs()` to inspect intercepted tool outputs.
6. Enable `logging.DEBUG` on `contextweaver.context` to trace pipeline stages (candidate counts, scores, drops, budget usage).
7. Enable `logging.DEBUG` on `contextweaver.routing` to trace beam search expansions and scoring.

## Adding a Feature

1. Identify the relevant module, modify it, add tests in `tests/test_<module>.py`.
2. Run `make ci` to verify (all declared targets must pass).
3. Update `CHANGELOG.md` and add docstrings to new public APIs.
4. Update agent-facing docs and examples if the pipeline or public API changed.
5. **If the feature can move recall@k / drops / dedup / token counts**: follow
   [`.github/prompts/add-eval.prompt.md`](.github/prompts/add-eval.prompt.md)
   to extend the gold set or scenarios and run `make benchmark-matrix &&
   make scorecard`. CI will post a sticky benchmark-delta comment on the PR.

For the full workflow and definition of done, see [docs/agent-context/workflows.md](docs/agent-context/workflows.md).

## Common Pitfalls

See [docs/agent-context/lessons-learned.md](docs/agent-context/lessons-learned.md) for durable recurring mistakes and how to avoid them.

## Documentation Map

| File | Role |
|---|---|
| `AGENTS.md` (this file) | Primary shared source of truth for all agents |
| `docs/agent-context/architecture.md` | Non-obvious architectural guidance and tradeoffs |
| `docs/agent-context/workflows.md` | Authoritative commands, sequencing, definition of done |
| `docs/agent-context/invariants.md` | Hard constraints and forbidden shortcuts |
| `docs/agent-context/lessons-learned.md` | Failure-capture workflow and durable lessons |
| `docs/agent-context/review-checklist.md` | Self-check and review gates |
| `docs/architecture.md` | Canonical architecture reference (full pipeline detail, diagrams) |
| `docs/concepts.md` | Core concept glossary (types, subsystems, phases) |
| `CONTRIBUTING.md` | Human contributor guide |

When architecture details conflict, `docs/architecture.md` is the canonical reference.

## Update Policy

- Update `AGENTS.md` when shared rules, conventions, or the module map change.
- Update `docs/agent-context/` files when their specific topic area changes.
- Any PR that changes the pipeline, public API, or project conventions must include doc updates.
- If two docs disagree, `AGENTS.md` is authoritative for agent guidance; `docs/architecture.md` is authoritative for architecture detail.
- See [docs/agent-context/workflows.md](docs/agent-context/workflows.md) for documentation governance rules.
