# Genesis: Versioned Build Plan — Safety-Ordered, Waterfall

**Status:** V3 Feature Complete | **Last updated:** 2026-04-12

> **V3 is feature complete.** All nine build phases (0-9) are done. The
> phase-ordered waterfall served its purpose — safety-first scaffolding for
> a system that now runs autonomously. Future work is scoped as V4 or V5,
> not as additional V3 phases. This document is now historical reference
> for the build sequence and design rationale.


> **Ordering principle:** Build what's safest, most testable, and least likely to break first.
> Each phase depends on the previous phases working correctly. No phase should be started until
> its dependencies are verified.
>
> **Versioning principle:** Each version is a **complete, independently valuable system** — not
> a half-built one. Later versions benefit from operational data generated by earlier ones.
> Features that fundamentally require operational data to avoid producing garbage belong in the
> version that will have that data.
>
> **Master design reference:** `genesis-v3-autonomous-behavior-design.md` remains the full
> architectural reference. This document says WHAT to build, WHEN, and in WHICH VERSION.
> The master doc says WHY and HOW each component works.
>
> **Runtime architecture:** `docs/plans/2026-03-07-agentic-runtime-design.md` defines the
> dual-runtime model: Claude Code as intelligence layer, Agent Zero as infrastructure.
> Phases 4+ (Perception onward) shift reflection engine ownership to CC background sessions.
> Task execution phases (8-9) use CC as orchestrator, not AZ's main agent.
>
> **Identity reference:** `genesis-v3-vision.md` defines who Genesis is and what it aspires
> to be. Every implementation decision should be consistent with that document.

---

## Version Overview

```
V3: The Complete Copilot          V4: The Self-Tuning Copilot       V5: The Autonomous Copilot
─────────────────────────         ──────────────────────────────    ──────────────────────────────
Perceives, remembers,             Same copilot, measurably          Proposes changes to itself,
learns, communicates              better. Meta-prompting,           anticipates needs, earns
(basic), has fixed                calibration loops, richer         higher autonomy, learns
boundaries. Ships with            outreach. Built on V3 data.       how to learn. Needs months
conservative defaults.            Features activate as data         of V4 data.
                                  accumulates (feature flags).

Knowledge Base: Parallel track — architecturally independent, can ship alongside any version.
```

| Version | Capability Plateau | Data Requirement |
|---------|-------------------|-----------------|
| **V3** | Complete working copilot | None — ships with conservative fixed defaults |
| **V4** | Self-tuning copilot | ~1-2 months of V3 operational data |
| **V5** | Autonomous copilot | ~3-6 months of V4 operational data |

**Why three versions?** They align with three genuine capability plateaus. More versions create
artificial boundaries. Fewer create a dumping ground where 2-week-data features mix with
6-month-data features.

---

# V3: The Complete Copilot

10 phases, ordered by safety. Each phase depends on its predecessors. V3 is a complete,
shippable system with conservative fixed defaults.

## V3 Phase Dependency Map

```
Phase 0: Data Foundation ✅
    ├── Phase 1: Awareness Loop ✅ ────────┐
    ├── Phase 2: Compute Routing ✅ ───────┤
    │                                      │
    │   Phase 3: Surplus Infrastructure ✅ ─┤  (needs Phase 0 + 2)
    │                                      │
    │                    Phase 4: Perception ✅
    │                         │
    │                    Phase 5: Memory Operations ✅
    │                         │
    │                    Phase 6: Learning Fundamentals ✅
    │                         │
    │                    Phase 7: Simple Deep Reflection ✅
    │                         │
    │                    Phase 8: Basic Outreach ✅
    │                         │
    │                    Phase 9: Basic Autonomy ✅ (needs Phase 6 + 8)
```

**Parallel tracks:** Phases 1, 2, and 3 can be built in parallel (all depend only on Phase 0).
Phase 4 can start once Phases 1 and 2 are done. The critical sequential path is Phases 4→5→6→7→8→9.

---

## V3 Phase 0: Data Foundation ✅ COMPLETE

**Risk: LOW** — Pure schemas and CRUD. No LLM calls. No user-facing behavior.

> **Status:** COMPLETE (2026-03-02). 13 tables, 13 CRUD modules, Qdrant wrapper (1024-dim),
> 4 MCP server stubs (24 tools), 170 tests. Committed on `main`.

### What to Build

| Component | Schema/Table | Design Doc Reference |
|-----------|-------------|---------------------|
| Memory storage | Episodic, semantic, procedural memory tables | §4 MCP Servers → memory-mcp |
| Observation storage | Observations with utility tracking (`retrieved_count`, `influenced_action`) | §Loop Taxonomy → Tier 4 |
| Execution traces | Trace table per Execution Trace Schema. Include `initiated_by` (user / awareness_loop / surplus / reflection) and `request_delivery_delta` (structured: original_request, discoveries, adjusted_scope, scope_communicated, delta classification, discovery attribution). See §Self-Learning Loop step 3. | §Task Execution Architecture |
| Surplus staging | `surplus_insights` (content, source_type, model, drive_alignment, confidence, TTL, promoted_to) | §Cognitive Surplus |
| Signal weights | Signal source weights with adaptation bounds | §Awareness Loop → Signal-Weighted Trigger |
| Capability gaps | Gap tracking (description, frequency, first/last seen, feasibility, revisit_after) | §Open Question #10 |
| Procedural memory | Procedures with confidence, invocation_count, success_rate, version | §Procedural Memory Design |
| User model cache | Structured user model JSON store | §Open Question #2 |
| Speculative claims | Hypothesis table (claim, speculative flag, evidence_count, expiry) | §Open Question #13 |
| Autonomy state | Per-category autonomy level tracking | §Autonomy Hierarchy |
| Outreach history | Outreach log with engagement tracking fields | §Proactive Outreach |
| Daily brainstorm log | Brainstorm outputs (upgrade-user, upgrade-self) with promotion status | §Cognitive Surplus |

### Workspace Files

Initialize workspace files alongside database schemas:

- **Cognitive state summary** (replaces JOURNAL.md) — Fixed-size regenerated summary
  (~600 tokens) of active context + pending actions. Stored in `cognitive_state` DB
  table, not a file. Written by: Deep reflection (Phase 7). Read by: every fresh
  20-30B+ context. Narrative continuity comes from this + memory-mcp episodic
  queries, not from an append-only journal. See §Cognitive State Summary in design doc.

All schemas include GROUNDWORK fields for V4/V5 features. Build the full schema now — some fields
stay empty until later versions populate them.

### MCP Server Interfaces

Define tool interfaces for all 4 servers. Implementation can be stubs initially.

- **memory-mcp**: store, retrieve (hybrid: embedding + activation), link, update_activation, list_by_type
- **recon-mcp**: store_finding, query_findings, schedule_job, list_scheduled
  - **Build note:** All HTTP fetches in recon-mcp should include the `Accept: text/markdown` header. Cloudflare-enabled sites (~20%+ of the web) will return clean Markdown instead of HTML, yielding ~80% token reduction. The `x-markdown-tokens` response header allows pre-checking content size before fetching the full body. Non-Cloudflare sites ignore the header and return HTML normally. Zero implementation cost — one header on every outbound request.
  - **Implementation candidate — Scrapling:** Evaluate [D4Vinci/Scrapling](https://github.com/D4Vinci/Scrapling) for recon-mcp's web monitoring/scraping. Key differentiators: adaptive element tracking that survives site redesigns (intelligent similarity-based relocation), TLS fingerprint impersonation, Cloudflare bypass via `StealthyFetcher`, and a Scrapy-like spider framework with pause/resume checkpoints. Offers three fetcher tiers (`Fetcher` for fast HTTP, `StealthyFetcher` for anti-detection, `DynamicFetcher` for full Playwright browser). Evaluate at implementation time against Playwright-only approach — Scrapling may be better for non-interactive scraping while Playwright remains necessary for interactive browser tasks.
- **health-mcp**: report_metric, query_health, get_error_rates, list_alerts
- **outreach-mcp**: queue_message, get_pending, record_engagement, get_channel_stats, list_channels

### Verification

- [x] Every table: create, read, update, delete
- [x] Schema supports every query pattern in the design doc
- [x] MCP tool interfaces accept and return expected types
- [x] Foreign key relationships are correct

---

## V3 Phase 1: The Metronome (Awareness Loop) ✅ COMPLETE

**Risk: LOW** — Purely programmatic. No LLM, no user-facing output. Dependencies: Phase 0.

> **Status:** COMPLETE (2026-03-03). Signal collector, composite scorer, depth classifier,
> critical-bypass, 5-minute tick scheduler with APScheduler. 117 tests (287 cumulative).
> Committed on `main`.

### What to Build

- **5-minute tick scheduler** with hybrid event-driven + calendar guardrails
- **Signal collector**: inbox, health-mcp, monitor, recon-mcp, calendar events
- **Composite urgency score calculator** (§Awareness Loop → Signal-Weighted Trigger)
- **Depth classifier**: score → Micro / Light / Deep
- **Critical event bypass**: urgent signals skip tick
- **Depth Escalation Protocol**: Awareness Loop stays sole coordinator; Reflection Engine sets
  escalation flags; critical override bypasses tick
- **Fixed signal weights** from the design doc (NOT adaptive — adaptation is V4)
- **Three-category scheduling**: Awareness Loop coordinates all three categories of scheduled
  work (§Three Categories of Scheduled Work in design doc):
  - Event-driven reflection: adaptive, signal-triggered, calendar floors as safety nets
  - Genesis's own rhythms: internal cadence timers (morning report, calibration cycles)
  - User-scheduled crons: recon-mcp + future cron infrastructure
  - Categories are distinct — different governing principles, same coordinator

### Verification

- [x] Known signal + known weight → expected composite urgency score
- [x] Score thresholds correctly classify to right depth
- [x] Critical events bypass the tick
- [x] Calendar guardrails enforce min/max intervals

---

## V3 Phase 2: Compute Routing ✅ COMPLETE

**Risk: LOW** — Infrastructure plumbing. No judgment calls. Dependencies: Phase 0.

> **Status:** COMPLETE (2026-03-04). Full `genesis.routing` package: types, config loader,
> circuit breaker (state machine + registry), retry with backoff, cost tracker, dead-letter
> queue, degradation tracker (L0-L5), router with fallback chains. YAML config at
> `config/model_routing.yaml` (13 providers, 23 call sites). 93 routing tests (380 cumulative).
> Committed on `main`.

### What AZ Already Provides

Agent Zero has provider abstraction via LiteLLM (`unified_call()` in
`models.py`) and a `RateLimiter` (token bucket) per model. Genesis does NOT
rebuild these — it layers on top:

| AZ provides | Genesis adds |
|------------|--------------|
| `unified_call()` — single LLM call path | Pre-call routing (fallback chain selection) |
| `RateLimiter` per model | Circuit breakers per provider (health-mcp) |
| LiteLLM's per-call cost tracking | Cost aggregation in SQLite (budget enforcement) |
| Two model slots (chat/utility) | 28-call-site routing registry with fallback chains |

### What to Build

Genesis wraps the call sites that invoke `unified_call()`, adding routing
and tracking. It does NOT replace `unified_call()` itself.

- **Call-site routing layer**: Wraps each Genesis LLM call site with
  primary → fallback chain logic from the model routing registry. On
  failure, try next model in chain. On success, record which model handled it.
- **Model availability detection**: health-mcp polls provider endpoints
  (local GPU, free APIs, paid APIs). This is Genesis-built — AZ has no
  equivalent.
- **Automatic fallback**: provider unreachable → next in fallback chain
- **Re-routing**: provider recovers → eligible for routing again
- **Model tracking**: which model handled each call (cost/quality analysis)
- **Cost accounting**: per-call cost (from LiteLLM), aggregated in SQLite
  `cost_events` table, enforced against `budgets` table
- **Model routing registry**: defines all 28 call sites with primary models,
  fallback chains, free compute sources, paid alternatives, and output
  validation contracts (see `docs/architecture/genesis-v3-model-routing-registry.md`)

### Compute Hierarchy

```
3B local (CPU)     → Embeddings, light extraction ONLY. No surplus, no reasoning.
20-30B local (GPU) → Micro/Light reflection, extraction, surplus tasks. PRIMARY workhorse.
Free APIs          → DEFAULT FALLBACK: Mistral, Groq, Gemini (privacy-ordered).
Sonnet-class       → Deep reflection, judgment calls, quality gates.
Opus-class         → Strategic reflection, identity proposals (V5).
```

### 3B Model Constraints

CPU only, must stay responsive for embeddings/extractions. CANNOT do: reflection, classification,
surplus tasks, meta-prompting, or anything requiring reasoning/judgment.

### Verification

- [x] Requests route to correct tier by call site
- [x] Fallback triggers when provider unreachable
- [x] Re-routing works when provider recovers
- [x] Every call tracked with model + cost in SQLite
- [x] 3B only receives embedding/extraction tasks
- [x] All LLM API calls use exponential backoff + jitter retry policy
- [x] Circuit breaker opens after consecutive failures per provider
- [x] Per-provider retry budget prevents retry storms
- [x] Cost events written for every paid call
- [x] Budget enforcement blocks calls when budget exceeded

See `genesis-v3-resilience-patterns.md` for tactical resilience design (backoff parameters,
circuit breaker thresholds, degradation levels, dead-letter staging).

See `genesis-v3-resilience-architecture.md` for system-level resilience: composite state
machine (cloud/memory/embedding/CC axes), deferred work queue with staleness policies,
embedding backlog recovery, CC budget management, recovery orchestration, and out-of-band
status file. Implemented in `src/genesis/resilience/` (7 modules).

---

## V3 Phase 3: Surplus Infrastructure ✅ COMPLETE

**Risk: LOW-MODERATE** — Scheduling and staging. Moved early to leverage free compute ASAP.
Dependencies: Phase 0, Phase 2.

> **Status (2026-03-13):** Surplus queue, staging area, deferred work infrastructure, and scheduler all implemented and tested.

### What to Build

- **Surplus task queue** with priority model:
  - Priority considers: drive weights (fixed), recency of last audit, user activity patterns
  - Task types: self-improvement, user-value-ideation, system-optimization
- **Idle cycle detection**: when no user interaction is active and compute is available
- **Compute preference** (not enforcement):
  - Surplus exists to capture free/cheap compute that would otherwise go to waste.
    Its routing chains default to free models first (local 30B, free-tier APIs).
  - This is a design preference, not a hard constraint. Genesis can override if it
    makes a case to the user, but that's an edge case — if a task genuinely needs
    paid compute, it probably belongs in regular operation, not surplus. Promote it.
  - No `never_pays` walls, no `ComputeTier.NEVER` rejection. Just chains ordered
    with free models first.
- **Surplus output staging area**: ALL outputs go to staging, never directly to production
  - Outputs await promotion by next reflection review or user approval
- **Local machine uptime tracking** (config-based schedule, not learned — learning is V4)
- **Daily brainstorming sessions** (mandatory, at least 2/day):
  1. **"How can I upgrade the user?"** — structured Light reflection on recent interactions,
     user model, and pending tasks. What opportunities is the user missing? What could be done
     better? What knowledge gap could be filled? → staging area
  2. **"How can I upgrade myself?"** — structured Light reflection on recent system performance,
     procedure effectiveness, observation utility. What's working? What's not? What should I
     try differently? → staging area
  - In V3, these use static prompt templates (Light depth, 20-30B or Gemini free).
    V4 upgrades them to meta-prompted sessions for higher quality.
  - Outputs go to staging. Promoted by next Deep reflection review or user review.
  - Each session writes a brainstorm observation to memory-mcp (outcomes + key ideas)
  - Run on free compute. If free compute is unavailable, these are the LAST tasks to skip.

### Why This Phase Is Early (Phase 3, Not Phase 6)

The surplus *infrastructure* is just a table, a queue, and a scheduler — trivially safe. Moving
it early means that from the moment Perception (Phase 4) exists, surplus can immediately run
extra micro/light reflections during idle cycles on free compute. As more phases come online,
surplus tasks get more sophisticated:

- **Phase 4+**: Surplus micro/light reflections on idle signals
- **Phase 6+**: Procedure auditing, memory scanning
- **V4**: Meta-prompted creative brainstorming, ongoing background dialogue

The daily brainstorming sessions run from day 1 of V3 with static prompts. They're simple
Light reflections — low risk, high learning value. The sophisticated meta-prompted versions
come in V4, but the infrastructure and habit are established now.

### Verification

- [x] Surplus tasks only execute on free/cheap compute
- [x] Cost-frequency rule enforced (free=always, threshold=never)
- [x] Staging area stores without promoting to production
- [x] Daily brainstorm sessions fire reliably (exactly 2/day minimum)
- [x] Brainstorm sessions write to brainstorm_log (memory-mcp integration in Phase 5)
- [x] Idle detection identifies available compute windows

---

## V3 Phase 4: Perception (Micro/Light Reflection)

**Risk: MODERATE** — First LLM calls, but low stakes. Dependencies: Phase 1, Phase 2.

### Known Gap: Signal Collectors (as of 2026-03-17)

8 of 11 awareness signal collectors are stubs returning hardcoded 0.0. This
means the awareness loop runs but triage receives near-zero signal data,
causing it to almost never escalate to micro/light/deep reflection. The
system looks "idle" when it's actually blind.

**Stub collectors** (return 0.0): ConversationCollector, TaskQualityCollector,
OutreachEngagementCollector, ReconFindingsCollector, BudgetCollector,
StrategicTimerCollector. (MemoryBacklogCollector removed 2026-04-11 — the
retrieval-coverage metric was being misread as reflection urgency by the
Deep depth scorer.)

**Real collectors**: CCSessions (queries DB), ErrorSpike (queries circuit
breakers), CriticalFailure (queries circuit breakers).

**Impact**: The perception pipeline (micro/light reflection) exists and works
when triggered, but is almost never triggered because signals stay near zero.
Fixing the collectors to query real data sources is required to make the
perception pipeline actually perceive.

### What to Build

- **Micro reflection**: routine ticks, low urgency. Quick structured extraction:
  - Tag signals, extract entities/topics, produce structured JSON summary
  - Model: 20-30B local or Gemini Flash fallback
- **Light reflection**: moderate urgency or accumulated micro signals:
  - Situation assessment, memory queries, drive-relevant interpretation, basic recommendation
  - Model: 20-30B local or Gemini Flash fallback
- **Prompt templates** for each depth (parameterized, with rotating prompt pool to prevent mode collapse)
  - **Convention:** All prompts at Light depth and above must include explicit chain-of-thought
    scaffolding ("Think through step by step: ..."). CoT significantly improves reasoning quality
    on judgment-heavy tasks (situation assessment, salience evaluation) at negligible token cost.
    Micro prompts should NOT use CoT — they are extraction tasks, not reasoning tasks.
- **Structured output parsing**: validate LLM output against schema, retry on malformed
- **Pre-Execution Assessment** prompt pattern in the main agent's system prompt:
  - LLM judgment call: "Does this request make sense? Better way? Missing info?"
  - Draws on: user model, memory, active context, procedural memory, open questions
  - Decision space: proceed / proceed with note / clarify / challenge / suggest alternative
  - Most requests pass through instantly (near-zero latency for clear requests)
  - Pushback is philosophically mandated, not signal-dependent
  - See §Pre-Execution Assessment in design doc
- **Identity context loading**: SOUL.md + user.md loaded into every 20-30B+ prompt
  - SOUL.md: `src/genesis/identity/SOUL.md` — who Genesis is (~1100 tokens, static)
  - user.md: `src/genesis/identity/user.md` — user's self-description (seed + overrides).
    Draft after Phase 4 implementation. Pull from v2 USER.md: timezone (EST),
    communication preferences ("brief first, detailed as context deepens", "don't offer
    next steps by default"), philosophical/first-principles thinking style, "critic when
    Jay locks onto one track" instruction. Omit: v2-specific tool references, v2 action
    policy (superseded by L1-L4 autonomy), relationship section (superseded by SOUL.md).
  - Context assembly must position identity docs early (high attention region)
- **Context assembly principles** (learned from v2 + design discussion):
  - v2 injected ~2200 tokens of identity files on every prompt (SOUL+USER+AGENTS+POLICY+MEMORY).
    v3 is more surgical: SOUL.md (~1100 tokens) + user.md (<300 tokens) + cognitive
    state (~600 tokens) = ~2000 tokens always loaded. Everything else on-demand via memory.
  - **Quality is non-negotiable; cost is managed through routing and pacing, never through
    context degradation.** ContextAssembler's job is to assemble the *best possible context*
    for the requested depth. It does NOT truncate, budget-gate, or degrade inputs to save
    tokens. Cost control belongs to the Router (cheaper models first) and the Surplus
    Scheduler (pacing — how often we reflect, not how well).
  - If assembled context significantly exceeds what's typical for a depth, that's a signal
    the depth classifier should have escalated — not a reason to truncate. Log it as a
    depth-mismatch signal for future classifier tuning (V4).
  - Per-depth context *scope* (not budget): Micro gets identity + signals. Light adds user
    profile + cognitive state + relevant memories. Deep/Strategic get richer context. This
    is about *relevance*, not cost — don't load irrelevant context, but never cut relevant
    context to save tokens.
  - "Context Triangulation" from v2: don't load full documents when a targeted extract
    suffices. The question is "what does this reflection need?" not "how much can we afford?"
- **Cognitive state summary infrastructure**:
  - `cognitive_state` table (schema + CRUD): stores regenerated summary of active
    context + pending actions (~600 tokens, regenerated after Deep reflections)
  - Rendered and loaded into 20-30B+ prompts alongside SOUL.md and user model
  - At Phase 4, generation is stubbed (manual or triggered by first Deep reflection
    in Phase 7). The table and rendering pipeline are built now so they're ready.
  - See §Cognitive State Summary in design doc

### Deferred to Phase 4b (Surplus Wiring)

Once the Reflection Engine is stable, a follow-up build wires surplus (Phase 3)
to trigger extra micro/light reflections during idle cycles on free compute. This
is a separate build step — Phase 4 proves the reflection pipeline works; Phase 4b
connects it to surplus for opportunistic use.

- Wire `SurplusExecutor` to call Reflection Engine for micro tasks (Light moved to CC pipeline)
- Surplus queue gets new task types: `surplus_micro_reflection`, `surplus_light_reflection`
- Idle-cycle reflections use the same prompt templates and output contracts as tick-driven ones
- Free compute only — surplus reflections never spend money

### Deferred: Observation Sweep (#28)

Call site #28 (Observation sweep — scan environment for noteworthy changes) is
deferred from Phase 4. It depends on having things to observe (recon findings,
filesystem changes, scheduled job results) which aren't fully wired yet. Place
in Phase 5 or later when memory operations and recon-mcp provide observable data.

### Verification

- [x] Micro produces valid structured tags from raw signals
- [x] Light produces coherent situation assessments
- [x] Output conforms to expected schema
- [x] Malformed output triggers retry (with limit)
- [x] Correct model tier handles each depth
- [x] Pre-Execution Assessment passes clear requests through with near-zero latency
- [x] Pre-Execution Assessment challenges requests with evidence when warranted
- [x] Pre-Execution Assessment does not over-interrogate (most requests proceed immediately)
- [x] SOUL.md loaded into prompt templates for 20-30B+ models
- [x] user.md loaded alongside SOUL.md (empty user.md produces no errors)
- [x] Cognitive state table exists with CRUD; renders to text for context loading
- [x] Context assembly positions identity docs in high-attention region

### Testing Strategy for LLM-Dependent Components

Phase 4 introduces the first LLM calls. LLM outputs are non-deterministic, which
requires a different testing approach than the pure data-layer tests in Phase 0.

**Unit tests (no real LLM):**
- Mock/stub the LLM call. Return canned responses.
- Test that the surrounding logic (prompt assembly, output parsing, retry on
  malformed, schema validation) works correctly.
- Test that bad LLM output (missing fields, wrong types, empty response) triggers
  proper error handling and retry.
- These run fast, are deterministic, and catch regressions in plumbing.

**Golden test sets (semi-deterministic):**
- Curated set of interaction summaries with expected triage depth assignments.
- Run against the real 3B model with `temperature=0` (or lowest available).
- Assert that depth assignment falls within an acceptable range (e.g., expected
  depth ±1), not exact match. The model's judgment may shift across versions.
- Maintain ~20-30 golden examples covering each depth level.
- Run as a separate test suite (`pytest -m integration`), not with unit tests.

**Distribution tests (aggregate behavior):**
- Feed a batch of ~100 varied interactions through triage.
- Assert distribution properties: "at least 40% are Depth 0 or 1" (trivial
  interactions should be common), "no more than 10% are Depth 3+" (deep
  reflections should be rare).
- This catches systematic bias (model always assigns high depth = expensive).

**Real LLM integration tests:**
- Separate test suite, run manually or in CI with `--run-integration` flag.
- Tests actual model endpoints (Ollama 3B, 20-30B if available).
- Slower, costs real compute, but catches model-specific issues.
- Not required to pass for every commit — run before releases.

**Key principle:** Test the system's behavior in aggregate and at boundaries,
not the exact text the LLM produces. If you're asserting exact strings from an
LLM, you're testing the wrong thing.

---

## V3 Phase 5: Memory Operations

**Risk: MODERATE** — Extension of existing memory system. Dependencies: Phase 0, Phase 4.

### What to Build

- **Activation scoring**: `base_score × recency_factor × access_frequency × connectivity_factor`
- **Hybrid retrieval**: embedding similarity + activation score for ranking
- **Observation storage** with utility tracking fields (retrieved_count, influenced_action)
- **Lightweight memory linking at storage time**: similarity search → link related memories
- **Memory-mcp full implementation**: replace Phase 0 stubs with real operations
- **GROUNDWORK for Knowledge Base**: multi-collection Qdrant support, `source` parameter on
  retrieval, `source_type` tagging (memory vs. reference)
- **Open question storage**: observations tagged `open_question` with domain and
  originating context. Participate in normal memory retrieval — surface when relevant
  information arrives. No special matching infrastructure needed; the memory system's
  existing semantic retrieval handles connection-making.
  See §Open Questions and Persistent Curiosity in design doc.
- **User model evolution**: v2 had a static USER.md (~300 tokens) injected every prompt.
  v3's user model is richer — it lives in the user model cache (Phase 0 schema) and gets
  synthesized by Light reflection (#11). Phase 5 builds the retrieval side: when Genesis
  needs user preferences (e.g., "does Jay prefer brief or detailed here?"), it pulls from
  the user model store, not a static file. user.md is the seed; the user model is the
  living version. v2 items to seed: timezone, communication style, autonomy preferences,
  thinking style, relationship expectations.

### AZ Memory System Internals (Discovered 2026-03-04)

Deep-dive into AZ's memory implementation reveals the exact surface area for the
FAISS→Qdrant swap:

**Current implementation:**
- `Memory` class (`python/helpers/memory.py`, ~570 lines) uses `MyFaiss` — a custom
  `FAISS` subclass with `InMemoryDocstore`. Three areas: MAIN, FRAGMENTS, SOLUTIONS.
- `VectorDB` class (`python/helpers/vector_db.py`, ~150 lines) wraps FAISS for
  knowledge/document queries. Used by `document_query.py`.
- `MemoryConsolidator` (`python/helpers/memory_consolidation.py`, ~780 lines) —
  merges/deduplicates memories using LLM judgment. Must continue working post-swap.
- Storage: `~/agent-zero/usr/memory/<subdir>/` with `index.faiss` and serialized
  docstore. Currently has live data in `default/`.

**Memory dashboard API contract** (`python/api/memory_dashboard.py`):
- Actions: `search`, `delete`, `bulk_delete`, `update`, `get_memory_subdirs`,
  `get_current_memory_subdir`
- Calls `Memory` class methods — confirmed it does NOT call FAISS directly
- Frontend formats: `id`, `area`, `timestamp`, `content_full`, `knowledge_source`,
  `source_file`, `file_type`, `consolidation_action`, `tags`, `metadata`

**Public API to preserve** (methods the dashboard and extensions call):
- `Memory.get()` / `Memory.get_by_subdir()` — factory
- `search_similarity_threshold(query, limit, threshold, filter)`
- `delete_documents_by_ids(ids)`
- `update_documents(docs)`
- `get_document_by_id(id)`
- `db.get_all_docs()` — returns dict of all documents (used for unfiltered listing)

**Migration consideration:** Existing FAISS data needs one-time migration to Qdrant.
Build a migration script that reads existing FAISS indices, re-embeds if needed
(dimension mismatch: FAISS may use different embeddings than our 1024-dim qwen3),
and inserts into Qdrant collections.

**`genesis-memory` plugin status:** Currently just a `plugin.yaml` with no code.
The actual swap implementation goes here.

### Verification

- [x] Activation scores produce reasonable rankings
- [x] Hybrid retrieval differs from pure embedding similarity
- [x] New memories link to related existing ones
- [x] Utility tracking fields increment on retrieval and action
- [x] Memory dashboard renders identically after FAISS→Qdrant swap — V3 built on Qdrant from day one, no FAISS layer
- [x] Memory consolidation works with Qdrant backend
- [DEFERRED:V4] Existing FAISS memories migrated without data loss — no FAISS data to migrate; V3 is greenfield on Qdrant

---

## V3 Phase 6: Learning Fundamentals

**Risk: MODERATE-HIGH** — First feedback-dependent behavior. Bad classification compounds.
Dependencies: Phase 4, Phase 5.

### What to Build

- **Post-interaction outcome classification**:
  - `approach_failure` → change behavior
  - `workaround_success` → store both failed path AND successful workaround as procedural
    memory. The workaround becomes the primary approach for future identical tasks.
    This is a positive outcome — Genesis is now better at this task type.
  - `capability_gap` → log, don't false-learn. **Requires workaround search exhaustion**
    (2-3 alternative approaches attempted or budget made further search uneconomical).
  - `external_blocker` → user-rectifiable / future feasibility (`revisit_after`) / permanent.
    **Same workaround search exhaustion requirement as `capability_gap`.** The bar for
    "permanent constraint" is high — prefer `revisit_after` when uncertain.
  - `success` → reinforce
- **Engagement signal extraction** (fixed per-channel heuristics):
  - Reply speed/length, reactions, follow-ups, explicit feedback
  - Channel-specific: WhatsApp reactions, email opens, web UI duration
- **Basic procedural memory**:
  - Store with confidence, retrieve top-match, version tracking
  - Failure modes stored with conditions (not bare strings) — "fails WHEN X" not just
    "fails." Include `transient` flag for conditions that may not persist.
  - Attempted workarounds (both successful and failed) stored on parent procedure with
    specific conditions. Failed workarounds are NOT blanket "never try this" signals.
  - Retrieval framing MUST surface failure conditions and workaround history alongside
    success data — nuance stripped at retrieval makes nuance in storage worthless.
  - No confidence decay in V3 (deferred to V4 — needs data to tune decay rate)
- **Real signal collectors** (Phase 1 stubs → real data):
  - BudgetCollector, ErrorSpikeCollector, CriticalFailureCollector, TaskQualityCollector
    — produce programmatic signals (computed metrics with thresholds),
    not raw data for the LLM to parse. Follows Trackio pattern: domain knowledge embedded
    in collector code, LLM receives pre-interpreted signals.
    (MemoryBacklogCollector removed 2026-04-11 — see note in Phase 1 stub list above.)
  - ConversationCollector, OutreachEngagementCollector, ReconFindingsCollector,
    StrategicTimerCollector — may remain stubs until their data sources exist (Phase 8).
- **Null hypothesis with fixed maturity milestones** (DATA VOLUME, not time):
  - `<50 procedures` = early (extract aggressively)
  - `50-200` = growing (moderate thresholds)
  - `200+` = mature (conservative, only novel patterns)
- **Basic speculative claim quarantine** (safety feature, not data-dependent):
  - Claims tagged `speculative: true` with TTL expiry
  - Quarantined from future context retrieval
  - Expired with no evidence → archived
  - Confirmation cycle (evidence counting, cross-referencing) deferred to V4
- **Retrospective triage** (Step 0 of Self-Learning Loop — runs on EVERY interaction):
  - Programmatic pre-filter: < ~100 tokens + zero tool calls → depth 0 (skip)
  - Everything else → 3B SLM classification (few-shot prompt + calibration rules)
  - Depth assigned by characteristics (complexity, blockers, effort, stakes), NOT by
    interaction type. A cron job or browser session can be depth 4; a trivially
    completed formal task might only be depth 1.
  - 3B prompt includes curated examples and calibration rules (initially hand-crafted,
    later maintained by the daily triage calibration cycle)
  - V3: all calibration is prompt-level (no fine-tuning)
- **Signal weight tiers** (bootstrap defaults — V3 uses fixed tiers, V4 calibrates):
  - Tier 1 (strong): direct user corrections, explicit feedback, rejected deliverables
  - Tier 2 (moderate): clear task success/failure, outreach engagement
  - Tier 3 (weak): behavioral inference, silence, override outcomes
  - Critical constraint: weak signals must NOT erode philosophical commitments (pushback)
  - See §Signal Weight Tiers in design doc
- **Daily triage calibration cycle** (companion job to Morning Report):
  - 20-30B model reviews sampled triage decisions from last 24h
  - Under-classification audit: reads actual chat logs for depth-0 decisions to check
    whether anything worth capturing was missed
  - Over-classification audit: checks whether depth 2+ retrospective outputs were
    subsequently used
  - Memory pattern review for salience shifts
  - Outputs updated few-shot examples + calibration rules for the 3B's triage prompt
  - Memory signals reach the 3B through the 30B's digested rules, NOT directly
- **Retrospective observations**: Interactions at depth 2+ that produce notable
  outcomes write a retrospective observation to memory-mcp — what happened, root cause
  classification, lessons extracted. Depth 2 = notable outcome (discovery, correction,
  or surprise). Depth 3+ = formal task outcome.

### Adaptive Obstacle Resolution (Procedural)
<!-- Added 2026-03-08: GL-1 post-implementation discussion -->

Genesis's philosophical commitment is "failure is not an option" — there is almost
always a workaround, the question is how much effort and creativity to spend finding
it. This applies to ALL obstacles: web fetching 403s, API rate limits, model
unavailability, tool failures, permission errors.

**V3 scope:** Static fallback chains for known obstacle types (e.g., web fetch:
direct → search cache → archive.org → headless browser). Hard-coded ordering,
but structured as procedure-like data so V4's learning system can take over ranking.

**Why V3 is static:** The adaptive system (procedures that learn which workarounds
work for which obstacles and update rankings) requires Phase 6's outcome
classification and procedural memory. V3 establishes the infrastructure; V4 makes
it adaptive.

**V4 target:** Each obstacle type becomes a learned procedure with a ranked list of
resolution methods. Genesis records which methods work for which contexts, updates
rankings as methods stop working or new ones emerge, and proactively researches new
tools/methods via surplus compute. The procedure's confidence scores reflect
real-world effectiveness, not static assumptions.

**Design principle:** Every tool in the fallback chain is valid — the question is
ordering by path of least resistance. A headless browser is overkill for most
fetches but valid when simpler methods fail. The learning system maintains this
ranking, not a developer.

### Tool Capability Discovery
<!-- Added 2026-03-09: research session — source discovery, obstacle resolution -->

The `tool_registry` table (Phase 0) stores tool metadata but is currently
unpopulated. Phase 6 populates it with real capability data and writes the
routing logic that enables content-type-based auto-routing.

**What to build:**
- Populate `tool_registry` with capability metadata for all available tools
  (MCP tools, Gemini API, Firecrawl, CC native tools, etc.). Each entry
  includes: content types handled, access requirements, cost tier, reliability.
- Content-type routing function: given a resource (URL, file, content type),
  query tool_registry for tools that can handle it, return ranked options.
- Integration with the Adaptive Obstacle Resolution fallback chains: when the
  primary tool fails, the routing function suggests the next capable tool.
- Cross-model routing table: YouTube → Gemini, web pages → Firecrawl → any LLM,
  images → Claude vision, PDFs → Claude/Gemini, code repos → CC native.
  See `2026-03-08-research-insights-and-followups.md` Section 3 for the full
  capability routing table.
- `capability_gaps` table (Phase 0) updated when a content type has no capable
  tool — feeds the proactive improvement surfacing in morning reports (Phase 8).

**Why Phase 6:** Tool capability discovery is learning infrastructure. Genesis
needs to know what its tools can do before it can autonomously resolve obstacles
or evaluate external sources. The static fallback chains (Adaptive Obstacle
Resolution above) provide the V3 structure; this populates them with real data.

### Sub-Agent Memory Harvesting
<!-- Added 2026-03-09: research session — CC session learnings -->

When Genesis dispatches CC sub-agent sessions (deep reflection, task execution,
surplus), incidental learnings are currently lost. Phase 6 implements two of the
three harvesting mechanisms (third in Phase 7):

**Mechanism 1 — Structured debrief (Phase 6):**
CC session system prompt requires a `learnings` section in final output. Genesis
parses this after session completion and feeds findings into memory operations
(store as observations, update/create procedures). Prompt engineering addition
to session-type CLAUDE.md content.

**Mechanism 2 — Auto-memory harvest (Phase 6):**
After CC session completion, Genesis reads the session's auto-memory directory
and ingests relevant items into the Genesis memory store. Richer than structured
debrief — captures things the LLM found worth remembering even if not in
explicit output.

**Mechanism 3 — Cross-run context injection (Phase 7):**
When launching a CC session, include relevant memories from previous runs of
similar tasks. Uses Phase 5 hybrid retrieval (query by task description, inject
top-k relevant memories into system prompt). Requires Phase 6 stored learnings.

See `2026-03-08-research-insights-and-followups.md` Section 11 for full design.

### Inbox Monitor Service
<!-- Added 2026-03-09: research session — async Obsidian inbox -->

A peripheral service (like the surplus scheduler) that monitors a user-configured
folder for new content. This is NOT part of the awareness loop — it's an external
task runner with its own schedule. The awareness loop is cognition; this is a hand.

**What to build:**
- `InboxMonitor` class with its own APScheduler instance (like `SurplusScheduler`)
- Configurable: folder path, check interval, response directory, enabled/disabled
- On each check: scan folder for new/modified files (mtime or content hash tracking)
- Item classification: link (→ research task), note (→ observe and store),
  ambiguous (→ queue question for user via outreach pipeline)
- Dispatches research tasks to the surplus queue (links → evaluate skill)
- Writes response files as Obsidian-compatible markdown to a configurable
  response subdirectory (e.g., `_genesis/`) within the watched folder
- Pending clarification items queued to outreach/message_queue — NOT processed
  until user responds through foreground channel
- User-configurable via dashboard or config file (path, interval, on/off)

**Architecture constraints:**
- The awareness loop does NOT know about Obsidian or the inbox folder
- The inbox monitor feeds INTO existing infrastructure (surplus queue,
  message_queue, outreach pipeline) — it does not bypass them
- Response files written atomically (temp file + rename) to avoid sync conflicts
- Processed items tracked by content hash to avoid re-processing

**Why Phase 6:** Requires outcome classification (is this a link? a note? ambiguous?)
and procedural memory (learn which inbox patterns are research vs noise). The
surplus scheduler infrastructure (Phase 3) provides the scheduler pattern.
CC session dispatch (Phase 7 session_config) handles the actual evaluation work.

**Minimum viable version (post-Phase 6):** Monitor folder, classify items, dispatch
link evaluation to CC sessions, write markdown responses. No tag-based routing,
no proactive research, no graph integration (those are V4).

See `docs/plans/2026-03-09-inbox-monitor-plan.md` for implementation plan.

<!-- ADDENDUM 2026-03-10: Inbox Monitor IMPLEMENTED -->
**Status: IMPLEMENTED (2026-03-10)**
Design updated: LLM-first classification (no heuristic pre-classification layer).
The LLM reads raw content and decides per-item whether to research, note, or
question. Batching: max 5 items per CC session, overflow creates additional
sessions. State tracked in `inbox_items` DB table (not JSON file). System prompt
at `src/genesis/identity/INBOX_EVALUATE.md`. AZ extension `_55_genesis_inbox.py`.
51 new tests (1067 cumulative).
<!-- /ADDENDUM -->

### Skill Wiring for Genesis Sessions
<!-- Added 2026-03-09: research session — skill availability -->

> **Skill conventions:** All Genesis skills MUST follow the conventions in
> `docs/reference/genesis-skill-conventions.md` — description format, writing
> style, size limits, progressive disclosure, Genesis-specific frontmatter
> fields. Read that doc before creating or modifying any skill.

AZ's skill system discovers `SKILL.md` files from plugin directories. Genesis
skills (e.g., `RESEARCH_EVALUATION.md`) must be wired into the AZ plugin's
skill directory for AZ-runtime discovery.

**What to build:**
- Copy/symlink Genesis skills from `src/genesis/skills/` into
  `usr/plugins/genesis/skills/<skill-name>/SKILL.md` with proper frontmatter
  (name, description, triggers matching Genesis's use cases).
- Ensure AZ agent can `skills_tool:load` Genesis skills when relevant signals
  trigger them (e.g., new tool discovered → load research-evaluation skill).
- Skills are living documents — procedural learning (this phase) can update
  skill files when Genesis learns better approaches.
- Adopt SKILL.md structure from Superpowers pattern: progressive disclosure
  (metadata always loaded, SKILL.md body on trigger, references/ on demand).
- The evaluate command (`.claude/commands/evaluate.md`) should be restructured as
  a Genesis plugin skill (`src/genesis/skills/evaluate/SKILL.md` + `references/`)
  for use in CC background sessions. Interactive CC command stays as-is.

**Why Phase 6:** Skills need to be discoverable before Genesis can autonomously
use them for evaluation or research. Phase 6's procedural learning also enables
skill self-updating — Genesis observes "this evaluation approach worked better"
and updates the skill file.

- **Skills inventory exercise**:
  <!-- Added 2026-03-09: research session — LangChain 25%→95%, Microsoft Agent Framework -->
  - Before building individual skills, enumerate ALL skills Genesis needs across all
    consumers: CC background sessions (evaluate, research, reflect, retrospective, debug),
    foreground conversation (morning-report, outreach), autonomous operation (task-planning,
    verification, self-assessment, obstacle-resolution), inbox monitor (classify, research).
  - LangChain demonstrated 25%→95% performance improvement from domain-specific skills.
    Skills are not optional — they are critical infrastructure.
  - Design each skill following the industry-standard SKILL.md pattern with progressive
    disclosure (advertise ~100 tokens, load <5000 tokens, read resources on demand).

### Perception Template Externalization
<!-- Added 2026-03-08: GL-1 post-implementation discussion -->

Phase 4's `PromptBuilder` has 6 reflection templates (3 micro, 3 light) hardcoded
as Python strings. GL-1 proved that externalizing prompts to CAPS markdown files
works better — easier to edit, version, audit, and test.

**Near-term cleanup:** Externalize the 6 templates to `src/genesis/identity/` as
CAPS markdown files (e.g., `MICRO_TEMPLATE_HEALTH.md`, `LIGHT_TEMPLATE_PATTERN.md`).
The `PromptBuilder` loads them the same way `CCReflectionBridge` loads reflection
prompts — with fallback to hardcoded strings if files are missing.

**Why:** Transparency — the user should be able to see and edit everything that
shapes how Genesis thinks. Hardcoded strings hide behavior in code that only
developers can audit. CAPS markdown files make it visible to anyone.

### Phase 4 Follow-Up: Reflection Template Expansion

Phase 4 ships with 3 micro templates and 3 light templates. By Phase 6, outcome
classification data reveals which templates produce useful observations vs noise.
Use this data to:
- Identify blind spots (signal patterns none of the 3 templates catch well)
- Identify redundancy (templates that overlap >40% of the time)
- Add templates to fill demonstrated gaps — target 5-7 micro once stabilized
- Do NOT add templates on a schedule. Only add when data justifies it.

### Critical Risk

**This is the highest-leverage failure point.** Bad classification → bad lessons → systematic
drift. Mitigation: null hypothesis = conservative by default. Monitor `capability_gap` vs
`approach_failure` ratio — heavy skew = classification probably wrong.

### Verification

- [x] Outcome classification matches human judgment on test set — `learning/classification/outcome.py` with typed outcomes
- [x] Engagement signals detect positive/negative correctly — `outreach/engagement.py` + `learning/signal_tiers.py`
- [x] Maturity thresholds switch at correct data volume — `learning/pipeline.py` signal tier routing
- [x] **Capability gaps are NOT learned as approach failures** (critical test) — `capability_gap` is a distinct outcome type in `learning/types.py`
- [x] `capability_gap` and `external_blocker` require evidence of workaround search
  exhaustion (genuinely different strategies explored, not variations of the same
  approach). A single failed attempt must NOT be classified as a gap — it's an
  incomplete workaround search. — `learning/classification/attribution.py` enforces evidence
- [x] `workaround_success` stores both the failed primary path AND the working
  workaround. Future identical tasks use the workaround as primary approach. — `learning/types.py` + `db/schema/_tables.py`
- [x] Request-delivery delta assessed for every task with full scope evolution chain:
  original request → discoveries during execution → adjusted scope → delta classification
  → discovery attribution. Critical test cases: — `learning/classification/delta.py` + `db/crud/execution_traces.py`
  - `success` + `acceptable_shortfall` + `external_limitation` routes differently than
    `success` + `acceptable_shortfall` + `genesis_capability`
  - `scope_communicated = false` is flagged as a process failure regardless of delta
  - `user_model_gap` attribution feeds "ask clarifying questions" heuristic, not procedures
  - `scope_was_underspecified` attribution feeds "proactively clarify scope" heuristic
- [x] Triage runs on every interaction — no interaction bypasses Step 0 — `pipeline/triage.py` + `learning/triage/`
- [x] Programmatic pre-filter correctly skips only definitionally trivial exchanges — `learning/triage/classifier.py`
- [x] 3B SLM triage assigns depth consistent with human judgment on test set — `learning/triage/classifier.py` with Ollama SLM
- [x] Daily triage calibration cycle produces updated few-shot examples and rules — `learning/triage/calibration.py`
- [x] Under-classification audit reads actual chat logs for depth-0 decisions — `learning/triage/calibration.py`
- [x] Memory signals reach triage through 30B calibration rules, not directly to 3B — `identity/TRIAGE_CALIBRATION.md` + calibration pipeline
- [DEFERRED:V4] Speculative claims quarantined from retrieval context — requires confidence-tagged memory retrieval filtering; V4 memory quality feature
- [x] Depth 2+ outcomes write retrospective observations to memory-mcp — `learning/observation_writer.py`
- [x] `tool_registry` populated with capability metadata for all available tools — `db/crud/tool_registry.py`
- [x] Content-type routing function returns ranked tool options for a given resource — `learning/tool_discovery.py`
- [x] Cross-model routing works end-to-end (e.g., YouTube URL → Gemini → summary) — `routing/model_profiles.py` + provider routing
- [x] `capability_gaps` updated when no tool can handle a content type — `db/crud/capability_gaps.py`
- [x] Structured debrief: CC session outputs include `learnings` section, parsed into memory — `learning/harvesting/debrief.py`
- [x] Auto-memory harvest: CC session auto-memory ingested into Genesis memory store — `runtime.py` auto_memory_harvest job
- [x] Genesis skills discoverable via AZ `skills_tool:load` — `learning/skills/wiring.py`
- [x] Skill files updated by procedural learning when better approaches are learned — `learning/skills/refiner.py`
- [x] Skills inventory completed — all required skills enumerated with consumers identified — `learning/skills/inventory.py`
- [x] Skill performance baseline measured (tasks with and without skills) — `learning/skills/effectiveness.py` baseline_success_rate
- [x] InboxMonitor runs on its own schedule, independent of awareness loop — `inbox/monitor.py` with own schedule
- [x] Inbox items classified correctly (link → research, note → store, ambiguous → ask) — `inbox/monitor.py`
- [x] Research tasks dispatched to surplus queue for CC session evaluation — `inbox/monitor.py` → surplus queue
- [x] Response files written as Obsidian-compatible markdown — `inbox/monitor.py`
- [x] Processed items tracked by hash — no duplicate processing — `inbox/monitor.py`
- [x] User can configure path, interval, and enable/disable via config — `inbox/config.py`

---

## V3 Phase 7: Simple Deep Reflection ✅ COMPLETE

**Risk: MODERATE** — LLM call at Sonnet-class, but single structured prompt (no multi-model
orchestration). Dependencies: Phase 4, Phase 5, Phase 6.

> **Status (2026-03-13):** Reflection scheduler, context gatherer, output router, learning stability monitor, CC session bridge — all implemented and tested.

### What to Build

- **Single structured Deep reflection** (Sonnet-class):
  - Triggered by Awareness Loop when composite urgency crosses Deep threshold
  - Only runs jobs with pending work (adaptive, NOT calendar-driven)
  - Covers when applicable:
    - Memory consolidation (if backlog of observations)
    - Cost reconciliation (if spend changed since last check)
    - Lessons extraction (if new retrospectives available)
    - Recon triage (if new findings accumulated)
    - **Review surplus staging area** — promote or discard brainstorm outputs
    - **Skill effectiveness review** — analyze skill-tagged session outcomes,
      trigger refiner for underperforming skills (see Skill Evolution section)
    - **Regenerate cognitive state summary** — model already has full picture in
      context; final step produces ~600-token summary of active context + pending
      actions, written to `cognitive_state` table
  - Reads current cognitive state summary for continuity ("what was I focused on?")
  - "Dream Cycle 2.0" — same jobs as current dream cycle, smarter trigger, single model call
- **Explicit memory consolidation in Deep reflection prompt**:
  <!-- Added 2026-03-09: research session — Google "Always On Memory Agent" pattern -->
  - Deep reflection prompt (REFLECTION_DEEP.md) must include explicit consolidation
    directives: deduplicate related memories, merge overlapping observations, restructure
    stale links, flag contradictions. Current prompt focuses on pattern recognition only.
  - "Dreaming" pattern: periodic LLM pass that reorganizes memory, not just reads it.
  - This is the v3 equivalent of the v2 dream cycle's memory cleanup jobs, but
    LLM-driven rather than programmatic.
- **Mandatory weekly self-assessment** (Sunday, configurable):
  - Runs as a Deep reflection job even if no other Deep triggers are pending
  - Evaluates 6 dimensions: reflection quality, procedure effectiveness, outreach calibration,
    learning velocity, resource efficiency, blind spots
  - Each dimension has a concrete data source (no vague self-congratulation)
  - Output: structured assessment → memory-mcp (episodic, tagged `self_assessment`)
  - See §Weekly Self-Assessment in design doc
- **Weekly quality calibration** (companion job to weekly self-assessment):
  - More capable model samples recent task outputs and assesses:
    - Quality gate strictness: were passes justified?
    - Pushback frequency: did Genesis challenge when a thoughtful person would have?
    - Quality drift: are standards slipping compared to earlier periods?
    - Signal weight compliance: did weak signals erode philosophical commitments?
    - **Procedure effectiveness**: do tasks where learned procedures were applied
      succeed more than baseline? Track per-procedure success rate trend.
    - **Learning velocity sanity**: is the rate of new procedures/observations
      reasonable? Contradictory observations at high rates = something wrong.
  - Output: quality calibration observations in memory-mcp, tagged `quality_drift` if
    drift detected. Retrieved during future Pre-Execution Assessments as counterweight.
  - See §Quality Calibration Cycle in design doc
- **Learning stability monitoring** (integrated into quality calibration + deep reflection):
  <!-- Added 2026-03-10: research session — TTT stability parallel, user-requested -->
  - **Problem**: same failure mode as Test-Time Training at neural level — unconstrained
    updates from learning can degrade system performance rather than improve it.
    Procedures learned from recent-but-atypical interactions can override sound general
    knowledge. Observations can accumulate contradictions without resolution.
  - **Procedure quarantine**: when weekly quality calibration detects a procedure's
    success rate declining (applied 3+ times, success rate below 40%), quarantine it:
    set a `quarantined` flag that excludes it from retrieval. Still stored, can be
    rehabilitated if circumstances change. Quarantine is a deep reflection output.
  - **Contradiction detection**: deep reflection memory consolidation explicitly checks
    for observations that contradict each other. When found: resolve (keep the one with
    stronger evidence), merge (synthesize a nuanced replacement), or flag for user
    review (when evidence is ambiguous). Don't let contradictions accumulate silently.
  - **Learning regression signal**: if weekly self-assessment's "procedure effectiveness"
    dimension trends downward for 2+ consecutive weeks, emit a `learning.regression`
    event via event bus AND include it as a cognitive state item. This makes the
    regression visible to every subsequent reflection and pre-execution assessment.
  - **Consolidation-as-defense**: deep reflection memory consolidation isn't just
    organization — it's the primary defense against memory pollution. Dedup, merge,
    and prune are safety operations, not housekeeping. Treat them with that gravity.
- **Research output documentation structure**:
  <!-- Added 2026-03-09: research session — Genesis needs to organize its own outputs -->
  - When Genesis performs Deep reflection, strategic evaluation, or autonomous research,
    it produces structured written findings. These need a defined storage location and
    organization scheme — Genesis can't just dump everything into a flat memory store.
  - Research during Phase 7 planning: evaluate options (dated files in docs/plans/,
    memory-mcp tagged entries, cognitive state journal, or a hybrid). Decide on a
    structure that supports both retrieval (memory system) and human audit (file system).
  - This is a design question to resolve during Phase 7 implementation, NOT to defer.
    The decision affects how all subsequent phases store their outputs.
- **CC session skill loading** (`session_config.py`):
  <!-- Added 2026-03-09: research session — GL-3 gap, skill availability -->
  - Phase 7 is when background CC sessions become the primary mechanism for Deep
    reflection. These sessions need skills injected based on session type.
  - `session_config.py` generates per-session-type configs: MCP servers, hooks,
    CLAUDE.md content, and skill references. Reflection sessions doing strategic
    evaluation get the research-evaluation skill. Task sessions get verification
    and debugging skills.
  - Mechanism: `--append-system-prompt` or `--system-prompt` flag in CCInvoker
    with skill content included for that session type.
  - See `2026-03-08-research-insights-and-followups.md` Section 13 for the full
    session type × capability matrix.
- **Cross-run context injection** (sub-agent memory harvesting mechanism 3):
  <!-- Added 2026-03-09: research session — memory across CC sessions -->
  - When launching a CC session for a task, include relevant memories from previous
    runs of similar tasks. Uses Phase 5 hybrid retrieval (query by task description,
    inject top-k relevant memories into system prompt).
  - Requires: Phase 6 stored learnings (structured debrief + auto-memory harvest
    must already be populating the memory store).
  - See `2026-03-08-research-insights-and-followups.md` Section 11.
- **Skill evolution system** (skills as learning artifacts, not static config):
  <!-- Added 2026-03-10: skill writing/refinement as core self-learning function -->
  - **Conventions baseline:** `docs/reference/genesis-skill-conventions.md` defines the
    structural and stylistic standards all skills must follow. The SkillRefiner must
    produce proposals that comply with these conventions (description format, imperative
    writing style, size limits, progressive disclosure structure).
  - **Design invariant:** Skills are not static configuration — they are learning artifacts
    governed by the same lifecycle as procedures (creation → usage → measurement → refinement
    → quarantine/retirement). The learning system and skill system are a single connected
    loop, not two disconnected ones. `skill_tags` on `CCInvocation` (added Phase 6) provide
    the data pipeline; this phase builds the analysis and action layers.
  - **SkillEffectivenessAnalyzer** (`genesis.learning.skills.effectiveness`):
    - Computes per-skill health metrics from `cc_sessions` (skill_tags in metadata) +
      outcome data from observations/debriefs. No new tables — computed on-the-fly.
    - `SkillReport`: usage_count, success_rate, baseline_success_rate (from untagged
      sessions doing similar work — answers "does this skill actually help?", per the
      LangChain 25%→95% evidence), failure_patterns, trend (IMPROVING/STABLE/DECLINING),
      tools_used vs tools_declared (mismatch = evolution signal), sessions_since_last_refined.
    - Trend computed from rolling window (last 10 uses vs prior 10).
  - **Skill typing** — each skill classified as `uplift` (extends what Genesis can do),
    `workflow` (describes how to do what it already can), or `hybrid`. Stored in SKILL.md
    frontmatter. Matters for evolution: uplift failures suggest capability gaps (need new
    tools/integrations); workflow failures suggest bad instructions (need prompt rewrites).
  - **SkillRefiner** (`genesis.learning.skills.refiner`):
    - LLM call: takes SkillReport + current SKILL.md → `SkillProposal` (proposed content,
      rationale, change_size classification, confidence, failure_patterns_addressed).
    - Change size: MINOR (description tweaks, examples, clarifying steps, <10 lines) |
      MODERATE (workflow steps, new sections, thresholds) | MAJOR (full restructure, new
      skill creation, retirement).
    - **Progressive disclosure enforcement**: if a skill grows past ~500 lines through
      evolution, propose restructuring into SKILL.md + `references/` subdirectory rather
      than appending more content (Microsoft three-stage model: advertise → load → read).
    - Classification is LLM-determined, sanity-checked by diff line count.
  - **SkillApplicator** (`genesis.learning.skills.applicator`):
    - Applies or stages based on change size × autonomy level:
      ```
      Autonomy L2 (V3 default):  MINOR auto-applies, MODERATE+ staged
      Autonomy L3 (earned):      MINOR + MODERATE auto-applies, MAJOR staged
      Autonomy L4 (earned):      All auto-apply
      ```
    - Auto-applied: write to SKILL.md, log as observation tagged `skill_evolution`.
    - Staged: write as observation tagged `skill_proposal`, include full diff + rationale.
    - **Validation step for MODERATE+ proposals**: second LLM call evaluates the proposal
      against the failure patterns it claims to address. GROUNDWORK(skill-ab-testing):
      V4 replaces this with actual A/B eval runs comparing old vs new skill on similar
      prompts.
    - All changes tracked in observations — the memory system IS the audit trail.
  - **Deep reflection integration**: "skill effectiveness review" added as a deep reflection
    job. Runs when new outcome data exists for skill-tagged sessions since last review, or
    when a skill's success rate drops below threshold. Produces SkillReports for active
    skills, triggers refiner for underperformers. Subject to same learning stability
    monitoring (quarantine, contradiction detection, regression signals).
  - **GROUNDWORK(skill-autonomy-graduation):** Autonomy graduation for skill evolution.
    Uses existing `autonomy_state` table — new category `skill_evolution`, starts at L2.
    Elevation criteria (V4 activates, V3 stores):
    - User approval rate of staged proposals > 80% over 20+ proposals
    - Zero regressions from auto-applied changes over 30-day window
    - No quarantined skills that were auto-modified
  - **GROUNDWORK(skill-ab-testing):** V4 adds actual A/B testing — run two skill versions
    on similar tasks, compare outcomes. Replaces the second-opinion LLM validation.
  - **GROUNDWORK(skill-auto-creation):** V4 adds new skill creation from scratch — pattern
    recognition across many sessions identifies repeated workflows worth crystallizing.
  - **GROUNDWORK(skill-manifest):** V4 adds structured capability manifests (HAND.toml →
    SKILL.toml pattern from OpenFang) — explicit tool/permission/metric declarations per
    skill. V3 tracks tools_used empirically; V4 makes declarations enforceable.
  - **Not in V3:** skill retirement (needs enough data), skill A/B testing (needs eval
    infrastructure), new skill creation from scratch (needs pattern recognition across many
    sessions), meta-skill learning (V5 — learning how to write better skills).
- **No meta-prompting** — uses a comprehensive static prompt. V4 replaces this with the
  3-step meta-prompting protocol for higher quality.
- **No Strategic reflection** — weekly MANAGER / monthly DIRECTOR reviews are V4.
- **No fresh-eyes review** on Deep outputs — V4 adds this for high-stakes outputs.

### Why This Exists (Not Just Micro/Light)

Without Deep reflection, V3 has no equivalent of the current nightly dream cycle. Micro/Light
handle real-time signal processing; Deep handles the periodic "step back and think about the
bigger picture" that consolidation, lesson extraction, and surplus review require. A single
Sonnet-class call with a comprehensive prompt is a massive upgrade over 13 fixed-interval
cron jobs that run every night regardless of whether there's anything to process.

### Verification

- [x] Deep reflection triggers only when warranted (pending work exists) — `reflection/scheduler.py` urgency threshold
- [x] Jobs with no pending work are skipped (not run for nothing) — `reflection/scheduler.py` adaptive scheduling
- [x] Surplus staging area outputs are reviewed and promoted/discarded — `reflection/output_router.py`
- [x] Cognitive state summary regenerated after each Deep reflection — `db/crud/cognitive_state.py` + reflection output
- [x] Cognitive state summary is ~600 tokens and contains active context + pending actions
- [x] Weekly self-assessment fires on schedule even during quiet weeks — `identity/SELF_ASSESSMENT.md` + reflection scheduler
- [x] Self-assessment queries real data sources (not generating fictional metrics) — `reflection/context_gatherer.py` pulls real health data
- [x] Output quality is reasonable (baseline for V4 meta-prompting comparison)
- [x] Quality calibration runs weekly and samples recent task outputs — `identity/QUALITY_CALIBRATION.md` + reflection scheduler
- [x] Quality drift detection flags declining standards with specific examples — `reflection/stability.py`
- [x] Quality calibration observations are retrievable during Pre-Execution Assessment — `perception/context.py` loads calibration observations
- [x] Deep reflection prompt includes explicit memory consolidation directives — `identity/REFLECTION_DEEP.md`
- [x] Memory consolidation deduplicates, merges, and restructures (not just pattern-matches) — `identity/REFLECTION_DEEP.md` consolidation directives
- [x] Procedure quarantine implemented: procedures with declining success rates excluded from retrieval — `db/crud/procedural.py` quarantine flag
- [x] Contradiction detection implemented: deep reflection identifies and resolves conflicting observations — `identity/REFLECTION_DEEP.md` contradiction resolution directives
- [x] Learning regression signal emits `learning.regression` event after 2+ weeks of declining procedure effectiveness — `reflection/stability.py`
- [x] Quality calibration tracks per-procedure success rate trends (not just aggregate) — `reflection/stability.py` per-procedure tracking
- [DEFERRED:V4] Research output documentation structure decided and implemented — research outputs currently go to memory/observations; structured documentation format is a V4 knowledge management feature
- [x] `session_config.py` generates per-session-type configs (MCP, hooks, CLAUDE.md, skills) — `cc/session_config.py` SessionConfigBuilder with reflection/task/surplus types
- [DEFERRED:V4] Reflection CC sessions receive research-evaluation skill when doing strategic work — skill injection per session type not yet wired; `session_config.py` has skill loading but no strategic-skill routing
- [x] Task CC sessions receive verification + debugging skills — `cc/session_config.py` build_task_config accepts skill_names
- [x] Cross-run context injection: CC sessions launched with relevant memories from prior runs — `cc/context_injector.py`
- [x] Session hook inheritance works (reflection = read-only, task = safety, surplus = budget) — `cc/session_config.py` disallowed_tools per type; GROUNDWORK for hook config
- [x] SkillEffectivenessAnalyzer computes per-skill metrics from cc_sessions + observations — `learning/skills/effectiveness.py`
- [x] SkillReport includes baseline_success_rate (with-skill vs without-skill comparison) — `learning/skills/effectiveness.py`
- [x] Skill typing (uplift/workflow/hybrid) present in SKILL.md frontmatter for all skills — `learning/skills/types.py` SkillType enum
- [x] SkillRefiner produces actionable proposals with change-size classification (MINOR/MODERATE/MAJOR) — `learning/skills/refiner.py` + `types.py` ChangeSize enum
- [x] SkillRefiner enforces progressive disclosure size budget (~500 lines, restructure to references/) — `learning/skills/refiner.py` 500-line warning
- [x] MINOR changes auto-apply at L2 autonomy; MODERATE+ staged for review — `learning/skills/applicator.py`
- [x] Applied skill changes logged as observations tagged `skill_evolution` — `learning/skills/applicator.py`
- [x] Staged skill proposals retrievable for user review (tagged `skill_proposal`) — `learning/skills/applicator.py`
- [x] Validation step runs for MODERATE+ proposals (second LLM call) — `learning/skills/applicator.py`
- [x] Skill evolution subject to same learning stability monitoring as procedures — `reflection/stability.py`
- [x] tools_used vs tools_declared tracked per skill (mismatch = evolution signal) — `learning/skills/effectiveness.py`
- [x] GROUNDWORK: `skill_evolution` autonomy category exists in autonomy_state, starts at L2 — GROUNDWORK comment in `learning/skills/applicator.py`; not yet a formal ContextCeiling category
- [x] GROUNDWORK: elevation criteria defined (approval rate, regression window) but inactive — GROUNDWORK tagged, inactive by design

---

## V3 Phase 8: Basic Outreach ✅ COMPLETE

**Risk: HIGH** — User-facing. Trust damage from bad outreach is hard to reverse.
Dependencies: Phase 3, Phase 6, Phase 7.

> **Status (2026-03-13):** Governance gate, fresh-eyes review, outreach pipeline, morning report, engagement tracker, calibration infrastructure, outreach MCP, recon MCP — all implemented and tested. Channel registration wired. Morning report failures observable via event bus.
>
> **Dashboard & UI (2026-03-13):** Genesis dashboard at `/genesis` with 4 panels (system health, pending actions, activity feed, config files). Provider activity tracker (`ProviderActivityTracker`) gives embedding call counts, latency, cache hit rates, error rates — exposed via `/api/genesis/provider-activity` and `health_status()` MCP tool. `EmbeddingProvider` instrumented with activity tracker + event bus events for fallback/failure. AZ UI rebranded with Genesis nav button. Outreach API blueprint wired into bootstrap. See dashboard design spec.

> **Pulled forward (pre-Phase-8):** Health MCP tool implementations
> (`health_status`, `health_errors`, `health_alerts`) and the Neural Monitor
> dashboard were built ahead of Phase 8. The shared `HealthDataService` backend
> serves both the human dashboard (Flask API at `/api/genesis/health`,
> dashboard at `/genesis/monitor`) and the health MCP tools (Genesis
> self-awareness). See neural monitor design spec.
> Runtime now exposes circuit_breakers, cost_tracker, dead_letter_queue,
> deferred_work_queue, cc_budget_tracker as properties.
>
> **Post-Phase-8 audit (2026-03-13):** Fixed 5 routing bugs (call site ID
> mismatch in retrospective triage, fresh_eyes method name, test references,
> Ollama model name). Changed circuit breaker to trip on PERMANENT errors
> (broken providers now show red, not green). Enriched HealthDataService with
> awareness heartbeat, queue ages, CC session durations, outreach rates, disk
> space, and Ollama model validation. New health alerts for tick overdue,
> stale dead letters, disk low, and Ollama model mismatch. Deferred signals
> documented in `docs/plans/health-signal-gaps-deferred.md`.

> **Knowledge autonomy reference:** `genesis-knowledge-autonomy.md` defines the
> principle that Genesis should handle all knowledge ingestion, organization,
> retrieval, and proactive integration without the user managing infrastructure.
> Phase 8 is where multi-channel ingestion, retrieval confidence gating, and
> ingestion feedback land. Read that doc when starting this phase.

### What to Build

- **Outreach pipeline**: staging → governance check → channel selection → timing → delivery → tracking
- **Governance gate** (deterministic, before every outreach):
  - Within autonomy permissions?
  - Passes salience threshold? (fixed threshold in V3)
  - Timing appropriate? (quiet hours from config)
  - Similar outreach sent recently? (dedup)
  - Budget check for paid channels
- **Outreach categories in V3**: **Blocker and Alert only**
  - Finding/Insight/Opportunity categories deferred to V4 (need calibrated user model)
- **Exactly 1/day surplus-driven outreach** from day 1 of autonomous behavior:
  - Sourced from daily brainstorm staging area (Phase 3)
  - Labeled as surplus-generated
  - Governance-gated and engagement-tracked
  - This is how the system learns to be proactive — it can't grow without opportunities
- **Engagement tracking per outreach**:
  - Delivered, opened, replied to, reply sentiment, action taken, ignored
  - Fed back to Learning Loop (Phase 6)
- **Fixed channel preferences** (config-driven, not learned — channel learning is V4)
- **Fresh-eyes review on outreach before sending**: cross-model check on the 1/day surplus outreach
- **Daily morning report**:
  - Trigger: first idle cycle after configured morning time (default: 7:00 AM), or first interaction of day
  - NOT a template checklist — Genesis decides what's worth saying based on cognitive state
    summary, overnight activity, system state, and pending items
  - **Question seam**: may include 1-2 questions from recent self-reflection — the most
    useful thing Genesis is currently uncertain about. Not a questionnaire, not a backlog.
    If unanswered, the outreach attempt expires but the question persists in memory as an
    open observation (see §Open Questions in design doc).
  - Delivered via the outreach pipeline (same governance, same channel selection)
  - Engagement-tracked — system learns what users find useful in morning reports
  - Model: Light depth (20-30B / Gemini free) — daily surplus task, not premium
  - V3: static prompt template. V4: meta-prompted for adaptive content selection.
  - See §Daily Morning Report in design doc

### Approval Dashboard UI

The task queue, surplus staging area, and outreach approval workflow need a
user-facing UI. Rather than building a separate "Mission Control" application,
Genesis builds this as an **AZ dashboard panel** — same Alpine.js, same Flask
API pattern, same auth as the existing memory dashboard.

The dashboard shows:
- **Pending proposals** — surplus insights, outreach drafts, task suggestions
  awaiting user approval
- **In-progress tasks** — background work Genesis is currently executing
- **Completed work** — recent results with outcomes and cost
- **Approval actions** — approve, reject, defer, modify scope

### Identity & Prompt File Audit Page
<!-- Added 2026-03-08: GL-1 post-implementation discussion -->

All CAPS markdown files in `src/genesis/identity/` that shape LLM behavior
(SOUL.md, USER.md, REFLECTION_DEEP.md, REFLECTION_STRATEGIC.md, future
perception templates) should be visible and editable from the dashboard.

**Why:** These files define how Genesis thinks. Hiding them in the codebase
breeds opacity — the user should be able to audit and modify anything that
isn't genuinely hardcoded. This is the difference between a system the user
trusts and a black box. If a behavior feels wrong, the user should be able
to trace it to a specific file and change it.

**Implementation:** A dashboard panel that discovers all `.md` files in the
identity directory, displays their content with syntax highlighting, and
allows inline editing with save. Changes are written to disk immediately
(no git commit required — the user can manage version control separately).

**Convention:** CAPS filenames (e.g., `SOUL.md`, not `soul.md`) signal
"user-editable, not internal implementation." Only constants that MUST be
hardcoded (schema definitions, protocol values, safety constraints) live in
Python code.

This is a V3 Phase 8 UI deliverable. The queue infrastructure is Phase 3
(surplus). The approval logic is Phase 9 (autonomy). The dashboard ties them
together for the user.

See `docs/reference/agent-zero-architecture-deep-dive.md` → Memory System →
Memory Dashboard for the pattern Genesis follows (Alpine.js + single Flask
endpoint + action-based API).

### CC Chat Widget
<!-- Added 2026-03-10: GL-3 implementation — web UI convergence strategy -->

The Genesis dashboard includes a CC-backed chat widget — the web UI equivalent
of the Telegram relay. Calls `ConversationLoop.handle_message()` via a new
Flask API endpoint (`genesis_chat`). Same session persistence, morning reset,
and intent parsing as Telegram.

**Why not rewire AZ's existing chat:** AZ's web UI chat is deeply coupled to
`AgentContext` — chat history, context management, tool call visualization,
sub-agent rendering. Replacing that backend with CC would mean reimplementing
most of the UI plumbing. Building a separate Genesis chat widget is cleaner,
has zero rebase liability, and naturally grows as phases land.

**Convergence:** AZ's chat tab becomes a debug/legacy tool as CC handles all
intelligent conversation. The Genesis dashboard is the primary user-facing UI.

### AZ UI Architecture Findings (Discovered 2026-03-04)

Deep-dive into AZ's web UI reveals the implementation approach for this dashboard:

**Approach: Option B + C** — Standalone Genesis dashboard modal (visibility) PLUS
Genesis config via MCP tools and chat (seamless on-the-fly changes). The dashboard
gives the user a place to see everything; chat-based config lets them change things
without leaving the conversation.

**Why NOT a new settings tab:**
- AZ's 6 settings tabs are **hardcoded** in `settings.html` — no plugin extensibility
- Settings fields are hardcoded in individual HTML templates
- Backend `Settings` TypedDict is a fixed schema in `settings.py`
- Adding a tab requires patching AZ core files (rebase liability)

**What IS extensible (use these):**
- **Modal system** (`window.openModal(path)`) — load any component as a modal
- **Component loader** (`<x-component path="...">`) — lazy-loads HTML + JS
- **Alpine.js stores** (`createStore("name", model)`) — reactive state management
- **API handlers** — add new Flask endpoints for Genesis data
- **Notification system** — already bridged via Genesis observability

**Implementation pattern:**
1. Add a Genesis button/icon to the sidebar or as a floating action button
   (minimal core touch — one line in sidebar HTML, or injected via extension)
2. Button opens Genesis dashboard modal via `window.openModal()`
3. Dashboard loads Genesis-specific Alpine.js components
4. Backend: new API handlers (`genesis_status`, `genesis_config_get`,
   `genesis_config_set`) following AZ's `ApiHandler` pattern
5. Config stored in Genesis DB tables, NOT in AZ's `settings.json`
6. MCP tools expose the same config surface for chat-based changes

**Technical requirements:**
- All API calls need CSRF token + Origin header (`http://localhost:5000`)
- Session cookie is scoped per runtime ID (`session_{runtime_id}`)
- Use `API.callJsonApi("endpoint_name", params)` from frontend JS
- Follow AZ's `ApiHandler` base class with `@requires_auth()` decorator

**Dashboard sections (refined):**
- **System Status** — health probes (DB, Qdrant, Ollama, scheduler)
- **Awareness Loop** — current depth, last tick, signal readings
- **Routing** — calls today, budget remaining, circuit breaker states
- **Surplus** — queue depth, LM Studio availability, brainstorm schedule
- **Memory** — Qdrant collection stats, entry counts
- **Pending Approvals** — proposals awaiting user action
- **Quick toggles** — enable/disable awareness, surplus, outreach

### Outreach Timing: Why Day 1

The 1/day surplus outreach starts the moment V3's autonomous behavior goes live — not after
a burn-in period. Rationale:
- V4 calibration needs engagement data. No outreach = no data = can't calibrate.
- V5 anticipatory intelligence needs engagement history. No early data = flying blind.
- The governance gate + engagement tracking ensure even early outreach is tracked and learned from.
- User can always disable if too noisy. Starting conservative but present > starting silent.
- 1/day is conservative enough that even mediocre early outreach won't overwhelm.

### Bayesian Calibration Infrastructure (AGI Gap Items 1-3)
<!-- Added 2026-03-12: AGI gap analysis — start logging in Phase 8 for data accumulation -->

Start prediction logging early so calibration data accumulates before Phase 9
needs it. This is pure instrumentation — zero LLM cost.

- **Prediction logging table** (`predictions`):
  - Schema: `{id, action_id, timestamp, prediction, confidence, confidence_bucket,
    domain, reasoning, outcome (nullable), correct (nullable), matched_at (nullable)}`
  - Every non-trivial decision logs a structured prediction with confidence
  - `confidence_bucket` is the binned value (e.g., "0.7-0.8") for calibration curves
  - `domain` enables per-domain calibration (outreach, triage, procedure, routing)
- **Prediction-outcome reconciliation** — batch job (surplus compute) that:
  - Finds predictions with no matched outcome
  - Matches against execution trace results, user engagement signals, procedure
    success/failure
  - Marks predictions as correct/incorrect/uncertain
  - Runs daily alongside morning report infrastructure
- **Calibration curve computation** — pure programmatic job:
  - Groups predictions by `confidence_bucket × domain`
  - Computes `actual_success_rate / predicted_confidence` per bucket
  - Produces calibration correction function per domain
  - Stores curves in `calibration_curves` table for Phase 9 context injection
- **Justification:** Google research on "Bayesian Teaching" (2026) confirmed LLMs
  hit a "one-and-done plateau" — they fail to update beliefs from sequential
  evidence. This calibration system is the architectural compensation: a symbolic
  system that tracks belief accuracy, paired with LLM context injection (Phase 9).
  See research doc §26.

### Verification

- [x] Governance gate blocks out-of-bounds outreach
- [x] Exactly 1/day surplus outreach (not 0, not 2+)
- [x] Surplus outreach clearly labeled
- [x] Engagement tracking attributes responses to triggering outreach
- [x] Alert/blocker bypasses normal pipeline, delivers immediately
- [x] Morning report fires daily at configured time
- [x] Morning report reads cognitive state summary for context
- [x] Prediction logging table exists and logs predictions on governance decisions
- [x] Reconciliation batch job matches predictions to outcomes
- [x] Calibration curves computed per domain with sufficient data (50+ predictions/bucket) — `calibration/curves.py` per-domain bucketed computation

---

## V3 Phase 9: Basic Autonomy ✅ COMPLETE

**Risk: HIGH** — Permission errors = trust damage. Dependencies: Phase 6, Phase 8.

### What to Build

- **Autonomy hierarchy L1-L4** with fixed defaults:
  - L1: Simple tool use → fully autonomous
  - L2: Known-pattern tasks → mostly autonomous
  - L3: Novel tasks → propose + execute with checkpoint
  - L4: Proactive outreach → threshold-gated + governance check
  - **L5-L7 deferred to V5** (need operational evidence to grant)
- **Context-dependent trust ceiling**:
  - Direct session: earned level (no cap)
  - Background cognitive: L3 max
  - Sub-agent: L2 for irreversible, earned for reversible
  - Outreach: L2 until engagement data proves calibration
- **Autonomy regression** (rule-based, works immediately):
  - 2 consecutive corrections → drop one level, re-earn
  - 1 user-reported harmful action → drop to default, full re-earn
  - Regression is announced, not silent
- **CLAUDE.md per-task isolation**: sub-agents do NOT share a CLAUDE.md write path
- **Hard verification gate** (Symphony Proof of Work pattern):
  <!-- Added 2026-03-09: research session — architectural enforcement -->
  - Task completion MUST include: all tests pass + lint clean (`ruff check .`) +
    diff review + structured explanation of changes + before/after state comparison.
  - This is architectural enforcement, not a skill suggestion. The CCInvoker
    post-session hook validates these artifacts before marking a task complete.
  - If verification fails, the task is NOT marked complete — it stays in-progress
    with the failure reason attached.
  - See `2026-03-08-research-insights-and-followups.md` Section 6 for full pattern.
- **Bugbot self-review** (auto-review on push to own repo):
  <!-- Added 2026-03-09: research session — Cursor Automations pattern -->
  - Genesis monitors pushes to its own repository (via webhook or git hook).
  - On each push: runs lint + tests + semantic review of the diff.
  - Flags regressions, style violations, and potential issues as observations.
  - This is Genesis dogfooding its own autonomous quality control.
  - See `2026-03-08-research-insights-and-followups.md` Section 10.
- **Autonomous obstacle resolution escalation**:
  <!-- Added 2026-03-09: research session — when to try vs when to escalate -->
  - Phase 6 builds static fallback chains and tool capability discovery.
  - Phase 9 adds the autonomy judgment: how many alternatives to try before
    escalating to the user, when to spend compute on creative workarounds vs
    accept a limitation, and how to surface the obstacle in outreach.
  - Uses autonomy level to gate escalation behavior: L1-L2 escalate quickly,
    L3-L4 exhaust more alternatives before involving the user.
- **No autonomy progression in V3** — levels are fixed by user. V4 adds evidence-based
  progression.

### Calibration-Informed Autonomy (AGI Gap Items 4-6)
<!-- Added 2026-03-12: AGI gap analysis — use Phase 8 calibration data for autonomy decisions -->

Phase 8 accumulates prediction data. Phase 9 uses it to inform autonomy decisions.

- **Calibration feedback injection** — context assembler includes calibration
  history in prompts: "When you report 80% confidence on outreach decisions,
  you're historically right ~60% of the time. Adjust accordingly." Per-domain
  curves from Phase 8's `calibration_curves` table. This is the "symbolic +
  neural" pairing validated by Google's Bayesian Teaching research.
- **Hard disagreement gates** — when cross-vendor review (call sites #17/#20)
  disagrees with primary model, the action BLOCKS until resolved:
  - Third model tiebreaker, OR
  - Escalation to user with both assessments
  - Promotes "disagreement-as-signal" from deferred pattern (design doc line
    2640) to core infrastructure
  - Disagreement rates tracked — if >30% disagreement, the system has calibration
    problems beyond what gates can fix
- **Decision trace verification** — for structured decisions (routing, triage,
  procedure selection), mechanically verify stated reasons match actual data:
  - "I chose depth 2 because token count > 500" → verify token count was > 500
  - "I routed to Sonnet because task is complex" → verify complexity score
  - No LLM needed — pure programmatic validation
  - Failed verifications logged as `calibration.trace_mismatch` observations

### Approval Timeout & Auto-Reject Policy
<!-- Added 2026-03-12: Pattern from Perplexity Personal Computer governance model -->

When Genesis proposes an action requiring user approval (L4 outreach, scope
changes, irreversible operations), the proposal cannot hang indefinitely:

- **Timeout window** — configurable per-action-type (default: 60 minutes for
  outreach, 24 hours for task proposals, no timeout for irreversible operations)
- **Auto-reject on timeout** — expired proposals are rejected, not silently
  executed. Genesis logs the timeout and may re-propose later if conditions
  still warrant it.
- **No auto-approve** — timeout never means "go ahead." Silence is absence of
  consent, not consent.
- **Kill switch** — user can cancel any pending or in-progress autonomous action
  via dashboard or chat command. Cancellation is immediate and logged.

This prevents the failure mode where Genesis queues up proposals during user
absence and executes a backlog when the user returns, or worse, assumes silence
means approval.

### PAI-Informed Additions (from Miessler deep dive, 2026-03-12)
<!-- Added 2026-03-12: Items from PAI 8-dimension analysis -->
<!-- Updated 2026-03-13: Scoping review — moved sentiment, work detection, 2 hooks to V4 -->

- **Mandatory task structure floor** — ANY task entering the task execution
  pipeline (background sessions, surplus tasks, dispatched CC work) MUST include:
  1. **Planning**: Define what you're doing and how.
  2. **Verification**: Confirm output matches plan.
  3. **Learning**: Extract what worked and what didn't.
  The LLM decides HOW MUCH of each, not WHETHER they happen. Foreground
  conversation is exempt (handled by Self-Learning Loop retrospective).
  Rationale: PAI's The Algorithm mandates 7 phases on every task. Our
  "proportionate effort" philosophy is correct for conversation but was letting
  the LLM cut corners on serious work. This is the minimum structural floor.

- **CC lifecycle hooks (3 of 5)** — Implement CC hooks for Phase 9:
  SessionStart (identity + steering + memory load), Stop (learning trigger +
  state persistence), SessionEnd (retrospective + cleanup). These are the
  guaranteed lifecycle injection points, not fallback. The remaining 2 hooks
  (PreToolUse for security invariants, UserPromptSubmit for work detection)
  are deferred to V4 — they require a security model and work detection
  pipeline not yet defined.

- **CC session continuity (Context Mode pattern)** — Adopt the session
  continuity hook architecture from Context Mode MCP server for Genesis's
  dispatched CC sessions. Key pattern: PostToolUse captures structured events
  (file/git/task/error) into SQLite, PreCompact builds a priority-tiered
  snapshot (≤2KB: active files, tasks, rules, errors, git ops), SessionStart
  restores state after compaction via FTS5-indexed event history + Session
  Guide. This extends CC session lifetime for long-running autonomous work
  (reflection, surplus, task execution). Complements the 3 lifecycle hooks
  above — those handle Genesis identity/learning injection; this handles
  working state preservation across compaction boundaries.
  Source: Context Mode MCP server (github.com/mksglu/context-mode),
  research 2026-03-14.

- **Work item detector** — **V4 scope** (moved from Phase 9 — this is an
  advanced autonomy feature). During or immediately after conversation, identify
  actionable items not being addressed now. Stage them in surplus queue with
  conversation context, priority, and originating exchange. Connects to:
  surplus staging → background CC execution → outreach delivery.
  Inspired by PAI's AutoWorkCreation hook.

- **Fast-path steering rules + STEERING.md** — When user gives unambiguous
  negative feedback ("never do X"), immediately create an observation tagged
  `steering_rule` that gets injected into future context. Don't wait for
  calibration cycles on clear corrections. STEERING.md is a first-class
  identity file loaded alongside SOUL.md — contains cross-cutting behavioral
  rules auto-populated from strong negative feedback.

- **Task state files** — Simple JSON/MD per task, written by Stop/SessionEnd
  hooks, read by SessionStart hook. Survives context compaction. Contains:
  task description, current phase, key decisions, blockers, outputs. Inspired
  by PAI's PRD (Persistent Requirements Document) pattern.

- **Time budget awareness** — Prompt-level injection in mandatory task
  structure: "You have N minutes. Allocate ~20% planning, ~60% execution,
  ~20% verification." LLM decides allocation; the structure enforces awareness.

- **`/btw` skill** — "By the way" command: store a quick note to memory without
  derailing current task. Simple skill wrapping memory store + continue signal.

### Verification

- [x] L1 executes autonomously, L4 requires governance gate — `autonomy/types.py` AutonomyLevel L1-L4 + `autonomy/classification.py`
- [x] Regression triggers on consecutive corrections — `autonomy/state_machine.py` consecutive_corrections tracking
- [x] Context ceiling caps appropriately — `autonomy/types.py` ContextCeiling + CONTEXT_CEILING_MAP
- [DEFERRED:V4] CLAUDE.md writes isolated per task — CC sessions use system_prompt injection, not per-task CLAUDE.md file generation; V4 will add task-scoped workspace isolation
- [x] Task pipeline enforces planning/verification/learning minimum — `autonomy/verification.py` TaskVerifier
- [x] CC hooks (SessionStart, Stop, SessionEnd) fire reliably — `cc/session_manager.py` + `cc/invoker.py`
- [x] Steering rules from strong negative feedback appear in next session context — `learning/pipeline.py` + `identity/loader.py` STEERING.md loading
- [x] Task state files written by Stop/SessionEnd, loaded by SessionStart — `cc/invoker.py` session state management
- [DEFERRED:V4] Time budget injected into task structure prompts — no time_budget field in session config; V4 resource-aware scheduling feature
- [x] STEERING.md loaded alongside SOUL.md in identity assembly — `identity/loader.py` + `identity/STEERING.md`
- [x] Hard verification gate: tasks without passing tests + lint are NOT marked complete — `autonomy/verification.py` TaskVerifier
- [DEFERRED:V4] Structured change summary produced for every autonomous task completion — CC session outputs captured but no structured change summary format; V4 reporting feature
- [DEFERRED:V4] Bugbot self-review triggers on push and flags regressions — no bugbot implementation; V4 CI integration feature
- [x] Obstacle escalation respects autonomy level (L1-L2 escalate faster than L3-L4) — `autonomy/escalation.py` should_escalate with autonomy_level
- [x] Autonomous actions log reasoning chain before execution (intent validation) — `autonomy/types.py` reasoning field in action types
- [x] Reasoning log is verifiable against policy/autonomy level — `autonomy/trace_verification.py` DecisionTraceVerifier
- [x] Calibration feedback injected into context assembly for autonomy decisions — `perception/context.py` calibration data loading
- [x] Hard disagreement gates block actions when cross-vendor review disagrees — `autonomy/disagreement.py` DisagreementGate; V3 stub returns no-disagreement, V4 wires cross-vendor review
- [x] Decision trace verification catches stated-reason vs actual-data mismatches — `autonomy/trace_verification.py`
- [x] Disagreement rates tracked and anomalous rates flagged — `autonomy/disagreement.py` disagreement_rate + is_anomalous
- [x] Approval timeout fires and auto-rejects expired proposals — `autonomy/classification.py` approval_timeouts config
- [x] Verified — not a bug: GenesisEvent signature compatibility between DisagreementGate/StateMachine — EventBus.emit() uses **details kwargs which handles both callers correctly
- [DEFERRED:V4] Kill switch cancels in-progress autonomous actions immediately — no kill switch implementation; V4 operational safety feature requiring process management
- [x] No auto-approve path exists — timeout always means reject — `autonomy/classification.py` timeout = reject by design

---

### CC Update Compatibility (Cross-Cutting)

<!-- Added 2026-03-09: research session — CC 2.0/2.1 as Genesis dependency -->
Claude Code is as critical a dependency as Agent Zero. CC updates (features, breaking
changes, deprecations) directly affect CCInvoker, reflection_bridge, session_manager,
and all CC background sessions.

**Tracking document:** `docs/reference/cc-compatibility.md` (to be created)
- Maps CC features → Genesis components that depend on them
- Maps CC version requirements → Genesis minimum compatible version
- Tracks CC capabilities Genesis is NOT yet using → evaluation queue
- Tracks CC deprecations → migration plans

**Process:** When CC updates, consult compatibility doc. Evaluate:
1. Does this affect our CCInvoker/reflection_bridge/session_manager?
2. Does it unlock something we're working around?
3. Does it obsolete something we built?
4. CC 2.1 features relevant to Genesis: hooks in frontmatter (Phase 7 session_config),
   forked skill context (Phase 6 skill wiring), wildcard permissions (session setup).

---

## V3 Summary

V3 delivers a **complete working copilot** that:
- Monitors signals and classifies urgency (Awareness Loop, fixed weights)
- Routes compute intelligently with local/cloud fallback and cost observability
- Leverages free compute from day 1 (surplus infrastructure early)
- Reflects at 3 depths: Micro, Light, Deep (static prompts)
- Maintains a cognitive state summary — regenerated after Deep reflections, loaded into every fresh context
- Stores and retrieves memory with activation scoring
- Classifies outcomes and builds procedural memory (null hypothesis defaults)
- Runs 2+ daily brainstorming sessions on free compute ("upgrade user" + "upgrade self")
- Sends a daily morning report (adaptive, not a checklist)
- Sends 1 proactive surplus outreach/day + alerts/blockers
- Runs a mandatory weekly self-assessment ("am I getting better?")
- Has fixed L1-L4 autonomy with automatic regression
- Quarantines speculative claims from context

**What V3 does NOT do** (by design, not by omission):
- No meta-prompting (uses static prompts — V4)
- No Strategic reflection (no MANAGER/DIRECTOR reviews — V4)
- No calibration loops (fixed weights, fixed thresholds — V4)
- No channel learning (config-driven — V4)
- No confidence decay on procedures (V4)
- No ISC (Ideal State Criteria) for task verification (V4 — needs Phase 9 data)
- No explicit SIGNALS rating capture (V4 — user rates interactions, feeds calibration)
- No inbox tag-based routing (V4 — requires vault-wide scanning + user model)
- No proactive inbox research (V4 — Genesis-initiated research based on user's notes)
- No graph-connected outputs (V4 — wiki links to user's existing vault notes)
- No natural language sentiment extraction (V4 — needs SIGNALS consumer + calibration data)
- No proactive context compaction (V4 — needs operational experience from Phase 9 tasks)
- No work item detection from conversation (V4 — UserPromptSubmit hook + detection pipeline)
- No autonomy progression (fixed levels — V5)
- No identity evolution (static identity — V5)
- No anticipatory intelligence (V5)
- No meta-learning (V5)
- No L5-L7 autonomy (V5)

---

### Retroactive Fix: Cost-as-Observability (Phases 0-3)

The "Quality over cost" principle (see CLAUDE.md) was established during Phase 4
design. Several Phase 0-3 components have automatic cost-based behavior that
violates this principle. These need updating before or during Phase 4:

1. **Router budget gating** (`routing/router.py`): Auto-skips paid providers when
   `BudgetStatus.EXCEEDED`. Change to: emit event, let Genesis + user decide.
   The Router should still *report* budget status; it should not *act* on it.
2. **`never_pays` constraint** (`routing/router.py`): Hard-filters call site chains
   to free-only. Change to: a routing *preference* in config, overridable by
   Genesis. Surplus tasks *prefer* free compute but aren't *forbidden* from paid.
3. **`ComputeTier.NEVER` rejection** (`surplus/queue.py`): Hard rejects tasks above
   a cost tier. Remove the NEVER tier concept — tasks go where compute is available.
4. **DegradationTracker auto-skip** (`routing/degradation.py`): Hardcoded skip lists
   for call sites at L2+ degradation. Change to: emit degradation event with
   affected call sites, let Genesis decide what to deprioritize. Note: degradation
   *level computation* (observability) is fine. Automatic *action* on it is not.
5. **Design doc language** (`build-phases.md`, `autonomous-behavior-design.md`):
   Audit for cost-frequency rules, "above cost threshold = NEVER" language, and
   budget-as-control framing. Rewrite as observability + user choice.

**Availability-driven filtering is fine** — if a model is down, you can't call it.
That's infrastructure reality, not cost control. CircuitBreaker health checks stay.

### Capability Modules Framework (Post-Phase 8, parallel with Phase 9)

External domain capabilities that plug into Genesis without modifying core.
Design spec: `docs/plans/capability-modules-design.md`

**Knowledge Pipeline** (`src/genesis/pipeline/`):
- Tiered research infrastructure: collect → triage → analyze → judge
- Research profiles (YAML) define per-domain signal collection
- Pluggable collector registry (web search built-in, extensible)
- Integrates with SurplusScheduler for free-tier compute

**Module Framework** (`src/genesis/modules/`):
- `CapabilityModule` protocol — pluggable register/deregister lifecycle
- `ModuleRegistry` — manages module loading and runtime wiring
- `GeneralizationFilter` — LLM quality gate between module-local learning
  and Genesis core. Only process/methodology/calibration lessons cross.

**First modules:**
- `prediction_markets/` — MarketScanner, CalibrationEngine (superforecasting),
  PositionSizer (fractional Kelly), OutcomeTracker (Brier scores)
- `crypto_ops/` — NarrativeDetector, PositionMonitor (exit signals),
  CryptoOutcomeTracker (P&L, narrative accuracy)

**Design principle:** "Hands, not brain" — modules use Genesis cognitive services
but don't modify core identity, reflection, or learning. Module outcomes are
isolated; only generalizable lessons are promoted through the quality gate.

**Tests:** 170 tests covering pipeline, framework, and both modules.

### V3 Closing Infrastructure (Post-Phase 9)

Before transitioning to V4, spin up:

- **Automated changelog system** — Every code change (by any agent or human)
  gets a professional changelog entry. Format TBD during implementation, but
  must be: machine-parseable, human-readable, categorized (feat/fix/refactor/
  docs), and attributable (who/what made the change). This becomes V4's audit
  trail for understanding what changed and why. Design during Phase 9, activate
  before V4 begins.

- **Free compute fallback for surplus** — When CC rate limits are hit, the
  surplus system should automatically route lower-priority work through free
  API alternatives (Gemini, Mistral, Groq free tiers — already cataloged in
  `docs/reference/models.md` Free Tier Terms). Not "free CC" — rather,
  alternative compute the surplus system can use when CC is unavailable.
  Priority-based: high-priority surplus waits for CC; medium-priority routes
  to free APIs; low-priority defers to next window. The existing router +
  free tier infrastructure already supports this — implementation is wiring
  the surplus system's rate-limit detection to the router's free tier pool.
  Source: competitive landscape research 2026-03-14.

- **cognitive_state schema migration** — The `cognitive_state` table has a CHECK
  constraint limiting `section` to three values: `active_context`,
  `pending_actions`, `state_flags`. This was defensive scaffolding from Phase 0.
  The cognitive loop hierarchy (Phase 2 wiring) stores `focus_next` in
  `state_flags` and `focus_next_week` in `pending_actions` — workable but
  fragile because `replace_section()` is destructive (deletes all rows for a
  section). V4's executor needs arbitrary section keys for the living working
  document. Fix: remove the CHECK constraint (or expand it), and allow
  freeform section names. The CRUD code (`cognitive_state.py`) already handles
  arbitrary section names — only the schema enforces the limit. Migration:
  `ALTER TABLE cognitive_state DROP CONSTRAINT ...` or recreate table without
  CHECK. Low risk, zero code changes beyond schema.

- **Free model escalation validation (batch 2.5)** — Test whether groq, mistral,
  and openrouter free models reliably produce `escalate_to_deep` JSON in light
  reflection output. Run 10-20 light reflections across free providers with
  signal data that should trigger escalation. If <80% reliability, upgrade
  light reflection's primary model path to Haiku (still cheap, much more
  reliable at structured output). If ≥80%, free models are adequate. Document
  results in `docs/reference/free-model-escalation-validation.md`. Best done
  after the system is running live with Phase 2-3 wiring merged.

### Codebase Audit Hardening (2026-03-20)

Items identified by the codebase audit that are real issues but were deferred
from the audit cleanup branch due to scope, risk, or V4 dependency.

### Deferred: `_bootstrapped` partial init guard
- **File:** `src/genesis/runtime.py:399`
- `_bootstrapped` is set `True` unconditionally even if subsystems failed.
  Components checking `is_bootstrapped` get True when outreach/autonomy/etc
  may be None.
- **Why deferred:** 10% breakage risk — FindingsBridge, health MCP, and
  shutdown all check this flag. Needs caller audit before changing.
- **Correct fix:** Set True only if no subsystem has "failed" in manifest.
  Audit all `is_bootstrapped` callers for graceful degradation.

### Deferred to V4: Hardcoded `/home/ubuntu` paths
- **Files:** `.mcp.json`, `.claude/settings.json`, `config/genesis-*.service`,
  and 10+ modules using `Path(__file__).parent` chains.
- All config paths, MCP commands, systemd services, and hook scripts assume
  `/home/ubuntu`. Migration to another machine = total system failure.
- **Why V4:** Genesis runs on exactly one machine in V3. Proper fix (env vars,
  `GENESIS_ROOT` discovery, templated configs) is migration prep work.

### Deferred to V4: Injection detection enforcement
- **File:** `src/genesis/security/patterns.py`
- Web content matching injection patterns is log-only — never blocks delivery
  to the LLM. Content safety hook emits advisory but never exit code 2.
- **Why V4:** False positive risk is real. Blocking requires a review/appeal
  workflow and confidence-based filtering not yet designed.

### Resolved: Items fixed in audit cleanup (2026-03-20)
For reference, the following audit findings were fixed in the
`fix/audit-cleanup` merge:
1. `str(Exception)` class bug in job health tracking (3 locations + 3 new methods)
2. `rate_limits.py` config path depth (3→4 parents)
3. OutcomeClassifier SUCCESS→UNKNOWN on LLM failure
4. DisagreementGate/StateMachine emit signature mismatch (sync→async bridge)
5. Approval poller task handle lifecycle (store + cancel in shutdown)
6. ALTER TABLE suppress blocks narrowed to "duplicate column" only (15 blocks)
7. Path traversal in ProtectedPaths `_normalize()` (added normpath)
8. Stuck surplus task recovery (`recover_stuck()` added)
9. Content safety hook silent parse failure (added stderr warning)
10. Telegram error reply suppression → logging (6 critical blocks)
11. Event bus shutdown suppress → logging
12. Event DB writer done callback (compensating for circular import)

---

# V4: The Self-Tuning Copilot

**Prerequisite:** ~1-2 months of V3 operational data.

## V4 Phase 0: Survivable Infrastructure (Guardian)

**Status:** Implemented (2026-03-26) | **Branch:** `feature/guardian`

The Guardian is V4's first priority — foundational safety that all other V4
features benefit from. A host VM health monitor running outside the container's
blast radius. When Genesis dies, the Guardian detects it, diagnoses root cause
via CC, recovers (with user approval), and alerts via Telegram.

**Form factor:** systemd timer (fires every 30s). Script runs <5s when healthy.
On failure detection, extends for the full confirm->diagnose->recover flow.

**Key components:**
- 5 health probes + 6 suspicious checks (host-side, stdlib-only)
- 10-state confirmation state machine with persistence
- Self-heal-first dialogue protocol (Genesis fixes itself when possible)
- CC-powered diagnosis with prime directive: "First, do no harm"
- Without CC: alert only, zero recovery actions (any signal can lie)
- Channel-agnostic alerting (Telegram first) with user approval links
- Snapshot manager + recovery engine (6 escalation levels)

**Deployment:** `scripts/install_guardian.sh` on host VM. See
`docs/architecture/genesis-v3-survivable-architecture.md` for full design.

## V4 Feature-Flag Activation

V4 is NOT sequential phases — it's a **feature-flag activation model**. Features enable
independently as V3 data accumulates, because different features need different data volumes.

### V4 Activation Model

Each V4 feature has:
1. **Data prerequisite check** (programmatic: query tables, count rows, compare minimums)
2. **Config flag** (even with sufficient data, user must explicitly enable)
3. **Optional shadow mode**: feature runs but outputs go to staging/logs, not production.
   Allows quality comparison against V3 baseline before going live.

> **Knowledge autonomy reference:** `genesis-knowledge-autonomy.md` — V4 is where
> proactive knowledge gap detection, interest pattern recognition, and channel-aware
> ingestion learning activate. These build on the Phase 8 ingestion infrastructure.

### V4 Features

| Feature | V3 Data Source | Min Data Volume | Shadow Mode? |
|---------|---------------|-----------------|-------------|
| **Meta-prompting protocol** | Simple deep reflection outputs | 20+ deep reflections | Yes — compare against static prompt baseline |
| **Strategic reflection** (MANAGER/DIRECTOR) | Weeks of full-stack data | 4+ weeks operation | Yes — log proposals without acting |
| **Signal weight adaptation** | Per-reflection utility tracking | 100+ reflection events | Yes — compute adapted weights, compare |
| **Drive weight adaptation** | Engagement + outcome data | 50+ outcomes, 30+ engagement | Yes — compute adapted weights, compare |
| **Salience threshold self-adjustment** | Outreach engagement per topic | 20+ engagement per category | Yes — bounded ±20% |
| **Finding/Insight/Opportunity outreach** | User model + calibrated engagement | 30+ user model evidence | No — live by nature |
| **Outreach frequency growth ramp** | Surplus engagement data | 20+ outreach w/ >40% engagement | No — live by nature |
| **Channel learning** | Per-channel engagement data | 10+ outreach per channel per type | No |
| **Speculative hypothesis confirmation** | Observation corpus | 10+ quarantined hypotheses | No |
| **Procedural memory confidence decay** | Procedure corpus with age variance | 50+ procedures w/ varying age | Yes — track what would decay |
| **Fresh-eyes review (non-outreach)** | High-stakes output corpus | 10+ config/identity proposals | Yes — review without blocking |
| **Prompt variation for reflection** | Evidence of mode collapse | 50+ same-depth reflections | No |
| **ISC (Ideal State Criteria)** | Phase 9 hard verification gate + prediction logs | 20+ autonomous task completions | Yes — compare ISC-gated vs ungated task quality |
| **Explicit SIGNALS rating capture** | Outreach engagement tracking | 30+ outreach interactions | No — live by nature. User rates interactions 1-10, low ratings auto-generate steering rules. Feeds calibration system. |
| **Natural language sentiment extraction** | Triage pass data + user engagement | 50+ triage passes | Yes — classify implicit 1-5 sentiment, compare to explicit SIGNALS ratings. Deferred from Phase 9: no consumer without SIGNALS. |
| **PreToolUse hook (security invariants)** | Phase 9 hook foundation + security model | Security model defined | No — requires security model not yet designed |
| **UserPromptSubmit hook (work detection)** | Phase 9 hook foundation + conversation data | 20+ conversations | No — work item detection pipeline is V4 scope |
| **Proactive context compaction** | Phase 9 task execution logs | 50+ long-running tasks | Yes — trigger compaction at phase boundaries when >60% context consumed |
| **Theater check (capability verification)** | Tool invocation logs | 30+ autonomous tasks | Yes — verify capabilities claimed in planning are actually invoked |
| **ISC-annotated task state files** | Phase 9 task state files | 20+ completed tasks with state files | Yes — add ISC binary criteria to task state, compare gated vs ungated |
| **Context recovery protocol** | Compaction + state file maturity | Stable compaction + state files | No — needs both compaction and state file maturity |
| **OPINIONS.md** | Reflection + learning corpus | 50+ deep reflections | No — graduated opinions from observation patterns |
| **Procedure graduation system** | Procedure corpus | 50+ procedures with varying confidence | Yes — track confidence decay, compare graduated vs ungradated |
| **Expanded hook coverage** | Phase 9 3-hook foundation | Stable hook infrastructure | No — adds PreToolUse, UserPromptSubmit to existing 3 hooks |
| **`/fork` and `/rewind` context management** | Session management logs | Stable session infrastructure | No — branch/restore conversation contexts |

### V4 Research-Driven Features (2026-03-14)

From competitive landscape research. Full specs in `docs/plans/v4-research-driven-features-spec.md`.

| Feature | Inspiration | Dependency |
|---------|-------------|------------|
| **Hot-reload tool discovery** | AWS Strands SDK directory-based tools | MCP server modification |
| **AI Functions / runtime capability expansion** | AWS Strands Labs AI Functions | Hot-reload tools + Phase 6 maturity |
| **Agent-to-Agent protocol (A2A)** | AWS Strands, Google ADK | A2A standard stabilization |
| **API-to-MCP gateway** | AWS AgentCore Gateway | Hot-reload tools + OpenAPI parser |
| **Tool Search API (deferred loading)** | Anthropic Tool Search Tool | Hot-reload tools + API-routed sessions |
| **Context-efficient CC sessions (sandbox)** | Context Mode MCP server | Phase 9 CC hooks + session continuity |

### V4 Daily Brainstorming Upgrade

V3's daily brainstorming sessions ("upgrade user" + "upgrade self") upgrade from static Light
prompts to **meta-prompted sessions**:
1. Cheap model (Gemini free) generates brainstorming questions based on recent data, user
   model, system performance
2. Capable model (Sonnet) explores the best questions with depth
3. Synthesis: actionable proposals → staging area

### V4 Morning Report Upgrade

V3's static-prompt morning report upgrades to **meta-prompted adaptive content selection**:
- Cheap model asks: "What does the user most need to hear this morning?" based on recent
  journal, user model, engagement patterns on previous morning reports
- Capable model generates the report with adaptive section selection
- Morning report engagement data feeds back into content selection model

### V4 Self-Assessment Integration

V3's standalone weekly self-assessment becomes an input to Strategic reflection (MANAGER role):
- MANAGER cross-references the self-assessment against system metrics
- Can propose parameter adjustments (drive weights, salience thresholds) based on assessment findings
- Self-assessment trends (improving/declining/stable) inform MANAGER's strategic recommendations

With sufficient free compute, these can expand into **ongoing background dialogues** — multiple
brainstorm rounds per day, each building on previous outputs. The cost-frequency rule still
applies: free=always, cheap=often, above threshold=never.

### V4 Summary

V4 makes V3 **measurably better** using operational evidence:
- Reflections improve through meta-prompting and prompt variation
- Parameters self-tune through calibration loops (bounded, auditable)
- Outreach categories expand (findings, insights, opportunities)
- Procedures decay without reinforcement (prevents stale knowledge)
- Brainstorming sessions become genuinely creative (meta-prompted, multi-round)

---

# V5: The Autonomous Copilot

**Prerequisite:** ~3-6 months of V4 operational data.

V5 features are **qualitatively different** from V4's parameter tuning. V4 makes the system
better at what it already does. V5 makes the system **capable of things it couldn't do before**:
proposing its own identity changes, anticipating needs, earning genuine autonomy, learning
how to learn.

> **Knowledge autonomy reference:** `genesis-knowledge-autonomy.md` — V5's
> anticipatory intelligence includes anticipatory knowledge acquisition: Genesis
> proactively deepens its knowledge in areas the user is trending toward,
> without being asked.

### V5 Features

| Feature | V4 Data Required | Min Data Volume | Why V5 |
|---------|-----------------|-----------------|--------|
| **L5-L7 autonomy levels** | Per-category success evidence | 10-20+ successes without correction | Trust requires time |
| **Evidence-based autonomy progression** | Success/correction tracking | Statistical significance | Cannot shortcut |
| **Identity evolution** (SOUL.md proposals) | Operational maturity | Months of stable operation | Identity is the user's call |
| **Anticipatory intelligence** | Rich user model + calibrated drives | 3+ months user model data | Design doc: Month 3+ |
| **Social simulation** ("imagine user reaction") | Deep user model + preferences | 100+ user model evidence | Shallow model = bad simulation |
| **Meta-learning** ("learn how to learn") | Reflection utility data | 200+ observations w/ utility | Need patterns in learning quality |
| **Self-detected error → self-regression** | Error pattern tracking | Enough to distinguish systematic | Premature self-regression harms |
| **"Silence ≠ approval" check-ins** | Active autonomy tracking | Running progression system | Needs progression first |
| **Relationship rhythm learning** | Temporal engagement patterns | 3+ months engagement data | Per design doc |

### V5 Autonomy Levels (New)

- L5: System configuration → propose only, user approves
- L6: Learning system modification → propose only, always user review
- L7: Identity evolution → draft only, user decides

Evidence-based progression: N successful executions without correction per category, explicit
user acknowledgment required (silence ≠ approval). Periodic check-in: "I've been handling X
autonomously with Y% success rate. Continue?"

### V5 Outreach: Anticipatory

The hardest capability. Requires rich user model + calibrated drives + procedural memory. Cross-
references new information against user model to identify things the user doesn't know they need.
Includes social simulation ("will the user find this useful?") backed by months of engagement data.

### V5 Summary

V5 is the **"cognitive partner" level** — the system that proposes changes to its own identity,
anticipates needs before the user knows them, earns and loses autonomy based on evidence, and
learns how to learn. This is where the vision document's progression from "aide" to "trusted
advisor" to "cognitive extension" becomes real.

---

# Knowledge Base: Parallel Track

Architecturally independent of the cognitive layer. Does NOT depend on V4/V5 data.

| KB Phase | Can Start After | Dependency |
|----------|----------------|------------|
| KB-1: Storage + manual ingestion | V3 Phase 5 (memory-mcp operational) | Multi-collection Qdrant, source_type |
| KB-2: Distillation pipeline | KB-1 | Needs storage working |
| KB-3: Transcription (audio/video) | KB-1 | Needs storage working |
| KB-4: Acquisition Agent (browser) | KB-2 | Needs distillation pipeline |

Full design: `post-v3-knowledge-pipeline.md` in project docs directory.

KB becomes more valuable once V4's meta-prompting can cross-reference KB content during
reflections, but KB-1/KB-2 are independently useful for task sub-agents.

---

# Cross-Cutting Concerns (All Versions)

### Testing Strategy

- **Per-phase verification**: Each phase has verification criteria (listed above)
- **Integration tests**: Does Phase N correctly use Phase N-1's output?
- **Regression tests**: Does adding Phase N break any Phase N-k behavior?
- **V4 shadow mode**: Compare feature output quality against V3 baseline before activation

### Observability

Every phase produces structured logs: what happened, why (triggering signal/score), which model,
outcome (success/failure/deferred).

### Rollback Plan

Each phase and each V4 feature is independently disableable via config. If Deep reflection
produces garbage, disable it and fall back to micro/light. If a V4 feature degrades quality,
disable its flag and revert to V3 behavior.

### Daily Brainstorm Sessions (Cross-Version)

| Version | Session Type | Prompt Style | Model | Frequency |
|---------|-------------|-------------|-------|-----------|
| V3 | "Upgrade user" + "Upgrade self" | Static Light template | 20-30B / Gemini free | 2/day minimum |
| V4 | Same + expanded topics | Meta-prompted (3-step) | Multi-model | 2/day minimum, more if free compute available |
| V5 | Same + anticipatory | Meta-prompted + user model cross-ref | Multi-model | Autonomous frequency |

These sessions are the LAST surplus tasks to skip when compute is constrained. They represent
the system's commitment to continuous improvement — even 1 simple brainstorm/day compounds.

---

# Design Doc Cross-Reference

| Build Phase | Master Design Doc Section |
|-------------|--------------------------|
| V3 Phase 0 | §4 MCP Servers, §Execution Trace Schema, §Procedural Memory Design |
| V3 Phase 1 | §Layer 1: Awareness Loop, §Signal-Weighted Trigger System, §Three Categories of Scheduled Work |
| V3 Phase 2 | §LLM Weakness Compensation → Pattern 1: Compute Hierarchy |
| V3 Phase 3 | §Cognitive Surplus |
| V3 Phase 4 | §Layer 2: Reflection Engine → Depth Levels (Micro, Light), §Pre-Execution Assessment, §Context Allocation by Model Tier, §Cognitive State Summary |
| V3 Phase 5 | §Memory Separation, §What We Learned (A-MEM, ACT-R gaps), §Open Questions and Persistent Curiosity |
| V3 Phase 6 | §Layer 3: Self-Learning Loop, §Procedural Memory, §LLM Weakness → Pattern 6, §Signal Weight Tiers |
| V3 Phase 7 | §Reflection Engine (Deep), §Cognitive State Summary (regeneration), §Weekly Self-Assessment, §Quality Calibration Cycle, current Dream Cycle jobs |
| V3 Phase 8 | §Proactive Outreach, §Daily Morning Report, §Bootstrap / Cold Start Strategy |
| V3 Phase 9 | §Self-Evolving Learning: The Autonomy Hierarchy (L1-L4 only) |
| V4 | §LLM Weakness → Patterns 2-5, §Loop Taxonomy → Tier 3 |
| V5 | §Autonomy Hierarchy (L5-L7), §Loop Taxonomy → Tier 4 |
| KB | `post-v3-knowledge-pipeline.md` |

---

## Pre-V4 Follow-Up Items (V3 complete, not yet wired)

These components were built as Phase 9 infrastructure but intentionally
deferred from V3's "Basic Autonomy" scope. They need wiring before V4.

### Task Autonomy System (TAS)
- **ApprovalManager** — `request_approval()` has zero call sites
- **TaskVerifier** — initialized with code validator but never called
- **AutonomyManager** — initialized but never accessed externally
- Wire into: surplus executor, outreach pipeline, task completion paths

### True E2E Cognitive Loop Test
- Current `tests/integration/test_cognitive_loop_e2e.py` covers segment only
- Need full: Awareness → Perception → Memory → Learning → Reflection → Outreach

### GROUNDWORK Tag Registry (as of 2026-03-22)

| Tag | File | Target Version |
|-----|------|---------------|
| `v4-executor` | `cc/reflection_bridge.py` | V4 |
| `cross-vendor-review` | `autonomy/disagreement.py` | V4 |
| `outreach-voice` | `outreach/pipeline.py` | V4 |
| `observation-feedback-loop` | `db/crud/observations.py` | V4 |
| `multi-person` | `db/schema/_tables.py` (8 tables) | V4+ |
| `v4-parallel-dispatch` | `surplus/scheduler.py` | V4 |
| `v4-surplus-tasks` | `surplus/types.py` | V4 |
| `v4-rate-tracking` | `surplus/compute_availability.py` | V4 |
| `skill-autonomy-graduation` | `learning/skills/applicator.py` | V4 |
| `user-model-synthesis` | `perception/writer.py` | V4 |
| `mcp-config` | `cc/session_config.py` | V4 |
| `hook-inheritance` | `cc/session_config.py` | V4 |
| `category-2-rhythms` | `learning/` | V4 |
| `category-3-crons` | `learning/` | V4 |
| `outreach-alerts` | `recon/cc_update_analyzer.py` | V4 |
| `outreach-pipeline` | `outreach/pipeline.py` | V4 |
| `provider-migration` | `learning/tool_discovery.py` | V4 |
| `unified-bridge` | `channels/` | V4 |
| `V4` | various | V4 |
| `pre-execution-gate` | `autonomy/` | V4 |
