# Context Retention Hardening (Extended Hourlies + Conversation Semantic Recall) Implementation Plan

> Status note (2026-03-08): Active roadmap order, blockers, and contributor priority live in the [Engram Feature Roadmap](https://github.com/users/joshuaswarren/projects/1). This file is historical design context and may describe scope that is only partially shipped.

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Make OpenClaw Engram retain context with near-zero knowledge loss by (1) richer hourly summaries and (2) optional semantic search over past conversation chunks.

**Architecture:** Extend Engram’s existing transcript + hourly summarizer pipeline with an “extended summary” mode, and add an optional conversation-chunk index (FAISS/QMD) that is searched on every prompt to inject relevant past context. Everything must be fail-open and privacy-safe.

**Tech Stack:** TypeScript (OpenClaw Engram plugin), existing transcript store (JSONL), existing memory store (Markdown + YAML), optional Python sidecar (FAISS + sentence-transformers) if `conversationIndexBackend=faiss`.

---

## Decisions (Locked)
- Cron integration must never silently create a broken cron job (`main/toolCall`) in installs that reject it.
- New features ship default-off (privacy + perf).
- All new recall paths must be timeboxed + fail-open (never block message handling).

---

### Task 1: Cron Integration Guard (No Auto-Rewrite By Default)

**Files:**
- Modify: `src/types.ts`
- Modify: `src/index.ts`
- Modify: `README.md`
- Test: `tests/hourly-cron-guard.test.ts`

**Step 1: Write failing test**
- Default config must not write to `~/.openclaw/cron/jobs.json`.
- Enabling `hourlySummaryCronAutoRegister=true` must build a job with:
- `sessionTarget: "isolated"`
- `payload.kind: "agentTurn"`
- `delivery.mode: "none"`

**Step 2: Implement config + guard**
- Add `hourlySummaryCronAutoRegister` (default `false`).
- If false:
- log one concise info line once per process start (not per session) with recommended cron snippet.
- If true:
- register an `isolated/agentTurn` job (never `main/toolCall`).

**Step 3: Docs**
- Document recommended cron configuration (copy/paste) in `README.md`.

**Step 4: Run tests**
Run: `npm test`
Expected: PASS.

---

### Task 2: Extended Hourly Summaries (Structured Output)

**Files:**
- Modify: `src/types.ts`
- Modify: `src/summarizer.ts`
- Modify: `src/transcript.ts`
- Modify: `src/tools.ts`
- Test: `tests/hourly-summaries-extended.test.ts`

**Step 1: Write failing tests**
- When `hourlySummariesExtendedEnabled=true`, summary output must include sections:
- Topics Discussed
- Decisions Made
- Action Items
- Rejected Ideas / Reversals
- Tools Used (counts) (only if enabled)
- Stats
- When disabled, behavior stays as-is.

**Step 2: Add config flags**
- `hourlySummariesExtendedEnabled` (default false)
- `hourlySummariesIncludeToolStats` (default false)
- `hourlySummariesIncludeSystemMessages` (default false)
- `hourlySummariesMaxTurnsPerRun` (default conservative)

**Step 3: Implement extended summarizer**
- Extend `HourlySummarizer` to optionally compute per-hour stats:
- user/assistant turn counts
- optional tool call counts by tool name (no args)
- optional system message count
- Produce an extended prompt and deterministic markdown formatting.
- Keep append-only hourly headings inside daily file.

**Step 4: Tool surface**
- Keep `memory_summarize_hourly` entrypoint.
- If extended enabled, write extended format; else existing format.

**Step 5: Run tests**
Run: `npm test`
Expected: PASS.

---

### Task 3: Conversation Chunk Index (QMD Backend, Optional, Default-Off)

**Files:**
- Create: `src/conversation-index/chunker.ts`
- Create: `src/conversation-index/indexer.ts`
- Create: `src/conversation-index/search.ts`
- Modify: `src/types.ts`
- Modify: `src/index.ts`
- Test: `tests/conversation-index-qmd.test.ts`

**Step 1: Write failing tests**
- Deterministic chunker: same transcript input produces same chunk boundaries.
- Indexer writes QMD docs for chunks under a known directory.
- Search returns top-K chunk docs for a query.

**Step 2: Add config flags**
- `conversationIndexEnabled` (default false)
- `conversationIndexBackend` = `qmd` (default if enabled)
- `conversationIndexRetentionDays` (default conservative)
- `conversationRecallTopK` (default small)
- `conversationRecallMaxChars` (cap injected text)
- `conversationRecallTimeoutMs` (timebox; fail-open)

**Step 3: Implement QMD chunk index**
- Store chunk docs with clear metadata header (sessionKey, ts range) and a snippet body.
- Use QMD to index/search those chunk docs.

**Step 4: Recall injection**
- In `before_agent_start`, if enabled:
- query the chunk index
- inject top results under a strict cap
- fail-open on errors/timeouts.

**Step 5: Run tests**
Run: `npm test`
Expected: PASS.

---

### Task 4: FAISS Backend (Optional Follow-On)

**Note:** This is optional and can be implemented after QMD backend works.

**Files:**
- Create: `scripts/conversation_indexer.py`
- Create: `src/conversation-index/faiss-adapter.ts` (shell-out wrapper)
- Test: `tests/conversation-index-faiss.test.ts` (skip if deps missing)

**Behavior**
- Python script maintains FAISS + metadata jsonl.
- TS wrapper shells out with a strict timeout and parses results.

---

### Task 5: Docs + Verification

**Files:**
- Modify: `README.md`
- Create: `docs/context-retention.md`

**Step 1: Docs**
- Document feature flags + safe defaults.
- Document privacy constraints (no tool args; caps; retention).
- Document operational guidance (disk growth, timeouts, how to disable).

**Step 2: Verification**
Run:
```bash
npm test
npm run check-types
npm run build
```
Expected: PASS.
