# v0.3 — Episodic Memory (Cognitive Phase 3 PR-9)

> Status: design-only (no code yet)
> Pinned to: `docs/ARCHITECTURE.md` §5.7.6 (memory scope layering)
> Tracks: `docs/handovers/v0.3-cognitive-layer-track.md` Phase 3
> Companion documents:
>   - `docs/ARCHITECTURE.md` §5.7.2 (cognitive middleware), §5.7.6 (scope layering)
>   - `docs/handovers/v0.3-cognitive-layer-track.md` (the parent track)
>   - `core/memory/conversation.py` (the existing turn-pair store this layers on)

## Why episodic memory exists

JAMES today persists **conversation turns** (user prompt + assistant
answer pairs in `core/memory/conversation.py`) and **long-term
summaries** (`core/memory/summaries.py`). What is missing is the
**intra-turn reasoning trail** — the retrieve / plan / reflect /
verify / synth events that produced one answer, kept in a form the
*next* turn in the same session can read.

The `audit_log` table captures every `TraceStep` for replay and
forensics, but its access pattern is *post-hoc* (`replay_trace.py`
joins by `trace_id`). The cognitive middleware needs an *online*
session-scoped view: "in this conversation, what did the planner
decompose the last question into, what did reflection flag, what
verification scores landed below threshold."

Reusing `audit_log` for that purpose conflates two concerns —
forensic audit (cross-session, append-only, retained) vs. working
session state (session-scoped, prunable, mutable until session
end). The split is intentional.

## Position in the scope hierarchy (§5.7.6)

```
┌─────────────────────────────────────────────────────────┐
│  system     — ontology, PolicyEngine rules, evolutions │
│       inherited read-only by ↓                          │
├─────────────────────────────────────────────────────────┤
│  workspace  — wiki entities, vector store, audit_log   │
│       inherited read-only by ↓                          │
├─────────────────────────────────────────────────────────┤
│  session    — episodic events, working memory, drafts  │  ← NEW (PR-9)
│       writes never escape upward                        │
└─────────────────────────────────────────────────────────┘
```

Episodic memory is the **session-scoped tier**. Two invariants pin it
to that row of the table:

1. **Writes never escape upward.** An episodic event written by
   session `S` is not visible to session `S'` and is not visible at
   the workspace tier. Cross-session "memory" promotion already has
   an explicit path — the user clicks "save as long-term memory" in
   the chat modal (PR #264) and `MemoryLoom` lifts the insight into
   workspace scope. Episodic does not invent a second promotion path.
2. **Session end clears.** A new session is a clean episodic slate.
   "Session end" today means a new `session_id` is allocated (frontend
   triggers via `newSession()` / `switchSession()` per PR #302); the
   episodic store keys on that id and discards old rows on a
   retention schedule.

## Schema

```python
@dataclass(frozen=True)
class EpisodicEvent:
    # ─── identity ──────────────────────────────────────────────
    event_id: str          # ULID — sortable + globally unique
    session_id: str        # the session this event belongs to
    turn_id: str           # the turn within the session (frontend-allocated)
    ts: float              # unix ts of write (server clock)

    # ─── content ───────────────────────────────────────────────
    stage: str             # "retrieve" | "plan" | "reflect" | "verify"
                           # | "synth" | "tool_call" | "error"
    summary: str           # short human-readable line (<=240 chars,
                           # truncated server-side via truncate_summary)
    score: float = 0.0     # stage-specific signal (0.0 if absent —
                           # retrieval similarity, verification confidence,
                           # reflection self-evaluation)
    extras: dict = field(default_factory=dict)
                           # stage-specific JSON payload, opaque to the
                           # store; consumers parse per stage
    trace_id: str = ""     # back-link to TraceStep via core/observability
                           # so replay can join episodic ↔ forensic
```

**Why not reuse `TraceStep`?** Two reasons:
- `TraceStep.stage` is reasoning-pipeline-stage (retrieve / graph /
  rerank / answer); `EpisodicEvent.stage` is cognitive-decision-stage
  (plan / reflect / verify). They overlap but are not equal —
  reflection emits multiple events from one TraceStep.
- `TraceStep` is the forensic record; modifying it would break the
  audit pipeline. Episodic mutates inside the session window
  (e.g. reflection updates its own row when a re-eval lands).

`trace_id` glues the two together for replay. A consumer that wants
the full reasoning trail joins on it.

## API surface

```python
class EpisodicMemory:
    """Session-scoped event store. One instance per process; scoped
    reads/writes happen via session_id arguments rather than per-session
    instances (matches MemoryStore's pattern in store.py).
    """

    def __init__(self, db_path: str = DEFAULT_EPISODIC_DB) -> None: ...

    # ─── write ──────────────────────────────────────────────────
    def record(
        self,
        *,
        session_id: str,
        turn_id: str,
        stage: str,
        summary: str,
        score: float = 0.0,
        extras: dict | None = None,
        trace_id: str = "",
    ) -> str:
        """Append one event. Returns the assigned event_id. Truncates
        summary to 240 chars. Validates stage against a known set —
        unknown stage raises (loud dev signal).
        """
        ...

    # ─── read ───────────────────────────────────────────────────
    def events_for_turn(
        self,
        session_id: str,
        turn_id: str,
    ) -> list[EpisodicEvent]:
        """Every event in one turn, sorted by event_id (ULID order)."""
        ...

    def recent_events(
        self,
        session_id: str,
        *,
        limit: int = 20,
        stages: tuple[str, ...] = (),
    ) -> list[EpisodicEvent]:
        """The last `limit` events in this session, optionally filtered
        by stage. Default 20 = ~5 turns of reasoning."""
        ...

    # ─── lifecycle ──────────────────────────────────────────────
    def clear_session(self, session_id: str) -> int:
        """Delete every row for this session. Returns rows removed.
        Called when the frontend cycles a session via newSession()
        and as part of retention sweep.
        """
        ...

    def prune_older_than(self, *, max_age_days: int) -> int:
        """Retention sweep. Default 7 days, matches
        JAMES_TRACE_RETENTION_DAYS (Axis 3 PR #84).
        """
        ...
```

**Read isolation** is enforced at the SQL layer: every read is
`WHERE session_id = ?`. The store never exposes a "list all sessions"
read on the user-facing path; that's an admin-only debugging API in
PR-9b and goes through `PolicyEngine` (`admin.episodic.list`).

## Persistence

SQLite, separate file: `james_episodic.db`. Reasons:

- The existing `james_data.db` is workspace-scoped (artifacts, wiki
  links). Episodic is session-scoped. Mixing them blurs the §5.7.6
  contract.
- Audit (`james_audit.db`) is retained forever for forensic replay.
  Episodic prunes at 7 days. Mixing forces one retention to win.
- A separate file makes the scope visible at the filesystem level —
  an operator can wipe episodic without touching workspace state.

Schema:

```sql
CREATE TABLE IF NOT EXISTS episodic_events (
    event_id    TEXT PRIMARY KEY,    -- ULID
    session_id  TEXT NOT NULL,
    turn_id     TEXT NOT NULL,
    ts          REAL NOT NULL,        -- unix
    stage       TEXT NOT NULL,
    summary     TEXT,
    score       REAL DEFAULT 0.0,
    extras_json TEXT,                 -- JSON-encoded dict
    trace_id    TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS ix_episodic_session_ts
    ON episodic_events(session_id, ts DESC);
CREATE INDEX IF NOT EXISTS ix_episodic_turn
    ON episodic_events(session_id, turn_id);
CREATE INDEX IF NOT EXISTS ix_episodic_trace
    ON episodic_events(trace_id);
```

Index choices match the three expected access patterns: most recent
N for the session, all events of one turn, and replay join.

## PolicyEngine integration

Two surfaces require gates:

1. **Cross-session read** — only the admin debugging endpoint
   (PR-9b) reads other sessions' rows. Gate: `admin.episodic.list`,
   admin-only by default.
2. **Promotion to workspace** — there is none in PR-9. Episodic is
   session-only. The existing "save as long-term memory" path
   (PR #264 modal → MemoryLoom) covers the only promotion we ship.

`PolicyEngine` does not need a new permission for PR-9 because every
read is `WHERE session_id = ?` and the session_id is bound to the
JWT subject by the existing chat handler. A user reading their own
session is not crossing a trust boundary.

## What PR-9 does NOT do

- **Wiring**: PR-9 ships `core/memory/episodic.py` + tests + the
  SQLite file scaffolding. Cognitive middleware does **not** start
  calling `record(...)` until PR-9b. This matches the L0/L1 split
  the backends track used (PR #283 shipped infrastructure; PR #284
  wired call sites).
- **Working memory** (`core/memory/working.py`): a separate file,
  separate PR (PR-10). Working memory holds intermediate state
  *during* one turn's reasoning loop and is cleared at turn end;
  episodic survives the turn. Conflating them in one PR risks
  the wrong abstraction.
- **Graph evolution** (event nodes in the entity graph): PR-11.
  Optional. Out of scope here.
- **Cross-session leakage tests against a malicious session_id**:
  PR-9 enforces `WHERE session_id = ?` at the SQL layer; a fuzzed
  malicious-session-id test belongs in the security suite, not the
  episodic module.

## Test plan (for PR-9a — code lands separately)

1. `record()` returns a sortable ULID
2. `events_for_turn()` returns chronological order
3. `recent_events()` respects `limit` and `stages` filter
4. `clear_session()` removes exactly that session's rows
5. **Session isolation**: writing to session A and reading session B
   returns `[]` — never leaks
6. **Stage validation**: unknown stage in `record()` raises
   `ValueError`
7. **Summary truncation**: a 5000-char summary lands as 240 chars
8. **Idempotent schema**: re-running `__init__` on an existing DB is
   a no-op
9. **trace_id is forwardable**: a record made with `trace_id="abc"`
   is retrievable by that trace_id (PR-9a does not ship the lookup
   API — but the index is in place for PR-9b to use)
10. **Retention sweep**: `prune_older_than(max_age_days=7)` removes
    rows older than 7 days, keeps newer ones

## PR sequence

| PR | Scope |
|---|---|
| **PR-9a** | `core/memory/episodic.py` + schema + tests. Wiring 0. The cognitive middleware does not yet call `record()`. |
| **PR-9b** | Wire `record()` calls into the cognitive stages (planner / reflect / verify / synth at minimum). Cross-turn read in the chat handler so a follow-up question sees the previous turn's reasoning trail. Admin endpoint `GET /admin/episodic/{session_id}` for debugging, gated by `admin.episodic.list`. |
| **PR-9c (optional)** | Retention sweep cron + `JAMES_EPISODIC_RETENTION_DAYS` env (default 7). Folds in if not done in PR-9b. |

## Open questions (resolve in PR-9a or before)

1. **Turn id source** — frontend `chat.js` does not currently emit a
   turn_id; today the backend infers from `SESSION_ID + utterance
   index`. PR-9a will derive `turn_id` server-side as
   `{session_id}:{utterance_index}` unless the frontend later starts
   sending one. Keeps the contract minimal.
2. **Multi-process safety** — SQLite WAL mode handles concurrent
   reads + one writer. Multiple JAMES processes per workspace (a v0.4
   topology) would need a different store; the schema and API are
   forward-compatible because session_id stays the namespace key.
3. **Storage cap** — 7-day retention + `~20 events/turn × ~10
   turns/session × N sessions` ≈ small (KB range per session). No
   hard cap in PR-9; revisit if a real workload pushes the DB past
   100 MB.

## Diff log

(Add entries below as the design evolves.)

| Date | Author | Change | PR |
|---|---|---|---|
| 2026-05-19 | Jiwon | Initial design (PR-9 of Cognitive Phase 3) | (this PR) |
