# v2.3 Memory Import & Export Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Add portable, round-trip import/export and safe local backups for Engram memory, without requiring a running gateway.

**Architecture:** Implement export/import as pure local filesystem ops + optional SQLite, driven by CLI subcommands. Default to privacy-safe behavior (exclude transcripts unless explicitly requested). Include versioned manifests and deterministic IDs to support round-trip.

**Tech Stack:** TypeScript (Node.js), `better-sqlite3`, filesystem APIs, existing `StorageManager`.

---

## Decisions (Locked)
- Transcripts excluded by default. Include only with `--include-transcripts`.
- SQLite export must be round-trip importable.

## Deliverables
- CLI:
- `openclaw engram export --format json|sqlite|md --out <path> [--include-transcripts] [--namespace <ns>]`
- `openclaw engram import --from <path> [--format auto] [--namespace <ns>] [--dry-run] [--conflict skip|overwrite|dedupe]`
- `openclaw engram backup --out-dir <dir> [--retention-days N] [--include-transcripts] [--namespace <ns>]`
- Documentation updates:
- `README.md` and a new `docs/import-export.md` describing formats, privacy, and round-trip guarantees.

## New Config (Optional, but likely useful)
- None required for v2.3 MVP. CLI flags drive behavior.
- If added, keep safe defaults and document them.

---

### Task 1: Add Export/Import Data Model Types

**Files:**
- Create: `src/transfer/types.ts`
- Modify: `src/types.ts` (if shared types are needed)
- Test: `tests/transfer-types.test.ts`

**Step 1: Write failing tests**
- Test that a manifest validates required fields (version, createdAt, pluginVersion, checksums).

**Step 2: Implement minimal types**
- Define:
- `ExportManifestV1`
- `ExportMemoryRecordV1` (frontmatter + content + relative path)
- `ExportBundleV1` (manifest + memories + optional state)

**Step 3: Run tests**
Run: `npm test`
Expected: PASS.

**Step 4: Commit**
Run:
```bash
git add src/transfer/types.ts tests/transfer-types.test.ts
git commit -m "feat: add v2.3 transfer bundle types"
```

---

### Task 2: Implement JSON Export (Bundle + Manifest + Checksums)

**Files:**
- Create: `src/transfer/export-json.ts`
- Modify: `src/storage.ts` (add helpers if needed: list files, read state)
- Modify: `src/cli.ts` (new `export` command)
- Test: `tests/export-json.test.ts`

**Step 1: Write failing test**
- Create a tmp memoryDir with 2 memory markdown files + `profile.md` + `state/topics.json`.
- Export to a temp directory.
- Assert:
- `manifest.json` exists and includes checksums.
- exported memory records match content.
- transcripts are absent unless `includeTranscripts=true`.

**Step 2: Implement `exportJson()`**
- Inputs:
- `memoryDir`, optional `namespace`, `includeTranscripts`, output path.
- Behavior:
- Walk memoryDir (or namespace directory once v3.0 exists; for now use memoryDir root).
- Include:
- memory markdown files under category dirs
- `profile.md`, `questions/*.json`, `entities/*.md`, `threads/*.json`, `summaries/*.md`, `state/*.json`
- Exclude by default:
- `transcripts/`
- Create `manifest.json` with:
- `format: "engram-json"`
- `schemaVersion: 1`
- `createdAt`
- `pluginVersion`
- file list + sha256 checksums
- Write bundle as:
- `manifest.json`
- `memories.json` (array of records) or `bundle.json` (single file)
- Recommendation: `bundle.json` single file for portability, plus manifest for quick introspection.

**Step 3: Run tests**
Run: `npm test`
Expected: PASS.

**Step 4: Commit**
```bash
git add src/transfer/export-json.ts src/cli.ts tests/export-json.test.ts
git commit -m "feat: v2.3 json export"
```

---

### Task 3: Implement JSON Import (Round-Trip + Conflicts + Dry Run)

**Files:**
- Create: `src/transfer/import-json.ts`
- Modify: `src/cli.ts`
- Test: `tests/import-json.test.ts`

**Step 1: Write failing tests**
- Import a previously exported JSON bundle into a clean temp memoryDir.
- Assert:
- memory files are written with identical content
- state/profile files are restored
- conflict policy:
- `skip` leaves existing untouched
- `overwrite` replaces existing
- `dedupe` avoids duplicates based on content hash and tags imported memory with `importedFrom`
- `--dry-run` makes no changes (verify by directory listing)

**Step 2: Implement `importJson()`**
- Auto-detect format:
- if `manifest.json` + `bundle.json` exists, treat as Engram JSON export.
- Apply conflicts:
- Use memory `id` as primary key (filename without `.md`).
- Dedupe:
- compute hash of normalized content (trim, collapse whitespace) for incoming and existing; skip if match.
- On actual import, write files preserving directory structure.

**Step 3: Run tests**
Run: `npm test`
Expected: PASS.

**Step 4: Commit**
```bash
git add src/transfer/import-json.ts src/cli.ts tests/import-json.test.ts
git commit -m "feat: v2.3 json import (round-trip)"
```

---

### Task 4: Implement SQLite Export (Round-Trip Schema v1)

**Files:**
- Create: `src/transfer/sqlite-schema.ts`
- Create: `src/transfer/export-sqlite.ts`
- Modify: `src/cli.ts`
- Test: `tests/export-sqlite.test.ts`

**Step 1: Write failing test**
- Create temp memoryDir with 2 memory files + relevance/negatives state.
- Export to `export.sqlite`.
- Assert tables exist and rows match expected.

**Step 2: Implement schema**
- `meta` (schemaVersion, pluginVersion, createdAt)
- `memories` (id, category, created, updated, source, confidence, confidenceTier, tags_json, entityRef, status, content, path_rel)
- `entities` (id, content, path_rel)
- `state_files` (path_rel, content, sha256)
- `feedback_relevance` (memoryId, up, down, lastUpdatedAt, notes_json)
- `feedback_negatives` (memoryId, notUseful, lastUpdatedAt, notes_json)
- Keep schemaVersion = 1.

**Step 3: Implement `exportSqlite()`**
- Use `better-sqlite3`.
- Insert rows transactionally.

**Step 4: Run tests**
Run: `npm test`
Expected: PASS.

**Step 5: Commit**
```bash
git add src/transfer/sqlite-schema.ts src/transfer/export-sqlite.ts src/cli.ts tests/export-sqlite.test.ts
git commit -m "feat: v2.3 sqlite export (schema v1)"
```

---

### Task 5: Implement SQLite Import (Round-Trip)

**Files:**
- Create: `src/transfer/import-sqlite.ts`
- Modify: `src/cli.ts`
- Test: `tests/import-sqlite.test.ts`

**Step 1: Write failing test**
- Export sqlite from a source memoryDir.
- Import into empty target.
- Assert files are reconstructed identically.

**Step 2: Implement `importSqlite()`**
- Read rows; write to filesystem.
- Respect conflict policies and `--dry-run`.

**Step 3: Run tests**
Run: `npm test`
Expected: PASS.

**Step 4: Commit**
```bash
git add src/transfer/import-sqlite.ts src/cli.ts tests/import-sqlite.test.ts
git commit -m "feat: v2.3 sqlite import (round-trip)"
```

---

### Task 6: Implement Markdown Bundle Export (Directory Copy)

**Files:**
- Create: `src/transfer/export-md.ts`
- Modify: `src/cli.ts`
- Test: `tests/export-md.test.ts`

**Behavior**
- Create output directory
- Copy included files
- Write `manifest.json` like JSON export (file list + checksums)

**Commit**
```bash
git add src/transfer/export-md.ts src/cli.ts tests/export-md.test.ts
git commit -m "feat: v2.3 md bundle export"
```

---

### Task 7: Implement Backup Command (Timestamped + Retention)

**Files:**
- Create: `src/transfer/backup.ts`
- Modify: `src/cli.ts`
- Test: `tests/backup.test.ts`

**Behavior**
- Create `outDir/YYYY-MM-DDTHHMMSSZ/`
- Use md export into that folder (or direct copy) to keep backup as plain files.
- If `--retention-days` provided, delete older timestamped dirs.

**Commit**
```bash
git add src/transfer/backup.ts src/cli.ts tests/backup.test.ts
git commit -m "feat: v2.3 backup command (retention)"
```

---

### Task 8: Docs and Verification

**Files:**
- Modify: `README.md`
- Create: `docs/import-export.md`

**Step 1: Document formats + privacy defaults**
- In `docs/import-export.md`, document:
- JSON bundle layout + manifest fields
- SQLite schema overview + round-trip guarantee
- MD bundle behavior
- Transcript default exclusion and `--include-transcripts`
- Conflict policies + `--dry-run`

**Step 2: Add README quickstart**
- Add a short section in `README.md` with:
- export examples
- import examples
- backup examples
- safety notes (`--dry-run`, conflict modes)

**Step 3: Verification**
Run:
```bash
npm test
npm run check-types
npm run build
```
Expected:
- tests pass
- typecheck passes
- build succeeds

**Step 4: Manual smoke test (no gateway required)**
- In a temp dir, initialize a minimal `memoryDir` fixture.
- Run export -> import -> export again.
- Verify checksums and file contents stable across the round-trip.

**Step 5: Commit**
```bash
git add README.md docs/import-export.md
git commit -m "docs: v2.3 import/export usage and formats"
```

---

### Task 9: Import Format Auto-Detect Router (CLI Glue)

**Files:**
- Modify: `src/transfer/import-json.ts`
- Modify: `src/transfer/import-sqlite.ts`
- Modify: `src/cli.ts`
- Test: `tests/import-autodetect.test.ts`

**Goal**
- `openclaw engram import --format auto` should detect:
- Engram JSON bundle (manifest + bundle)
- SQLite file (`.sqlite` magic or extension)
- MD bundle directory (manifest + files)

**Verification**
Run: `npm test`
Expected: PASS.
Run:
```bash
npm test
npm run check-types
npm run build
```

**Commit**
```bash
git add README.md docs/import-export.md
git commit -m "docs: v2.3 import/export/backup"
```
