# RFD 031: Durable Conversation Storage with Workspace Projection

- **Status**: Discussion
- **Category**: Design
- **Authors**: Jean Mertz <git@jeanmertz.com>
- **Date**: 2026-03-05
- **Required by**: [RFD 046]

## Summary

This RFD changes conversation storage so that user-local storage is the source
of truth for all conversations.
Workspace storage (`.jp/conversations/`) becomes a projection — a copy that
exists for git visibility.
Non-local conversations are written to both locations; local conversations are
written only to user-local.
The `local` property becomes a runtime-derived indicator (whether the workspace
copy exists), not a stored flag.

This makes conversations durable across workspace directory deletion - the
primary pain point for git worktree users; without requiring JP to be aware of
git.

## Motivation

JP stores conversations in two locations: workspace storage
(`.jp/conversations/` inside the project directory) and user-local storage
(`~/.local/share/jp/workspace/<name>-<id>/conversations/`).
Today, these are mutually exclusive — a conversation lives in one place or the
other, controlled by the `conversation.user` flag (exposed as `--local`).

This creates a data loss problem for git worktree users.
A typical worktree workflow looks like:

```sh
cd /path/to/project
git worktree add my-feature
cd my-feature
# ... work, use jp query, accumulate conversations ...
cd ..
git worktree remove my-feature   # conversations in .jp/conversations/ are gone
```

Any non-local conversation that was not committed to git is permanently lost.
The user-local storage for that worktree also becomes orphaned, because the
directory name is encoded in the user-local path (`my-feature-<workspace-id>`),
and no future worktree will reconnect to it unless it happens to have the same
directory name.

This is not a theoretical problem — it's a regular occurrence in active
development with multiple worktrees.
The current mitigations (commit conversations to git, or remember to `--local`
every time, or manually move conversations before deleting the worktree) are all
fragile and rely on the user remembering to act before destruction.

The underlying issue is that workspace storage is the *only* copy for non-local
conversations.
If the workspace directory disappears, the data is gone.

## Design

### Storage Model

Every conversation is always persisted to user-local storage.
This is the durable copy.
For non-local conversations, a second copy is additionally written to the
workspace `.jp/conversations/` directory.
This workspace copy is a projection — it exists so that conversations are
visible to `git status` and can be committed alongside code.

| Conversation type   | User-local       | Workspace      |
| ------------------- | ---------------- | -------------- |
| Non-local (default) | ✓ (durable copy) | ✓ (projection) |
| Local (`--local`)   | ✓ (durable copy) | —              |

When both copies exist and their contents differ, the most recently modified
copy takes precedence (see [Manual Editing and Conflict
Resolution](#manual-editing-and-conflict-resolution)).

User-local storage remains optional in `jp_storage`.
When it is `None` (e.g., a headless server setup, or tests that don't need
durability), the dual-write logic is skipped and JP falls back to single-write
workspace storage — the same behavior as today.
The `--local` flag and `local` derivation are unavailable without user-local
storage; all conversations are workspace-only.

> [!TIP]
> [RFD 039] adds tree-structured parent-child relationships via a `parent_id`
> field in each conversation's metadata, while retaining the flat directory
> layout described here.

### Shared User-Local Storage

Today, user-local storage is keyed by both the worktree directory name and
workspace ID: `~/.local/share/jp/workspace/<name>-<id>/`.
This means each worktree gets its own user-local silo, and removing a worktree
orphans its silo.

This RFD changes the user-local path to be keyed by workspace ID only:

```
~/.local/share/jp/workspace/<workspace-id>/conversations/
```

All worktrees for the same repository share a single user-local store.
This aligns with [RFD 020], which already places session mappings and locks at
`~/.local/share/jp/workspace/<workspace-id>/`.

The `with_user_storage` method on `Storage` drops the `name` parameter.
The rename-on-mismatch logic in `with_user_storage` is replaced by a one-time
migration that moves existing `<name>-<id>` directories to `<id>`.

### The `local` Property

The `conversation.user` field (serialized as `"local"` in `metadata.json`) is
removed from the `Conversation` struct.
The `local` indicator shown in `jp conversation ls` becomes a runtime-derived
property:

- **`local N`**: The conversation exists in the current workspace's
  `.jp/conversations/`.
  It is projected.
- **`local Y`**: The conversation exists only in user-local storage.
  It is not projected into this workspace.

This derivation is a filesystem check: does a directory matching this
conversation's ID exist in `<workspace>/.jp/conversations/`?

### Persistence Behavior

When JP persists a conversation:

1. Always write to user-local storage.
2. Check if a workspace copy exists (directory present in `.jp/conversations/`).
   - If yes, also write to workspace storage (update the projection).
   - If no, do not create a workspace projection.

For new conversations:

- `jp query --new` → create in both user-local and workspace.
- `jp query --new --local` → create in user-local only.

This means the workspace projection is created once (at conversation creation
time for non-local conversations, or explicitly via `jp conversation edit`), and
then maintained as long as the workspace directory survives.
If the workspace directory is deleted and recreated, the projection is gone, and
the conversation appears as `local Y` until explicitly re-projected.

### Toggling Projection

`jp conversation edit --local` toggles the workspace projection:

- **`local Y` → `local N`** (project into workspace): Copy the conversation
  from user-local to workspace `.jp/conversations/`.
- **`local N` → `local Y`** (remove from workspace): Delete the workspace copy.
  The user-local copy remains.

### Conversation Origin

When JP creates a conversation, it stores the worktree directory name as an
`origin` field in the conversation metadata.
This is purely informational — it has no behavioral impact — but provides
context in `jp conversation ls` for identifying which worktree a conversation
came from, especially when viewing conversations that were created in a worktree
that no longer exists.

```json
{
  "origin": "feature-a",
  "last_activated_at": "2025-07-20T10:00:00.000Z"
}
```

The `origin` field is set once at creation time and never updated.

### Manual Editing and Conflict Resolution

JP uses plain, pretty-printed JSON files (`events.json`, `metadata.json`)
specifically so that users can edit them by hand.
Tweaking a conversation's context window — removing a noisy tool call, editing
a response, trimming history — is a supported workflow.
The dual-write model must not break this.

The rule is **last-write-wins based on file modification time (mtime)**.
When JP loads a conversation that exists in both user-local and workspace
storage, it compares the mtime of the two `events.json` files (and separately,
the two `metadata.json` files).
Whichever was modified more recently is the version JP loads into memory.

On persist, JP writes to both locations, bringing them back into sync.
After a successful persist, both copies are identical with fresh mtimes.

The full load-persist cycle:

1. **Load**: For each conversation file (`events.json`, `metadata.json`), if
   both copies exist, compare mtimes.
   Load the newer one.
2. **Run**: Execute the query, tool calls, etc. The in-memory state reflects the
   user's edits.
3. **Persist**: Write to user-local, then write to workspace (if projected).
   Both copies are now identical.

If only one copy exists (local-only conversation, or workspace copy was deleted
by a worktree removal), there is nothing to compare — JP loads what's there.

This means a user can edit either copy between JP runs:

- Edit `.jp/conversations/<id>/events.json` in the workspace (the common case —
  the file is right there in the project directory, visible in the editor's file
  tree).
- Edit `~/.local/share/jp/workspace/<id>/conversations/<id>/events.json` in
  user-local (less common, but works the same way).

In both cases, JP picks up the edit on the next run and propagates it to the
other copy on persist.

### Loading Conversations

When loading the conversation list (e.g., `jp conversation ls`), JP reads from
both user-local and workspace storage:

1. Load all conversation IDs from user-local (authoritative).
2. Load all conversation IDs from workspace `.jp/conversations/`.
3. Merge: conversations in user-local are the primary set.
   Conversations that exist only in the workspace (not in user-local) are
   included but marked as workspace-only (see [External
   Conversations](#external-conversations)).
4. Derive `local` for each conversation: check whether a workspace copy exists.

### External Conversations

Conversations can appear in the workspace that do not exist in user-local.
This happens when another contributor commits a conversation to git and you pull
it.

These conversations are shown in `jp conversation ls` with a distinct indicator
(e.g., `origin: <their-worktree-name>` from the stored metadata, or simply the
absence of a user-local copy).
They are read-only from JP's perspective until the user interacts with them.

On first write interaction (`jp query --id=<id>`, `jp conversation edit`, or any
operation that modifies the conversation), the conversation is imported: copied
from workspace to user-local.
From that point on, it follows the normal dual-write persistence rules.

Non-write interactions (`jp conversation show`, `jp conversation print`, `jp
conversation ls`) read directly from the workspace without importing.

## Drawbacks

**Double disk I/O for non-local conversations.** Every persist operation writes
to two locations.
For JSON conversation files this is negligible, but it is strictly more work
than the current single-write model.

**Divergence risk.** If JP crashes between writing to user-local and workspace
(or vice versa), the two copies can diverge.
The mtime-based resolution (see [Manual Editing and Conflict
Resolution](#manual-editing-and-conflict-resolution)) handles this gracefully —
whichever copy was written last is used on next load, and the next successful
persist re-syncs both copies.
However, if the crash happens after writing user-local but before writing
workspace, the workspace copy may be stale until the next JP run.
Users who inspect workspace files directly between JP runs could see outdated
content.

**Conversation accumulation in user-local.** Because user-local is the durable
store, conversations accumulate there indefinitely (across worktree lifetimes).
The existing ephemeral conversation cleanup (`expires_at`) helps, but users who
create many conversations across many worktrees may eventually want a dedicated
cleanup command or policy.
This is not new — user-local conversations already accumulate today — but the
volume increases when all conversations go through user-local.

## Alternatives

### Keep current model, add evacuation command

A `jp conversation evacuate` command that moves all workspace conversations to
user-local before worktree removal.
This requires the user to remember to run it (or configure a git hook).

Rejected because it doesn't solve the problem — it shifts it from "remember to
use `--local`" to "remember to run evacuate."
Forgetting either one results in data loss.

### Store conversations at the bare repo level

For worktree setups, store conversations in a shared location above the
worktrees (e.g., `/path/to/project/.jp/conversations/`).
All worktrees read from and write to this shared directory.

Rejected because it requires JP to understand git worktree topology (resolving
the common git dir), which violates the goal of keeping JP git-unaware.
It also introduces write contention between concurrent worktrees.

### Sync conversations across workspaces

Keep per-worktree storage but replicate conversations bidirectionally across all
workspaces with the same ID.

Rejected because bidirectional sync is the wrong model.
Conversations are contextual to a branch or task — syncing a feature-a
conversation into the feature-b worktree is undesirable.
The problem is durability, not distribution.

### `.jp` as a file (like `.git` in worktrees)

Make `.jp` a pointer file that redirects to shared storage, similar to how git
worktrees use a `.git` file pointing to the common git dir.

Rejected because `.jp/conversations/` needs to contain real files for `git
status` visibility.
A pointer file would make conversations invisible to git, defeating one of the
two core requirements.

### Default all conversations to `--local`

Make `Conversation::default()` set `user: true`, so all conversations go to
user-local by default.
Users who want git-visible conversations opt in explicitly.

Rejected because it inverts the current expectation that conversations are
visible in the workspace by default.
The dual-write approach preserves the default behavior (workspace-visible) while
adding durability.

## Non-Goals

- **Git awareness.** JP does not detect whether it is running inside a git
  worktree, does not resolve the common git directory, and does not read or
  write git-specific metadata.
  The design works for any workflow where the workspace directory might be
  deleted — worktrees, temporary clones, or manual cleanup.

- **Three-way merge conflict resolution.** If both copies are edited between JP
  runs (e.g., a user edits the workspace copy while a background process edits
  the user-local copy), the older edit is silently overwritten by the newer one.
  JP does not attempt to merge concurrent edits.
  In practice, users edit one copy (almost always the workspace one), so this is
  unlikely to cause issues.

- **Cross-machine sync.** User-local storage is local to the machine.
  This RFD does not address synchronizing conversations across machines.

- **Conversation garbage collection policy.** This RFD does not define when or
  how old conversations in user-local should be cleaned up.
  The existing `expires_at` mechanism continues to work.
  A dedicated cleanup UX is future work.

## Risks and Open Questions

### Migration of existing user-local directories

Renaming `<name>-<id>` directories to `<id>` needs to handle the case where
multiple worktrees have already created separate user-local directories (e.g.,
`main-otvo8` and `feature-a-otvo8`).
The migration must merge their contents without losing conversations.
If two directories contain a conversation with the same ID (unlikely but
possible if both worktrees were used concurrently), the most recently modified
copy should win.

### Workspace copy freshness

The workspace copy is updated on every persist, but only if the workspace copy
already exists.
If a user manually deletes a conversation from `.jp/conversations/` (e.g., via
`git checkout` or `git clean`), JP will not recreate it — the conversation
silently becomes local-only.
This is arguably correct behavior (the user deleted it), but might surprise
users who expect workspace conversations to be self-healing.

### Performance of `local` derivation

Deriving the `local` indicator requires checking the filesystem for each
conversation when listing.
For workspaces with many conversations, this could add latency to `jp
conversation ls`.
In practice, the number of conversations per workspace is small (tens, not
thousands), so this is unlikely to be a problem.
If it becomes one, the check can be batched into a single directory listing.

## Implementation Plan

### Phase 1: Shared User-Local Storage

Change the user-local storage path from `<name>-<id>` to `<id>`.
Add migration logic to merge existing per-worktree directories.
Update `with_user_storage` to drop the `name` parameter.

Depends on: nothing.
Can be merged independently.

### Phase 2: Dual-Write Persistence

Change `persist_conversations_and_events` to always write to user-local.
For conversations where a workspace copy exists (or for new non-local
conversations), also write to the workspace.
Remove the `conversation.user` field from the `Conversation` struct.
Derive the `local` property at runtime from workspace filesystem state.

Depends on: Phase 1.
Can be merged independently.

### Phase 3: Conversation Origin Metadata

Add the `origin` field to `Conversation`, populated from the worktree directory
name at creation time.
Surface it in `jp conversation ls` and `jp conversation show`.

Depends on: nothing (can be done in parallel with Phase 1/2).
Can be merged independently.

### Phase 4: External Conversation Import

Add lazy-import logic: when JP encounters a conversation in the workspace that
does not exist in user-local, and the user performs a write operation on it,
copy it to user-local first.
Update `jp conversation ls` to display workspace-only conversations with
appropriate indicators.

Depends on: Phase 2.
Can be merged independently.

### Phase 5: Update `jp conversation edit --local`

Change `--local` toggling to copy-to-workspace / delete-from-workspace instead
of the current move-between-storage-locations behavior.

Depends on: Phase 2.
Can be merged independently.

## References

- [RFD 020: Parallel Conversations][RFD 020] — removes
  `active_conversation_id`, introduces per-session conversation tracking and
  conversation locks.
  This RFD's shared user-local storage aligns with RFD 020's session and lock
  storage paths.

[RFD 020]: 020-parallel-conversations.md
[RFD 039]: 039-conversation-trees.md
[RFD 046]: 046-nested-workspace-projection.md
