# RFD 052: Workspace Data Store Sanitization

- **Status**: Implemented
- **Category**: Design
- **Authors**: Jean Mertz <git@jeanmertz.com>
- **Date**: 2026-03-18
- **Required by**: [RFD 073]

## Summary

This RFD introduces a `Workspace::sanitize()` method that validates and repairs
the `.jp` data store on every CLI invocation.
Conversations that fail validation are moved to `.jp/conversations/.trash/` with
a `TRASHED.md` file explaining the error.
The method guarantees four invariants before any command touches workspace
state: at least one conversation exists, the active conversation ID resolves,
every conversation directory is well-formed, and every conversation has valid
`metadata.json` and `events.json` files.

## Motivation

The `.jp/conversations/` directory is the primary data store for JP.
Every CLI command depends on loading this state correctly.
When any piece is corrupt — a truncated `events.json`, a directory with an
unparseable name, a `metadata.json` that references a deleted conversation —
the CLI either crashes or silently operates on empty default state.

JP's data store is intentionally transparent: plain JSON files in a
well-documented directory layout.
Users are expected to inspect, edit, and script against this data — renaming
conversations, pruning events, copying conversation directories between
workspaces, or integrating with external tools.
This is a feature, not an accident.
But it also means the data store will encounter states that JP itself would
never produce: partial edits, hand-written JSON with typos, directories left
behind by interrupted scripts, or files modified by tools that don't understand
the full schema.
The sanitizer must handle these gracefully — trashing what it can't load,
explaining what went wrong, and continuing to operate on the data that is still
valid.
A single bad file should never prevent the user from accessing the rest of their
workspace.

Today the code handles some of these cases, but the defenses are scattered and
inconsistent:

- `load_conversations_from_disk` has a retry loop that falls back through
  conversation IDs when the active conversation's `metadata.json` is missing.
  But a corrupt (not missing) `metadata.json` is a hard error that propagates
  up.

- The active conversation's `events.json` is loaded eagerly.
  If it's corrupt or missing, the load fails entirely (the FIXME at
  `workspace/lib.rs:222` documents this).
  Non-active conversations are lazily loaded via `OnceCell` and silently skipped
  on failure — different failure behavior for the same kind of data.

- In `cli/lib.rs`, `load_conversations_from_disk` errors are caught and logged,
  but execution continues with default (empty) state.
  The `persist()` call in `Workspace::Drop` is safe in this case — the empty
  `TombMap` has no dead keys, so no existing conversations are deleted, and the
  active conversation has no events, so nothing is written.
  If the user runs a query after the failed load, the new conversation is
  persisted alongside the existing ones.
  No data is destroyed, but the user experience is poor: the load is
  all-or-nothing, so a single corrupt conversation hides the entire conversation
  list.
  A workspace with hundreds of valid conversations shows just one (the fresh
  default) in `jp conversation ls`.
  The user sees an `ERROR` log but has no way to identify which conversation is
  broken or fix it without manually inspecting JSON files on disk.

- Conversation directories with unparseable names (e.g. manually created, left
  over from a bug) are logged as warnings and ignored forever.

The blast radius is the core problem: one bad file takes down the whole
conversation list.
The defenses that exist are either too aggressive (hard error on corrupt active
conversation) or too lenient (silent skip on corrupt non-active conversation),
and none of them tell the user what went wrong or how to fix it.

Issue [#404] is one symptom — a clean install fails because
`load_conversations_from_disk` can't find a conversation that `metadata.json`
references.
The fix in [PR #450][#450] addresses the specific case of a fresh workspace, but
the general problem of corrupt data causing cascading failures remains.

We need a single, systematic pass that enforces data store invariants before any
command runs.
When something is broken, it should be fixed or moved aside with an explanation
— not silently ignored or allowed to corrupt other data.

## Design

### Invariants

The sanitizer enforces four invariants on the `.jp/conversations/` directory
(and its user-storage counterpart):

1. **Well-formed directories.** Every entry in `conversations/` is either a
   known file (`metadata.json`) or a directory whose name starts with a valid
   `ConversationId` (parseable via `try_from_dirname`).

2. **Valid conversation data.** Every conversation directory contains a
   `metadata.json` that deserializes as `Conversation` and an `events.json` that
   is structurally valid JSON (see [Lightweight events.json
   validation](#lightweight-eventsjson-validation)).

3. **Valid conversations metadata.** The `conversations/metadata.json` file (the
   global metadata that holds `active_conversation_id`) is structurally valid
   JSON that deserializes as `ConversationsMetadata`.
   If the file is missing, the system uses a default.
   If the file is corrupt, the sanitizer deletes it so that subsequent loads use
   a default.

4. **Resolvable active conversation.** The `active_conversation_id` from the
   conversations metadata resolves to a conversation that passes checks 1 and 2.
   This check runs unconditionally — not only when conversations are trashed,
   but also when the metadata points to a conversation ID that has no
   corresponding directory on disk (e.g., stale metadata from a previous
   session, or a corrupt metadata file that was just reset to default).

5. **Graceful empty state.** If all conversations are trashed (or none existed),
   the stale metadata file is removed and `load_conversations_from_disk` handles
   this as a fresh workspace with default state.
   No conversation is written to disk during sanitization.

### User experience

When sanitization trashes a conversation, the CLI logs a warning:

```txt
WARN Trashed corrupt conversation 17457886043-my-chat
     (see .jp/conversations/.trash/17457886043-my-chat/TRASHED.md)
```

If the active conversation was trashed and a fallback was selected:

```txt
WARN Active conversation was corrupt, switched to 17636257528-other-chat
```

If all conversations were trashed and a new default was created:

```txt
WARN No valid conversations found, created a new empty conversation
```

Users can inspect `.trash/` to understand what went wrong and manually recover
data if the underlying files are salvageable.

### Trash directory

Conversations that fail validation are moved to
`.jp/conversations/.trash/<original-dirname>/`.
The leading dot makes the trash directory invisible to the rest of the system:
`load_conversation_id_from_entry` skips entries whose names start with `.`, so
the trash directory is never attempted as a `ConversationId` parse and generates
no warnings.

Each trashed conversation gets a `TRASHED.md` file written into its directory:

```markdown
# Trashed Conversation

This conversation was moved here because it failed workspace sanitization.

**Error:** metadata.json: expected value at line 3 column 1
**Date:** 2026-03-18T14:22:07Z

The original conversation files are preserved alongside this file.
If the data is recoverable, you can fix the issue and move the
directory back to `.jp/conversations/`.
```

The format is intentionally human-readable Markdown, not JSON.
This is a recovery aid for the user, not machine-readable state.

If a conversation with the same directory name already exists in `.trash/` (from
a previous sanitization run), the new trashed version gets an integer suffix
(e.g.
`17457886043-my-chat-1`).

### API

A new public method on `Workspace`:

```rust
impl Workspace {
    /// Validate and repair the `.jp/conversations/` data store.
    ///
    /// Trashes conversations that fail validation and ensures the workspace
    /// invariants hold. Returns a report of actions taken.
    pub fn sanitize(&mut self) -> Result<SanitizeReport>;
}
```

The `SanitizeReport` captures what happened:

```rust
pub struct SanitizeReport {
    /// Conversations moved to `.trash/`.
    pub trashed: Vec<TrashedConversation>,

    /// Whether the active conversation was reassigned.
    pub active_reassigned: bool,

    /// Whether a new default conversation was created because none remained.
    pub default_created: bool,
}

pub struct TrashedConversation {
    /// The original directory name.
    pub dirname: String,

    /// The error that caused the conversation to be trashed.
    pub error: String,
}

impl SanitizeReport {
    /// Returns `true` if any repairs were made.
    pub fn has_repairs(&self) -> bool {
        !self.trashed.is_empty() || self.active_reassigned || self.default_created
    }
}
```

### Sanitization logic

The method operates directly on the filesystem, before in-memory state is
constructed.
It runs against both the workspace storage root and the user storage root (if
configured).

```txt
for each storage root (workspace, user):
    for each entry in {root}/conversations/:
        skip if entry is a file (metadata.json, etc.)
        skip if entry name starts with "." (.trash/)

        1. try parse ConversationId from dirname
           → on failure: trash with "invalid directory name: {name}"

        2. try deserialize {entry}/metadata.json as Conversation
           → on missing: trash with "missing metadata.json"
           → on parse error: trash with "metadata.json: {error}"

        3. try lightweight validation of {entry}/events.json
           (valid JSON, array of objects with `timestamp` fields)
           → on missing: trash with "missing events.json"
           → on parse error: trash with "events.json: {error}"

load conversations/metadata.json:
    → on missing: use default ConversationsMetadata
    → on corrupt JSON: delete the file, use default ConversationsMetadata
    → on I/O error: propagate (not a data-corruption issue)

if no valid conversations remain and nothing was trashed:
    return (fresh workspace, nothing to repair)

resolve active_conversation_id:
    if the referenced conversation doesn't exist among the valid IDs:
        pick the most recent valid conversation (by ConversationId sort order,
            descending)
        update conversations/metadata.json
        set active_reassigned = true

if no valid conversations remain:
    remove stale conversations/metadata.json if present
    set active_reassigned = true
    set default_created = true
    (load_conversations_from_disk handles this as a fresh workspace)
```

After sanitization completes, `load_conversations_from_disk` is called as usual.
Because the filesystem is now clean, the load should succeed.
If it still fails, the error propagates — sanitization is best-effort repair,
not an infinite retry.

### Call site in `cli/lib.rs`

The sanitization runs between workspace construction and conversation loading:

```rust
let mut workspace = load_workspace(cli.globals.workspace.as_ref())?;

let report = workspace.sanitize()?;
if report.has_repairs() {
    for trashed in &report.trashed {
        tracing::warn!(
            dirname = trashed.dirname,
            error = trashed.error,
            "Trashed corrupt conversation"
        );
    }
    if report.active_reassigned {
        tracing::warn!("Active conversation was corrupt, switched to fallback");
    }
    if report.default_created {
        tracing::warn!("No valid conversations found, created new default");
    }
}

if let Err(error) = workspace.load_conversations_from_disk() {
    tracing::error!(error = ?error, "Failed to load workspace.");
}
```

The existing `persist()` behavior after a failed load is actually safe — the
empty `TombMap` has no dead keys, so no existing conversations are deleted from
disk.
If the user runs a query, the new conversation is written alongside the existing
ones.
No data is lost in either case.

With sanitization in place, load failures after sanitize should be rare
(permission errors, races with other processes).
If one does occur, the CLI continues with a fresh default conversation and the
user's session work is persisted normally.

Note that the metadata resolution runs unconditionally — not only when
conversations are trashed, but whenever the `active_conversation_id` doesn't
match a valid conversation on disk.
This covers stale metadata that points to a conversation whose directory was
manually deleted, or a corrupt metadata file that was just reset to its default
value.

### Lightweight `events.json` validation

`ConversationStream` deserialization is expensive — event files can be
thousands of lines.
The `OnceCell`-based lazy loading in `Workspace` exists specifically to defer
this cost until a conversation is actually accessed.

Sanitization must not defeat this design.
Instead of deserializing the full `ConversationStream`, it uses a lightweight
structural check modeled on the existing `load_count_and_timestamp_events`
function in `jp_storage::load`:

```rust
#[derive(Deserialize)]
struct RawEvent {
    timestamp: Box<serde_json::value::RawValue>,
}

fn validate_events_file(path: &Utf8Path) -> Result<(), String> {
    let file = fs::File::open(path)
        .map_err(|e| format!("cannot open events.json: {e}"))?;
    let reader = BufReader::new(file);
    let _events: Vec<RawEvent> = serde_json::from_reader(reader)
        .map_err(|e| format!("events.json: {e}"))?;
    Ok(())
}
```

This confirms:

1. The file exists and is readable.
2. The content is valid JSON.
3. The top-level structure is an array.
4. Each element is an object with a `timestamp` field.

This does not validate individual event variants (tool calls, chat requests,
config deltas).
Variant-level errors are caught later during lazy deserialization, where
per-conversation failure handling already exists (the `maybe_init_events`
functions log warnings and skip broken conversations).

### Simplifying `load_conversations_from_disk`

`load_conversations_from_disk` currently has its own fallback loop that retries
loading through conversation IDs when the active conversation is missing.
With sanitization running first, this fallback logic is dead code — sanitize
has already trashed conversations with missing or corrupt metadata and
reassigned the active conversation ID if needed.
The fallback loop should be removed to keep the load path simple and avoid two
code paths that handle the same failure mode differently.

The FIXME at `workspace/lib.rs:222` (corrupt `events.json` for the active
conversation causing a hard failure) is also resolved: sanitize validates
`events.json` before load ever runs.
The load method can assume the filesystem is clean and treat any remaining
errors as unexpected failures.

The global `conversations/metadata.json` is also validated during sanitization.
If it contains corrupt JSON, the sanitizer deletes it (causing
`load_conversations_metadata` to return a default) and proceeds with the normal
resolution logic.
I/O errors reading the file still propagate — those indicate a system problem,
not data corruption.

## Drawbacks

**I/O cost on every invocation.** Sanitization scans all conversation
directories, deserializes each `metadata.json`, and performs a lightweight
structural check on each `events.json` (see [Lightweight events.json
validation](#lightweight-eventsjson-validation)).
The lightweight check avoids full `ConversationStream` deserialization — it
parses the array structure and `timestamp` fields using `RawValue`, which is
substantially cheaper than deserializing every event variant.

For most workspaces (tens of conversations), this adds negligible latency.
For large workspaces (hundreds of conversations), the cost is dominated by file
opens and JSON tokenization.
If this becomes measurable, a future optimization could skip validation for
conversations whose `events.json` mtime hasn't changed since the last successful
sanitization run.

**Data loss via trashing.** Moving a conversation to `.trash/` makes it
invisible to the CLI.
If the corruption was a transient issue (e.g. a concurrent write from another
process), the user loses access to their conversation.
The `.trash/` directory and `TRASHED.md` file mitigate this — the data is
recoverable — but it requires manual intervention.

## Alternatives

### Sanitize inside `load_conversations_from_disk`

Merge validation and repair into the existing load method.
Every load call would also fix corruption.

Rejected because it conflates two responsibilities — loading state and
repairing state — and makes the load method harder to reason about.
Callers don't expect a "load" to have filesystem side effects like moving
directories.
A separate method makes the repair explicit and lets the caller decide when and
whether to run it.

### Storage-level validation

Push validation into `jp_storage`.
Each `load_*` method gains a "validate and repair" mode.

Rejected because `jp_storage` is a generic persistence layer.
It shouldn't know domain-level invariants like "a conversation must have both
metadata.json and events.json" or "the active conversation ID must resolve."
Those invariants belong in `jp_workspace`, which understands the conversation
model.

### Delete instead of trash

Remove corrupt conversations instead of moving them.

Rejected because deletion is irreversible.
Conversations may contain hours of LLM interaction that the user values.
The `.trash/` directory is cheap (no additional disk usage beyond the move) and
gives users a recovery path.

## Non-Goals

- **Automatic recovery of corrupt JSON.** If `events.json` is truncated, the
  sanitizer does not attempt to parse what it can and reconstruct a partial
  stream.
  That level of recovery is complex and error-prone.
  The user can attempt manual recovery from `.trash/`.

- **Crash-safe writes.** Preventing corruption in the first place (e.g. via
  atomic writes with rename) is a separate concern.
  This RFD handles the consequences of corruption, not its causes.
  Atomic writes would be a worthwhile follow-up.

- **Trash management.** Automatically cleaning up old `.trash/` entries (e.g.
  after 30 days) is deferred.
  The trash directory can grow without bound, but conversation data is small and
  this is unlikely to be a problem in practice.

- **User-facing `jp workspace repair` command.** A CLI command that explicitly
  triggers sanitization (and possibly more aggressive repair) is a natural
  follow-up but out of scope here.
  The sanitizer runs automatically; the command would be for manual use.

## Risks and Open Questions

### Performance with large workspaces

The lightweight `events.json` validation avoids full deserialization, but still
requires opening and tokenizing every events file.
For a workspace with hundreds of conversations, this could add measurable
startup latency.
Implementation should benchmark with a realistic workspace (50+ conversations
with large event files) to confirm the cost is acceptable.

### User-storage and workspace-storage interaction

Conversations can live in either the workspace storage (`.jp/conversations/`) or
user-local storage (`$XDG_DATA_HOME/jp/workspace/<name>-<id>/conversations/`).
The sanitizer must scan both roots.
The `conversations/metadata.json` (which holds `active_conversation_id`) lives
in user storage when configured.
If the active conversation is in workspace storage but has been trashed, the
user-storage metadata needs updating.
The implementation must handle both roots consistently.

### Concurrent access

If two JP processes run simultaneously (e.g. in different terminals), one
process's sanitization could trash a conversation that the other process is
actively writing to.
This is an existing problem (no file locking today), and sanitization doesn't
make it worse — but it doesn't fix it either.
[RFD 020] addresses concurrent access with file locking; sanitization should be
aware of locks if/when they exist.

## Implementation Plan

### Phase 1: Trash infrastructure

Add the `.trash/` directory support to `jp_storage`:

- `Storage::trash_conversation(id, dirname, error)` — moves a conversation
  directory to `.trash/` and writes `TRASHED.md`.
- Handle the case where a trashed conversation with the same name already
  exists.
- Unit tests with temp directories.

**Dependency:** None.
**Mergeable:** Yes.

### Phase 2: `Workspace::sanitize()`

Implement the sanitization logic in `jp_workspace`:

- `SanitizeReport` and `TrashedConversation` types.
- The validation loop: parse dirname, deserialize metadata, validate events.
- Global `conversations/metadata.json` validation and recovery from corrupt
  JSON.
- Active conversation reassignment (runs unconditionally, not only after
  trashing).
- Fresh-workspace detection: if no conversations exist on disk and nothing was
  trashed, return immediately without touching metadata.
- Remove the fallback loop and FIXME in `load_conversations_from_disk` —
  sanitize makes them redundant.
- Unit tests covering each validation failure mode, including stale and corrupt
  global metadata.

**Dependency:** Phase 1.
**Mergeable:** Yes.

### Phase 3: CLI integration

Wire sanitization into `cli/lib.rs`:

- Call `workspace.sanitize()` after workspace construction, before
  `load_conversations_from_disk`.
- Log the sanitization report as warnings when repairs were made.

**Dependency:** Phase 2.
**Mergeable:** Yes.

## References

- [Issue #404][#404] — clean install fails due to unresolvable conversation ID.
- [PR #450][#450] — partial fix for fresh workspace handling.
- [RFD 006] — `ConversationStream::sanitize()`, the event-level precedent for
  this pattern.
- [RFD 020] — parallel conversations and file locking (relevant to concurrent
  access risk).
- [RFD 023] — resumable conversation turns, interaction with `sanitize()`.

[#404]: https://github.com/dcdpr/jp/issues/404
[#450]: https://github.com/dcdpr/jp/pull/450
[RFD 006]: 006-turn-scoped-mutations.md
[RFD 020]: 020-parallel-conversations.md
[RFD 023]: 023-resumable-conversation-turns.md
[RFD 073]: 073-layered-storage-backend-for-workspaces.md
