# Filesystem Storage Patterns

## Overview

The backend uses filesystem-based storage for all document and annotation data, with PostgreSQL used only for user authentication.

All runtime paths are derived from `SemiontProject` (imported from `@semiont/core/node`), which resolves XDG base directories at construction time. No runtime files are written into the project directory.

## Storage Architecture

```
$XDG_STATE_HOME/semiont/{project}/      (~/.local/state/semiont/{project}/)
├── events/                              # Immutable event log (system of record)
│   └── {shard}/
│       └── {resourceId}/
│           └── events-{seq}.jsonl
├── views/                               # Materialized current state
│   └── {shard}/
│       └── {resourceId}.json
└── jobs/                                # Job queue
    ├── pending/
    ├── running/
    ├── complete/
    ├── failed/
    └── cancelled/
```

Resources reference their content via `storageUri` (e.g. `file://README.md`) — Semiont reads files where they live in the working tree rather than copying them into a separate store.

## Event Storage

Events are stored as append-only JSONL files — the system of record for all domain events:

### File Structure
```jsonl
{"event":{"id":"evt-123","type":"yield:created",...},"metadata":{...}}
{"event":{"id":"evt-124","type":"mark:added",...},"metadata":{...}}
```

### Sharding Strategy
- 65,536 shards via jump consistent hash keyed by resource ID
- File rotation at 10,000 events per shard

See [@semiont/event-sourcing](../../packages/event-sourcing/docs/) for implementation details.

## View Storage

Materialized views stored as JSON files — derived from the event log, rebuildable at any time:

### Resource View
```json
{
  "@context": "https://schema.org/",
  "@id": "http://localhost:4000/resources/doc-123",
  "name": "Document Title",
  "storageUri": "file://README.md",
  "annotations": [...]
}
```

## Job Queue Storage

Filesystem-based job queue with atomic state transitions:

### State Transitions
```
pending/ → running/ → complete/
                   ↘ failed/
                   ↘ cancelled/
```

State transitions use file moves for atomicity — no database dependency, no external broker.

See [@semiont/jobs](../../packages/jobs/docs/API.md) for implementation.

## Path Resolution

All paths are resolved through `SemiontProject`:

```typescript
import { SemiontProject } from '@semiont/core/node';

const project = new SemiontProject(projectRoot);

// Event log and views — state dir
project.stateDir   // $XDG_STATE_HOME/semiont/{project}/

// Job queue — also state dir
project.jobsDir    // $XDG_STATE_HOME/semiont/{project}/jobs/

// Generated configs (e.g. envoy.yaml) — config dir
project.configDir  // $XDG_CONFIG_HOME/semiont/{project}/

// Runtime (pid files, sockets) — runtime dir
project.runtimeDir // $XDG_RUNTIME_DIR/semiont/{project}/

// Persistent data (database files) — data dir
project.dataHome   // $XDG_DATA_HOME/semiont/{project}/
```

On macOS, `$XDG_RUNTIME_DIR` is not set by default; the fallback is `$TMPDIR/semiont/{project}/`.

## Backup Strategy

### What to Backup
1. **Critical**: `$XDG_STATE_HOME/semiont/{project}/events/` — complete history, system of record
2. **Optional**: `$XDG_STATE_HOME/semiont/{project}/views/` — can rebuild from events

Views can always be regenerated by replaying the event log:
```bash
semiont rebuild --service backend --environment local
```

## Disaster Recovery

### Rebuild from Events
```typescript
// Rebuild all views from event log
await eventStore.rebuildAllViews();

// Rebuild graph from events
await graphConsumer.rebuildAll();
```

## Monitoring

### Disk Usage
```bash
# Monitor storage growth
du -sh ~/.local/state/semiont/*/

# Check event shard distribution
find ~/.local/state/semiont/my-project/events -name "*.jsonl" | wc -l
```

## Related Documentation

- [Event Store Architecture](../../packages/event-sourcing/docs/ARCHITECTURE.md)
- [Job Queue Patterns](../../packages/jobs/docs/API.md)
- [Database Guide](./administration/DATABASE.md) — User authentication only
- [Configuration Guide](./administration/CONFIGURATION.md) — XDG path configuration
