# Data Persistence and Storage Guide

SousChef v3.6+ includes comprehensive data persistence capabilities to store analysis history, conversion results, and generated assets across container restarts.

## Features

### Persistent Storage
- **SQLite Database**: Stores analysis results, conversion history, and metadata
- **PostgreSQL Database**: Production-grade relational database for shared deployments
- **Blob Storage**: Stores generated Ansible playbooks, roles, and supporting files
- **Cache Layer**: Automatically caches AI analysis results to save costs and time
- **Historical Tracking**: View and download past analyses and conversions

### Database Schema

The persistence layer uses SQLite or PostgreSQL with the following tables:

#### Analysis Results
- Cookbook name, version, and metadata
- Complexity ratings and effort estimates
- AI provider and model used
- Full analysis data (JSON)
- Cache keys for efficient lookups
- **Source archive reference** (v3.7+): Blob storage key for uploaded cookbook
- Timestamps

#### Conversion Results
- Output type (playbook/role/collection)
- Status (success/partial/failed)
- Files generated count
- Blob storage reference
- Full conversion data (JSON)
- Links to original analysis

### Storage Backends

#### Local Storage (Default)
By default, SousChef stores data under a `.souschef` directory inside the system temporary directory (for example `/tmp/.souschef` on Linux/macOS):
- `<system-temp>/.souschef/data/souschef.db` - SQLite database
- `<system-temp>/.souschef/storage/` - Generated artefacts

```python
from souschef.storage import get_storage_manager, get_blob_storage

# Uses default local storage (system temporary directory, e.g. /tmp/.souschef)
storage = get_storage_manager()
blob = get_blob_storage("local")
```

#### S3-Compatible Storage
Use AWS S3, MinIO, or any S3-compatible service:

```python
from souschef.storage import get_blob_storage

# AWS S3
blob = get_blob_storage(
    backend="s3",
    bucket_name="souschef-artefacts",
    region="us-east-1"
    # access_key and secret_key from environment or explicit
)

# MinIO
blob = get_blob_storage(
    backend="s3",
    bucket_name="souschef",
    endpoint_url="http://minio:9000",
    access_key="admin",
    secret_key="password"
)
```

## Using the History Page

The History page in the UI provides comprehensive access to all past analyses and conversions.

### Analysis History
1. Navigate to **History** in the main navigation
2. Click the **Analysis History** tab
3. Filter by cookbook name (optional)
4. View detailed metrics:
   - Complexity ratings
   - Manual vs AI-assisted effort estimates
   - Time saved
   - AI provider/model used
5. Click on individual analyses to see full details and recommendations

**Source Archive Storage** (v3.7+):
- When you upload a cookbook archive (tar/zip), it's automatically stored alongside the analysis
- Stored archives enable re-conversion without re-uploading
- Archives are referenced by `cookbook_blob_key` in the database
- Future versions will enable one-click re-conversion from stored archives

### Conversion History
1. Navigate to **History** in the main navigation
2. Click the **Conversion History** tab
3. Filter by cookbook name or status
4. View conversion details:
   - Success/failure status
   - Files generated
   - Output types (playbook/role/collection)
5. Download previously generated artefacts with one click

**Automatic Archive Creation** (v3.6+):
- Conversion artefacts are automatically stored as tar/zip archives during conversion
- Both roles and repository archives are saved to blob storage
- Archives are immediately available for download from the History page
- No need to regenerate - just click to download pre-created archives

### Statistics Dashboard
View overall metrics:
- Total analyses performed
- Unique cookbooks analysed
- Total conversions and success rate
- Average time saved with AI assistance
- Total files generated

## Automated Caching

SousChef automatically caches AI analysis results to:
- **Reduce costs**: Avoid re-running expensive AI analyses
- **Improve speed**: Serve cached results instantly
- **Maintain consistency**: Same cookbook + AI settings = cached result

### Cache Invalidation
Cache is automatically invalidated when:
- Cookbook content changes (detected via content hash)
- Different AI provider or model is used
- Manual cache clear (future feature)

### Cache Keys
Cache keys are generated from:
```
SHA256(cookbook_path + ai_provider + ai_model + content_hash)
```

## Mass Analysis and Conversion

### Batch Analysis
For entire Chef estates:

1. **Prepare Archive**:
   ```bash
   # Create archive of all cookbooks
   tar czf chef_estate.tar.gz /path/to/cookbooks/*
   ```

2. **Upload to SousChef UI**:
   - Navigate to Cookbook Analysis
   - Upload the archive
   - Click "Analyse ALL Cookbooks"

3. **View Results**:
   - Results are automatically saved to database
   - Access anytime from History page
   - Download comprehensive reports

### Parallel Processing
For large estates, SousChef processes cookbooks in parallel:
- Checks cache before analysis (instant for cached cookbooks)
- Analyses new cookbooks in batches
- Saves results incrementally
- Shows progress for long-running operations

## API Usage

### Saving Analysis Results

```python
from souschef.storage import get_storage_manager

storage = get_storage_manager()

# Save analysis
analysis_id = storage.save_analysis(
    cookbook_name="nginx",
    cookbook_path="/cookbooks/nginx",
    cookbook_version="1.0.0",
    complexity="Medium",
    estimated_hours=16.0,
    estimated_hours_with_souschef=8.0,
    recommendations="Use Ansible Galaxy nginx role...",
    analysis_data={"details": "..."},
    ai_provider="anthropic",
    ai_model="claude-3-5-sonnet-20241022",
    cookbook_blob_key="cookbooks/nginx/nginx-1.0.0.tar.gz"  # Optional: v3.7+
)
```

### Retrieving Cached Results

```python
# Check for cached analysis
cached = storage.get_cached_analysis(
    "/cookbooks/nginx",
    ai_provider="anthropic",
    ai_model="claude-3-5-sonnet-20241022"
)

if cached:
    print(f"Found cached analysis from {cached.created_at}")
    print(f"Complexity: {cached.complexity}")
    print(f"Recommendations: {cached.recommendations}")
```

### Saving Conversion Results

```python
# Save conversion with blob storage reference
# Note: In UI mode (v3.6+), this happens automatically during conversion
conversion_id = storage.save_conversion(
    cookbook_name="nginx",
    output_type="role",
    status="success",
    files_generated=12,
    conversion_data={
        "parsed_result": {...},
        "roles_blob_key": "conversions/nginx/roles_20250202_123456",
        "repo_blob_key": "conversions/nginx/repo_20250202_123500",
        "timestamp": "20250202_123456"
    },
    analysis_id=analysis_id,
    blob_storage_key="conversions/nginx/roles_20250202_123456"
)
```

**Automatic Conversion Storage (UI)**:
- When you convert cookbooks in the UI, artefacts are automatically:
  1. Uploaded to blob storage as tar/zip archives
  2. Saved to database with blob storage keys
  3. Made available for download from the History page
- Both roles and repository archives are stored separately
- Archives include timestamps for version tracking

### Uploading to Blob Storage

```python
from pathlib import Path
from souschef.storage import get_blob_storage

blob = get_blob_storage("local")

# Upload directory (automatically zipped)
storage_key = blob.upload(
    Path("/tmp/nginx_role"),
    "conversions/nginx_20250202_123456"
)
print(f"Uploaded to: {storage_key}")

# Download later
blob.download(
    storage_key,
    Path("/tmp/restored_nginx_role")
)
```

## Environment Configuration

Configure storage via environment variables:

```bash
# Default local storage (no config needed)

# Default SQLite database (no config needed)

# PostgreSQL database
export SOUSCHEF_DB_BACKEND=postgres
export SOUSCHEF_DB_HOST=postgres
export SOUSCHEF_DB_PORT=5432
export SOUSCHEF_DB_NAME=souschef
export SOUSCHEF_DB_USER=souschef
export SOUSCHEF_DB_PASSWORD=souschef
export SOUSCHEF_DB_SSLMODE=require
# For local development against a PostgreSQL instance without TLS, you may override:
# export SOUSCHEF_DB_SSLMODE=disable

# For S3 storage
export SOUSCHEF_STORAGE_BACKEND=s3
export SOUSCHEF_S3_BUCKET=souschef-artefacts
export SOUSCHEF_S3_REGION=us-east-1
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret

# For MinIO
export SOUSCHEF_STORAGE_BACKEND=s3
export SOUSCHEF_S3_BUCKET=souschef
export SOUSCHEF_S3_ENDPOINT=http://minio:9000
export SOUSCHEF_S3_ACCESS_KEY=admin
export SOUSCHEF_S3_SECRET_KEY=password
```

## Docker Deployment

### Persistent Volumes

Mount volumes to persist data across container restarts:

```yaml
# docker-compose.yml
version: '3.8'

services:
  souschef-ui:
    image: souschef-ui:latest
    volumes:
      # Persist database and local storage
      - souschef-data:/tmp/.souschef/data
      - souschef-storage:/tmp/.souschef/storage
      # Optional: mount S3 credentials
      - ~/.aws:/root/.aws:ro
    environment:
      - SOUSCHEF_DB_BACKEND=sqlite
      - SOUSCHEF_STORAGE_BACKEND=local
    ports:
      - "8501:8501"

volumes:
  souschef-data:
    driver: local
  souschef-storage:
    driver: local
```

### With PostgreSQL and MinIO

```yaml
version: '3.8'

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: souschef
      POSTGRES_USER: souschef
      POSTGRES_PASSWORD: souschef
    volumes:
      - postgres-data:/var/lib/postgresql/data

  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"

  souschef-ui:
    image: souschef-ui:latest
    depends_on:
      - postgres
      - minio
    environment:
      - SOUSCHEF_DB_BACKEND=postgres
      - SOUSCHEF_DB_HOST=postgres
      - SOUSCHEF_DB_PORT=5432
      - SOUSCHEF_DB_NAME=souschef
      - SOUSCHEF_DB_USER=souschef
      - SOUSCHEF_DB_PASSWORD=souschef
      - SOUSCHEF_STORAGE_BACKEND=s3
      - SOUSCHEF_S3_BUCKET=souschef
      - SOUSCHEF_S3_ENDPOINT=http://minio:9000
      - SOUSCHEF_S3_ACCESS_KEY=minioadmin
      - SOUSCHEF_S3_SECRET_KEY=minioadmin
    volumes:
      - souschef-data:/tmp/.souschef/data
      - souschef-storage:/tmp/.souschef/storage
    ports:
      - "8501:8501"

volumes:
  minio-data:
  postgres-data:
  souschef-data:
  souschef-storage:
```

## Best Practices

### 1. Regular Backups
```bash
# Backup SQLite database
cp ~/.souschef/data/souschef.db ~/backups/souschef_$(date +%Y%m%d).db

# For S3, enable versioning on the bucket
aws s3api put-bucket-versioning \
  --bucket souschef-artefacts \
  --versioning-configuration Status=Enabled
```

### 2. Cache Management
- Cache grows over time - monitor disk usage
- Cached analyses are automatically reused
- Different AI models create separate cache entries

### 3. Large Estates
- Upload cookbooks in batches if possible
- Use S3 storage for large numbers of conversions
- Enable parallel processing for faster analysis

### 4. Cost Optimisation
- Caching dramatically reduces AI API costs
- Analyse similar cookbooks in the same session
- Review cached results before re-analysing

## Troubleshooting

### Database Locked Errors
SQLite may show "database is locked" under heavy concurrent use:
```python
# Configure in storage manager (future enhancement)
# For now, retry operations or use PostgreSQL backend
```

### Storage Path Errors
Ensure the storage directory is writable:
```bash
# Check permissions
ls -la ~/.souschef/
chmod 700 ~/.souschef/
```

### S3 Connection Issues
Verify credentials and connectivity:
```python
from souschef.storage import get_blob_storage

try:
    blob = get_blob_storage("s3", bucket_name="test-bucket")
    keys = blob.list_keys()
    print(f"Connected successfully, found {len(keys)} objects")
except Exception as e:
    print(f"Connection failed: {e}")
```

### Cache Not Working
Verify cache key generation:
```python
from souschef.storage import get_storage_manager

storage = get_storage_manager()
cache_key = storage.generate_cache_key(
    "/path/to/cookbook",
    ai_provider="anthropic",
    ai_model="claude-3-5-sonnet-20241022"
)
print(f"Cache key: {cache_key}")

# Check if cached entry exists
cached = storage.get_cached_analysis(
    "/path/to/cookbook",
    ai_provider="anthropic",
    ai_model="claude-3-5-sonnet-20241022"
)
print(f"Cached: {cached is not None}")
```

## Future Enhancements

Planned features for upcoming releases:

- [ ] Manual cache invalidation controls
- [ ] Export/import database snapshots
- [ ] Azure Blob Storage support
- [ ] Google Cloud Storage support
- [ ] Retention policies for old analyses
- [ ] Data deduplication for similar cookbooks
- [ ] Full-text search across analysis history
- [ ] Collaborative analysis sharing
- [ ] Enhanced PostgreSQL features (connection pooling, replication)

## Migration from Previous Versions

If you're upgrading from SousChef v3.5 or earlier:

1. **No migration needed**: The database is created automatically
2. **Existing data**: Previous session state data is lost (by design)
3. **New analyses**: Will be automatically persisted going forward
4. **Re-analyse**: Consider re-analysing important cookbooks to populate history

---

For more information, see:
- [Architecture Documentation](../ARCHITECTURE.md)
- [API Reference](../api-reference/server.md)
- [Docker Deployment Guide](../docker-deployment.md)
