# do-memory-storage-redb

[![Crates.io](https://img.shields.io/crates/v/do-memory-storage-redb.svg)](https://crates.io/crates/do-memory-storage-redb)
[![Documentation](https://docs.rs/do-memory-storage-redb/badge.svg)](https://docs.rs/do-memory-storage-redb)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Purpose**: High-performance embedded cache layer for do-memory-core

## Overview

`do-memory-storage-redb` implements the high-performance cache layer for AI agent episodic memory using [redb](https://www.redb.org), an embedded key-value database. It provides blazing fast reads for hot-path operations with minimal overhead.

## Features

- **Blazing Fast**: Sub-microsecond read latency for cached data
- **Embedded**: No separate server process required
- **LRU Caching**: Automatic eviction of least recently used entries
- **TTL Support**: Configurable time-to-live for cache entries
- **Safe Concurrency**: Multiple readers with single writer (MVCC)
- **Crash Safe**: ACID guarantees with automatic recovery
- **Zero Copy**: Direct memory mapping for minimal overhead

## Key Modules

| Module | Purpose |
|--------|---------|
| `storage` | Core storage operations and cache management |
| `tables` | Table definitions and schema management |
| `cache` | Cache-specific operations (LRU, TTL, eviction) |

## Installation

Add this to your `Cargo.toml`:

```toml
[dependencies]
do-memory-storage-redb = "0.1"
```

## Quick Start

```rust
use memory_storage_redb::RedbStorage;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create cache with default settings
    let cache = RedbStorage::new("./data/cache.redb")?;

    // Configure cache size (max episodes)
    let cache = RedbStorage::with_config(
        "./data/cache.redb",
        1000,  // max_cache_size
    )?;

    // Cache is ready to use
    Ok(())
}
```

## ⚠️ IMPORTANT: v0.1.7+ Breaking Change (Current: v0.1.13)

### Postcard Serialization (NOT bincode)

**Version v0.1.7 introduced a breaking change**: The serialization format changed from **bincode** to **postcard** (current version: v0.1.13).

### What Changed?
- **Old (< v0.1.7)**: Used `bincode` for serialization
- **New (≥ v0.1.7)**: Uses `postcard` for serialization (v0.1.13)
- **Why**: Postcard provides better memory safety and is more Rust-idiomatic

### Migration Requirements

If you have existing cache files from v0.1.6 or earlier:

```bash
# 1. Backup your existing cache (if needed)
cp ./data/cache.redb ./data/cache.redb.backup

# 2. Clear the cache to force re-build from Turso
rm ./data/cache.redb

# 3. Restart your application
# The cache will be repopulated from Turso storage
```

### No Data Loss

The cache is a **secondary storage layer**. Your primary data is stored in Turso. Clearing the cache simply forces a rebuild from the durable storage layer.

## Configuration

### Basic Configuration

```rust
use memory_storage_redb::{RedbStorage, RedbConfig};

let config = RedbConfig {
    max_cache_size: 1000,        // Maximum episodes to cache
    ttl_seconds: Some(3600),      // 1 hour TTL
    enable_compression: true,     // Enable value compression
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;
```

### Environment Variables

```bash
# Optional: Configure cache settings
export REDB_CACHE_PATH="./data/cache.redb"
export REDB_MAX_CACHE_SIZE="1000"
export REDB_TTL_SECONDS="3600"
```

## Cache Tables

The cache maintains four tables for different data types:

- **episodes**: Full episode data for fast retrieval
- **patterns**: Extracted patterns with similarity keys
- **embeddings**: Vector embeddings (if enabled)
- **metadata**: Cache statistics and health info

## Performance

Optimized for extremely fast read operations:

| Operation | Latency | Throughput |
|-----------|---------|------------|
| Cache Read | < 10µs | 100K+ ops/s |
| Cache Write | < 100µs | 10K+ ops/s |
| LRU Eviction | < 1ms | Background |
| TTL Cleanup | < 10ms | Background |

### Real-world Performance

In production with do-memory-core (v0.1.13):
- **Cache hit rate**: 85-95% (varies by workload)
- **Average read latency**: 5-8µs (cache hits)
- **Average write latency**: 50-80µs
- **Throughput**: 50K+ reads/s, 8K+ writes/s

## LRU Cache Strategy

The cache uses an LRU (Least Recently Used) eviction policy:

1. Cache fills up to `max_cache_size`
2. New entries trigger eviction of oldest entries
3. Access updates entry timestamp
4. TTL expiration runs in background

### LRU Behavior

```rust
// Entries are tracked by last access time
cache.get("episode-123")?;  // Updates access time

// When cache is full, oldest entry is evicted
cache.insert("episode-456", data)?;  // May trigger eviction
```

## TTL Management

Time-to-live (TTL) automatically expires old entries:

```rust
use memory_storage_redb::{RedbConfig, RedbStorage};

let config = RedbConfig {
    ttl_seconds: Some(3600),  // Expire after 1 hour
    ..Default::default()
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;

// TTL cleanup runs every 60 seconds in background
// Expired entries are automatically removed
```

### TTL Settings

| Setting | Description | Default |
|---------|-------------|---------|
| `ttl_seconds` | Time-to-live in seconds | `None` (no TTL) |
| `ttl_check_interval` | Cleanup interval in seconds | 60 |

## Usage with do-memory-core

```rust
use memory_core::SelfLearningMemory;
use memory_storage_turso::TursoStorage;
use memory_storage_redb::RedbStorage;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Turso for durable storage
    let turso = TursoStorage::new(
        std::env::var("TURSO_DATABASE_URL")?,
        std::env::var("TURSO_AUTH_TOKEN")?,
    ).await?;

    // redb for fast caching
    let redb = RedbStorage::new("./data/cache.redb")?;

    // Hybrid storage for best performance
    let memory = SelfLearningMemory::with_storage(turso, redb).await?;

    Ok(())
}
```

## Cache Synchronization

The cache automatically synchronizes with Turso:

- **Write-through**: Writes go to both Turso and redb
- **Read-aside**: Reads check cache first, then Turso
- **Invalidation**: Cache can be refreshed from Turso
- **Reconciliation**: Periodic sync ensures consistency

```rust
// Manual cache refresh (force sync from Turso)
memory.sync_memories().await?;

// Get cache statistics
let stats = cache.get_stats()?;
println!("Cache hit rate: {:.2}%", stats.hit_rate * 100.0);
println!("Total entries: {}", stats.total_entries);
println!("Cache size: {} bytes", stats.cache_size_bytes);
```

## Background Tasks

The cache runs background tasks for maintenance:

- **TTL Cleanup**: Removes expired entries (every 60s)
- **LRU Eviction**: Maintains size limits (on write)
- **Compression**: Optional value compression (configurable)
- **Health Checks**: Monitors cache health (on demand)

### Background Task Configuration

```rust
use memory_storage_redb::{RedbConfig, RedbStorage};

let config = RedbConfig {
    ttl_check_interval: 30,      // Check TTL every 30s
    enable_compression: true,     // Compress values
    compression_threshold: 1024, // Compress values > 1KB
    ..Default::default()
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;
```

## File Management

redb stores all data in a single file:

```bash
# Check cache file size
ls -lh ./data/cache.redb

# Backup cache
cp ./data/cache.redb ./backups/cache-$(date +%Y%m%d).redb

# Clear cache (forces rebuild from Turso)
rm ./data/cache.redb

# Monitor cache growth
watch -n 5 'ls -lh ./data/cache.redb'
```

### Cache File Structure

- **Single file database**: All tables and data in one file
- **Memory mapped**: Direct access for zero-copy reads
- **Crash recovery**: Automatic recovery from unclean shutdown
- **Incremental writes**: Only modified data is written

## Compression

Optional value compression reduces cache size:

```rust
let config = RedbConfig {
    enable_compression: true,          // Enable compression
    compression_threshold: 1024,        // Compress values > 1KB
    compression_level: 6,               // Compression level (0-9)
    ..Default::default()
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;
```

### Compression Benefits

- **Reduced memory usage**: 40-60% smaller cache files
- **Faster disk I/O**: Less data to read/write
- **Trade-off**: Slightly higher CPU usage

## Monitoring & Statistics

### Cache Statistics

```rust
let stats = cache.get_stats()?;
println!("=== Cache Statistics ===");
println!("Hit rate: {:.2}%", stats.hit_rate * 100.0);
println!("Total entries: {}", stats.total_entries);
println!("Cache size: {} bytes", stats.cache_size_bytes);
println!("Reads: {}", stats.total_reads);
println!("Writes: {}", stats.total_writes);
println!("Evictions: {}", stats.total_evictions);
```

### Health Check

```rust
let health = cache.health_check()?;
println!("Cache healthy: {}", health.is_healthy);
println!("Disk available: {} bytes", health.available_disk_space);
println!("Cache file: {} bytes", health.cache_file_size);
```

## Testing

Run tests with an in-memory database:

```bash
cargo test -p do-memory-storage-redb
```

For integration tests with a real database file:

```bash
cargo test -p do-memory-storage-redb -- --ignored
```

## Dependencies

### Core Dependencies
- **redb**: Embedded key-value database
- **tokio**: Async runtime
- **async-trait**: Async trait support
- **anyhow**: Error handling
- **serde**: Serialization framework
- **postcard**: Serialization format (v0.1.7+, current: v0.1.13)

### Breaking Change Dependencies
- **postcard** (v0.1.7+, current: v0.1.13): Replaces bincode for safer serialization

## Documentation

Full API documentation: [docs.rs/do-memory-storage-redb](https://docs.rs/do-memory-storage-redb)

## Best Practices

### Production Deployment
- Enable TTL for automatic cleanup
- Set appropriate `max_cache_size` based on memory constraints
- Monitor cache hit rate to optimize size
- Enable compression for large datasets
- Run periodic health checks

### Development
- Use smaller cache sizes for local development
- Disable compression for easier debugging
- Monitor cache file growth
- Clear cache frequently during development

### Performance Tuning
- Adjust `max_cache_size` to balance memory and hit rate
- Use TTL for time-sensitive data
- Enable compression for memory-constrained environments
- Monitor eviction rate to detect undersized cache

### Cache Size Guidelines

| Workload | Max Cache Size | Expected Hit Rate |
|----------|----------------|-------------------|
| Low traffic | 100-500 | 85-90% |
| Medium traffic | 500-2000 | 90-95% |
| High traffic | 2000-10000 | 95-98% |

## Troubleshooting

### Cache Not Populating
```rust
// Check Turso connection first
let turso = TursoStorage::new(url, token).await?;
let episodes = turso.get_all_episodes().await?;
println!("Turso has {} episodes", episodes.len());

// Force cache sync
memory.sync_memories().await?;
```

### High Eviction Rate
```rust
let stats = cache.get_stats()?;
if stats.total_evictions > stats.total_writes / 2 {
    // Increase cache size
    let config = RedbConfig {
        max_cache_size: stats.total_entries * 2,
        ..Default::default()
    };
}
```

### Low Cache Hit Rate
```rust
let stats = cache.get_stats()?;
if stats.hit_rate < 0.8 {
    // Check if cache is being used correctly
    // Verify do-memory-core is configured to use cache
    println!("Consider increasing cache size or TTL");
}
```

## License

Licensed under the MIT License. See [LICENSE](../LICENSE) for details.

## Project

Part of the [rust-self-learning-memory](https://github.com/d-o-hub/rust-self-learning-memory) project.

## Version History

- **v0.1.13** (Current): Postcard serialization, improved performance
- **v0.1.7**: Postcard serialization introduced
- **v0.1.6**: Initial release with bincode serialization
