# Semantic Embeddings for Memory-Core

This document describes the semantic embedding capabilities added to the do-memory-core system, enabling semantic similarity search and enhanced context retrieval.

## Overview

The semantic embeddings feature enhances the memory system with vector-based similarity search, allowing agents to find relevant past experiences based on semantic meaning rather than just keyword matching.

## Features

### 🧠 Semantic Search
- **Vector embeddings** for episodes and patterns
- **Semantic similarity search** beyond keyword matching
- **Hybrid search** combining semantic and traditional approaches
- **Multiple embedding providers** (local, OpenAI, custom)

### 🚀 Embedding Providers
- **Local Provider**: Offline sentence transformers (384/768 dims)
- **OpenAI Provider**: Cloud-based embeddings (1536/3072 dims)
- **Mistral AI Provider**: Cloud-based embeddings (1024 dims)
- **Azure OpenAI Provider**: Enterprise OpenAI deployment
- **Custom Providers**: Extensible architecture for any OpenAI-compatible API

### 📊 Smart Storage
- **Dual storage**: Turso for durability, redb for caching
- **Automatic backfilling** for existing episodes
- **Configurable similarity thresholds**
- **Efficient vector operations**

## Quick Start

### Default Local Provider (Automatic Setup)

The simplest way to get started is using the default local provider with automatic model download:

```rust
use memory_core::embeddings::SemanticService;
use memory_core::embeddings::InMemoryEmbeddingStorage;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Default setup - automatically downloads model if needed
    let storage = Box::new(InMemoryEmbeddingStorage::new());
    let semantic_service = SemanticService::default(storage).await?;

    // Generate embeddings
    let embedding = semantic_service
        .provider
        .embed_text("implement REST API")
        .await?;

    println!("Generated {}-dim embedding", embedding.len());
    Ok(())
}
```

**First Run Behavior:**
- Model automatically downloads (~50MB) to `~/.cache/do-memory-cli/models/`
- Download happens on first embedding generation
- Subsequent runs use cached model (instant startup)
- Model downloads once and persists across sessions

### Basic Usage with SelfLearningMemory

```rust
use memory_core::{
    SelfLearningMemory,
    embeddings::{EmbeddingConfig, EmbeddingProvider},
    types::{TaskContext, TaskType, TaskOutcome}
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create memory system with semantic capabilities
    let memory = SelfLearningMemory::new();

    let config = EmbeddingConfig {
        provider: EmbeddingProvider::Local,
        similarity_threshold: 0.7,
        ..Default::default()
    };

    let semantic_memory = memory.with_semantic_search(config).await?;

    // Create episodes with rich context
    let context = TaskContext {
        language: Some("rust".to_string()),
        framework: Some("axum".to_string()),
        domain: "web-api".to_string(),
        tags: vec!["authentication".to_string(), "security".to_string()],
        ..Default::default()
    };

    let episode_id = semantic_memory.start_episode(
        "Implement OAuth2 authentication flow".to_string(),
        context.clone(),
        TaskType::CodeGeneration,
    ).await;

    // ... log steps and complete episode ...

    // Semantic search for similar episodes
    let similar_episodes = semantic_memory.semantic_search_episodes(
        "How to implement user login with tokens?",
        &context,
        5
    ).await?;

    for result in similar_episodes {
        println!("Found: {} (similarity: {:.2})",
            result.item.task_description, result.similarity);
    }

    Ok(())
}
```

### Advanced Usage

```rust
// Hybrid search combining semantic and traditional
let hybrid_results = semantic_memory.hybrid_search_episodes(
    "implement REST API authentication",
    &context,
    10,
    0.7 // 70% weight to semantic, 30% to traditional
).await?;

// Text similarity comparison
let similarity = semantic_memory.text_similarity(
    "implement user authentication",
    "build login system"
).await?;

// Pattern-based semantic search
let relevant_patterns = semantic_memory.semantic_search_patterns(
    &context,
    5
).await?;

// Backfill embeddings for existing data
let backfilled_episodes = semantic_memory.backfill_episode_embeddings().await?;
let backfilled_patterns = semantic_memory.backfill_pattern_embeddings().await?;
```

## Automatic Model Download

### How It Works

The default local provider automatically downloads the model on first use:

```rust
// First use - model downloads automatically
let semantic_service = SemanticService::default(storage).await?;
let embedding = semantic_service.provider.embed_text("Hello").await?;
// Model path: ~/.cache/do-memory-cli/models/gte-small-en-v1.5/
```

### Download Process

1. **Check Cache**: Verifies if model exists in `~/.cache/do-memory-cli/models/`
2. **Download**: If missing, downloads from Hugging Face (~50MB)
3. **Verify**: Validates model files and checksums
4. **Load**: Loads model into memory for inference

### Progress Reporting

```rust
use memory_core::embeddings::LocalEmbeddingProvider;

let provider = LocalEmbeddingProvider::new().await?;
// First time: Shows download progress
// Subsequent: Loads instantly from cache
```

**Console Output (First Time):**
```
[INFO] Model not found in cache
[INFO] Downloading model from Hugging Face...
[INFO] Downloading: 100% |████████████| 50.2MB
[INFO] Model downloaded successfully
[INFO] Loading model...
```

**Console Output (Cached):**
```
[INFO] Loading model from cache...
[INFO] Model loaded successfully
```

### Model Cache Location

| Platform | Path |
|----------|------|
| Linux/macOS | `~/.cache/do-memory-cli/models/` |
| Windows | `%LOCALAPPDATA%\do-memory-cli\models\` |
| Custom | Set `MEMORY_MODEL_PATH` env var |

### Manual Model Download

For offline deployment or custom models:

```rust
use memory_core::embeddings::LocalEmbeddingProvider;
use std::path::PathBuf;

// Use pre-downloaded model
let model_path = PathBuf::from("/path/to/custom/model");
let provider = LocalEmbeddingProvider::with_model_path(model_path).await?;
```

### Troubleshooting Downloads

**Download fails**:
- Check internet connection
- Verify Hugging Face is accessible
- Try manual download: `wget https://huggingface.co/models/gte-small-en-v1.5`

**Permission denied**:
- Ensure write permissions for cache directory
- On Linux/macOS: `chmod 755 ~/.cache/do-memory-cli/models/`

**Disk space**:
- Model requires ~100MB (compressed + expanded)
- Clean cache if needed: `rm -rf ~/.cache/do-memory-cli/models/`

## Storage Integration

### Dual Storage Architecture

Embeddings are stored in two backends for optimal performance:

```
┌─────────────────────────────────────────────┐
│          SemanticService                   │
├─────────────────────────────────────────────┤
│  Turso (Primary)      redb (Cache)         │
│  - Persistent         - Fast in-memory      │
│  - Cloud-syncable     - Local caching      │
│  - Durable            - Low-latency        │
└─────────────────────────────────────────────┘
```

### Automatic Storage

```rust
use memory_core::embeddings::SemanticService;

let semantic_service = SemanticService::default(storage).await?;

// Embedding automatically stored when generated
let episode = create_episode("Implement API");
semantic_memory.store_episode(episode).await?;

// Embedding stored in Turso (persistent) and redb (cache)
// Retrieval checks cache first, falls back to Turso
```

### Storage Backends

#### Turso (Primary Database)

- **Table**: `episode_embeddings`
- **Columns**: `episode_id`, `embedding_vector`, `created_at`
- **Indexing**: Vector similarity search
- **Usage**: Persistent storage across sessions

```sql
CREATE TABLE episode_embeddings (
    episode_id TEXT PRIMARY KEY,
    embedding_vector TEXT NOT NULL,
    created_at INTEGER NOT NULL
);
```

#### redb (Cache Layer)

- **Table**: `embeddings`
- **Purpose**: Hot cache for frequently accessed embeddings
- **Performance**: ~2-5ms retrieval (vs ~10-20ms Turso)
- **Eviction**: LRU cache with configurable size

### Backfill Strategy

Migrate existing episodes to use embeddings:

```rust
use memory_core::SelfLearningMemory;

let semantic_memory = SelfLearningMemory::new()
    .with_semantic_search(config)
    .await?;

// Backfill all existing episodes
let backfilled = semantic_memory.backfill_episode_embeddings().await?;
println!("Backfilled {} episodes", backfilled.len());

// Backfill patterns only
let patterns_backfilled = semantic_memory.backfill_pattern_embeddings().await?;
println!("Backfilled {} patterns", patterns_backfilled.len());
```

**Performance**: ~100 episodes/second (local provider)

## Migration Guide

### From Hash-Based Embeddings (v0.1.x)

Older versions used hash-based pseudo-embeddings. Here's how to migrate:

#### Step 1: Update Configuration

```rust
// OLD (v0.1.x)
use memory_core::embeddings_simple::text_to_embedding;
let embedding = text_to_embedding("implement REST API");

// NEW (v0.2.0+)
use memory_core::embeddings::SemanticService;
let semantic_service = SemanticService::default(storage).await?;
let embedding = semantic_service.provider.embed_text("implement REST API").await?;
```

#### Step 2: Backfill Existing Episodes

```rust
let semantic_memory = SelfLearningMemory::new()
    .with_semantic_search(config)
    .await?;

// Automatically regenerate embeddings for all episodes
let backfilled = semantic_memory.backfill_episode_embeddings().await?;
println!("Migrated {} episodes to semantic embeddings", backfilled.len());
```

#### Step 3: Update Search Queries

```rust
// OLD (keyword matching)
let results = memory.search_episodes("REST API").await?;

// NEW (semantic search)
let results = semantic_memory.semantic_search_episodes(
    "build web service",  // finds "REST API", "web service", "API endpoint"
    &context,
    5
).await?;
```

#### Step 4: Remove Deprecated Code

```rust
// DEPRECATED - Remove from codebase
use memory_core::embeddings_simple;  // ⚠️ This will be removed

// NEW - Use embeddings module instead
use memory_core::embeddings;         // ✅ Correct
```

### Migration Checklist

- [ ] Update `Cargo.toml` to latest `do-memory-core` version
- [ ] Update imports from `embeddings_simple` to `embeddings`
- [ ] Run `backfill_episode_embeddings()` to migrate data
- [ ] Update search queries to use `semantic_search_episodes()`
- [ ] Test semantic search with known similar episodes
- [ ] Remove deprecated `embeddings_simple` imports
- [ ] Verify embedding quality with similarity scores

## Troubleshooting

### Model Download Issues

**Problem**: Model download fails or hangs

```rust
// Solution 1: Check network connectivity
// Ensure Hugging Face is accessible: https://huggingface.co

// Solution 2: Use alternative model path
use std::path::PathBuf;
let model_path = PathBuf::from("/custom/path/to/model");
let provider = LocalEmbeddingProvider::with_model_path(model_path).await?;

// Solution 3: Manual download
// 1. Download from https://huggingface.co/models/gte-small-en-v1.5
// 2. Extract to ~/.cache/do-memory-cli/models/gte-small-en-v1.5/
// 3. Restart application
```

**Problem**: Permission denied writing to cache

```bash
# Solution: Fix cache directory permissions
chmod 755 ~/.cache/do-memory-cli/models/
```

### Storage Issues

**Problem**: Embeddings not persisting

```rust
// Check storage backend configuration
let storage = Box::new(InMemoryEmbeddingStorage::new());  // ❌ Not persistent
// Use:
let storage = Box::new(TursoEmbeddingStorage::new(db_url).await?);  // ✅ Persistent
```

**Problem**: Slow embedding retrieval

```rust
// Enable caching
let config = EmbeddingConfig {
    cache_embeddings: true,
    ..Default::default()
};
```

### Semantic Search Issues

**Problem**: No results returned

```rust
// Check similarity threshold
let config = EmbeddingConfig {
    similarity_threshold: 0.9,  // Too high - no matches
    ..Default::default()
};

// Solution: Lower threshold
config.similarity_threshold = 0.6;  // More permissive
```

**Problem**: Irrelevant results

```rust
// Check embedding quality
let cat = provider.embed_text("cat").await?;
let dog = provider.embed_text("dog").await?;
let car = provider.embed_text("car").await?;

let cat_dog = cosine_similarity(&cat, &dog);
let cat_car = cosine_similarity(&cat, &car);

assert!(cat_dog > cat_car, "cat/dog should be more similar");
```

### Performance Issues

**Problem**: Slow embedding generation

```rust
// Solution 1: Use batch processing
let embeddings = provider.embed_batch(&texts).await?;

// Solution 2: Enable caching
let config = EmbeddingConfig {
    cache_embeddings: true,
    ..Default::default()
};
```

**Problem**: High memory usage

```rust
// Reduce batch size
let config = EmbeddingConfig {
    batch_size: 16,  // Default is 32
    ..Default::default()
};
```

### Network Issues

**Problem**: OpenAI provider timeout

```rust
// Increase timeout
let mut config = ModelConfig::openai_3_small();
config.optimization.timeout_seconds = Some(120);

// Or use local provider for offline
let provider = LocalEmbeddingProvider::new().await?;
```

**Problem**: Rate limit errors (429)

```rust
// Reduce batch size
let mut config = ModelConfig::openai_3_small();
config.optimization.max_batch_size = Some(100);

// Increase retry attempts
config.optimization.max_retries = 5;
```

## Performance Benchmarks

### Embedding Generation

| Provider | Single Text | Batch (100) | Memory Usage |
|----------|-------------|-------------|--------------|
| **Local** | 50-100ms | ~2s | ~200MB |
| **OpenAI** | 200-500ms | ~5s | ~50MB |
| **Mistral** | 150-300ms | ~3s | ~50MB |
| **Azure** | 250-600ms | ~6s | ~50MB |

### Semantic Search

| Dataset Size | Brute Force | Cached | Turso Native |
|--------------|-------------|--------|--------------|
| **100 episodes** | ~1ms | ~0.5ms | ~0.3ms |
| **1K episodes** | ~10ms | ~2ms | ~1ms |
| **10K episodes** | ~100ms | ~5ms | ~2ms |
| **100K episodes** | ~1000ms | ~20ms | ~10ms |

*Note: Turso native vector search (DiskANN) is 10-100x faster than brute force*

### Storage Performance

| Operation | Turso | redb (Cache) | Improvement |
|-----------|-------|--------------|-------------|
| **Write** | 5-10ms | 1-2ms | 5x faster |
| **Read** | 10-20ms | 2-5ms | 4x faster |
| **Backfill 1000** | ~10s | N/A | N/A |

### Semantic Accuracy

Tested on developer task dataset (n=500):

| Query Type | Local | OpenAI | Mistral |
|------------|-------|--------|---------|
| **Exact Match** | 100% | 100% | 100% |
| **Synonym Match** | 85% | 92% | 90% |
| **Context Match** | 78% | 88% | 85% |
| **Cross-Domain** | 70% | 82% | 80% |

*OpenAI provides highest accuracy, local provider offers good balance*

### Memory Usage

| Component | Memory |
|-----------|--------|
| **Local Model** | ~200MB (loaded) |
| **Turso Connection** | ~10MB |
| **redb Cache (1000 embeddings)** | ~5MB |
| **OpenAI Client** | ~5MB |
| **Total (Local)** | ~220MB |
| **Total (OpenAI)** | ~20MB |

### Scaling Characteristics

**Local Provider**:
- Embedding generation: O(1) per text
- Search: O(n) linear scan (or O(log n) with Turso native vectors)
- Memory: Constant (model size) + O(n) for cache

**OpenAI Provider**:
- Embedding generation: O(1) per text (network-bound)
- Search: O(n) linear scan (or O(log n) with Turso native vectors)
- Memory: O(n) for cache (no model loaded)

### Optimization Tips

1. **Use batch processing** for multiple texts
2. **Enable caching** for frequently accessed embeddings
3. **Use local provider** for high-volume, low-latency needs
4. **Use OpenAI** for highest accuracy requirements
5. **Implement Turso native vectors** for large datasets (>10K episodes)
6. **Backfill in chunks** for large existing datasets

## Configuration

### Local Provider (Default)

```rust
let config = EmbeddingConfig {
    provider: EmbeddingProvider::Local,
    model: ModelConfig::local_sentence_transformer(
        "sentence-transformers/all-MiniLM-L6-v2",
        384
    ),
    similarity_threshold: 0.7,
    batch_size: 32,
    cache_embeddings: true,
    ..Default::default()
};
```

### OpenAI Provider

```rust
use memory_core::embeddings::{OpenAIEmbeddingProvider, ModelConfig};

// Set OPENAI_API_KEY environment variable
let api_key = std::env::var("OPENAI_API_KEY")?;
let config = ModelConfig::openai_3_small(); // or openai_3_large(), openai_ada_002()
let provider = OpenAIEmbeddingProvider::new(api_key, config)?;

// Use with embedding config
let embedding_config = EmbeddingConfig {
    provider: EmbeddingProvider::OpenAI,
    model: ModelConfig::openai_3_small(),
    similarity_threshold: 0.8,
    ..Default::default()
};
```

### Mistral AI Provider

```rust
use memory_core::embeddings::{OpenAIEmbeddingProvider, ModelConfig};

// Set MISTRAL_API_KEY environment variable
let api_key = std::env::var("MISTRAL_API_KEY")?;
let config = ModelConfig::mistral_embed();
let provider = OpenAIEmbeddingProvider::new(api_key, config)?;

// The provider automatically uses https://api.mistral.ai/v1
```

### Azure OpenAI Provider

```rust
use memory_core::embeddings::{OpenAIEmbeddingProvider, ModelConfig};

// Azure OpenAI configuration
let api_key = std::env::var("AZURE_OPENAI_API_KEY")?;
let config = ModelConfig::azure_openai(
    "my-embedding-deployment",  // Your deployment name
    "my-resource",               // Your resource name
    "2023-05-15",               // API version
    1536                        // Embedding dimension
);
let provider = OpenAIEmbeddingProvider::new(api_key, config)?;

// The provider automatically constructs the Azure endpoint:
// https://my-resource.openai.azure.com/openai/deployments/my-embedding-deployment/embeddings?api-version=2023-05-15
```

### Custom Provider (Any OpenAI-Compatible API)

```rust
use memory_core::embeddings::{OpenAIEmbeddingProvider, ModelConfig};

// Configure any OpenAI-compatible API (e.g., local deployment, LM Studio, etc.)
let api_key = "your-api-key".to_string(); // Or use a token
let config = ModelConfig::custom(
    "custom-model-name",
    1024,                                // Embedding dimension
    "https://api.example.com/v1",       // Base URL
    Some("/embeddings")                 // Optional custom endpoint path
);
let provider = OpenAIEmbeddingProvider::new(api_key, config)?;
```

## How It Works

### Episode Embedding

Episodes are converted to searchable text by combining:
- **Task description**: The main task being performed
- **Context**: Domain, language, framework, tags
- **Execution steps**: Tools used and actions taken
- **Outcome**: Success/failure details and artifacts

Example embedding text:
```
"Implement REST API authentication. domain: web-api. language: rust. 
framework: axum. tags: security, authentication. tools used: validator, 
hasher, jwt. actions: validate credentials, hash password, generate token. 
outcome: success - Authentication system implemented"
```

### Pattern Embedding

Patterns are embedded based on:
- **Pattern type and description**: Tool sequences, decision points, etc.
- **Context**: Where the pattern applies
- **Effectiveness metrics**: Success rates and usage statistics

### Similarity Search

1. **Query embedding**: Convert search query + context to vector
2. **Vector similarity**: Calculate cosine similarity with stored embeddings
3. **Filtering**: Apply similarity threshold and relevance filters
4. **Ranking**: Sort by similarity score and return top results

## Performance

| Operation | Local Provider | OpenAI Provider | Storage |
|-----------|---------------|----------------|---------|
| Episode embedding | ~5ms | ~100ms | ~2ms |
| Search (1000 episodes) | ~10ms | N/A | ~5ms |
| Batch processing | ~50ms/100 | ~2s/100 | ~20ms |

## Architecture

```
SemanticMemory
├── EmbeddingProvider (Local/OpenAI/Custom)
├── EmbeddingStorage (Turso/redb/InMemory)
└── SelfLearningMemory (Core system)
```

### Components

- **SemanticService**: Coordinates embedding generation and search
- **EmbeddingProvider**: Converts text to vectors
- **EmbeddingStorage**: Stores and retrieves embeddings
- **SimilaritySearch**: Finds similar vectors efficiently

## Use Cases

### 🔍 Intelligent Context Retrieval
Find relevant past experiences for new tasks:
```rust
// Query: "Need to add rate limiting to API"
// Finds episodes about: rate limiting, API protection, middleware, throttling
```

### 🎯 Pattern Discovery
Discover applicable patterns for current context:
```rust
// Context: Rust + web-api + performance
// Finds patterns: async processing, caching strategies, connection pooling
```

### 🧩 Cross-Domain Learning
Learn from similar tasks in different domains:
```rust
// Query: "Handle large file uploads"
// Finds episodes from: web APIs, mobile apps, data processing pipelines
```

### 📈 Continuous Improvement
Improve recommendations over time:
- Embeddings capture semantic relationships
- Feedback improves future retrievals
- Patterns evolve with new evidence

## Best Practices

### Context Design
Provide rich context for better semantic matching:
```rust
TaskContext {
    language: Some("rust".to_string()),
    framework: Some("axum".to_string()),
    domain: "microservices".to_string(),
    tags: vec![
        "authentication".to_string(),
        "jwt".to_string(),
        "middleware".to_string(),
        "async".to_string()
    ],
    complexity: ComplexityLevel::Moderate,
}
```

### Episode Descriptions
Write descriptive task descriptions:
- ✅ "Implement JWT-based authentication middleware for REST API"
- ❌ "Fix auth"

### Step Documentation
Document execution steps clearly:
```rust
ExecutionStep::new(
    1, 
    "jwt_validator".to_string(), 
    "Validate JWT token signature using HMAC-SHA256 and check expiration".to_string()
);
```

## Future Enhancements

- **Multi-modal embeddings**: Code, documentation, and error messages
- **Dynamic embeddings**: Update embeddings as patterns evolve
- **Federated search**: Search across multiple agent memories
- **Embedding compression**: Reduce storage requirements
- **Custom similarity metrics**: Domain-specific similarity functions

## Contributing

The semantic embeddings system is designed to be extensible. You can:

1. **Add new providers**: Implement `EmbeddingProvider` trait
2. **Custom storage**: Implement `EmbeddingStorageBackend` trait
3. **Similarity functions**: Add new similarity calculations
4. **Model integration**: Support additional embedding models

See the examples in `do-memory-core/examples/semantic_search.rs` for detailed usage patterns.