src — database

Module: src-database Cohesion: 0.80 Members: 0

src — database

The src/database module is the core persistence layer for Code-Buddy, providing robust and efficient data storage using SQLite. It manages the database connection, schema, migrations, and offers a structured way to interact with various data entities like memories, sessions, code embeddings, analytics, and a general-purpose cache.

This documentation aims to provide developers with a comprehensive understanding of the module's architecture, key components, and how to effectively use and contribute to it.

1. Module Overview

The src/database module centralizes all data storage for Code-Buddy. It replaces previous file-based JSON storage with a single, transactional SQLite database, offering improved performance, data integrity, and query capabilities.

Key Features:

2. Architecture

The database module is structured in layers, promoting separation of concerns and maintainability. 

graph TD
    subgraph Application Layer
        OtherModules[Other Code-Buddy Modules]
    end

    subgraph Database Module
        DI[DatabaseIntegration]
        subgraph Repositories
            MR[MemoryRepository]
            SR[SessionRepository]
            AR[AnalyticsRepository]
            ER[EmbeddingRepository]
            CR[CacheRepository]
        end
        DM[DatabaseManager]
        S[Schema & Migrations]
    end

    subgraph External Dependencies
        EP[EmbeddingProvider]
        PL[PersistentLearning]
        PA[PersistentAnalytics]
    end

    OtherModules --> DI
    DI -- orchestrates --> Repositories
    DI -- uses --> DM
    DI -- interacts with --> EP
    DI -- interacts with --> PL
    DI -- interacts with --> PA

    Repositories -- uses --> DM
    DM -- manages --> S

Explanation of Layers:

3. Key Components

3.1. DatabaseManager (src/database/database-manager.ts)

The DatabaseManager is the central hub for all database operations. It's a singleton class that ensures only one connection to the SQLite database is active at any time.

Responsibilities:

Usage: 

import { getDatabaseManager, initializeDatabase } from './database-manager.js';

async function setupDatabase() {
  const dbManager = await initializeDatabase({ inMemory: true }); class="hl-cmt">// Or specify dbPath
  console.log('Database initialized:', dbManager.isInitialized());

  const stats = dbManager.getDatabaseStats();
  console.log(dbManager.formatStats());

  class="hl-cmt">// Get the raw database instance for direct queries (use with caution)
  const db = dbManager.getDatabase();
  db.exec('CREATE TABLE IF NOT EXISTS test (id INTEGER PRIMARY KEY)');

  dbManager.close();
}

3.2. schema.ts

This file defines the entire database schema using SQL CREATE TABLE statements and corresponding TypeScript interfaces. It's the single source of truth for the database structure.

Key Contents:

Contribution Note: When adding new tables or modifying existing ones, update SCHEMA_VERSION, add a new entry to MIGRATIONS with the necessary ALTER TABLE or CREATE TABLE statements, and update/add corresponding TypeScript interfaces.

3.3. DatabaseMigration (src/database/migration.ts)

The DatabaseMigration class is responsible for migrating data from Code-Buddy's legacy JSON file-based storage to the new SQLite database. This is a critical step for users upgrading from older versions.

Responsibilities:

Migration Flow:

  1. initializeDatabaseIntegration() (or initializeDatabaseSystem()) calls needsMigration().
  2. If migration is needed, runMigration() is invoked.
  3. DatabaseMigration.migrate() orchestrates the migration of:

  1. Each migration step uses the appropriate repository (MemoryRepository, SessionRepository, etc.) to insert the data.

Usage: 

import { needsMigration, runMigration, getMigrationStatus } from './migration.js';

async function performMigrationIfNeeded() {
  if (needsMigration()) {
    console.log('Legacy data detected. Starting migration...');
    const status = getMigrationStatus();
    console.log('Files to migrate:', status.files.filter(f => f.exists).map(f => f.path));

    const migrationResult = await runMigration({ verbose: true, deleteAfterMigration: true });
    if (migrationResult.success) {
      console.log('Migration complete:', migrationResult.migratedItems);
    } else {
      console.error('Migration failed:', migrationResult.errors);
    }
  } else {
    console.log('No migration needed.');
  }
}

3.4. Repositories (src/database/repositories/)

The repositories directory contains specialized classes for interacting with specific data entities. Each repository provides a clear API for CRUD operations and domain-specific queries. They all depend on DatabaseManager to get the underlying better-sqlite3 instance.

Common Patterns:

3.4.1. MemoryRepository (memory-repository.ts)

Manages Memory entities, which represent persistent knowledge.

3.4.2. SessionRepository (session-repository.ts)

Manages Session and Message entities, representing conversation history.

3.4.3. AnalyticsRepository (analytics-repository.ts)

Manages Analytics, ToolStats, and RepairLearning entities, tracking usage and learning.

3.4.4. EmbeddingRepository (embedding-repository.ts)

Manages CodeEmbedding entities, specifically for indexing and searching code.

3.4.5. CacheRepository (cache-repository.ts)

Provides a general-purpose key-value cache with optional Time-To-Live (TTL) and semantic search capabilities.

3.5. DatabaseIntegration (src/database/integration.ts)

The DatabaseIntegration class acts as a high-level facade, simplifying interactions with the database system for other parts of Code-Buddy. It orchestrates calls to multiple repositories and integrates with external modules.

Responsibilities:

Usage: 

import { getDatabaseIntegration, initializeDatabaseIntegration } from './integration.js';

async function useDatabaseIntegration() {
  const dbIntegration = await initializeDatabaseIntegration({
    dbPath: './my-codebuddy.db',
    autoMigrate: true,
    embeddingProvider: 'local',
  });

  class="hl-cmt">// Memory operations
  const memory = await dbIntegration.addMemory('How to write a good commit message', {
    type: 'instruction',
    generateEmbedding: true,
  });
  const searchResults = await dbIntegration.searchMemories('commit best practices', { semantic: true, limit: 3 });

  class="hl-cmt">// Session operations
  const session = dbIntegration.createSession({ name: 'Refactoring session' });
  dbIntegration.addMessage(session.id, 'user', 'Refactor this code...');

  class="hl-cmt">// Cache operations
  const cachedValue = await dbIntegration.getOrComputeCached('my_expensive_computation', async () => {
    class="hl-cmt">// Simulate expensive computation
    await new Promise(resolve => setTimeout(resolve, 100));
    return { data: 'computed result' };
  }, 60000); class="hl-cmt">// Cache for 1 minute

  class="hl-cmt">// Code embedding operations
  await dbIntegration.indexCodeChunk('my-project-id', 'src/main.ts', 0, 'function foo() { ... }', {
    symbolType: 'function',
    symbolName: 'foo',
  });

  console.log(dbIntegration.formatStats());
}

4. Usage Patterns

4.1. Initializing the Database System

The recommended way to initialize the entire database system is through the initializeDatabaseSystem function from src/database/index.ts. This ensures the DatabaseManager is set up, and all repositories are ready. 

import { initializeDatabaseSystem } from '../database/index.js';

async function startApp() {
  await initializeDatabaseSystem({
    dbPath: process.env.CODEBUDDY_DB_PATH || undefined,
    inMemory: process.env.NODE_ENV === 'test',
    verbose: process.env.DEBUG_DB === 'true',
  });
  console.log('Database system ready.');
  class="hl-cmt">// Now you can safely access repositories or DatabaseIntegration
}

4.2. Accessing Repositories

Once initialized, you can get singleton instances of any repository: 

import { getMemoryRepository, getSessionRepository } from '../database/index.js';

const memoryRepo = getMemoryRepository();
const sessionRepo = getSessionRepository();

const memories = memoryRepo.find({ type: 'fact' });
const sessions = sessionRepo.getRecentSessions(5);

4.3. Using DatabaseIntegration

For higher-level operations that might span multiple data types or involve external services (like embedding generation), use DatabaseIntegration: 

import { getDatabaseIntegration } from '../database/index.js';

const dbIntegration = getDatabaseIntegration(); class="hl-cmt">// Assumes initializeDatabaseSystem was called

async function performComplexOperation() {
  const query = 'how to handle errors in async functions';
  const searchResults = await dbIntegration.searchMemories(query, { semantic: true, limit: 5 });
  console.log('Semantic memory search results:', searchResults);

  await dbIntegration.recordToolUsage('code_fixer', true, 1500, false, 'my-project-id');
}

4.4. Resetting for Testing

For unit and integration tests, it's crucial to reset the database state between tests. The module provides resetDatabaseSystem() for this purpose. 

import { initializeDatabaseSystem, resetDatabaseSystem } from '../database/index.js';

describe('Database tests', () => {
  beforeEach(async () => {
    class="hl-cmt">// Ensure a clean in-memory database for each test
    await initializeDatabaseSystem({ inMemory: true });
  });

  afterEach(() => {
    resetDatabaseSystem(); class="hl-cmt">// Clears all singletons and closes DB
  });

  test('should add and retrieve a memory', async () => {
    const dbIntegration = getDatabaseIntegration();
    const memory = await dbIntegration.addMemory('Test content', { type: 'fact' });
    expect(memory.content).toBe('Test content');
  });
});

5. Data Model (Schema)

The src/database/schema.ts file defines the following key tables:

All tables include created_at and updated_at timestamps where appropriate, and many use TEXT fields for JSON storage of metadata. Embeddings are stored as BLOB (binary data) and deserialized into Float32Array by the repositories.

6. Integration Points

The src/database module is a foundational component, integrated across various parts of the Code-Buddy application:

7. Contributing

When contributing to the src/database module, keep the following guidelines in mind:

  1. Schema Changes:

  1. Repository Logic:

  1. DatabaseIntegration:

  1. Singleton Access:

  1. Error Handling:

By adhering to these guidelines, we can maintain a robust, scalable, and easy-to-understand database layer for Code-Buddy.