src — sync

Module: src-sync Cohesion: 0.80 Members: 0

src — sync

The src/sync module provides comprehensive capabilities for managing data synchronization and backup, primarily focusing on cloud integration but also offering a generic state synchronization framework. It is designed to ensure data consistency, handle conflicts, and provide robust backup and restore functionalities for application data.

This module is divided into two main parts:

  1. src/sync/cloud: Focuses on file-level synchronization and backup with various cloud storage providers (S3, GCS, Azure, Local).
  2. src/sync/index.ts: Offers a more abstract, state-based synchronization mechanism using vector clocks for conflict detection and resolution, suitable for synchronizing arbitrary application data structures.

1. Cloud Synchronization and Backup (src/sync/cloud)

This sub-module provides high-level APIs for managing file-based synchronization and backups to and from cloud storage. It abstracts away the complexities of interacting with different cloud providers, handling encryption, compression, and conflict resolution.

1.1 Architecture Overview

The cloud sync and backup system is built around a few core components: 

graph TD
    A[createCloudSyncSystem] --> B(CloudSyncManager)
    A --> C(BackupManager)
    B --> D(CloudStorage)
    C --> D
    D --> E{CloudConfig}
    E --> F(LocalStorage)
    E --> G(S3Storage)
    E --> H(GCSStorage)
    E --> I(AzureBlobStorage)

1.2 Cloud Storage Abstraction (src/sync/cloud/storage.ts)

The CloudStorage class provides a unified interface for interacting with various cloud storage services. It handles common concerns like client-side encryption and key derivation.

Key Features:

Implementations:

1.3 Cloud Sync Manager (src/sync/cloud/sync-manager.ts)

The CloudSyncManager is responsible for synchronizing local files and directories with cloud storage. It supports automatic synchronization, various sync directions, and conflict resolution strategies.

Configuration (SyncManagerConfig, SyncConfig, SyncItem):

Core Synchronization Flow (sync() method):

  1. State Management: Updates internal SyncState and emits sync_started event.
  2. Item Iteration: Loops through all enabled SyncItems.
  3. Delta Calculation (calculateDelta):

  1. Execution based on direction:

  1. Conflict Resolution (resolveConflict): Applies the configured conflictResolution strategy. For 'manual', conflicts are left for external handling via resolveConflictManually.
  2. Data Transfer (uploadFiles, downloadFiles):

  1. Cleanup: Updates lastSync for each item.
  2. Event Emission: Emits sync_completed or sync_failed events.

Eventing:

CloudSyncManager extends TypedEventEmitterAdapter, providing type-safe events for various sync lifecycle stages (e.g., sync:started, sync:completed, sync:item_uploaded, sync:conflict_detected). It also maintains backward compatibility by emitting generic 'sync-event' and specific legacy events.

Versioning:

1.4 Cloud Backup Manager (src/sync/cloud/backup-manager.ts)

The BackupManager handles the creation, restoration, and management of application backups in cloud storage. It supports automatic backups, compression, and retention policies.

Configuration (BackupManagerConfig, BackupConfig):

Core Backup Flow (createBackup() method):

  1. Backup ID Generation: A unique ID is generated using date and UUID.
  2. Item Collection (collectItem, collectDirectory):

  1. Compression (compress): Compresses the combined data using gzip.
  2. Manifest Creation: Generates a BackupManifest containing metadata about the backup (ID, creation time, items, sizes, checksums, encryption status).
  3. Upload (uploadBackup, uploadSplitBackup):

  1. Cleanup (cleanupOldBackups): Deletes older backups based on the maxBackups retention policy.
  2. Event Emission: Emits backup_started, backup_progress, and backup_created events.

Other Operations:

1.5 Convenience Functions (src/sync/cloud/index.ts)

The src/sync/cloud/index.ts file serves as the main entry point for cloud-related sync and backup features. It re-exports all public APIs and provides a powerful factory function:

2. Generic State Synchronization (src/sync/index.ts)

This module provides a lower-level, more abstract synchronization framework designed for arbitrary application states (e.g., JSON objects). It implements a distributed synchronization model using vector clocks to track causality and resolve conflicts.

2.1 Key Concepts

2.2 Vector Clock Operations

The module provides a set of utility functions for working with vector clocks:

2.3 State Management

2.4 Conflict Detection and Resolution

2.5 Sync Manager (SyncManager)

The SyncManager class orchestrates the entire state synchronization process.

Configuration (SyncConfig):

Core Functionality:

  1. State Storage: Internally maintains a Map> for all local states.
  2. Pending Operations: Stores a list of SyncOperations representing local changes that need to be synchronized with other nodes.
  3. Persistence (save(), load()): Uses UnifiedVfsRouter to persist the manager's internal state (all SyncState objects and pendingOperations) to a local JSON file. This ensures state is preserved across application restarts.
  4. CRUD Operations: Provides createState(), getState(), updateState(), and deleteState() methods for managing local SyncState objects. Each modification automatically adds a SyncOperation to pendingOperations and triggers a save().
  5. applyRemoteState(remoteState: SyncState): This is the heart of reconciliation for a single state.

  1. reconcile(remoteStates: SyncState[]): Orchestrates the application of multiple remote states, calling applyRemoteState for each. It returns a ReconciliationResult detailing applied states, conflicts, and pending operations.
  2. resolveConflict(conflict: SyncConflict, resolution: ConflictResolution, customData?: T): Allows for manual resolution of conflicts, providing options to choose local, remote, or a custom merged data.
  3. Auto-Sync: startAutoSync() and stopAutoSync() manage an interval timer that emits auto-sync-tick events, allowing external logic to trigger reconciliation.
  4. Export/Import: exportState() and importState() facilitate transferring the entire set of local states.
  5. Eventing: Extends EventEmitter and emits events like status-changed, state-created, state-updated, conflict-detected, conflict-resolved, saved, loaded, etc.

2.6 Singleton Access

3. Integration and Usage

The src/sync module offers two distinct but complementary synchronization paradigms:

Developers can choose the appropriate manager based on their data type and synchronization requirements. For example, a Code Buddy application might use CloudSyncManager to back up user-generated code files and SyncManager to synchronize internal application settings or session metadata across different devices.

The src/sync/cloud/index.ts module also provides createDefaultSyncItems() and createDefaultBackupItems() which define common paths for Code Buddy's internal data (e.g., .codebuddy/sessions, .codebuddy/memory, .codebuddy/settings), demonstrating how these modules are intended to be used within the application.

Example Usage (Conceptual)

import { createCloudSyncSystem, createS3Config, createDefaultSyncItems, createDefaultBackupItems } from './sync/cloud/index.js';
import { getSyncManager } from './sync/index.js';

async function setupSync() {
  class="hl-cmt">// 1. Cloud File Sync & Backup
  const cloudConfig = createS3Config({
    bucket: 'my-codebuddy-data',
    region: 'us-east-1',
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    encryptionKey: 'my-super-secret-key',
  });

  const cloudSystem = createCloudSyncSystem({
    cloud: cloudConfig,
    sync: {
      autoSync: true,
      syncInterval: 60000, class="hl-cmt">// Every minute
      direction: 'bidirectional',
      items: createDefaultSyncItems(),
    },
    backup: {
      autoBackup: true,
      backupInterval: 3600000, class="hl-cmt">// Every hour
      maxBackups: 5,
      items: createDefaultBackupItems(),
    },
  });

  cloudSystem.sync.onTypedSyncEvent('sync:completed', (event) => {
    console.log(`Cloud sync completed: ${event.result.itemsSynced} items.`);
  });
  cloudSystem.backup.on('backup_created', (event) => {
    console.log(`Cloud backup created: ${event.backup.id}`);
  });

  cloudSystem.startAll();
  console.log('Cloud sync and backup started.');

  class="hl-cmt">// 2. Generic State Synchronization
  interface AppSettings {
    theme: string;
    fontSize: number;
    lastOpenedFile: string;
  }

  const settingsSyncManager = getSyncManager<AppSettings>({
    nodeId: &#39;user-desktop-client&#39;,
    conflictStrategy: &#39;last-write-wins&#39;,
    persistPath: &#39;.codebuddy/sync/settings&#39;,
  });

  settingsSyncManager.on(&#39;state-updated&#39;, (state) => {
    console.log(`Settings updated: ${state.id} - ${JSON.stringify(state.data)}`);
  });

  class="hl-cmt">// Create or update a setting
  let mySettings = settingsSyncManager.getState(&#39;app-settings&#39;);
  if (!mySettings) {
    mySettings = settingsSyncManager.createState({
      theme: &#39;dark&#39;,
      fontSize: 14,
      lastOpenedFile: &#39;/home/user/project/main.ts&#39;,
    });
  } else {
    settingsSyncManager.updateState(mySettings.id, {
      ...mySettings.data,
      fontSize: 16,
    });
  }

  class="hl-cmt">// Simulate receiving remote states (e.g., from another client)
  const remoteSettings: AppSettings = {
    theme: &#39;light&#39;,
    fontSize: 14,
    lastOpenedFile: &#39;/home/user/project/index.js&#39;,
  };
  const remoteSyncState = settingsSyncManager.createState(remoteSettings); class="hl-cmt">// In a real scenario, this would come from a remote source

  class="hl-cmt">// Reconcile with remote states
  const reconciliationResult = await settingsSyncManager.reconcile([remoteSyncState]);
  if (reconciliationResult.conflicts.length > 0) {
    console.warn(&#39;Conflicts detected during state reconciliation:&#39;, reconciliationResult.conflicts);
  }

  class="hl-cmt">// Clean up on shutdown
  process.on(&#39;beforeExit&#39;, () => {
    cloudSystem.dispose();
    settingsSyncManager.dispose();
    console.log(&#39;Sync systems disposed.&#39;);
  });
}

setupSync().catch(console.error);