# Source Code Splitter Architecture

## Overview

The Source Code Splitter transforms source code files into hierarchical, concatenable chunks that preserve semantic structure while enabling effective code search. The system uses tree-sitter for precise syntax tree parsing to detect semantic boundaries, creating context-aware chunks that respect language structure.

## Design Philosophy

### Tree-sitter Semantic Boundaries

The splitter uses tree-sitter parsers to identify semantic boundaries in source code, providing:

- **Precision**: Syntax tree analysis ensures accurate boundary detection
- **Language Awareness**: Native support for JavaScript, TypeScript, JSX, and TSX
- **Semantic Chunking**: One chunk per function/method/class with proper hierarchy
- **Hierarchical Structure**: Chunks maintain proper nesting relationships

### Documentation-Focused Chunking

The chunking strategy prioritizes indexing public interfaces and maintaining comment-signature relationships:

- **Public Interface Emphasis**: Captures outward-facing class methods and top-level functions
- **Comment Preservation**: Documentation comments stay with their associated code
- **Hierarchical Paths**: Enable semantic search within code structure
- **Concatenable Chunks**: Chunks can be reassembled to reconstruct original context

## Architecture Components

```mermaid
graph TD
    subgraph "Tree-sitter Parsing"
        A[Source Code File]
        B[Language Detection]
        C[Tree-sitter Parser]
        D[Syntax Tree Generation]
    end

    subgraph "Boundary Extraction"
        E[Semantic Boundary Detection]
        F[Hierarchical Path Construction]
        G[Boundary Classification]
    end

    subgraph "Chunk Processing"
        H[Content Extraction]
        I[TextSplitter Delegation]
        J[Hierarchical Chunk Creation]
    end

    subgraph "Output Generation"
        K[ContentChunk Array]
    end

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J
    J --> K

    style A fill:#e1f5fe
    style K fill:#f3e5f5
```

## Core Components

### 1. Tree-sitter Language Parsers

The system supports multiple languages through dedicated tree-sitter parsers:

**Supported Languages:**

- **JavaScript**: ES6+ classes, functions, arrow functions, JSX elements
- **TypeScript**: Interfaces, types, enums, namespaces, decorators, generics, TSX
- **JSX/TSX**: React component parsing with TypeScript integration

**Language Registry:**
The `LanguageParserRegistry` automatically selects the appropriate parser based on file extension and content analysis, falling back to `TextDocumentSplitter` for unsupported languages.

### 2. Semantic Boundary Detection

Tree-sitter parsers identify structural elements through syntax tree traversal:

**Primary Boundaries:**

- **Classes**: Complete class definitions with all members
- **Functions**: Top-level function declarations and expressions
- **Methods**: Class and interface method definitions
- **Interfaces**: TypeScript interface declarations
- **Namespaces**: TypeScript namespace blocks
- **Types**: Type alias definitions

**Boundary Extraction:**
Each boundary includes start/end positions, basic type classification, and optional name for context. The system focuses on structural boundaries rather than detailed metadata extraction.

### 3. Hierarchical Chunking Strategy

The chunking approach creates semantic units that respect code structure:

```mermaid
graph LR
    subgraph "Chunk Hierarchy"
        A[File Level] --> B[Class Level]
        B --> C[Method Level]
        A --> D[Function Level]
        B --> E[Property Level]
    end

    subgraph "Path Structure"
        F["['UserService.ts']"] --> G["['UserService.ts', 'UserService']"]
        G --> H["['UserService.ts', 'UserService', 'getUser']"]
        F --> I["['UserService.ts', 'calculateSum']"]
    end

    style A fill:#e1f5fe
    style G fill:#f3e5f5
    style H fill:#e8f5e8
```

**Chunking Rules (Canonical Ruleset):**

Core Principles:

- **Semantic Fidelity**: Boundaries align with grammar-level constructs (namespace/module, class, interface, enum, type alias, function/method/constructor).
- **Hierarchical Integrity**: Every chunk has a full path (e.g. `['File.ts', 'Namespace', 'ClassName', 'methodName']`).
- **Perfect Reconstructability**: Concatenating chunks in emission order reproduces the exact original file bytes.
- **Retrieval Granularity**: Prefer the smallest semantically meaningful unit (do not merge adjacent declarations).

Boundary Emission:

- Emit a primary chunk for each declaration that introduces a named scope or executable unit:
  - Namespaces / modules
  - Classes / interfaces / enums / type aliases
  - Top-level functions (regular / async / arrow assigned via `const`)
  - Methods / constructors (including `static` / `private` / accessor forms)
- Do NOT emit chunks for internal control-flow (`if`, `for`, `switch`, etc.) or nested local helper functions inside another function/method body (these remain part of the parent body chunk).
- Transparent wrappers (`export`, `declare`, modifiers) never suppress boundary emission; they are included in the declaration chunk content.

Classification (Dual Typing):

- `boundaryType: "structural"` for: namespace/module, class, interface, enum, type alias, import/export units.
- `boundaryType: "content"` for: function, method, constructor, arrow function, variable declaration introducing executable code.
- All chunks include `types: ['code']` for backward compatibility; semantic classification augments, not replaces, existing type labels.

Documentation & Signature Association:

- Preceding contiguous documentation comments (JSDoc / multi-line block / meaningful line comments) are merged into the declaration chunk.
- Documentation scan crosses transparent wrappers (e.g. `export` before `class`).
- Chunk `startLine` and `startByte` are adjusted to the first doc line when docs are present.

Atomicity & Non-Merging:

- Never merge two siblings (e.g. two methods, function + next function, class + first method).
- Never merge a structural declaration with its first child declaration.
- A method/function body is treated as a single atomic content region unless size splitting (see Size Management) is triggered.

Size Management (Universal Max Size Enforcement):

- No emitted chunk may exceed the configured maximum size (bytes or token estimate surrogate). This rule applies to:
  - Declaration (structural) segments (e.g. large class/interface/enum declarations with decorators or long heritage clauses)
  - Function/method/constructor bodies
  - Interstitial / global content between declarations
  - Any trailing or leading whitespace/comment regions
- Oversized segments are **delegated** to the `TextSplitter` AFTER semantic boundary determination so structural intent is preserved.
- Sub-chunks produced from delegation:
  - Preserve strict original ordering
  - Inherit the parent path (adding a deterministic ordinal suffix only if needed for uniqueness, e.g. `MyClass/doWork#1`, `MyClass/doWork#2`)
  - Are all classified as `boundaryType: "content"` unless they correspond to the first structural declaration slice (which remains `structural` if it contains the signature/docs)
  - Never duplicate signature or doc lines
- Structural declaration chunk strategy:
  - Signature + docs remain in the first chunk (always under max size due to early split trigger on large bodies/content)
  - Body content beyond the first segment is delegated in slices as needed
- Guarantees:
  - Perfect reconstructability (concatenation of all chunks == original file)
  - Deterministic chunk boundaries for identical inputs/config
  - No late greedy merging stage; only size-driven subdivision

Path & Hierarchy:

- Each emitted chunk path is the full ancestry of structural declarations.
- Sub-chunks produced by size delegation extend that path deterministically (ordinal suffix or segmented identifier) without inventing new semantic ancestors.

Wrapper & Modifier Handling:

- `export`, `default`, `abstract`, `async`, visibility (`public|private|protected`), and decorators remain within the declaration chunk.
- Multiple stacked modifiers do not produce multiple chunks.

Content Integrity:

- No duplication: bytes belong to exactly one chunk except optional intentional signature/body split (if implemented).
- No gaps: every byte of the original file is covered by exactly one chunk (structural + body subdivision collectively).

Suppression Rules:

- Suppress nested function-like declarations only when they are local (declared inside another function/method body) AND not part of the public structural surface (they remain embedded implementation details).
- Do not suppress functions declared directly inside namespaces or classes (those emit boundaries).

Fallback & Error Resilience:

- On parser failure / zero boundaries: fall back to line-based splitter preserving reconstructability.
- On partial parse (ERROR nodes): still emit any confidently parsed boundaries; malformed regions become part of surrounding content.

Determinism:

- Given identical source input and configuration, chunk boundaries (names, byte ranges, hierarchy) are deterministic.

Extensibility:

- New structural kinds (e.g. `trait`, `record`, future TS constructs) must specify:
  - Classification (structural vs content)
  - Inclusion in documentation merging rules
  - Whether they introduce a hierarchical path segment.

Implementation Notes:

- Boundary traversal is single-pass with documentation accumulation.
- Size threshold should be configurable (env / constructor option).
- Delegated sub-chunks MUST NOT exceed the max size individually.
- Token estimation (if used) should degrade gracefully to byte length.

Testing Guidelines:

- Assert reconstructability (join == original).
- Assert presence + uniqueness of expected paths.
- Assert doc block capture (start line alignment).
- Assert large body subdivision ordering & naming.
- Assert no emission for nested local helpers.
- Assert classification correctness (`structural` vs `content`).

Summary:

- Structural nodes = skeleton for hierarchical reassembly.
- Content nodes = precise retrieval targets.
- Large bodies subdivided AFTER boundary identification without erasing the semantic anchor.

### 4. Content Processing Pipeline

```mermaid
sequenceDiagram
    participant TSS as TreesitterSourceCodeSplitter
    participant LP as LanguageParser
    participant BE as BoundaryExtractor
    participant TS as TextSplitter

    TSS->>LP: Parse source code
    LP->>BE: Extract semantic boundaries
    BE->>TSS: Return boundary positions

    loop For each boundary
        TSS->>TSS: Extract boundary content
        TSS->>TS: Delegate large content sections
        TS->>TSS: Return subdivided chunks
    end

    TSS->>TSS: Assign hierarchical paths
    TSS->>TSS: Create ContentChunk array
```

## Processing Flow

### Semantic Chunking Process

```mermaid
flowchart TD
    A[Source Code Input] --> B{Language Supported?}
    B -->|Yes| C[Tree-sitter Parsing]
    B -->|No| D[TextDocumentSplitter Fallback]

    C --> E[Boundary Extraction]
    E --> F[Content Sectioning]
    F --> G[Hierarchical Path Assignment]
    G --> H[Chunk Generation]

    D --> I[Line-based Chunks]
    H --> J[ContentChunk Array]
    I --> J

    style A fill:#e1f5fe
    style J fill:#e8f5e8
    style D fill:#fff3e0
```

### Chunk Generation Strategy

The system creates chunks that balance semantic meaning with search effectiveness:

**Chunk Types:**

- **Structural Chunks**: Class/function/interface signatures with documentation
- **Method Chunks**: Complete method implementations including comments
- **Content Chunks**: Code sections between structural boundaries
- **Delegated Chunks**: Large content sections processed by TextSplitter

#### Dual Typing & Boundary Classification

Each emitted `ContentChunk` for source code now uses a dual typing strategy:

- `types` array always includes `code` (backward compatibility for existing retrieval / embedding logic).
- A secondary semantic classification is derived from boundary analysis:
  - `boundaryType: "structural"` for declarations that introduce _named, nestable scopes_ (classes, interfaces, enums, namespaces/modules, type aliases, import/export units).
  - `boundaryType: "content"` for executable or implementation-level units (functions, methods, constructors, arrow functions, variable/lexical declarations that carry behavior, etc.).

Design goals:

1. Hierarchical Assembly: `structural` nodes act as stable anchors for subtree reconstruction (e.g. return an entire class when a method matches).
2. Precision in Retrieval: `content` nodes map to the most specific executable region, improving ranking granularity.
3. Non-Destructive Enrichment: Existing consumers relying only on `types.includes("code")` remain unaffected.
4. Future Extensibility: Additional refinement layers (e.g. `interface-signature`, `public-api`, `test-code`) can be layered without breaking current contracts.

Why not merge structural + content nodes pre-index?

- Merging obscures which textual spans define navigational structure vs. executable implementation.
- Preserving both classifications enables context-dependent expansion policies (e.g. include whole class vs. just matched method).

Boundary integrity + dual typing replace the need for post-hoc greedy size normalization.

**Path Inheritance:**
Each chunk inherits the hierarchical path of its containing structure, enabling context-aware search and proper reassembly.

## Error Handling & Fallback Strategy

### Graceful Degradation

The splitter handles various scenarios through layered fallback mechanisms:

```mermaid
graph TD
    A[Input Source Code] --> B{Language Supported?}
    B -->|No| C[TextDocumentSplitter]
    B -->|Yes| D{Tree-sitter Parse Success?}
    D -->|No| C
    D -->|Yes| E{Boundaries Extracted?}
    E -->|No| C
    E -->|Yes| F[Semantic Chunking]

    F --> G[ContentChunk Array]
    C --> H[Line-based Chunks]
    G --> I[Output]
    H --> I

    style A fill:#e1f5fe
    style I fill:#e8f5e8
    style C fill:#fff3e0
```

**Fallback Scenarios:**

- **Unsupported Languages**: Automatic delegation to TextDocumentSplitter
- **Parse Errors**: Graceful fallback for malformed syntax
- **Boundary Detection Failures**: Line-based processing for complex edge cases
- **Large File Handling**: For files exceeding a universal 32KB limit in the underlying parser, the system provides graceful degradation. It performs a full semantic parse on the initial ~32KB and falls back to `TextDocumentSplitter` for the remainder of the file, ensuring that no content is lost

### Error Recovery

The system maintains robust operation through:

- **Parse Error Isolation**: Errors in one section don't affect others
- **Content Preservation**: All source content is retained in chunks
- **Consistent Interface**: All fallback paths produce compatible ContentChunk arrays

## Language Extensibility

### Current Language Support

The tree-sitter implementation provides comprehensive support for web development languages:

**JavaScript (ES6+):**

- Classes with methods and properties
- Functions (regular, async, arrow functions for top-level declarations)
- JSX elements and React components
- Import/export statements

**TypeScript:**

- All JavaScript features plus type system constructs
- Interfaces and type alias definitions
- Namespaces and modules
- Enums and decorators
- Generic type parameters
- TSX (TypeScript + JSX)

**Python:**

- Classes, functions, and methods
- `import` and `from...import` statements
- Decorators and `async` functions

### Architecture for Extension

The modular design supports future language additions through:

**Parser Registry System:**

- Automatic language detection by file extension
- Fallback mechanisms for unsupported languages
- Consistent interface across all language parsers

**Tree-sitter Integration:**

- Leverages existing tree-sitter grammar ecosystem
- Language-specific parsers implement common boundary extraction interface
- Shared infrastructure for syntax tree traversal and boundary detection

**Future Language Candidates:**

- Java (packages, classes, methods)
- C# (namespaces, classes, properties, methods)
- Go (packages, structs, methods, functions)
- Rust (modules, structs, impl blocks, functions)

## Integration with Pipeline System

### Source Code Processing Pipeline

The tree-sitter splitter integrates seamlessly with the existing content processing infrastructure:

```mermaid
graph LR
    subgraph "Source Code Processing"
        A[SourceCodePipeline] --> B[TreesitterSourceCodeSplitter]
        B --> C[ContentChunk Array]
        C --> D[Embedding Generation]
        D --> E[Vector Storage]
    end

    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style E fill:#e8f5e8
```

### Rationale: Omission of Greedy Size-Based Merging

The previous design showed an additional GreedySplitter phase for size normalization. This has been intentionally removed for source code because:

- Structural Fidelity: Treesitter-derived chunks encode precise `{level, path}` hierarchies required for hierarchical reassembly and subtree reconstruction. Greedy merging collapses boundaries and degrades that signal.
- Retrieval Quality: HierarchicalAssemblyStrategy relies on intact structural units (class, method, function). Artificially merged aggregates reduce precision and introduce unrelated code into a match context.
- Reassembly Guarantees: The splitter already guarantees concatenability. Additional size-based merging provides negligible token efficiency gains compared to the semantic loss.
- JSON / Structured Parity: The same reasoning applies to JSON and other strictly nested formats—each structural node is meaningful even if its textual size is small.
- Simplicity & Predictability: A single semantic splitter reduces mental overhead, improves test determinism, and avoids edge cases where mixed-level metadata must be reconciled.

If future optimization is needed, it should be: (a) post-retrieval context window packing, or (b) language-aware micro-chunk collapsing that preserves explicit structural node boundaries—never generic greedy adjacency merging.

### System Benefits

**Enhanced Search Quality:**

- Semantic chunks respect code structure boundaries
- Hierarchical paths enable context-aware retrieval
- Documentation comments stay associated with relevant code
- Function and method retrieval maintains complete context

**Performance Characteristics:**

- Tree-sitter parsing provides linear time complexity
- Memory efficient processing without large intermediate structures
- Robust error handling prevents pipeline failures
- Maintains compatibility with existing chunk optimization systems

**Developer Experience:**

- Search results respect semantic boundaries
- Retrieved chunks include necessary surrounding context
- Hierarchical structure aids in understanding code relationships
- Consistent interface with other document processing pipelines
