# OpenContracts v3.0.0.b4 Release Notes

**Release Date**: January 2026
**Previous Version**: v3.0.0.b3

---

## Executive Summary

v3.0.0.b4 introduces **corpus-isolated structural annotations** and **multimodal embedding support**. This release prioritizes data isolation and prepares the platform for advanced AI capabilities including image understanding.

Key changes:
- **Corpus-Isolated Structural Annotations** - Each corpus maintains independent structural annotation sets
- **Multimodal Embeddings** - Support for image tokens and visual content in documents
- **Multi-Embedder Architecture** - Each corpus can use different embedding models

> **Breaking Change**: Structural annotations are no longer shared across corpus copies. See [Migration Notes](#migration-notes) below.

---

## Table of Contents

1. [New Features](#new-features)
   - [Corpus-Isolated Structural Annotations](#1-corpus-isolated-structural-annotations)
   - [Multimodal Embedding Support](#2-multimodal-embedding-support)
2. [Architecture Changes](#architecture-changes)
3. [Migration Notes](#migration-notes)
4. [Bug Fixes](#bug-fixes)

---

## New Features

### 1. Corpus-Isolated Structural Annotations

Structural annotations (headers, sections, paragraphs, tables, figures) are now **duplicated per corpus** instead of shared across corpus copies.

#### Why This Change?

The previous shared approach created complications for:
- **Multi-embedder support**: Each corpus can use different embedding models, requiring consistent per-corpus vector spaces
- **Data isolation**: Shared data across corpus boundaries complicated permission models and lifecycle management
- **Vector search**: Shared annotations required complex queries to select the right embedding for each corpus's embedder

#### How It Works Now

```
Document uploaded to Corpus A:
  → Document #1 created
  → StructuralAnnotationSet #1 created (content_hash: "abc123_corpus_a_id")
  → 5000 structural annotations linked to Set #1

Same document added to Corpus B:
  → Document #2 created (corpus-isolated copy)
  → StructuralAnnotationSet #2 created (content_hash: "abc123_corpus_b_id")
  → 5000 structural annotations DUPLICATED to Set #2

Result: Complete isolation between corpuses
```

#### Key Benefits

| Benefit | Description |
|---------|-------------|
| **Complete Isolation** | Each corpus is a self-contained unit with no cross-corpus data sharing |
| **Multi-Embedder Ready** | Each corpus can use different embedding models without conflict |
| **Simplified Permissions** | Corpus boundaries are hard permission boundaries |
| **Clean Lifecycle** | Deleting a corpus cleanly removes all its data |

#### Trade-offs

- **Storage**: O(n) storage for structural annotations where n = number of corpus copies
- **Processing**: Document parsing may occur multiple times for same content

---

### 2. Multimodal Embedding Support

Documents can now include **image tokens** for visual content understanding.

#### What's New

- **Image Token Model**: Unified representation for images in PAWLs format
- **Multimodal Embedders**: Support for embedding models that process both text and images
- **Content Modality Enum**: Explicit tracking of TEXT, IMAGE, and MULTIMODAL content types
- **Image Retrieval Tools**: LLM agents can now retrieve and analyze document images

#### Image Token Format

Images are now represented as special tokens in the PAWLs format:

```json
{
  "page": 0,
  "tokens": [
    {"text": "Introduction", "x": 100, "y": 50, ...},
    {
      "text": "[IMG]",
      "x": 100, "y": 200,
      "width": 400, "height": 300,
      "is_image": true,
      "image_path": "images/page_0_img_1.png"
    }
  ]
}
```

#### Supported Embedders

| Embedder | Modality | Notes |
|----------|----------|-------|
| OpenAI (text-embedding-*) | TEXT | Text-only embeddings |
| Voyage AI (voyage-multimodal-*) | MULTIMODAL | Text + image embeddings |
| Sentence Transformers | TEXT | Local text embeddings |

---

## Architecture Changes

### StructuralAnnotationSet Model

The `content_hash` field format changed to support corpus isolation:

| Version | Format | Max Length |
|---------|--------|------------|
| v3.0.0.b3 | `{sha256_hash}` | 64 chars |
| v3.0.0.b4 | `{sha256_hash}_{corpus_id}` | 128 chars |

### Document.source_document Field

The `source_document` field is **no longer set** when adding documents to a corpus. Each corpus copy is an independent entity with no provenance tracking to the source.

### Database Schema Changes

New migration applied:
- `annotations/0056_alter_structuralannotationset_content_hash` - Extends content_hash to 128 chars

---

## Migration Notes

### Automatic Migration

The database migration runs automatically and extends the `content_hash` field. No manual intervention required.

### Behavior Change

If your application relied on:
- **Shared structural annotations** across corpus copies → They are now duplicated
- **`source_document` provenance tracking** → This field is no longer set on import
- **Content-based deduplication** → Each upload creates independent documents

### Storage Impact

Deployments with documents appearing in multiple corpuses will see increased storage usage proportional to the number of corpus copies.

---

## Bug Fixes

- Fixed migration using incorrect related_name for document relationships
- Fixed embeddings task signal using wrong filter relationship (`documents=` → `document_paths__document=`)
- Fixed annotation queries to use `structural_annotation_set.structural_annotations` relationship
- Fixed test suite alignment with corpus-isolated annotation behavior

---

## Documentation Updates

Updated documentation to reflect corpus-isolated architecture:

- `docs/architecture/structural_vs_non_structural_annotations.md` - Complete rewrite
- `docs/architecture/document_versioning.md` - Added multi-embedder rationale
- `docs/architecture/data_model/documents_and_annotations.md` - Updated StructuralAnnotationSet description
- `docs/migrations/v3_upgrade_guide.md` - Updated structural annotation section
- `docs/architecture/corpus_export_import_v2.md` - Updated export format comments

---

## Related Documentation

- [Structural vs Non-Structural Annotations](../architecture/structural_vs_non_structural_annotations.md)
- [Document Versioning](../architecture/document_versioning.md)
- [Core Data Model](../architecture/data_model/documents_and_annotations.md)
