# File Understanding Architecture

## Overview

File Understanding is the independent layer between ChatKit uploads and Agent file usage.

The new flow no longer treats `StorageFile` as the business model for chat attachments, and the Agent no longer depends on ad-hoc full-text loading through `LoadFileCommand` at runtime.

Flow:

1. ChatKit uploads the original file.
2. The upload layer stores bytes and keeps the legacy `StorageFile` record.
3. File Understanding creates a `FileAsset`, parses the file, and creates artifacts, chunks, and citation anchors.
4. Chat messages persist the new `fileAssets` relation.
5. Agent runtime reads files through summaries, search chunks, read tools, and workspace tools.

This turns uploaded files into parsed, searchable, citable context assets.

## Goals

- Keep `StorageFile` as the object-storage compatibility layer only.
- Use `FileAsset` and `AgentFile` as the new business handles.
- Keep ChatKit responsible only for upload, status display, and file-handle submission.
- Keep Agent runtime independent from raw storage implementation.
- Use CQRS handlers as the application logic boundary instead of thin service delegation.
- Keep parsers, indexing, citations, and workspace tools isolated from chat persistence.
- Avoid putting large file bodies into the prompt.

## Module Layout

Backend module:

```text
packages/server-ai/src/file-understanding/
```

Structure:

```text
file-understanding/
  domain/       # types, status, capabilities, artifact kinds
  entities/     # TypeORM entities
  commands/     # CQRS write model
  queries/      # CQRS read model
  parsers/      # parser adapters
  middlewares/  # built-in Agent middleware
  tools/        # Agent tools
```

The module is registered by:

```text
packages/server-ai/src/file-understanding/file-understanding.module.ts
packages/server-ai/src/app.module.ts
packages/server-ai/src/index.ts
```

## Data Model

### `FileAsset`

Table: `file_asset`

The new platform-level file understanding asset.

Important fields:

- `storageFileId`: legacy stored-object reference.
- `conversationId`, `threadId`, `projectId`, `xpertId`: context links.
- `tenantId`, `organizationId`, `userId`: access boundary.
- `sha256`: content hash.
- `purpose`: `chat_attachment`, `workspace`, or `knowledge`.
- `parseMode`: `auto`, `fast`, `deep`, or `none`.
- `status`: `uploaded`, `scanning`, `parsing`, `ready`, `partial`, or `failed`.
- `capabilities`: `preview`, `search`, `read`, `table_query`, `vision`, `ocr`, `workspace`.
- `summary`: parsed summary.
- `workspacePath`: projected workspace path. It is present only after the original file has been written into the runtime volume, and can be passed to `sandbox_file` or shell tools.

### `FileArtifact`

Table: `file_artifact`

Structured parser output.

Common kinds:

- `summary`
- `text`
- `page_text`
- `table`
- `sheet`
- `slide`
- `image_metadata`
- `ocr_text`
- `code_tree`
- `file_manifest`

Artifacts can carry anchors such as page, sheet, slide, or path.

### `FileChunk`

Table: `file_chunk`

Searchable chunks created from artifacts.

Important fields:

- `fileAssetId`
- `artifactId`
- `orderNo`
- `content`
- `tokenCount`
- `anchor`

The current implementation searches file chunk vectors first, then falls back to database text matching when embeddings or the vector store are unavailable, when vector search returns no usable matches, or when chunk hydration fails.

### `FileCitationAnchor`

Table: `file_citation_anchor`

Stable citation anchors for file-grounded answers.

Examples:

- `page:3:chunk:12`
- `sheet:Sheet1:chunk:4`
- `slide:2:chunk:8`
- `path:src/index.ts:chunk:6`
- `chunk:0`

### `FileEmbedding`

Table: `file_embedding`

Metadata for vector indexing:

- `provider`
- `model`
- `vectorId`
- `metadata`

`metadata` stores the collection name, collection scope, embedding fingerprint, dimensions, chunk order, and anchor. Collection names use `file-understanding:<scope>:<scopeId>:<embeddingFingerprint>`, scoped by `xpertId + embeddingFingerprint` first, then falling back to `projectId`, then `tenantId`.

### `ConversationFileLink`

Table: `conversation_file_link`

Links a file asset to a conversation so Agents cannot read files across conversation or tenant boundaries.

### `ChatMessage.fileAssets`

`ChatMessage` now has a `fileAssets` relation through:

```text
chat_message_file_asset
```

New messages should persist files here. Legacy `attachments: StorageFile[]` remains only for old message compatibility.

## Upload Flow

### ChatKit

ChatKit uses the new `AgentFile` handle:

```ts
export interface AgentFile {
  id: string
  fileId: string
  storageFileId: string
  objectKey?: string
  url?: string
  fileUrl?: string
  thumbUrl?: string
  originalName?: string
  size?: number
  mimeType?: string
  status: AgentFileStatus
  parseStatus: AgentFileStatus
  parseMode?: 'auto' | 'fast' | 'deep' | 'none'
  purpose?: 'chat_attachment' | 'workspace' | 'knowledge'
  capabilities?: string[]
  summary?: string
  workspacePath?: string
}
```

Upload request:

```ts
const formData = new FormData()
formData.append('file', file, file.name || 'upload')
formData.append('purpose', 'chat_attachment')
formData.append('parseMode', 'auto')
formData.append('threadId', stream.threadId)

const uploaded = await stream.client.contexts.fetch<AgentFile>('/contexts/file', {
  method: 'POST',
  body: formData
})
```

Raw HTTP path with the normal API prefix:

```text
POST /api/ai/contexts/file
```

### Backend Entry

Entry file:

```text
packages/server-ai/src/ai/context.controller.ts
```

`POST /api/ai/contexts/file`:

1. Runs `UploadFileCommand`.
2. Persists the legacy `StorageFile` for `storage` targets.
3. Creates a new `FileAsset`.
4. Links the file to conversation/thread context when provided.
5. Parses according to `parseMode`.
6. Returns `AgentFile`.

Returned ids:

- `id`: `FileAsset.id`
- `fileId`: `FileAsset.id`
- `storageFileId`: legacy `StorageFile.id`

### Message Submission

ChatKit submits file handles, not parsed content:

```json
{
  "files": [
    {
      "id": "file-asset-id",
      "fileId": "file-asset-id",
      "storageFileId": "storage-file-id",
      "originalName": "report.pdf",
      "mimeType": "application/pdf",
      "parseStatus": "ready",
      "capabilities": ["preview", "read", "search", "ocr"]
    }
  ]
}
```

`xpert/commands/handlers/chat.handler.ts` persists new files into `ChatMessage.fileAssets`. Only legacy `StorageFile` payloads are written into `ChatMessage.attachments`.

## Parsing Flow

Core handler:

```text
packages/server-ai/src/file-understanding/commands/handlers/parse-file-asset.handler.ts
```

Flow:

1. Load `FileAsset`.
2. Resolve `StorageFile`.
3. Resolve the stored file path.
4. Pick a parser through `FileParserRegistry`.
5. Persist artifacts.
6. Split indexable artifacts into chunks.
7. Rebuild citation anchors.
8. Update status, summary, capabilities, and metadata.

Default chunking:

```text
chunk size: 3600 characters
chunk overlap: 240 characters
```

Indexable artifact kinds:

- `text`
- `page_text`
- `sheet`
- `slide`
- `table`
- `code_tree`
- `file_manifest`
- `ocr_text`

## Parser Support

| Parser                  | File types               | Current capabilities                    | Notes                                 |
| ----------------------- | ------------------------ | --------------------------------------- | ------------------------------------- |
| `TextFileParser`        | txt, md, json, log, code | summary, text, search, read             | UTF-8 text                            |
| `PdfFileParser`         | pdf                      | page text, page anchors, OCR capability | `partial` if no text is extracted     |
| `OfficeFileParser`      | doc, docx, ppt, pptx     | text or slide artifacts                 | LangChain loaders                     |
| `SpreadsheetFileParser` | csv, tsv, xlsx, xls      | sheet/table artifacts, table query      | xlsx includes headers and sample rows |
| `ImageFileParser`       | common image formats     | image metadata, vision/OCR capability   | OCR/VLM integration is the next step  |
| `ArchiveFileParser`     | zip                      | manifest, code tree, text indexing      | indexes selected text files           |

Parser selection currently uses MIME type and extension. Add magic-byte sniffing in parser `supports(source)` or the registry rather than in controllers.

## API

Assuming the global API prefix is `/api`.

```text
POST   /api/ai/contexts/file
DELETE /api/ai/contexts/file/:storageFileId

GET    /api/ai/files/:fileId
GET    /api/ai/files/:fileId/status
POST   /api/ai/files/:fileId/parse/retry
POST   /api/ai/files/:fileId/search
GET    /api/ai/files/:fileId/preview
POST   /api/ai/files/:fileId/read
DELETE /api/ai/files/:fileId

GET    /api/ai/conversations/:conversationId/files
```

Search request:

```json
{
  "query": "revenue forecast",
  "limit": 8
}
```

Read request:

```json
{
  "chunkId": "chunk-id"
}
```

or:

```json
{
  "orderNo": 0
}
```

## Agent Runtime

Message assembly:

```text
packages/server-ai/src/shared/agent/message.ts
```

Behavior:

- Resolve new `FileAsset` through `fileId` / `fileAssetId` first.
- Fall back to legacy `StorageFile` only when needed.
- Inject compact file cards and available anchors, not raw chunk bodies.
- Avoid injecting large full text.
- Let Agents use `file_search` / `file_read` when exact content is needed.
- Fall back to `LoadFileCommand` for failed parsing or legacy-only files.

`LoadFileCommand` must distinguish two path kinds when it reads original bytes:

- Absolute paths are legacy or already-resolved server-side paths and can be passed directly to LangChain / Node file loaders.
- Workspace-relative paths, for example `files/wechat/<integrationId>/<messageId>/<fileName>`, must not be opened relative to the process `cwd`. They must be resolved through `FileAsset.metadata.workspace.catalog`, `scopeId`, and `tenantId` / `userId`, then mapped with `VolumeClient.resolve(scope).path(relativePath)`.

Workspace-backed assets should keep this metadata:

```ts
{
  source: 'platform.workspace.files',
  catalog: 'projects' | 'users' | 'knowledges' | 'skills' | 'xperts',
  scopeId: '<projectId | userId | knowledgeId | rootId | xpertId>',
  relativePath: 'files/wechat/.../technical-notice.docx',
  workspacePath: 'files/wechat/.../technical-notice.docx'
}
```

The shared resolver lives in:

```text
packages/server-ai/src/file-understanding/domain/workspace-file.ts
```

`resolveWorkspaceVolumeScope` / `inferWorkspaceVolumeScope` own the catalog-to-`VolumeScope` mapping. `resolveFileAssetWorkspaceRelativePath` / `resolveFileAssetWorkspaceVolumeScope` read the path and scope from `FileAsset`. `LoadFileHandler`, the workspace file runtime capability, and workspace-backed `FileAsset` creation should reuse these helpers instead of maintaining command-handler-local catalog branches.

With this contract, plugins and external connectors only need to provide a relative path plus catalog/scope metadata. Production does not depend on the API process `cwd` being the workspace root, and messages do not need to persist environment-specific absolute paths such as `/sandbox/...` or `/workspace/...`.

Agent tools and built-in middleware:

```text
packages/server-ai/src/file-understanding/tools/file-understanding.tools.ts
packages/server-ai/src/file-understanding/middlewares/file-understanding.middleware.ts
```

The tools are produced by `FileUnderstandingMiddleware`. At runtime the subgraph creates a hidden required middleware entry, so conversation files are available without manually connecting a middleware node on the Agent canvas. The tool names stay stable while execution goes through the standard middleware tool path.

Tools:

- `file_search(fileIds, query, limit)`
- `file_read(fileId, chunkId, orderNo)`
- `file_table_query(fileId, query, limit)`
- `file_preview(fileId)`
- `workspace_list()`
- `workspace_read(fileId | path, chunkId, orderNo)`
- `workspace_search(query, limit)`

Agents should search first, then read exact chunks when more context is needed. `file_preview` returns a compact preview with business file fields, a short summary, artifact indexes, and chunk anchors; it does not return tenant, organization, audit fields, artifact full text, chunk full text, or raw parser metadata. If a file card or `workspace_list` includes `workspacePath`, Agents with `sandbox_file` or shell tools may read the original file by that path. `workspace_read` resolves by `fileId` or `workspacePath`, but reads parsed chunks rather than binary file bytes. Citations should use the returned anchors.

### PDF text-first and image-on-demand best practice

For PDFs that are mostly text with a small number of illustrations, use parsed `page_text` chunks as the primary source of truth. Inspect rendered `page_image` artifacts only when the user asks about visual content or when text search points to a page that likely depends on an image. This keeps normal PDF questions cheap and fast while still preserving access to charts, screenshots, diagrams, and scanned visual details.

Recommended Agent behavior:

- Use `file_search` first for normal text questions.
- Use `file_read` after search when exact surrounding text is needed.
- Use `file_page_images(fileId, pageStart, pageEnd)` when the user asks about charts, screenshots, diagrams, figures, visual layout, OCR, or page-specific image details.
- If the user does not specify a page, search for the user query plus visual-marker terms such as `figure`, `chart`, `diagram`, `screenshot`, `图`, `见下图`, and `流程图`; then inspect the returned page anchors with `file_page_images`.

Prompt snippet for Agents:

```text
When answering questions about uploaded PDFs, prefer parsed text first.
Use file_search to locate relevant page_text chunks, then file_read for exact text.
Do not inspect PDF page images for ordinary text questions.
Use file_page_images only when the user asks about visual content, page layout,
charts, screenshots, diagrams, figures, OCR, or when text search returns a page
that references nearby visual material such as "figure", "chart", "diagram",
"screenshot", "图", "见下图", or "流程图".
When the user does not provide a page number, search for the question and these
visual-marker terms, use the chunk anchors to identify pages, then inspect only
those pages with file_page_images.
Always cite the returned page, sheet, slide, path, or chunk anchors.
```

`parseMode: deep` currently controls whether parsing runs asynchronously. Do not present it as automatic full-PDF OCR or VLM processing. If image content must become searchable in a future implementation, run selective OCR or VLM only on relevant pages or image assets and store the result as anchored text artifacts such as `ocr_text`, not only as file summaries.

Workspace projection:

ChatKit sends the current `xpertId` during `/contexts/file` upload. When the message is submitted and the file is attached to a conversation, the backend projects the stored original file into:

```text
sessions/{conversationId}/files/{fileAssetId}/{originalName}
```

After a successful write, `FileAsset.workspacePath` and the `workspace` capability are updated. Docker sandboxes usually see `/workspace/sessions/...`; local shell sandboxes see the local runtime volume path.

## Compatibility

- Old `StorageFile` rows remain valid.
- Old `ChatMessage.attachments` remain readable.
- New messages use `ChatMessage.fileAssets`.
- `LoadFileCommand` first tries File Understanding artifacts, then falls back to legacy loaders.
- `DELETE /api/ai/contexts/file/:storageFileId` removes both the file asset and the storage file.

## Upgrade Guide

### Add a parser

1. Add a parser adapter under `parsers/`.
2. Implement `supports(source)` and `parse(source)`.
3. Return artifacts, capabilities, summary, metadata, and optional status.
4. Register it in `FileParserRegistry` and `FileParsers`.
5. Add new artifact kinds to `domain/types.ts`.
6. Add indexable kinds to `CHUNKABLE_ARTIFACT_KINDS`.
7. Preserve anchors for citations.

### Add OCR or VLM

Recommended places:

- `ImageFileParser` for OCR text, captions, and labels.
- `PdfFileParser` for scanned pages.

Store results as artifacts, not only as summary text. Keep page or path anchors.

### Add vector search

`SearchFileChunksQuery` now uses vector-first retrieval with text fallback:

1. Create embeddings after chunks are built.
2. Create the vector store through `RagCreateVStoreCommand`; the underlying store is supplied by plugin-backed `IVectorStoreStrategy`.
3. Store vector ids in `file_embedding.vectorId`.
4. Scope collections by `xpertId + embeddingFingerprint`, falling back to `projectId` and then `tenantId`.
5. Query the vector store first, then hydrate chunks by id and file permissions; fall back to database text matching when needed.
6. Keep Agent tool schemas unchanged.

### Add production migrations

When TypeORM synchronize is disabled, add migrations for:

- `file_asset`
- `file_artifact`
- `file_chunk`
- `file_citation_anchor`
- `file_embedding`
- `conversation_file_link`
- `chat_message_file_asset`

Keep legacy tables and message structures.

## Maintenance Rules

- Do not add new file-understanding fields to `StorageFile`.
- Do not persist parsed full text into chat messages.
- Do not let Agent tools bypass `FileAsset` and read arbitrary storage paths.
- Do not make ChatKit depend on parser artifact internals.
- Do not write new `AgentFile` payloads into legacy `attachments`.

## Verification

Backend:

```bash
pnpm nx build server-ai
pnpm check:typeorm-columns
```

ChatKit:

```bash
pnpm --filter @xpert-ai/chatkit-ui types
```

Functional checks:

- Text files become `ready` and searchable.
- PDF chunks include page anchors.
- XLSX/CSV can be queried through `file_table_query`.
- Images can be `partial` with `vision` and `ocr` capabilities.
- ZIP files expose manifest/code tree search.
- Large files are not injected into prompts as full text.
- Legacy `StorageFile` attachments still work.
- Deleted files cannot be read through the new APIs.
