# Worker Uploads (REST API)

The worker upload system provides a REST API for external pipelines and scripts
to push pre-processed documents -- with annotations, labels, and embeddings
already attached -- directly into a corpus. Unlike the other upload methods,
worker uploads bypass OpenContracts' built-in parsing pipeline entirely because
the documents have already been processed externally.

## When to Use Worker Uploads

- **External processing pipelines** -- your organization has a custom NLP or
  document processing pipeline that produces structured output
- **Bulk ingestion** -- migrating thousands of pre-annotated documents from
  another system
- **Pre-computed embeddings** -- documents where text extraction, layout
  analysis, and embedding generation have already been performed

## How It Works

Worker uploads use a token-scoped REST API with a database-backed processing
queue:

1. An admin creates a **WorkerAccount** (a service account with no web login)
2. A corpus owner creates a **CorpusAccessToken** scoped to a specific corpus
3. The external pipeline sends documents via `POST /api/worker-uploads/documents/`
   with the token in the `Authorization: WorkerKey <token>` header
4. Uploads are staged in a database queue and processed asynchronously by a
   Celery batch processor
5. Each upload becomes a Document with annotations, labels, and embeddings in
   the target corpus

## Key Differences from Other Upload Methods

| Feature | Worker Uploads | Other Methods |
|---------|---------------|---------------|
| Authentication | Scoped API token (`WorkerKey`) | Session/Auth0 |
| Processing pipeline | Skipped (data pre-processed) | Runs parser, thumbnailer, embedder |
| Annotations | Included in upload metadata | Created during parsing or added manually |
| Embeddings | Can be pre-computed and included | Generated by the embedder pipeline stage |
| Folder assignment | Via `target_folder_path` in metadata | Via UI or mutation parameter |
| Document ownership | Corpus creator (not the worker) | Uploading user |

## REST API Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/worker-uploads/documents/` | Upload a document with metadata |
| GET | `/api/worker-uploads/documents/<upload-id>/` | Check status of a single upload |
| GET | `/api/worker-uploads/documents/list/` | List uploads for this token (paginated) |

All endpoints require the `Authorization: WorkerKey <token>` header.

### Upload Request

The upload is a `multipart/form-data` POST with two fields:

| Field | Type | Description |
|-------|------|-------------|
| `file` | File | The document file (PDF, DOCX, TXT, etc.) |
| `metadata` | String | JSON string with document text, annotations, labels, and embeddings |

### Required Metadata Fields

| Field | Type | Description |
|-------|------|-------------|
| `title` | string | Document title |
| `content` | string | Full extracted text |
| `page_count` | integer | Number of pages |
| `pawls_file_content` | array | PAWLs token data (one entry per page with page dimensions and tokens) |

### Optional Metadata Fields

| Field | Type | Description |
|-------|------|-------------|
| `description` | string | Document description |
| `file_type` | string | MIME type (defaults to `application/pdf`) |
| `target_folder_path` | string | Folder path to place the document in (auto-created if missing) |
| `doc_labels` | array of strings | Document-level label names to apply |
| `doc_labels_definitions` | object | Label definitions for document-level labels |
| `text_labels` | object | Label definitions for text annotations |
| `labelled_text` | array | Text annotations (same format as corpus export annotations) |
| `relationships` | array | Annotation-to-annotation relationships |
| `embeddings` | object | Pre-computed vectors with `embedder_path`, optional `document_embedding`, and `annotation_embeddings` map |

## Upload Lifecycle

```
PENDING  -->  PROCESSING  -->  COMPLETED
                   |
                   +-------->  FAILED
```

- **PENDING** -- staged, waiting for the batch processor
- **PROCESSING** -- batch processor has claimed this upload
- **COMPLETED** -- document created and added to the corpus
- **FAILED** -- check the `error_message` field for details

!!! warning "A Celery worker must drain the queue"
    The batch processor (`process_pending_uploads`) runs on the **`worker_uploads`
    Celery queue**, and thumbnail generation runs on the default `celery` queue.
    The target deployment **must** run a Celery worker consuming both
    (the stock images start it with `-Q celery,worker_uploads`) plus Celery Beat.
    With no worker draining `worker_uploads`, uploads are accepted (HTTP 202) but
    stay `PENDING` indefinitely and **no documents are created** -- a silent
    failure for self-hosted/custom deployments. See
    [Celery Configuration for Worker Uploads](worker_celery_setup.md) for the
    full operations reference (queues, Beat, scaling, verification).

Uploads stuck in PROCESSING for longer than `WORKER_UPLOAD_STALE_MINUTES`
(default: 15 minutes) are automatically reset to PENDING by a recovery task
(`recover_stalled_uploads`, scheduled by Celery Beat).

## Security Model

- **Token storage**: Only a SHA-256 hash is stored; the plaintext is shown
  exactly once at creation
- **Scope**: Each token is scoped to exactly one corpus
- **Worker accounts**: Django users with unusable passwords -- cannot log in via
  web UI
- **Ownership**: Documents are owned by the corpus creator, not the worker
  account
- **Revocation**: Revoking a token immediately prevents authentication; existing
  queued uploads continue processing

## Rate Limiting

Rate limits are set per token via `rate_limit_per_minute` (0 = unlimited).
When exceeded, the API returns HTTP 429 with a `Retry-After: 60` header. The
check is best-effort; for strict enforcement, use a reverse proxy.

## Configuration

| Setting | Default | Description |
|---------|---------|-------------|
| `MAX_WORKER_UPLOAD_SIZE_BYTES` | 256 MiB | Max file size per upload |
| `MAX_WORKER_METADATA_SIZE_BYTES` | 500 MiB | Max metadata JSON size |
| `WORKER_UPLOAD_BATCH_SIZE` | 50 | Uploads claimed per batch processor run |
| `WORKER_UPLOAD_STALE_MINUTES` | 15 | Minutes before stalled uploads reset to PENDING |

## Managing Worker Accounts and Tokens

Worker accounts are created by superusers. Corpus access tokens can be created
by superusers or corpus owners. Both are manageable through the UI (Admin
Settings for accounts, Corpus Settings for tokens) and via GraphQL mutations.

For the complete setup walkthrough with examples, UI screenshots, and the full
metadata schema reference, see the
[Worker Upload System Walkthrough](../worker_uploads/walkthrough.md).

## Higher-Level Driver: The Remote Ingest Worker

You usually do not call this REST API by hand. The
[Remote Ingest Worker](remote_ingest_worker.md) (`scripts/remote_ingest`) is a
turnkey, resumable CLI + docker-compose bundle that runs the **real** Docling
parser and embedder on your own hardware, optionally calculates extra metadata
and annotations, and streams the finished documents to this endpoint -- so the
upload metadata above is produced faithfully for you rather than assembled by
hand. Also mints tokens with one command: `python manage.py mint_worker_token`.
