# Changelog

All notable changes to OpenContracts will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- **Service-layer foundation — `BaseService`, `ServiceResult`, and IDOR-safe `get_for_user_or_none`** (`opencontractserver/shared/services/__init__.py`, `opencontractserver/shared/services/base.py`, `opencontractserver/shared/services/conventions.py`, `opencontractserver/tests/test_base_service.py`). Phase 1 of the service-layer centralization roadmap (`docs/refactor_plans/2026-05-19-service-layer-centralization-design.md`). Introduces the shared machinery every per-app service package will inherit in later phases: `ServiceResult` — a frozen generic write-result envelope (`success` / `failure` constructors, derived `ok`, `(value, error)` tuple-unpacking for backward compatibility with the legacy `(obj, error)` convention); `get_for_user_or_none(model, pk, user, permission=READ, *, request=None)` — an IDOR-safe single-object lookup that returns `None` indistinguishably for not-found, permission-denied, and malformed-pk; and `BaseService`, a staticmethod-based base class exposing `get_or_none` (IDOR-safe lookup), `filter_visible` (permission-filtered queryset via `visible_to_user`), `require_permission` (uniform write-operation gate returning `""` or a denial string), and `log_action` (structured who-did-what logging). No existing callers are migrated in this phase — it ships the primitives only, covered by 23 unit tests. `get_for_user_or_none` takes a single-query fast path for the high-frequency READ case (via `visible_to_user`), falling back to fetch-then-`user_can` only for non-READ permissions. Permission checks delegate to the existing `Model.objects.user_can` / `visible_to_user` manager API; no permission semantics change.

- **Phase F — authorization-invariant gap-fill + permissioning docs refresh** (issue #1660; final phase of the #1637 permission-centralization plan). Closes the remaining gaps in the `user_can` invariant suite and re-points the permissioning docs at the new API.
  - **Invariant suite** (`opencontractserver/tests/permissioning/test_authorization_invariants.py`). Added a top-of-file fixture-matrix audit comment and these focused tests: group-shared READ equivalence for `Corpus` (`CorpusAuthorizationInvariantsTestCase.test_group_shared_read_equivalence`); `is_public` write-asymmetry for `Note` / `Annotation` / `Conversation` / `UserFeedback` (`test_is_public_does_not_grant_writes` — a non-creator/non-shared user gets READ but never UPDATE/DELETE on a public instance, matching the pre-existing `Corpus` / `Document` pins); a `created_by_extract` annotation added to `AnnotationAuthorizationInvariantsTestCase`'s matrix so the equivalence mixin covers the extract privacy-recursion branch in-class; and `ChatMessageAuthorizationInvariantsTestCase.test_chat_message_does_not_inherit_corpus_context` pinning CHAT/THREAD bifurcation at the message layer.
  - **Audit finding — group-perm filter/check drift.** The audit's group-shared coverage surfaced a genuine per-model drift: `Manager.user_can` honours group object-permissions (`_default_user_can` resolves with `include_group_permissions=True`) but `QuerySet.visible_to_user` in `opencontractserver/shared/QuerySets.py` only consults the *user* object-permission table, never the `*groupobjectpermission` table. `Corpus` is unaffected (its queryset handles groups); `Document`, `Annotation`, and `Note` drift. The current behaviour is pinned by `DocumentAuthorizationInvariantsTestCase.test_group_shared_known_drift_user_can_vs_visible_to_user` so it cannot change silently. Closing the drift is a query-path behaviour change left out of this test/docs-only sweep.

### Removed

- **Deleted the deprecated `user_has_permission_for_obj` shim** (issue #1661 — final cleanup of the #1637 permission-centralization plan; `opencontractserver/utils/permissioning.py`). The helper was deprecated in Phase A (#1655) and kept for one release as a `DeprecationWarning`-emitting shim that delegated to `Manager.user_can`, protecting any plugin / external code that imported the symbol. With the grace period elapsed and every in-repo caller migrated in Phases B/C/D/E (#1656, #1657, #1658, #1659), the shim and its per-call-site warning-dedup machinery are now gone — `Model.objects.user_can(user, obj, permission)` / `obj.user_can(user, permission)` is the only single-object authorization API. The ~20 test modules that still imported the symbol for parity testing were migrated to call `obj.user_can(...)` directly; the two shim-only tests (`ShimDeprecationWarningTestCase` in `test_authorization_invariants.py`, `ShimTypeErrorGuardTestCase` in `test_authorization_invariants_coverage.py`) were deleted. Doc references in `CLAUDE.md`, `docs/permissioning/consolidated_permissioning_guide.md`, and the architecture docs were re-pointed at the `user_can` API.

### Security

- **Phase D — IDOR-safe `get_for_user_or_none(model_cls, pk, user)` helper + mutation migration** (`opencontractserver/utils/permissioning.py`, `opencontractserver/tests/permissioning/test_get_for_user_or_none.py`, `opencontractserver/users/models.py`, plus the affected `config/graphql/*_mutations.py` files; closes #1658). Adds a generic non-corpus-scoped lookup helper that returns `None` whenever the pk doesn't exist OR the caller can't READ the row, so mutation call sites render a single unified `"<resource> not found or you don't have permission..."` message and stop leaking object existence via differential error text (CLAUDE.md "IDOR Prevention" — formerly `Model.objects.get(pk=X)` raising `DoesNotExist` vs a separate `user_has_permission_for_obj` permission denial returned distinguishable error states). Helper is generic over the model TypeVar so callers keep concrete `.fieldset` / `.is_public` / etc. attribute typing across the gate, and catches `ValueError`/`TypeError` from garbage pk inputs so untrusted GraphQL ids can't bubble up as 500s.
  - **READ is now a precondition for any other permission op.** Granting UPDATE / DELETE / PERMISSION on an object without an explicit READ codename used to "work" on a handful of mutations because they fetched the row with a bare `Model.objects.get(pk=...)` and then checked the higher permission directly. After Phase D every migrated mutation routes the lookup through `visible_to_user` first, so the caller must be able to see the row before any non-READ check fires. Two pinning tests were updated to grant `[READ, UPDATE]` / `[READ, PERMISSION]` instead of the higher permission alone (`test_user_with_permission_perm_can_change_visibility`, `test_non_owner_with_explicit_update_permission_can_create`, `test_remove_labels_allows_non_owner_with_explicit_update_permission`). `test_invalid_id_format_returns_error` was updated to assert the unified `ok=False` + `"not found or you don't have permission"` response now that the helper catches malformed-pk `ValueError`s instead of letting them bubble as GraphQL `errors`.
  - **Superuser bypass added to `UserProfileManager.visible_to_user`** (`opencontractserver/users/models.py:51`). Mirrors `BaseVisibilityManager`'s superuser branch so admin / moderation surfaces (e.g. badge awarding) can route the recipient lookup through `get_for_user_or_none` and still reach private-profile users — without dropping the IDOR-safe pattern for non-superuser callers (who still only see their own profile + public profiles).
  - **Migrated** (non-corpus-scoped sites; `get_for_user_or_none`): `extract_mutations.py` — `UpdateExtractMutation` (Extract / Corpus / Fieldset lookups) and `CreateExtractIteration` (source Extract). `analysis_mutations.py` — `DeleteAnalysisMutation` (previously raised three different exception types depending on existence / read-permission / delete-permission). `agent_mutations.py` — `CreateAgentConfigurationMutation`, `UpdateAgentConfigurationMutation`, `DeleteAgentConfigurationMutation`. `corpus_mutations.py` — `SetCorpusVisibility`, `UpdateCorpusMutation` preferred_embedder pre-check (previously leaked "this corpus has documents" via the early-exit), `UpdateCorpusDescription`, `DeleteCorpusMutation` `is_personal` pre-check, `StartCorpusFork`, `ReEmbedCorpus`, `CreateCorpusAction`, `AddTemplateToCorpus`, `ToggleCorpusMemory`, `AddDocumentsToCorpus`, `RemoveDocumentsFromCorpus`. `badge_mutations.py` — `AwardBadgeMutation` Badge / Corpus / User-recipient lookups. `label_mutations.py` — `DeleteMultipleLabelMutation` (creator-based permissions on `AnnotationLabel`), `CreateLabelForLabelsetMutation`, `RemoveLabelsFromLabelsetMutation`. `conversation_mutations.py` — `CreateThreadMutation` (Corpus, Document), `CreateThreadMessageMutation`, `ReplyToMessageMutation`, `DeleteConversationMutation`, `DeleteMessageMutation`. `document_mutations.py` — `RetryDocumentProcessing`. `document_relationship_mutations.py` — `CreateDocumentRelationship` (Corpus + source/target Document), `UpdateDocumentRelationship` (Corpus).
  - **Corpus-scoped document lookups** continue to flow through `CorpusObjsService.get_corpus_document_by_id` / `is_document_in_corpus` per PR #1685; the helper's docstring cross-references the service so the two IDOR-safe surfaces don't drift.
  - **`from_global_id` decode pre-guarded at every migrated site.** `get_for_user_or_none` catches malformed-pk `ValueError`/`TypeError` *after* the ORM call, but cannot help if `from_global_id` itself raises (a `binascii.Error` on malformed base64) before the helper is reached — in that case the outer mutation `except Exception` returned a generic message instead of the unified IDOR-safe envelope. The `from_global_id` calls in `agent_mutations.py` (`Create`/`Update`/`Delete AgentConfiguration`) and `conversation_mutations.py` (`CreateThreadMutation` corpus + document arms, `CreateThreadMessageMutation`, `ReplyToMessageMutation`, `DeleteConversationMutation`, `DeleteMessageMutation`) are now wrapped so a malformed id returns the same `"not found / no permission"` response as a genuinely missing or inaccessible resource. `CreateAgentConfigurationMutation`'s malformed-id and not-found branches were unified to emit the identical string. `AddDocumentsToCorpus` / `RemoveDocumentsFromCorpus` now decode the corpus id and the document ids in separate guards so a malformed *document* id no longer returns a misleading corpus-scoped error message.

### Changed

- **Permissioning documentation refresh — `user_can` is the canonical authorization API** (issue #1660; `docs/permissioning/consolidated_permissioning_guide.md`, `CLAUDE.md`, `mkdocs.yml`). Documents the multi-phase permission-centralization outcome:
  - **`Manager.user_can` / `obj.user_can` API** (shipped by PR #1663). The canonical single-object authorization check, paired with the `visible_to_user` queryset filter and pinned to agree by `test_authorization_invariants.py`. The consolidated guide's structural-enforcement banner, "single source of truth" section, inheritance references, overview tables, and Agent/LLM section now point at `AnnotationManager.user_can` / `RelationshipManager.user_can` (`opencontractserver/shared/Managers.py`) instead of `user_has_permission_for_obj` at stale `permissioning.py` line numbers. New "The `user_can` Authorization API" section added with a checklist for extending `user_can` to a new visibility-managed model.
  - **Two-tier permission cache** (PR #1665). New guide subsection: Tier-1 per-instance memoization (always on, used by Celery/agent/internal code) vs. Tier-2 request-scoped optimizer (opt-in via `request=info.context`). Convention: pass `request=` in GraphQL resolvers/mutations, omit it everywhere else.
  - **Service-layer split** (PR #1685). New guide subsection pointing corpus-scoped reads at `CorpusObjsService` and document-level operations at `DocumentService`, with the IDOR rationale for not hand-fusing `corpus.get_documents()` with `Document.objects.visible_to_user(user)`.
  - **Deprecated shim.** `user_has_permission_for_obj` is documented as a `DeprecationWarning`-emitting shim that delegates to `Manager.user_can`; removal is tracked in issue #1661 (one release after Phase E).
  - **Creator-status semantic widening** (Phase A). `user_can` honours creator access uniformly — a creator who never received an explicit guardian grant now passes the check, matching the read-side `visible_to_user` filter. Affected paths: annotation recursive-privacy checks (`AnnotationManager.user_can` honours the creator of the source `Analysis`/`Extract`), non-Corpus mutations that previously called bare `user_has_permission_for_obj`, the CAML article edit gate (`caml_article.py`), and the image tools (`image_tools.py`); the guide's Agent/LLM tool-author guidance now tells tool authors not to re-add bare `creator_id == user.pk` OR-branches.
  - **`docs/permissioning/REMEDIATION_GUIDE.md` deleted** along with its `mkdocs.yml` nav entry — a point-in-time remediation tracker whose `user_has_permission_for_obj`-era findings are fully superseded by the centralized API.

### Fixed

- **`QuerySet.visible_to_user` now honours group object-permissions for Document / Annotation / Note** (issue #1714; `opencontractserver/shared/QuerySets.py`, `opencontractserver/tests/permissioning/test_permissioned_querysets.py`). The Phase F authorization-invariant audit (#1660) surfaced a genuine per-model filter/check drift. `Manager.user_can` resolves group object-permissions — `_default_user_can` runs with `include_group_permissions=True` — but `PermissionQuerySet.visible_to_user`, `DocumentQuerySet.visible_to_user`, and `AnnotationQuerySet.visible_to_user` only joined the *user* object-permission table (`*userobjectpermission`), never `*groupobjectpermission`. A user whose only READ grant on a Document / Annotation / Note was via a **group** therefore passed `obj.user_can(user, READ)` yet was excluded from `Model.objects.visible_to_user(user)` — authorized for a single object but never able to see it in any list query. `Corpus` was unaffected (its `PermissionedTreeQuerySet` resolves group perms via guardian's `get_objects_for_user`). Fix: each affected queryset now ORs in a subquery against the model's `*groupobjectpermission` table filtered by the requesting user's group ids — `DocumentQuerySet` / `PermissionQuerySet` add one `Q(id__in=<group subquery>)`, `AnnotationQuerySet` adds group subqueries to both its document- and corpus-visibility predicates. `NoteQuerySet` needs no code change: it composes `Document.objects.visible_to_user` / `Corpus.objects.visible_to_user`, so it inherits the fix transitively. Group ids are resolved as a lazy `values_list` subquery, so the filter stays a single round-trip — query count is independent of how many groups the user belongs to (pinned by `test_group_lookup_uses_subquery_not_per_group_round_trips`). New `GroupObjectPermissionVisibilityTest` pins filter/check equivalence for a group-shared user across Document, Annotation, Note, and the generic `PermissionQuerySet` body, plus the read/write asymmetry (a group READ grant never widens to UPDATE/DELETE) and non-member exclusion.

- **Stop `EnabledComponentsMutationTestCase` poisoning the `PipelineSettings` singleton cache for later test classes** (`opencontractserver/tests/test_pipeline_settings.py`). The class's `setUp` sets `default_embedder=""`; a rejection-path mutation under test then calls `PipelineSettings.get_instance(use_cache=True)`, which caches the empty-embedder instance in the process-global, non-transaction-aware `LocMemCache`. The DB row rolls back on teardown but the cached Python object does not, so the next test class on the same xdist worker inherited the poisoned singleton and agent / vector-store tests failed with "resolved no embedder_path". Added a `tearDown` that clears `PipelineSettings.CACHE_KEY` after every test in the class, scoping the fix to the one offending class rather than a global autouse fixture.

- **Restore MIN-permission semantic on the GraphQL `CorpusType.documents` resolver — stop private documents leaking through readable corpora** (issue #1682, security; `opencontractserver/corpuses/corpus_objs_service.py`, `config/graphql/corpus_types.py`, `CLAUDE.md`, `opencontractserver/tests/test_corpus_objs_service.py`, `opencontractserver/tests/test_caml_pipeline_coverage.py`). PR #1699's CAML migration silently flipped `corpus_types.py:resolve_documents` from the MIN-permission semantic (a `Document.objects.visible_to_user(user)` intersection — the resolver's documented behaviour) to the corpus-as-gate `CorpusObjsService.get_corpus_documents`. The flip meant any document with an active path in a corpus the requester could merely READ — including a private document inside a shared or public corpus that the requester had no document-level access to — surfaced through the GraphQL `documents` field, contradicting the `MIN(document_permission, corpus_permission)` model documented in `CLAUDE.md` and `docs/permissioning/consolidated_permissioning_guide.md`. The fix follows issue #1682's recommended Option B: corpus-as-gate stays the documented service default for pipeline-facing callers (MCP, discovery, badge/analysis tasks), and a new explicit `CorpusObjsService.get_corpus_documents_visible_to_user(user, corpus, *, include_deleted=False, include_caml=False, request=None)` carries the MIN-permission semantic — it gates corpus READ, then intersects the corpus document set with `Document.objects.visible_to_user(user)`. `resolve_documents` now delegates to the new method, restoring its pre-#1699 behaviour. The two semantics are separately named so no caller can silently swap one for the other; the decision is documented in `CLAUDE.md`'s Permission System section. New `TestGetCorpusDocumentsVisibleToUser` covers the service method (owner / collaborator / stranger / anonymous / superuser, plus `include_caml` / `include_deleted` toggle composition and a corpus-as-gate contrast assertion); new `CorpusResolveDocumentsRespectsDocumentVisibilityTest` is a GraphQL-level regression guard pinning that a collaborator cannot see a private document through `corpus.documents`.

- **`handleDeleteCorpus` now honours the `ok=False` envelope** (`frontend/src/views/Corpuses.tsx`). Phase D converted `DeleteCorpusMutation` from `DRFDeletion` to a `graphene.Mutation` that returns `ok=False` + `message` (instead of raising a `GraphQLError`) when a delete is rejected — e.g. attempting to delete a personal "My Documents" corpus. The view's `.then()` handler unconditionally fired `toast.success("SUCCESS! Deleted corpus.")` on any resolved promise, so a rejected delete resolved successfully and showed a false success toast. The handler now branches on `data.deleteCorpus.ok` and surfaces `data.deleteCorpus.message` on failure.

- **Mobile layout of discussion thread header — stop description overflow and shrink the breadcrumb pill** (`frontend/src/components/threads/ThreadDetail.tsx`, `frontend/src/components/routes/CorpusThreadRoute.tsx`). The `TitleRow` in `ThreadDetail` was a single non-wrapping flex row holding the QUESTION badge, the serif title, and the description, with the description set to `flex: 1; white-space: nowrap; text-overflow: ellipsis`. On a ~540 px phone viewport the description's single-line ellipsis collapsed to zero width but still rendered its first words past the right edge of the white card, producing the cut-off "Discussing purpose of this[overflow]" seen in the bug screenshot. Added `flex-wrap: wrap; row-gap: 0.375rem` to `TitleRow`, made `Title` `flex-basis: 100%` with `white-space: normal; word-break: break-word; font-size: 1.0625rem` under `@media (max-width: 768px)` so the title gets its own line, and switched `Description` to a 2-line `-webkit-line-clamp` block (`flex: 1 1 100%; overflow-wrap: anywhere`) so subtitles wrap below the title instead of overflowing. `MetaRow` gains `flex-wrap: wrap; row-gap: 0.25rem` so the "author · time · message count" row can break cleanly on narrow widths. `Header` gets `min-width: 0` to honour the wrap, and slightly more vertical padding on mobile (`0.75rem 1rem`). In `CorpusThreadRoute` the breadcrumb's `CorpusLink` could grow to 2–3 wrapped lines holding a long case title; wrapped the title in a new `CorpusLinkLabel` span with `overflow: hidden; text-overflow: ellipsis; white-space: nowrap`, gave `CorpusLink` `min-width: 0; max-width: 100%` plus `flex: 1 1 auto` on mobile, and pinned `flex-shrink: 0` on the folder icon — so the pill now truncates to a single line with ellipsis on narrow screens (full title still available via the existing `title=` tooltip).

- **Restore `.with_tree_fields()` for superusers on `PermissionedTreeQuerySet.visible_to_user`** (issue #1668; `opencontractserver/shared/QuerySets.py`, `opencontractserver/tests/test_resolvers.py`, `opencontractserver/tests/test_visibility_managers.py`, `opencontractserver/tests/permissioning/test_authorization_invariants_coverage.py`). Phase A follow-up. The superuser branch previously returned `self.all().order_by("created")`, which silently dropped `.with_tree_fields()` from the queryset — superusers were missing the `tree_depth` / `tree_path` annotations that every other branch supplied via `with_tree_fields()`. Now returns `self.all().with_tree_fields()`, matching the non-superuser branches. As a hygiene side-effect, the asymmetric `order_by("created")` is also removed from the superuser branch of `BaseVisibilityManager.visible_to_user`; ordering is owned by the resolver / caller, consistent with the rest of the codebase. Two superuser `order_by` assertions in `test_resolvers.py:71` and `test_visibility_managers.py:83` updated to assert no manager-imposed ordering. Also deletes the now-redundant `DocumentManager.user_can` passthrough (MRO falls through to `BaseVisibilityManager.user_can` → `UserCanMixin.user_can` → `_default_user_can` for identical behaviour) and tightens `Any` annotations on `user_can` signatures across six managers and the mixin.

- **Eliminate N+1 permission loops in `get_analysis_annotations` / `get_extract_datacells`** (issue #1692; `opencontractserver/annotations/query_optimizer.py:1259-1271`, `:1443-1455`; `opencontractserver/tests/permissioning/test_query_optimizer_methods.py`). Phase C cleanup follow-up. Both methods previously iterated `analysis.analyzed_documents.all()` / `extract.documents.all()` in Python, calling `doc.user_can(user, PermissionTypes.READ)` per document — O(N) permission checks with full-row deserialisation on each loop step. Replaced both loops with a single `Document.objects.visible_to_user(user).filter(id__in=<m2m subquery>.values("id")).values_list("id", flat=True)` queryset intersection, which already encodes the full visibility model (creator + is_public + guardian + corpus membership) and matches the pattern in use elsewhere in the same file (`query_optimizer.py:745-752`). Behaviour preserved: same readable-doc-id set, same early `Annotation.objects.none()` / `Datacell.objects.none()` exit when nothing is readable. New `DocumentPermissionFilterQueryCountTestCase` pins behaviour parity against a mix of readable / non-readable documents and asserts a bounded query count for the visibility filter regardless of N.

### Changed

- **Test suite optimization Phase 3 — stop `BaseFixtureTestCase` reloading the 18 MB fixture on every test** (issue #1711; `opencontractserver/tests/base.py`, `opencontractserver/tests/fixtures/test_data.json`, `opencontractserver/tests/test_fixture_base_classes.py`, `docs/test_scripts/regenerate_structural_test_data_fixture.md`, and 6 dependent test files). `BaseFixtureTestCase` declared `fixtures = ["…/test_data.json"]` **and** extended `TransactionTestCase`, so Django's `loaddata` re-loaded the 7,584-object / 18 MB fixture before *every* test method instead of once per class, and `setUp()` re-ran a structural-annotation migration loop plus per-document file/summary work each time.
  - **`base.py` split into a `TestCase` default + a `TransactionTestCase` variant.** Shared setup moved into a `_FixtureSetupMixin`. `BaseFixtureTestCase` is now `TestCase`-backed: the fixture and the derived corpus/document state (`add_document` corpus isolation, guardian grants) build **once per class** in `setUpTestData`, inside the class transaction; each test still runs in its own savepoint. The new `TransactionFixtureTestCase` is `TransactionTestCase`-backed for tests that need committed-data semantics; `WebsocketFixtureBaseTestCase` and `CeleryEagerModeFixtureTestCase` now extend it. Only the fixture-file copy into `MEDIA_ROOT` stays per-test — the autouse `media_storage` fixture repoints `MEDIA_ROOT` at a fresh tmpdir for every test, so the files must be re-materialised into it.
  - **`test_data.json` regenerated in the current schema and slimmed** (7,584 → 4,284 objects; procedure documented in `docs/test_scripts/regenerate_structural_test_data_fixture.md`). The 1,344 structural annotations and 472 structural relationships were migrated from the legacy `Annotation.document` schema to `StructuralAnnotationSet` (4 sets added, one per document); `Document.md_summary_file` was populated with the existing `files/md_summaries/{pk}_summary.md` paths; and the 3,304 `guardian.userobjectpermission` rows for `annotations.relationship` were dropped — relationships carry no individual object permissions in the current model (they inherit from document + corpus). This makes the per-test structural-migration loop and the convention-based `md_summary` discovery / placeholder-regeneration block in `setUp()` dead code; both were deleted.
  - **Variant assignment driven by what each file's tests actually need, verified by running them.** `async def` agent tests fail under `TestCase` with `psycopg2.InterfaceError: connection already closed`, and `transaction.on_commit`-dispatched analyzer tasks produce no `DocumentAnalysisRow` rows when the test transaction never commits — so those files keep `TransactionTestCase` semantics. Moved to `TransactionFixtureTestCase`: `test_extract_analyzer_tools.py` (async `sync_to_async(thread_sensitive=False)` tests), `test_structured_response_api.py` / `test_structured_response_simple.py` / `permissioning/test_public_and_permissions.py` (async agent tests), `test_individual_extract_tasks.py` (`celery_task_with_async_to_sync` task), `test_agentic_highlighter_task.py` (analysis dispatched via `transaction.on_commit`). The synchronous GraphQL/ORM files `test_note_tree.py` and `test_export_mutations.py` stay on the `TestCase`-backed `BaseFixtureTestCase`.
  - **Every dependent file benefits, including those kept on `TransactionFixtureTestCase`:** the slimmer fixture (44 % fewer objects → fewer `loaddata` INSERTs per test) and the deleted per-test `setUp()` work (no 1,344-annotation + 472-relationship migration loop, no per-document summary-file generation).
  - **Measured delta:** for `test_note_tree.py` the per-test `setup` under the old `TransactionTestCase` base was 129 s / 28 s (the smoke run's two test-class first methods, fixture reloaded before every test); under the new `TestCase` base the fixture loads once per class (~18 s, attributed to each class's first test) after which per-test `setup` is negligible.
  - **AC pinned by `test_fixture_base_classes.py`** — a `setUpTestData` load counter asserts the fixture/corpus build runs once per class for `TestCase`-backed subclasses, alongside base-class wiring assertions (`BaseFixtureTestCase` → `TestCase`; `TransactionFixtureTestCase` / `WebsocketFixtureBaseTestCase` → `TransactionTestCase`).

- **Test suite optimization Phase 2 — convert 160 needlessly-`TransactionTestCase` classes to `TestCase`** (issue #1710; 28 files under `opencontractserver/tests/` and `opencontractserver/tests/permissioning/`). `TransactionTestCase` `TRUNCATE`s every table and re-emits the `post_migrate` signal (`create_permissions`, `create_contenttypes`, django-guardian setup) between *every* test method; `TestCase` instead wraps each test in a transaction and rolls it back — far cheaper per-test overhead, and the highest-leverage structural lever for bringing the ~1 hr backend suite under 20 minutes. All 235 classes that extend `TransactionTestCase` directly were audited against the patterns that genuinely require transaction semantics, and the 160 pure-synchronous ORM / GraphQL / permission / management-command classes were reclassified to `django.test.TestCase`.
  - **Highest-yield files:** `test_corpus_objs_service.py` (63 classes, via 3 converted base classes plus direct extenders), `test_corpus_export_import_v2.py` (15), `permissioning/test_authorization_invariants.py` (10), `test_v3_migration.py` (10), `permissioning/test_authorization_invariants_coverage.py` (9), `permissioning/test_permission_optimizer.py` (8), `test_corpus_fork_round_trip.py` (5), plus `test_badges.py`, `test_notifications.py`, `test_worker_uploads.py`, `test_corpus_forking.py`, `test_personal_corpus.py`, `test_analyzer_app_coverage.py`, and ~16 other files.
  - **Kept on `TransactionTestCase` (75 classes):** every class with an `async def` test method / `await` body, OS-thread races (`Thread(` / `ThreadPoolExecutor` — `EmbeddingManagerConcurrentTest`, `Tier1CacheThreadSafetyTestCase`, `TestConcurrentPersonalCorpusCreation`), and genuine `transaction.on_commit` tests (`InstallGremlinOnCreationTests`, `TestThreadCorpusActionSignals`, `CorpusForkOrphanedBlobGCTest`, `TestMoveDocumentAsync`). `test_blob_retention.py` is left entirely untouched — its blob garbage-collection assertions depend on real `on_commit` deletion callbacks firing, which `TestCase` (no commit inside the test) silently skips.
  - **`async_to_sync` failure mode pinned by running the suite.** A first conversion pass also flipped the `async_to_sync`-of-Django-ORM classes; running them under `pytest -n 4 --dist loadscope` surfaced `psycopg2.InterfaceError: connection already closed` — `async_to_sync` of ORM code recycles the DB connection mid-test, destroying the long-lived transaction `TestCase` depends on (`TransactionTestCase`'s autocommit-per-statement model tolerates it). This is broader than the issue's original grep checklist, which only named `async def` / `WebsocketCommunicator` / `database_sync_to_async`. Those classes were reverted to `TransactionTestCase`: the memory-CRUD tests in `test_agent_memory.py` (which call `async_to_sync` directly) and the whole of `test_extract_tasks.py`, `test_individual_extract_tasks.py`, `test_pii_scanner_analyzer_task.py`, and the async-wrapper tests in `test_caml_review_tools.py` (whose code under test runs `async_to_sync` internally). `test_multimodal_integration.py` was likewise left as-is — it could not be runtime-verified in this environment (real Docling I/O).
  - **No behaviour or coverage change** — `TestCase` vs `TransactionTestCase` only changes DB isolation. `transaction.atomic()` blocks inside the code under test become savepoints, so the existing rollback / mocked-`IntegrityError` regression tests (`TestErrorPaths_*AtomicRollback`, `RollbackAndEdgeCaseTests`, `TestMoveDocumentIntegrityRecovery`, …) execute identical paths and assert identical outcomes.
  - **Measured delta:** an A/B run of `test_corpus_objs_service.py` (the largest converted file, 63 classes / ~200 tests) under `pytest -n 4 --dist loadscope --reuse-db` drops from **244 s on `TransactionTestCase` to 29 s on `TestCase`** — an ~8× reduction, all tests passing both ways. Under `TransactionTestCase` even a trivial `assertRaises(ValueError)` test costs ~5 s because of the inter-test `TRUNCATE` + `post_migrate` re-emit; under `TestCase` the same test is ~0.01 s.
  - **`conftest.py::make_create_permissions_xdist_safe` retained.** That fixture papers over the cross-worker `auth_permission` `IntegrityError` race introduced by `TransactionTestCase._fixture_teardown`'s `flush` → `emit_post_migrate_signal`; it can only be deleted once *no* `TransactionTestCase` classes remain, and 75 still do.

- **Skip coverage instrumentation on pull-request CI runs** (`.github/workflows/backend.yml`). The Backend CI `pytest` invocation now passes `--cov --cov-report=xml` only on `push` events (merges to a protected branch); pull-request runs execute the suite without coverage. coverage.py's trace function — the C tracer, since the test image runs Python 3.11 and `sys.monitoring` / `COVERAGE_CORE=sysmon` requires 3.12+ — adds noticeable wall time to the full ~6.4k-test suite, paid on every PR push. Coverage is still measured on every merge, so Codecov's tracked metric on `main` is unchanged; only the per-PR coverage delta is no longer computed. The `Verify Coverage File Exists` and `Upload Coverage Reports to Codecov` steps are gated to `push` accordingly. **Branch-protection note:** if a Codecov `patch` / `project` status for the `backend` flag is configured as a *required* status check, it must be made non-required (or `informational`) before this merges — `.codecov.yml` currently leaves `coverage.status.patch.default` non-informational. First step of a broader effort to bring the backend suite under 20 minutes.

- **Sweep stale `Corpus.get_documents()` docstrings and test fixtures** (issue #1684; `opencontractserver/shared/QuerySets.py`, `opencontractserver/tests/test_document_path_migration.py`, `opencontractserver/tests/test_corpus_fork_round_trip.py`, `opencontractserver/tests/test_corpus_export.py`, `opencontractserver/tests/test_corpus_forking.py`, `opencontractserver/tests/test_bulk_document_upload.py`, `opencontractserver/tests/test_versioning_migration.py`, `opencontractserver/tests/test_agentic_highlighter_task.py`, `opencontractserver/tests/test_extract_analyzer_tools.py`). Migrated 32 fixture-style `corpus.get_documents()` call sites across 8 test files to `corpus._get_active_documents()` — the blessed internal alias the deprecated wrapper already delegates to (identical semantics, same active-path predicate). Eliminates `DeprecationWarning` noise from setup helpers without touching the per-user `CorpusObjsService` path reserved for request-context code. Also updated two docstrings in `QuerySets.py` (lines 34 and 355) that named the predicate via the deprecated public method. `test_corpus_get_documents_deprecation.py` is intentionally left unchanged — it exercises the deprecation warning. Out of scope: the `DocumentFolderService` back-compat shim and the production GraphQL / agent-tool call sites that still call `DocumentFolderService.X()` / `corpus.get_documents()` directly; those move when the alias is removed.

- **Thread GraphQL context through optimizer helpers for Tier-2 cache hits** (issue #1691; `opencontractserver/annotations/query_optimizer.py`, `opencontractserver/documents/query_optimizer.py`, `config/graphql/{action_queries,annotation_queries,corpus_queries,extract_queries,extract_types}.py`). Follow-up to Phase C (#1689). Five `AnnotationQueryOptimizer` / `AnalysisQueryOptimizer` / `ExtractQueryOptimizer` helpers — `get_corpus_annotations`, `check_analysis_permission`, `get_visible_analyses`, `check_extract_permission`, `get_visible_extracts` — plus the intermediate `DocumentActionsQueryOptimizer.get_document_actions` now accept an optional `context: Optional[Any] = None` kwarg and thread it into every `user_can(...)` call as `request=context`. The Tier-2 request-scoped permission cache introduced in PR #1665 was dormant on these helpers because their GraphQL callers had `info.context` in scope but did not propagate it. After this PR every GraphQL caller (`annotation_queries`, `corpus_queries`, `extract_queries`, `extract_types` get_node, `action_queries`) passes `info.context` through. Backward-compatible: `None` default keeps test callers (`test_query_optimizer_methods.py`, `test_corpus_annotations_query.py`) and any Celery / internal sync code on the Tier-1 instance cache. No behaviour change beyond cache-hit rate.

- **Retire `DocumentFolderService` back-compat shim — migrate all importers to `DocumentService` / `CorpusObjsService`** (issue #1680). PR #1685 split the monolithic `DocumentFolderService` along its natural seam and kept the original class as a `class DocumentFolderService(CorpusObjsService, DocumentService)` shim in `opencontractserver/corpuses/folder_service.py` so the ~18 existing importers could migrate later. This change finishes that migration:
  - **Production call sites (9 files):** `config/graphql/{corpus_folder_mutations,corpus_mutations,corpus_queries,document_mutations,document_types}.py`, `opencontractserver/tasks/import_tasks.py`, `opencontractserver/llms/tools/core_tools/documents.py`, `opencontractserver/mcp/server.py`, `opencontractserver/utils/import_v2.py` — every `DocumentFolderService.X(...)` rewritten to `CorpusObjsService.X(...)` based on the partition documented in `test_service_split_method_partition` (`opencontractserver/tests/test_corpus_objs_service.py:46`). None of these call sites hit a `DocumentService`-routed method.
  - **Test call sites (3 files):** `opencontractserver/tests/test_permanent_deletion.py`, `opencontractserver/tests/test_zip_import_integration.py`, `opencontractserver/tests/permissioning/test_permission_optimizer.py` — same surgical rewrite to `CorpusObjsService`.
  - **Test suite merge:** `opencontractserver/tests/test_document_folder_service.py` (4423 lines) and the existing `opencontractserver/tests/test_corpus_objs_service.py` (523 lines) collapsed into a single `opencontractserver/tests/test_corpus_objs_service.py` (4925 lines). The two test classes that exercise `DocumentService` methods (`TestEdgeCases_UploadQuota`, `TestValidateFileType`) now call `DocumentService.check_user_upload_quota` / `DocumentService.validate_file_type` directly and inherit from `TransactionTestCase` rather than the folder-test base. The folder-test base itself was renamed from `DocumentFolderServiceTestBase` to the underscore-prefixed `_CorpusObjsServiceFolderTestBase` so it no longer clashes with the existing public `CorpusObjsServiceTestBase` fixture for the convenience-method tests.
  - **Shim removed:** `opencontractserver/corpuses/folder_service.py` deleted along with the `TestServiceSplit_BackwardsCompatAlias` class in `opencontractserver/tests/test_document_service.py` and the surrounding docstring bullet that explained the alias. The disjoint-partition test (`TestServiceSplit_MethodPartition`) survives — it still pins the invariant that `DocumentService.__dict__` and `CorpusObjsService.__dict__` share no method names, which is exactly what made the two-parent inheritance unambiguous in the first place and remains the load-bearing assertion behind the split.
  - **Stale docstring sweep:** comment / docstring mentions of "DocumentFolderService" / "folder_service" in `opencontractserver/corpuses/{models.py,corpus_objs_service.py}`, `opencontractserver/mcp/resources.py`, `opencontractserver/tests/{test_document_stats.py,test_import_utils.py,test_move_document_tool.py}`, the `CLAUDE.md` follow-up bullet at line 245, and the active architecture / permissioning docs (`docs/architecture/{bulk-import.md,data_model/documents_and_annotations.md}`, `docs/permissioning/REMEDIATION_GUIDE.md`, `docs/plans/documentation_alignment_review.md`) reworded to name the new services. The three historical standalone docs (`docs/architecture/document_folder_service{,_plan,_migration_inventory}.md`) and the mkdocs nav entry pointing at the first of those deleted along with the shim — they document a class that no longer exists. `docs/releases/v3.0.0.b3.md` is intentionally untouched (release notes are an immutable snapshot of the release that shipped); `docs/typing/mypy_baseline.txt` carries stale entries that point at the renamed test file but will be regenerated by the next mypy run.

- **CAML-aware corpus-objs service method + caml_article tooling migration** (issue #1681; `opencontractserver/corpuses/corpus_objs_service.py`, `opencontractserver/llms/tools/core_tools/caml_article.py`, `config/graphql/corpus_types.py`, `opencontractserver/tests/test_corpus_objs_service.py`). Follow-up to PR #1685's service split: the two CAML-aware production paths that were deliberately left calling `Corpus._get_active_documents(include_caml=True)` directly now route through `CorpusObjsService`. New `CorpusObjsService.get_corpus_caml_articles(user, corpus, *, request=None)` returns the CAML article documents (`title="Readme.CAML"`, `file_type="text/markdown"`) under the same corpus-as-gate semantic as the rest of the service surface — empty queryset on permission denial, IDOR-safe. Plural shape mirrors `get_corpus_documents` so future multi-article designs do not require a signature change. `caml_article.py:_load_caml_document_for_user` now delegates to the new method, dropping the manual `Corpus.objects.visible_to_user(user)` + `_get_active_documents(include_caml=True)` + `Document.objects.visible_to_user(user)` fusion (and the now-unused `MARKDOWN_MIME_TYPE` import); the IDOR-safe identical-`ValueError` contract is preserved. `corpus_types.py:resolve_documents` similarly delegates to `CorpusObjsService.get_corpus_documents(user, self, include_caml=True, request=info.context)`, retiring its own copy of the fusion and threading the Tier-2 request-scoped permission cache through. Net result: every CAML-aware corpus read passes through one entry point, the `Corpus.get_documents()` deprecation surface stays cold on these paths, and the manual-fusion IDOR pattern CLAUDE.md warns against is gone from the codebase.

- **Phase E permission centralization — migrate field resolvers, agent tools, Celery tasks, websocket consumer, and import services to `obj.user_can(...)`** (issue #1659; `config/graphql/document_types.py`, `config/graphql/extract_types.py`, `config/websocket/consumers/unified_agent_conversation.py`, `opencontractserver/llms/agents/agent_factory.py`, `opencontractserver/llms/tools/image_tools.py`, `opencontractserver/llms/tools/core_tools/caml_article.py`, `opencontractserver/llms/tools/core_tools/extracts_and_analyzers.py`, `opencontractserver/llms/tools/core_tools/memory.py`, `opencontractserver/tasks/doc_tasks.py`, `opencontractserver/tasks/extract_orchestrator_tasks.py`, `opencontractserver/document_imports/services.py`, `opencontractserver/tests/test_caml_review_tools.py`). The remaining 20 `user_has_permission_for_obj` call sites that lived outside GraphQL mutations now route through `obj.user_can(...)`. After Phase E the in-repo call count of `user_has_permission_for_obj` drops to zero; the shim survives one release cycle for plugin/external code (deletion tracked in #1661). Net effect:
  - **Cache plumbing.** Six GraphQL field resolvers (`DocumentType.resolve_can_restore` x2, `DocumentType.resolve_can_view_history`, `DocumentType.resolve_can_retry`, `ExtractType.resolve_document_count`, `ExtractType.resolve_full_document_list`) now pass `request=info.context` so the Tier-2 request-scoped cache from PR #1665 fires on the per-row permission checks that previously ran cold. Agent tools, Celery tasks, the websocket consumer, and import services have no GraphQL request in scope and omit the kwarg — they degrade gracefully to Tier-1 instance caching.
  - **Creator-honoring uniform across agent tools.** The `_apply_caml_article_edit` site drops the legacy `locked_doc.creator_id == user.pk` OR-branch because `user_can` now honours creator access uniformly per Phase A. The `image_tools.py`, `extracts_and_analyzers.py`, and `memory.py` sites similarly switch from "do I have READ/UPDATE/CRUD on this resource" to "can I (per the new uniform creator-honoring path) act"; a creator who never received an explicit guardian grant now passes the gate, matching the read-side `visible_to_user` filters. Pinned by `test_creator_without_explicit_perm_can_edit` in `test_caml_review_tools.py`.
  - **Group-permission widening.** Five call sites (`unified_agent_conversation.py` x2, `agent_factory.py`, `memory.py`, `caml_article.py`, `services.py` x2) previously relied on the shim's legacy `include_group_permissions=False` default. The new `obj.user_can(...)` defaults to `True`, matching `_default_user_can` and every other `Manager.user_can` / `obj.user_can` surface. Users who hold the relevant guardian permission via a group on Corpuses / Documents / Extracts now pass these websocket/agent/import gates; this matches the documented per-model policy.
  - **No remaining `user_has_permission_for_obj` calls in any of the migrated files** — verified by grep. The handful of surviving references in `opencontractserver/utils/permissioning.py`, `opencontractserver/shared/`, `opencontractserver/annotations/query_optimizer.py`, etc. are docstring / comment mentions describing the shim and its Phase A / B / C / E migration history.

- **xdist-safe `auth_permission` teardown — kill the cross-worker `IntegrityError` cascade** (`conftest.py:15-67`). Every `TransactionTestCase._fixture_teardown` calls Django's `flush` which fires `emit_post_migrate_signal` → `django.contrib.auth.management.create_permissions` → `Permission.objects.bulk_create(perms)`. Under `pytest -n 4 --dist loadscope`, workers race on the shared `auth_permission` table (postgres content_type ids overlap across the per-worker test DBs because they're assigned serially during migrate), and the `(content_type_id, codename)` unique constraint produces `UniqueViolation` for whichever worker loses. The loser's teardown then dies un-rolled-back, the next test's `setUp` finds an empty fixture, and a cascade of `User.DoesNotExist: User matching query does not exist` setUp failures follows — surfacing as ~20+ apparently-random failures in `test_agent_permission_escalation.py`, `test_unified_agent_consumer.py`, etc. Phase E itself didn't cause the race; it concentrated enough fixture-heavy classes into one dispatch to make it routinely fire. Fix is a session-scoped `autouse=True` fixture in the root `conftest.py` that re-registers the `post_migrate` handler with a thin wrapper: `with transaction.atomic(using=using): create_permissions(...)` plus `except IntegrityError: pass`. The conflicting rows are exactly the permissions we'd have inserted anyway (winning worker already wrote them), so the swallow is a true no-op and the atomic block guarantees no half-inserted state leaks past the rollback. The fixture restores the original handler at session teardown, so no production import path is affected.

- **Phase C permission centralization — migrate optimizer call sites to `obj.user_can(...)`** (issue #1657; `opencontractserver/annotations/query_optimizer.py`, `opencontractserver/extracts/query_optimizer.py`). The 24 remaining `user_has_permission_for_obj` call sites across `AnnotationQueryOptimizer` (analysis/extract/datacell/corpus-annotation/document-relationship paths) and `MetadataQueryOptimizer` (`_compute_effective_permissions`, `get_corpus_metadata_columns`, `get_documents_metadata_batch`, `check_metadata_mutation_permission`) now route through `obj.user_can(...)` — completing the substitution PR #1663 started in `AnnotationQueryOptimizer._compute_effective_permissions`. Net effect:
  - **Cache plumbing.** Three of the migrated sites (`AnnotationQueryOptimizer.get_document_annotations` analysis check, `RelationshipQueryOptimizer.get_document_relationships` analysis check, `MetadataQueryOptimizer.get_documents_metadata_batch` corpus check) now pass `request=context` so the Tier-2 request-scoped cache (PR #1665) fires for the optimizer's existing analysis/corpus dereferences. The 21 other migrated sites live in helpers without a GraphQL request in scope (`check_analysis_permission`, `get_visible_extracts`, `_compute_effective_permissions` on the extract path, etc.) and degrade gracefully to Tier-1 instance caching. The annotation optimizer's bespoke `context._effective_perms_cache` (Phase A) is untouched — it continues to sit on top of `permission_cache` / `permission_optimizer`.
  - **Behavioral preservation.** The MIN(doc, corpus) bodies in both `_compute_effective_permissions` methods are unchanged; only the per-call shim invocations move. BACON-mode (`corpus.allow_comments`) branch ordering in the annotation optimizer is preserved verbatim per the issue's pin. The legacy `include_group_permissions=True` argument drops because `obj.user_can(...)` defaults to `True` (matching every other `Manager.user_can` / `obj.user_can` surface, and matching what these call sites were already passing explicitly).
  - **No remaining `user_has_permission_for_obj` calls in either optimizer file** — verified by grep. The two surviving references in `annotations/query_optimizer.py` are docstring mentions describing the Phase A migration.

- **Bulk permission collapse in `MetadataQueryOptimizer._compute_effective_permissions`** (issue #1690; `opencontractserver/extracts/query_optimizer.py:34-126`, `opencontractserver/tests/permissioning/test_metadata_query_optimizer.py`). The Phase C migration left the method issuing 8 sequential `user_can` calls (4 per object × 2 objects); since `get_document_metadata` and `get_metadata_completion_status` don't thread a GraphQL `request`, the Tier-2 `PermissionQueryOptimizer` never fires and each call paid full cold-path overhead. The method now fetches the guardian codename set once per object via `get_users_permissions_for_obj` (Tier-1 cached) and resolves the 4 permission types via set membership (`"read_document" in granted`, etc.). Creator short-circuit is preserved verbatim to match `_default_user_can` semantics on instances without explicit guardian grants. Two new `assertNumQueries` regression tests (`test_compute_permissions_query_count_superuser_is_zero` and `test_compute_permissions_query_count_collaborator_is_bounded`) lock the budget so any future reintroduction of per-permission fan-out fails CI. Tier-2 batching across both objects is deferred to sister issue #1691, which threads `request` into this code path.

- **Phase B permission centralization — replace compound `user_has_permission_for_obj` checks with `obj.user_can(...)`** (issue #1656; `config/graphql/{base,agent_mutations,analysis_mutations,annotation_mutations,badge_mutations,conversation_mutations,corpus_mutations,document_mutations,document_relationship_mutations,extract_mutations,ingestion_source_mutations,label_mutations,smart_label_mutations,voting_mutations}.py`, `opencontractserver/documents/query_optimizer.py`). Every mutation site that previously composed `user.is_superuser or obj.creator_id == user.id or user_has_permission_for_obj(user, obj, perm, include_group_permissions=True)` now calls the single `obj.user_can(user, perm, request=info.context)` surface introduced in PRs #1663 and #1665. Net effect:
  - **Cache plumbing.** Each migrated check passes `request=info.context` so the Tier-2 request-scoped permission cache from PR #1665 actually fires; previously the cache stayed dormant across mutations. Every `set_permissions_for_obj_to_user(...)` call in the mutation layer also now passes `request=info.context` (plus `is_new=True` on fresh-creation paths) so Tier-1 / Tier-2 cache invalidation lands on grant / revoke.
  - **Creator-status widening across non-Corpus mutations.** A handful of legacy call sites passed `include_group_permissions=False` (the shim's default). The new `obj.user_can(...)` defaults to `True`, matching `_default_user_can` and every other `Manager.user_can` / `obj.user_can` surface. Users who hold the relevant guardian permission via a group on Documents / Extracts / Annotations / Notes / Conversations / Labels / Badges now pass the relevant mutation gate; this matches the documented per-model policy and the read-side `visible_to_user` queryset filters.
  - **`DocumentActionsQueryOptimizer._check_document_permission` / `_check_corpus_permission` deleted** (`opencontractserver/documents/query_optimizer.py`); five call sites in `get_document_actions`, `get_corpus_actions_for_corpus`, `get_extracts_for_document`, `get_analysis_rows_for_document`, and `DocumentRelationshipQueryOptimizer.get_relationships_for_document` now call `document.user_can(user, READ)` / `corpus.user_can(user, READ)` directly. `DocumentRelationshipQueryOptimizer.user_has_permission` similarly drops its three compound checks. The two callsites with no GraphQL request in scope omit `request=` and degrade gracefully to Tier-1 instance caching.
  - **`_resolve_annotation_parents` and `_clone_fieldset_for_iteration` gained `request=None` kwargs** so their callers thread `info.context` through, keeping the cache lit on the helper paths that were previously dark.
  - **No behaviour change for the corpus-level checks** that already used `include_group_permissions=True` — those collapse to a single call against the same default-True API.
  - Out of scope (deferred to Phase D / E per issue #1658 / #1659): IDOR-unified-error refactors, field-resolver / agent-tool / Celery / import-service migrations, and the websocket consumer in `unified_agent_conversation.py`.

- **Permission cache hardening — review follow-ups on PR #1665** (`opencontractserver/utils/{permissioning,permission_optimizer}.py`, `opencontractserver/shared/user_can_mixin.py`, `opencontractserver/constants/permissioning.py`, `opencontractserver/conversations/models.py`, `opencontractserver/tests/permissioning/test_permission_optimizer.py`, `opencontractserver/shared/Managers.py`).
  - Aligned `get_users_permissions_for_obj`'s `include_group_permissions` default to `True`, matching `PermissionQueryOptimizer.get_granted`, `_default_user_can`, and every `Manager.user_can` / `obj.user_can` surface. One default, one answer — eliminates the latent footgun where a caller mixing the Tier 1 helper and the Tier 2 optimizer would get different group-permission semantics without passing the flag. The legacy `user_has_permission_for_obj` shim deliberately keeps its `False` default (its ~170 callers were written against that behaviour).
  - Added a `threading.Lock()` around `PermissionQueryOptimizer._cache` reads/writes/iterates. Cheap insurance against future shapes that violate the one-request-one-thread assumption (multi-user delegation, background threads spun off mid-request, service-account sharing). The DB-bound `get_users_permissions_for_obj` call still runs outside the lock so guardian lookups never serialise on it.
  - Replaced the raw Tier 1 `dict` with `_InstancePermsCache(dict)` — a thread-safe subclass exposing `drop_for_user(user_id)` that holds an internal `Lock` for the compound iterate-then-delete sweep in `set_permissions_for_obj_to_user`. Individual `cache[key]` reads/writes still ride CPython's GIL atomicity; the lock exists for the compound op only. `_get_or_create_instance_perms_cache` uses `instance.__dict__.setdefault` so two threads first-touching the same instance converge on a single cache rather than racing via `getattr`+`setattr`. Closes the Tier 1 ASGI / async-view exposure flagged in PR review. New `Tier1CacheThreadSafetyTestCase` pins the contract under N=4 reader-thread contention against repeated invalidate sweeps.
  - Backwards-compat fallback in `set_permissions_for_obj_to_user`: if an instance carries a plain-`dict` cache (e.g. unpickled from an older worker or attached by a test fixture), the helper falls back to a snapshot-then-`.pop()` invalidation pattern so `RuntimeError` cannot escape even when the thread-safe wrapper is absent.
  - Added explicit `request: Any = None` annotation on `ConversationManager.user_can` and `ChatMessageManager.user_can` (previously inferred as `None` by type checkers, which would reject real request objects at the type level).
  - `InstanceUserCanMixin.__getstate__` strips `INSTANCE_PERMS_CACHE_ATTR` before pickling. Any `BaseOCModel` (or `Corpus` / `CorpusFolder`) passed verbatim to a Celery task — the well-known anti-pattern — can no longer carry a stale Tier 1 frozenset to the worker; the footgun is impossible rather than merely documented. Producer-side instance keeps its cache; only the serialised state is scrubbed.
  - Documented in code that `ContentType.objects.get_for_model` is backed by `ContentType._cache`, so `_resolve_content_type_id` is effectively a dict read after the first call per process — pinning the reliance for future readers.
  - Documented that creator-based (non-guardian) models on the Tier 1 cache path are always cache-safe: `set_permissions_for_obj_to_user` only mutates guardian tables, which don't exist for these models, so the cached frozenset can't drift from the standard mutation path.
  - Threaded the `request=None` kwarg through `AnnotationManager.user_can` → `_check_annotation_privacy_recursion` → `Analysis.objects.user_can` / `Extract.objects.user_can` and through `_compute_annotation_effective_permission` → `AnnotationQueryOptimizer._compute_effective_permissions(context=request)` so a paginated annotation list resolver that calls `obj.user_can` per row shares one request-scoped Tier 2 cache across the privacy-recursion source-object checks AND the MIN(doc, corpus) Document/Corpus lookups, instead of re-querying guardian per row.

- **UserFeedback visibility now follows the commented annotation's full visibility model** (`opencontractserver/shared/Managers.py:353-401`, `opencontractserver/shared/QuerySets.py:169-217`, `opencontractserver/tests/test_feedback.py:334-510`, `opencontractserver/tests/permissioning/test_authorization_invariants.py:1211-1310`, `opencontractserver/tests/permissioning/test_permissioning.py:783-893`). The Phase A `UserFeedbackManager.user_can` and `UserFeedbackQuerySet.visible_to_user` previously granted READ when `commented_annotation.is_public=True` rather than when the annotation itself was visible to the requesting user. This produced two divergences from the intended "feedback inherits permissioning from the annotation it comments on" policy: (1) anonymous users could read feedback on a non-structural `is_public=True` annotation that `Annotation.objects.visible_to_user(AnonymousUser())` does not surface, and (2) authenticated users with a guardian READ grant on a private annotation could read the annotation but not the feedback on it. Both surfaces now route through `Annotation.objects.visible_to_user(user).filter(pk=commented_annotation_id).exists()` (and the equivalent `commented_annotation_id__in=...` subquery on the queryset path), keeping the manager check and the queryset filter aligned with each other and with the annotation's own visibility model. Inherited grant remains READ-only; non-READ permissions continue to require creator status or an explicit guardian grant on the feedback row itself.

- **Documents.tsx decomposition — coverage follow-up on PR #1677** (`frontend/src/utils/__tests__/files.test.ts`, `frontend/tests/Documents.ct.tsx`). The decomposition PR's three new presentational components (`DocumentsGridView`, `DocumentsListView`, `DocumentsCompactView`) and the extracted `getDocumentTypeBadge` helper landed under codecov's frontend patch target (61.15% vs target 69.89%). The pre-existing component tests only ever drove the list and compact view branches (and even then, the list test switched to list view immediately, leaving grid-view selection/context-menu untested). Two test additions push patch coverage above target:
  - **Unit tests for `frontend/src/utils/files.ts`.** Exhaustive `getDocumentTypeBadge` branch coverage (canonical fileTypes, mixed case, legacy `doc` alias, generic uppercase fallback, title-extension fallback, and the final `"PDF"` safety net), plus the previously-untested `isDocxFileType` / `isSpanBasedFileType` / `formatFileSize` exports.
  - **`tests/Documents.ct.tsx > should exercise grid view selection and context menu`.** Stays in the default grid view, clicks the per-card checkbox (drives `DocumentsGridView.onSelect` and the `isSelected` branch on the `CardCheckbox` visibility state), then clicks the kebab menu (drives the second `onContextMenu` trigger path — the row right-click is the first).
- **`AnnotationManager.user_can` refactored into helpers** (`opencontractserver/shared/Managers.py:630-846`). The 183-line method is now decomposed into `_anonymous_read_via_visible_to_user` (line 728), `_check_annotation_privacy_recursion` (line 744), and `_compute_annotation_effective_permission` (line 807) — load-bearing branch order on the top-level method is unchanged, behavior is identical, but the main body now reads as a scannable sequence of branches without scrolling. Phase B activation note added to the docstring re: per-call Analysis/Extract dereferences in the privacy-recursion branch (issue #1655 follow-up).
- **`_user_has_permission_for_obj_warned` dedup set documented as bounded** (`opencontractserver/utils/permissioning.py:546-558`). Existing comment expanded to make the bounded-by-call-sites invariant explicit, plus a pointer to `ShimDeprecationWarningTestCase.setUp` as the documented test-isolation hook.
- **`ShimDeprecationWarningTestCase` clears the warning dedup set before each test** (`opencontractserver/tests/permissioning/test_authorization_invariants.py:1313-1342`). Without this, when a previous test in the same `pytest --keepdb` run triggered a warning from the same `(filename, lineno)` site, `catch_warnings` would record nothing and the assertion would fail silently.
- **`RelationshipNonOwnerCreatorTestCase` docstring clarifies intended behavior** (`opencontractserver/tests/permissioning/test_authorization_invariants_coverage.py:627-742`). Reframed the class docstring as a POLICY note rather than "addresses the latent invariant violation flagged on PR #1663" — relationship creators retain READ/UPDATE/DELETE via the creator branch after explicit doc/corpus guardian grants are revoked **by design**, mirroring `RelationshipQuerySet.visible_to_user`'s `Q(creator=user)` predicate to keep manager and queryset surfaces aligned.
- **`permission_cache.py` module docstring notes nested-scope inheritance for Phase B** (`opencontractserver/shared/permission_cache.py:33-51`). The current allocation-on-entry behavior means nested scopes do not inherit parent entries; a "join existing scope" variant may better suit Phase B's wire-up pattern. Documented as a follow-up consideration without behavior change.

- **Extract creation chokepoint — analyzer-style auto-CRUD for Extracts** (`opencontractserver/utils/extract.py` (new), `opencontractserver/utils/analysis.py`, `opencontractserver/llms/tools/core_tools/extracts_and_analyzers.py`, `opencontractserver/tasks/corpus_tasks.py`, `opencontractserver/tests/test_extract_utils.py` (new), `opencontractserver/tests/test_analysis_utils.py`). The analyzer framework already had `create_and_setup_analysis` as the canonical Analysis-creation chokepoint that grants the creator guardian CRUD; the Extract side had no equivalent — every implementer (`start_extract` agent tool, `process_corpus_action`, the GraphQL mutations) had to remember to call `set_permissions_for_obj_to_user` separately, and `process_corpus_action` had silently been omitting the grant. New `create_and_setup_extract(user_id, *, corpus, fieldset, name=None, document_ids=None, corpus_action=None, mark_started=False)` mirrors `create_and_setup_analysis`: atomic create, creator-CRUD grant via `set_permissions_for_obj_to_user(..., is_new=True)`, optional document linking, optional started-timestamp marking. `start_extract` now routes through the helper instead of hand-rolling the create + grant; `process_corpus_action`'s `get_or_create` path now inlines the grant on first creation with a comment pointing at the framework helper. The docstring on `create_and_setup_analysis` was strengthened to pin the CRUD-grant contract as part of the framework guarantee so a future refactor can't quietly drop it. New `test_extract_utils.py` covers creation, the CRUD-grant invariant, document linking, started-timestamp marking, corpus-action lineage, and name defaulting; `test_analysis_utils.py` gains a matching CRUD-grant invariant test for the analysis helper.
- **Extract & analyzer agent tool hardening** (`opencontractserver/llms/tools/core_tools/extracts_and_analyzers.py`, `opencontractserver/llms/tools/tool_registry.py`, `opencontractserver/constants/tools.py`, `opencontractserver/tests/test_extract_analyzer_tools.py`).
  - **Status vocabulary unified across `start_analysis` and `list_recent_analyses`.** `start_analysis` previously returned `EXTRACT_STATUS_QUEUED` (`"queued"`) while `list_recent_analyses` passed the raw model status through (`JobStatus.QUEUED.value` → `"QUEUED"`). An LLM polling the listing right after dispatch saw the status appear to change without any state transition. New `_normalize_analysis_status` lowercases every status going through the listing tool; `start_analysis` now returns `_ANALYSIS_STATUS_QUEUED` (derived from `JobStatus.QUEUED.value`) so the two surfaces agree on a single lowercase vocabulary that matches the extract tools. `EXTRACT_STATUS_QUEUED` is no longer used outside the extract path.
  - **`analysis_input_data` privilege-escalation hardening.** New `ANALYSIS_INPUT_DATA_RESERVED_KEYS` enumerates the kwargs the analyzer task framework owns (`analysis_id`, `user_id`, scoping IDs, etc.); `start_analysis` strips them from the agent-supplied payload before dispatch and logs a warning. The task remains the canonical validation boundary; this is defense-in-depth on the uniquely-exposed agent path (LLM payloads can be adversary-controlled via prompt injection through document content).
  - **`list_analyzers` truncates oversized `input_schema` payloads.** `Analyzer.input_schema` is an unbounded `JSONField`. New `ANALYZER_INPUT_SCHEMA_MAX_INLINE_CHARS = 4_000` cap replaces schemas whose JSON dump exceeds the limit with a placeholder pointer; `start_analysis` itself does not need the schema (the task validates) and the LLM still discovers the analyzer. `list_analyzers` also now orders by `-created` rather than the CharField `id` so the listing is meaningful (newest first) rather than lexicographic across arbitrary dotted IDs.
  - **`list_fieldsets` defaults to slim payloads + `extractable` flag.** Returns column names only by default (`column_names: [...]`), with the full long-form column definitions (`query`, `match_text`, `instructions`, `output_type`) available behind a new `include_columns=True` flag. Bounds the context-window cost on a corpus admin with many fieldsets of many long-form columns. The listing also surfaces an `extractable` flag so the LLM knows up-front that manual-only or empty fieldsets will be rejected at `start_extract`. Listing is ordered by `-created` (not `-modified`) to match `list_recent_extracts` and avoid silent position shifts when a fieldset is edited mid-session.
  - **`_resolve_target_document_ids` pushes the intersection into SQL.** Previously materialised the entire corpus document ID set into a Python `set` on every dispatch. Now a `Document` queryset filter on the corpus's active document set when `requested_ids` are supplied, or an `exists()` membership check for a document-agent's `document_id`. The corpus-agent default path still walks the full ID list (necessary for the response payload) but at least streams via `values_list(flat=True)` rather than re-collecting from a set.
  - **Document-agent scope mismatch now fails loudly.** A document agent whose injected `document_id` is not a member of the working corpus previously fell back to the full corpus scope with only a logged warning. Now raises `ValueError` so a single-doc agent cannot silently expand into a corpus-wide dispatch. The `start_extract` / `start_analysis` tool descriptions also warn that omitting `document_ids` on a corpus agent dispatches against the entire corpus, so admins working over corpora with thousands of documents pass an explicit subset.
  - New tests pin each contract: status-normalisation in `list_recent_analyses` (including all four lifecycle states for the extract surface), reserved-key strip + warning emission in `start_analysis`, untouched legitimate payload in `start_analysis`, oversized-schema truncation in `list_analyzers`, the `extractable` flag in the manual-only-fieldset listing-vs-dispatch invariant, the slim-vs-full `list_fieldsets` payload toggle, the `limit` pagination clamp end-to-end, and the new outside-corpus `ValueError` for document agents.

### Added

- **Extract & Analyzer agent tools — let LLM agents dispatch configured extracts/analyses** (`opencontractserver/llms/tools/core_tools/extracts_and_analyzers.py` (new), `opencontractserver/llms/tools/core_tools/__init__.py`, `opencontractserver/llms/tools/tool_registry.py`, `opencontractserver/tests/test_extract_analyzer_tools.py` (new), `docs/architecture/llms/README.md`, `docs/corpus_actions/intro_to_corpus_actions.md`, `docs/extract_and_retrieval/data_extraction.md`). Six new tools slot into the existing registry alongside notes/summaries/annotations: `list_fieldsets`, `start_extract`, `list_recent_extracts`, `list_analyzers`, `start_analysis`, `list_recent_analyses`. Discovery tools are read-only and filter via the model manager's `visible_to_user`; `start_extract` and `start_analysis` are `requires_approval=True` + `requires_write_permission=True` so they pass through the standard approval gate (`ToolConfirmationRequired` → `ApprovalNeededEvent` → `resume_with_approval`) and the agent-factory's write-permission filter. New `ToolCategory.EXTRACTS` and `ToolCategory.ANALYZERS` keep them browsable in the available-tools GraphQL surface. **The agent picks from existing Fieldsets/Analyzers** — there is no schema-creation surface; this matches the user's requirement that the tool not invent column definitions from scratch. **Document scope** follows the agent context: corpus agents default to the full corpus (via `corpus.get_documents()`, which already excludes CAML and soft-deleted docs); document agents default to the current document; any `document_ids` the LLM passes are intersected with the corpus's active document set so the agent cannot reach documents outside its scope. **Dispatch reuses existing code paths** — `start_extract` creates the `Extract` row, adds documents, grants creator CRUD via `set_permissions_for_obj_to_user`, and queues `run_extract.s(extract_id, user_id).apply_async()` on `transaction.on_commit`; `start_analysis` delegates entirely to `process_analyzer` so chat-agent runs and auto-CorpusAction runs produce identical `Analysis` records. Both tools accept `corpus_action_id` (auto-injected by `build_inject_params_for_context`) to keep lineage tracking intact when an agent corpus action delegates onward. **Opt-in for CorpusActions**: deliberately *not* added to `DEFAULT_DOCUMENT_ACTION_TOOLS` — admins must list them on a CorpusAction's `pre_authorized_tools` to skip approval in the automation path. Test coverage in `test_extract_analyzer_tools.py` exercises registry integration & flag wiring (`TestExtractAnalyzerRegistryIntegration`), visibility filtering for discovery tools (`TestListFieldsets`, `TestListAnalyzers`, `TestListRecentExtracts`, `TestListRecentAnalyses`), document-scope resolution and permission gating for the write tools (`TestStartExtract`, `TestStartAnalysis`), and the approval gate firing on the `PydanticAIToolWrapper` (`TestApprovalGate`). Docs updated in three places: a new "Extract & Analyzer Tools" subsection in `docs/architecture/llms/README.md` (with a usage example and the opt-in CorpusAction pattern), a callout in `docs/corpus_actions/intro_to_corpus_actions.md` pointing at the delegation path, and a cross-link from `docs/extract_and_retrieval/data_extraction.md` so people browsing the extract docs find the agent route.
- **Service-layer split — review follow-ups on PR #1685** (`opencontractserver/corpuses/corpus_objs_service.py`, `opencontractserver/users/types.py`, `opencontractserver/llms/agents/core_agents.py`, `opencontractserver/tests/test_corpus_objs_service.py`, `opencontractserver/tests/test_document_folder_service.py`, `opencontractserver/tests/test_llms_typing_coverage.py`, `opencontractserver/tests/test_user_types_resolver.py` (new)).
  - **Unified `get_corpus_documents(include_deleted=True/False)` paths.** Pre-fix, the True branch hand-rolled a `DocumentPath` query that silently bypassed CAML filtering altogether; the False branch delegated to `Corpus._get_active_documents()` which excluded CAML. New `_build_corpus_documents_queryset` helper is the single source of truth — both branches now compose `include_deleted` and `include_caml` toggles uniformly so CAML can never sneak through the trash-view path. Added `include_caml=False` to the public method signature so callers can opt in explicitly. New `TestGetCorpusDocuments_CamlAndDeleted` fixture pins all four flag combinations including the regression guard `test_include_deleted_still_excludes_caml`.
  - **Moved `_resolve_user_or_anon` from `core_agents.py` to shared `users/types.py`** as the public `resolve_user_or_anon(user_id)` helper. Centralising avoids duplicate `User.DoesNotExist` handling and gives the docstring (re: must be called inside the same `sync_to_async` block) one place to live. `core_agents.py` retains both call sites, imported from the shared module. New `test_user_types_resolver.py` pins the three branches (existing id, `None` → `AnonymousUser`, unknown id → `DoesNotExist` rather than silent demotion).
  - **Pytest fixes:** `test_document_folder_service.py` mocks of `post_save` were patching `opencontractserver.corpuses.folder_service.post_save` which no longer exists in the shim module after the split — repointed to the actual home in `corpus_objs_service`. `test_llms_typing_coverage.py::TestCorpusAgentContextInitialize` mocked `Corpus.get_documents` but `CorpusAgentContext.initialize` now routes through `CorpusObjsService.get_corpus_documents` — updated the patch target so the test continues to drive both the "empty list triggers fetch" and "non-empty list short-circuits" branches.

- **PR #1685 third-pass review polish** (`opencontractserver/corpuses/corpus_objs_service.py`, `config/graphql/corpus_types.py`, `opencontractserver/llms/agents/core_agents.py`, `opencontractserver/llms/tools/core_tools/caml_article.py`). Addresses the third Claude review batch on the service-layer split:
  - **`move_documents_to_folder` N+1 eliminated**. The validation loop that ran one ``DocumentPath.objects.filter(...).exists()`` per document via ``check_document_in_corpus`` is gone. The atomic block's existing ``select_for_update`` on ``DocumentPath`` (filtered by ``document_id__in=document_ids, corpus=..., is_current=True, is_deleted=False``) already proves membership, so the missing-set is now computed as ``set(document_ids) - {p.document_id for p in current_paths}`` inside the locked transaction. Removes O(N) round-trips and folds the membership check into the TOCTOU-coherent atomic step. The error message now enumerates every missing id rather than only the first.
  - **`check_document_in_corpus` → `_check_document_in_corpus`** to match its "NO PERMISSION CHECK" contract. The method has three internal callers (all of which already enforce corpus READ upstream) and zero external callers; the underscore makes the footgun less reachable from autocomplete and signals service-internal use.
  - **CAML production callsites migrated to ``Corpus._get_active_documents(include_caml=True)``** (`opencontractserver/llms/tools/core_tools/caml_article.py:180`, `config/graphql/corpus_types.py:resolve_documents`, `config/graphql/corpus_types.py:resolve_annotations`). Each site was already gated by an upstream corpus READ check (``Corpus.objects.visible_to_user(user).get(...)`` in caml_article, parent-query visibility for the CorpusType sub-resolvers), so calling the deprecated user-facing ``get_documents()`` alias was emitting a ``DeprecationWarning`` on every live CAML resolve / corpus document render. The internal helper is the correct entry point for already-gated sites.
  - **Hoisted the two function-local ``CorpusObjsService`` imports in `opencontractserver/llms/agents/core_agents.py`** to module level, completing the second-pass hoist work that left these two sites behind. No circular import exists (the service is a pure consumer of model classes).

- **PR #1685 second-pass review polish** (`config/graphql/{og_metadata_queries,document_relationship_mutations,extract_mutations}.py`, `opencontractserver/corpuses/{corpus_objs_service,models}.py`, `opencontractserver/users/types.py`, `opencontractserver/tests/test_document_service.py`). Addresses the latest Claude review on the service-layer split:
  - **Hoisted three function-local `from … corpus_objs_service import CorpusObjsService` imports to module level** in `og_metadata_queries.py`, `document_relationship_mutations.py`, and `extract_mutations.py`. Verified no circular dependency exists at module-load time — the inline form was cargo-cult, not load-bearing.
  - **`resolve_user_or_anon` stale-id contract is now explicit in the docstring** — `User.DoesNotExist` propagates by design; callers that need a "treat deleted creator as anonymous" fallback must catch at their own boundary. Silent demotion would broaden access (the deleted creator's private content would suddenly resolve through AnonymousUser visibility).
  - **`Corpus._get_active_documents` API note** explains why the underscore-prefixed name is the blessed cross-module helper for non-user-context callers (badge / corpus / analyzer tasks, signal handlers, LLM tool batches) despite Python convention pointing the other way. The underscore distinguishes the bypass-permission path from the deprecated user-facing `get_documents()` alias.
  - **`CorpusObjsService.check_document_in_corpus` docstring now leads with "NO PERMISSION CHECK"** — the method is a raw `DocumentPath` membership probe; callers must gate corpus READ before invoking. Suggests `get_corpus_document_by_id` as the higher-level alternative for new resolvers.
  - **`og_metadata_queries.resolve_og_corpus_document` now documents that the `is_public=True` filter on the corpus lookup is load-bearing** — without it, an `AnonymousUser` call to `CorpusObjsService.get_corpus_documents` would resolve documents for any public corpus. The service's anonymous READ path is permissive by design; the gate must live in the resolver's `Corpus.objects.get(... is_public=True)`.
  - **`_build_corpus_documents_queryset` `.distinct()` defensiveness is now documented**: `DocumentPath` has no DB constraint preventing multiple `is_current=True` rows per `(corpus, document)`, only the `(corpus, path)` uniqueness — so an aliased current path or an `include_deleted=True` query that catches both a soft-deleted and a non-deleted current row could surface the same `document_id` twice. `.distinct()` collapses that defensively.
  - **`get_folder_tree` `O(N)` query loop is now flagged with a `TODO(#1685 follow-up)`** documenting the path-from-parent-map / annotate-Count optimisation. Pre-existing behaviour; not changing in this PR to keep the service-split surface stable.
  - **`DocumentService` codecov coverage lifted** from 42.68% to ~95% by adding behavioural test cases for each method's branches (quota over/under, validate_file_type plaintext/binary/disallowed-MIME, create_document PDF/TXT/unsupported/quota-blocked/validation-failed/atomic-rollback, get_document_by_id IDOR matrix, set_document_permissions owner / PERMISSION / denied / exception fall-through). The previous test file only verified structural assertions (method partition, backwards-compat alias surface).
  - **Known follow-up — `corpus_objs_service.py` SRP**: the file is ~2,700 lines and now monolith-scale on its own. Splitting along the sub-seams (folder CRUD, document lifecycle, corpus-document membership) is tracked for a future PR; not scope-creep on this split.

### Added

- **Service-layer split: `DocumentService` + `CorpusObjsService` as the canonical entry points** (`opencontractserver/documents/document_service.py` (new), `opencontractserver/corpuses/corpus_objs_service.py` (new), `opencontractserver/corpuses/folder_service.py` (converted to back-compat shim), `opencontractserver/corpuses/models.py` (added `_get_active_documents` + `DeprecationWarning` wrapper on `get_documents`), `CLAUDE.md`, plus migrations across `opencontractserver/mcp/tools.py` (5 sites), `opencontractserver/mcp/resources.py` (2 sites), `opencontractserver/llms/agents/core_agents.py` (2 sites), `opencontractserver/discovery/views.py` (3 sites in `_search_within_corpus`), `config/graphql/og_metadata_queries.py`, `config/graphql/extract_mutations.py`, `config/graphql/document_relationship_mutations.py`, `opencontractserver/tasks/badge_tasks.py`, `opencontractserver/tasks/corpus_tasks.py`, `opencontractserver/utils/sharing.py`, `opencontractserver/utils/analyzer.py`, `opencontractserver/llms/tools/core_tools/{pii.py,annotations.py,document_indexing.py}`). The single-class `DocumentFolderService` has been split along the natural seam revealed by the PR review of `opencontractserver/mcp/tools.py`: methods where the **document** is the noun and corpus context is incidental (`create_document`, `check_user_upload_quota`, `validate_file_type`, `get_document_by_id`, `set_document_permissions`) now live on `DocumentService`; methods of the form *"give me X inside corpus Y for user Z"* (every folder operation, the corpus-scoped doc lifecycle helpers `soft_delete_document` / `restore_document` / `permanently_delete_document` / `empty_trash`, the upload/add/remove corpus mutations, and `get_corpus_documents`) now live on `CorpusObjsService`. The new corpus-objs service is designed to grow to other corpus-linked object types — notes, annotations, relationships, extracts, analyses — and is the canonical extension point for future "corpus-scoped X" patterns. `DocumentFolderService` survives as a multiple-inheritance subclass of both new services so the ~18 existing importers continue to work unchanged (cleanup tracked in follow-up issue). New convenience methods on `CorpusObjsService` close the buggy fusion pattern that motivated the split: `get_corpus_document_by_slug(user, corpus, slug)`, `get_corpus_document_by_id(user, corpus, doc_id)`, and `is_document_in_corpus(user, corpus, doc_id)` — all corpus-as-gate, all IDOR-safe (same `Document.DoesNotExist` for not-found, not-in-corpus, and no-permission). All 15 user-context call sites that previously composed `corpus_doc_ids = corpus.get_documents().values_list("id", flat=True)` + `Document.objects.visible_to_user(user).get(id__in=corpus_doc_ids, slug=...)` now flow through the service. The previous pattern silently mixed two different permission semantics (corpus-membership AND document `visible_to_user`) that the canonical service had standardised on as corpus-as-gate; the fusion produced inconsistent behaviour between MCP, discovery, and the GraphQL CAML resolver. The single service entry point now guarantees every caller asks the same question with the same answer. CLAUDE.md item 7 declares the service layer as the canonical entry point for any user-context code touching documents. `Corpus.get_documents()` is now a thin `DeprecationWarning`-emitting wrapper around the new internal helper `Corpus._get_active_documents(include_caml=...)`; internal/Celery callers (`badge_tasks.py:234`, `corpus_tasks.py:72`, `utils/sharing.py:make_corpus_public`, `utils/analyzer.py:run_analysis`, the framework-private LLM helpers in `pii.py` / `annotations.py` / `document_indexing.py`) opt out by calling `_get_active_documents()` directly. Production CAML article paths (`opencontractserver/llms/tools/core_tools/caml_article.py:180`, `config/graphql/corpus_types.py:resolve_caml_articles`) are deliberately left on `corpus.get_documents(include_caml=True)` and will emit the warning at runtime — they are tracked under follow-up issues (CAML-aware service method; MIN-semantic convergence). **Phase-B caching follow-up — `request=` kwarg + N+1 fix:** the new `CorpusObjsService` and `DocumentService` methods that perform `corpus.user_can(...)` / `document.user_can(...)` permission gates now accept and forward a kw-only `request=None` kwarg (mirroring the `DocumentFolderService` signature from PR #1665). Every GraphQL caller (`config/graphql/extract_mutations.py`, `config/graphql/document_relationship_mutations.py`, `config/graphql/og_metadata_queries.py`) now passes `request=info.context` so the two-tier `PermissionQueryOptimizer` cache spans the full corpus-scoped lookup path, eliminating the N+1 regression that the service split would otherwise have introduced. The `get_folder_tree` `O(N)` query loop flagged with `TODO(#1685 follow-up)` is also fixed: descendant counts are now resolved with a single `.annotate(direct_children_count=Count("children", filter=Q(deleted_at__isnull=True), distinct=True))` pass instead of a `.count()` per node.

- **Two-tier permission caching for `Manager.user_can` (closes #1640)** (`opencontractserver/constants/permissioning.py` (new), `opencontractserver/utils/permission_optimizer.py` (new), `opencontractserver/utils/permissioning.py`, `opencontractserver/shared/user_can_mixin.py`, `opencontractserver/corpuses/models.py`, `opencontractserver/corpuses/folder_service.py`, `config/graphql/{corpus_queries,corpus_mutations,corpus_folder_mutations,document_mutations}.py`, `opencontractserver/tests/permissioning/test_permission_optimizer.py` (new), `docs/architecture/query_permission_patterns.md`). Follow-up to PR #1637 which centralized authorization in `_default_user_can`. The cold path (non-superuser, non-creator, non-public-READ) used to hit the DB on every `user_can` call — fine for Phase 0 (Corpus only, one check per folder-service entry point) but linear in the row count once paginated list resolvers start calling `user_can` per row.

  **Tier 1 — per-instance memoization.** `get_users_permissions_for_obj` now caches its result on the instance as `_oc_granted_perms_cache: dict[(user_id, include_group_permissions_bool), frozenset[str]]`. Transparent to every caller (including the deprecated `user_has_permission_for_obj`). Anonymous / unauthenticated users bypass the cache. Cache lifetime equals the instance lifetime; `refresh_from_db` does NOT clear it — long-lived Celery instances must `del instance._oc_granted_perms_cache` after out-of-band perm changes.

  **Tier 2 — request-scoped `PermissionQueryOptimizer`.** Mirrors `ConversationQueryOptimizer` (`opencontractserver/conversations/query_optimizer.py`). Lazily attached to the request as `request._permission_query_optimizer` via `get_request_optimizer(request)`. Keyed by `(user_id, content_type_id, instance_pk, include_group_permissions)` so multiple instances of the same model in one request share a cache. Opt-in via the new `request=` kwarg threaded through `Manager.user_can` / `obj.user_can` / `_default_user_can`. `get_request_optimizer(None)` returns a one-shot optimizer for Celery / management-command contexts.

  **Composition with PR #1663's `permission_cache_scope`.** PR #1663 introduced a dormant `ContextVar`-backed boolean cache wrapping `_default_user_can` (Phase A — opt-in via `permission_cache_scope()`, not yet wired in production). The two layers compose cleanly: scope cache wraps the cold-path lookup at the boolean level, Tier 2 optimizer/Tier 1 instance memo wrap the underlying granted-set computation. Outside a scope (today's default), Tier 1 and Tier 2 do all the deduplication; once Phase B activates `permission_cache_scope()` via middleware, both layers continue to compose without invalidation hazards (the scope is read-only and Tier 2's `set_permissions_for_obj_to_user` invalidation only touches Tier 1/Tier 2, not the scope dict).

  **Wiring.** Plumbed `request=info.context` through:
  - `DocumentFolderService` classmethods (25+ entry points in `corpuses/folder_service.py`) and their callers in `config/graphql/{corpus_queries,corpus_mutations,corpus_folder_mutations,document_mutations}.py`.
  - `CorpusFolder.user_can` override (`corpuses/models.py:1945`) so folder permission checks share the corpus's cache entry.
  - `UserCanMixin` and `InstanceUserCanMixin` in `shared/user_can_mixin.py`.

  **Mutation invalidation.** `set_permissions_for_obj_to_user` gained a `request=None` kwarg and now self-invalidates both tiers for the affected `(user, instance)` pair when called inside the HTTP lifecycle. The Phase-0 `Corpus` mutation call sites (`corpus_mutations.py:180,562,1007,1025,1056,1468`; `document_mutations.py:655,1024,1508`) pass `request=info.context`. Out-of-band call sites (Celery tasks, fixtures, importing utilities) continue to call without a request — Tier 2 is skipped, Tier 1 is still scrubbed for the target user so reused instances see the new grant.

  **Coverage.** New `tests/permissioning/test_permission_optimizer.py` covers per-instance memoization (zero queries on repeat, no leakage across `include_group_permissions`, fast-path short-circuits don't populate cache, defensive copy contract), the request-scoped optimizer (lazy attach, `None`-safe, distinct users keyed independently, per-user / per-instance / total invalidation, anonymous skip), the mutation invalidation contract (cache invalidates mid-request when `request=` is supplied; cache is scrubbed for the target user even when `request=None`), and `Manager.user_can` / `obj.user_can` request passthrough.
- **Decompose `Documents.tsx` into thinner composer + sibling files** (`frontend/src/views/Documents.tsx`, plus new siblings `Documents.styles.ts`, `DocumentsGridView.tsx`, `DocumentsListView.tsx`, `DocumentsCompactView.tsx`, and an extension to `frontend/src/utils/files.ts`). Final slice of the multi-PR decomposition tracked by #1446 (after #1459 / `ModernDocumentItem`, #1578 / `Corpuses`, #1586 / `ChatMessage`, #1580 / `DocumentKnowledgeBase` partial, #1639 / `ChatTray`). `frontend/src/views/Documents.tsx` had grown to **1,725 lines**. Five extractions land it at **915 lines (-47%)**:
  - **`Documents.styles.ts`** *(new, 581 lines)*: All 50 styled-components moved to a sibling file as named exports. Cross-component selectors (`${DocumentCardWrapper}:hover &` referenced by `CardCheckbox` and `CardMenuButton`; `${ListItem}:hover &` referenced by `ListItemActions`) require same-module scope, so all styles stay in one file (same rule that kept `ModernDocumentItem.styles.ts` monolithic in PR #1459). Renamed three styled-components to free up names taken by the new React components — `DocumentsListContainer` (outer section wrapper) → `DocumentsSection`; `DocumentsListView` (inner list table wrapper) → `DocumentsListContainer`; `DocumentsCompactView` (inner compact list wrapper) → `DocumentsCompactContainer`. Dropped the unused `MetaSeparator` styled-component (declared but never rendered in any view branch — CLAUDE.md "no dead code" rule).
  - **`DocumentsGridView.tsx`** *(new, 145 lines)*: The grid-view branch — `DocumentCardWrapper` cards with checkbox / preview / body / footer. Props: `documents`, `selectedIds`, `activeContextMenuDocId`, `onDocumentClick`, `onSelect`, `onContextMenu`.
  - **`DocumentsListView.tsx`** *(new, 121 lines)*: The list-view branch — table-shaped grid including `ListHeader` with select-all checkbox. Adds `onSelectAll` and pre-computed `allSelected: boolean` to the shared prop shape.
  - **`DocumentsCompactView.tsx`** *(new, 91 lines)*: The compact-view branch — flat row layout. Same props as `DocumentsGridView`.
  - **`frontend/src/utils/files.ts`** *(extended, +27 lines)*: Adds `getDocumentTypeBadge(fileType, title)`, the file-extension/type normalizer previously inlined in `Documents.tsx`. Lives in the existing shared `files.ts` utility module per CLAUDE.md's "utility functions belong in utility files" rule. Consumed by `DocumentsGridView` and `DocumentsListView`.
  - Behavior is preserved verbatim — same handler bodies, same DOM, same Apollo reactive-var reads (all 11 stay at the composer; props drilled down to children, matching `ExtractsTabContent.tsx` precedent which receives data via props rather than reading reactive vars directly). Public API (`Documents` named export) is unchanged; importers (`frontend/src/App.tsx:49`, `frontend/tests/Documents.ct.tsx:8`, `frontend/tests/DocumentsTestWrapper.tsx:7`) need no changes. The hero section, search box, filter tabs row, advanced-filters popup, stats grid, section header with view-mode toggle, fetch-more sentinel, empty-state + error branches, context menu, modals, and the `DocumentIcon` SVG fallback stay inline in `Documents.tsx`. All 10 component tests in `frontend/tests/Documents.ct.tsx` and the 4 cases in `frontend/src/views/Documents.refetch-shape.test.ts` pass without modification.

  Closes #1446.

- **Surface OC_SUBTREE_GROUP relationships in semantic search + doc-viewer deep-link** (closes #1645). End-to-end wiring that turns the materialised `OC_SUBTREE_GROUP` rows from #1646 into a first-class semantic-search surface and lets the document viewer jump directly to a surfaced block.

  - **Block-context augmentation on annotation hits** (`opencontractserver/llms/vector_stores/core_vector_stores.py`, `opencontractserver/constants/annotations.py`, `opencontractserver/llms/vector_stores/pydantic_ai_vector_stores.py`). New `BlockContext` dataclass on `VectorSearchResult` carries the smallest-enclosing `OC_SUBTREE_GROUP` relationship for each hit: `relationship_id`, `source_annotation_id`, `source_text`, `target_annotation_ids`, plus a newline-joined `block_text` capped at `SUBTREE_GROUP_BLOCK_TEXT_MAX_CHARS` (16k chars). `_attach_block_context_sync` runs as a single `Relationship.objects.filter(target_annotations__in=hit_ids, relationship_label__text="OC_SUBTREE_GROUP", structural=True).annotate(descendant_count=Count(...))` join after reranking, picks the smallest enclosing group per hit (ties broken on lowest relationship pk for determinism), and constructs `block_text` from one cached `IN`-query over the referenced annotations. Wired into `search()`, `async_search()`, `hybrid_search()`, `async_hybrid_search()`, and `global_search()` so every public retrieval entry-point benefits. `PydanticAIVectorSearchResponse.from_core_results` / `.async_from_core_results` pass `block_context` through as a plain dict on the agent tool surface.

  - **Relationship as a first-class vector target** (`opencontractserver/annotations/models.py`, `opencontractserver/annotations/migrations/0073_embedding_relationship.py`, `opencontractserver/shared/Managers.py`, `opencontractserver/tasks/embeddings_task.py`, `opencontractserver/utils/subtree_groups.py`, `opencontractserver/llms/vector_stores/core_relationship_vector_store.py`). Added polymorphic `Embedding.relationship` FK alongside the existing document/annotation/note/conversation/message slots, with a partial-unique constraint on `(embedder_path, relationship)`. `Relationship` now mixes in `HasEmbeddingMixin` (`get_embedding_reference_kwargs -> {"relationship_id": self.pk}`); `EmbeddingManager.store_embedding` accepts `relationship_id`. New Celery task `calculate_embeddings_for_relationship_batch` mirrors the per-annotation dual-embedding strategy (default embedder + corpus-preferred embedder when distinct), keyed on `synthesize_relationship_block_text(relationship)` — the same bounded source+targets concatenation the GraphQL surface returns, so the embedded payload and the snippet clients see never drift. `build_subtree_groups_for_document` dispatches the task after each successful materialisation (`_dispatch_relationship_embeddings`), fanning out per-corpus when a document belongs to multiple corpora with distinct preferred embedders. New `CoreRelationshipVectorStore` runs the same cosine-distance pass against `Embedding` filtered by `relationship__isnull=False`, scoped by `Relationship.objects.visible_to_user(user)` + corpus/document-aware Q-OR (mirrors the annotation store's structural-set inclusion logic).

  - **GraphQL surface** (`config/graphql/social_types.py`, `config/graphql/graphene_types.py`, `config/graphql/search_queries.py`). New `BlockContextType` exposed as a `block_context` field on `SemanticSearchResultType`. New `semanticSearchRelationships` resolver returns `SemanticSearchRelationshipResultType` (relationship pk, similarity, label, source/target annotation pks, block_text, document/corpus pks). Permission flow mirrors `resolve_semantic_search` — empty-list response for missing-or-denied document/corpus prevents IDOR enumeration; corpus-scoped queries derive the embedder from `Corpus.preferred_embedder` so relationship vectors stay in the corpus's frozen embedding space.

  - **Frontend deep-link + jump-to-relationship** (`frontend/src/graphql/{queries,cache}.ts`, `frontend/src/utils/{navigationUtils,__tests__/navigationUtils.test}.ts`, `frontend/src/routing/CentralRouteManager.tsx`, `frontend/src/components/knowledge_base/document/document_kb/useJumpToRelationship.ts`, `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`, `frontend/src/components/annotations/CorpusAnnotationCards.tsx`). New `SEMANTIC_SEARCH_RELATIONSHIPS` query and `blockContext { … }` field added to the existing `SEMANTIC_SEARCH_ANNOTATIONS` selection. New URL parameter `?rel=<relationship_pk>` (raw Django PK, not a Relay global ID) consumed by `CentralRouteManager` and stored on a new `selectedRelationshipId` reactive var. New `useJumpToRelationship` hook, mounted unconditionally in `DocumentKnowledgeBase`, watches `selectedRelationshipId` + `allRelationsAtom` (combined user-editable + structural relations) and, on a match, sets `selectedRelationsAtom` and scroll-into-views the source annotation — applied exactly once per id change via a ref so subsequent local edits aren't fought. `CorpusAnnotationCards` annotation click now includes `relationshipId` in the navigation query params when the hit carries a `block_context`, so users land on the doc with both the leaf annotation selected (existing `?ann=` flow) AND the containing block's relation line rendered. `buildQueryParams` learns to emit the new `rel` segment; navigation-utils tests cover the round-trip.

  - **Tests** (`opencontractserver/tests/test_subtree_group_vector_search.py`, `frontend/src/utils/__tests__/navigationUtils.test.ts`). Backend suite covers: smallest-enclosing-group selection on a 4-level structural tree, root-level hit returns `None`, empty input is a pass-through, `block_text` truncation at `SUBTREE_GROUP_BLOCK_TEXT_MAX_CHARS`, end-to-end relationship vector retrieval with the same embedding the helper synthesised, and IDOR denial returning `[]` for a user lacking corpus visibility. Helper-text synthesis tests cover deterministic id-ordering of source+targets, cap-enforcement, and empty-component skipping. Frontend test pins the `rel=<pk>` URL-builder contract.

- **PII scanner task-based analyzer + analyzer-app coverage lift** (`opencontractserver/tasks/doc_analysis_tasks.py`, `opencontractserver/tests/test_pii_scanner_analyzer_task.py` (new), `opencontractserver/tests/test_analyzer_app_coverage.py` (new), `docs/test_scripts/pii_scanner_analyzer_smoke.md` (new)). A new `@async_doc_analyzer_task`-decorated function, `pii_scanner_privacy_filter`, wraps the existing `adetect_pii` privacy-filter client and exposes PII scanning as a first-class analyzer that can be invoked via `startAnalysisOnDoc` (i.e. the same UI/GraphQL path as any other analyzer), rather than only through the chat-sidebar agent tool `scan_and_annotate_pii`. The task chunks via the underlying client (40 KB chunks with 500-char overlap and `(start, end, group)` de-dup) and produces SPAN_LABEL annotations for text/markdown and TOKEN_LABEL annotations for PDFs via the standard `async_doc_analyzer_task` post-processing. Labels are pre-created with the canonical text/colour/icon from `opencontractserver.llms.tools.core_tools.pii.ENTITY_GROUP_LABELS` (e.g. `"PII: Email"` / `#1f77b4` / `mail`) so visual parity with the agent-tool path is preserved. A single configurable knob `min_score` (default 0.5) is surfaced through `input_schema`. The analyzer is auto-registered by `auto_create_doc_analyzers` (migration `0009_auto_load_doc_analyzers` plus the startup hook in `opencontractserver/analyzer/startup.py`); no new migration. Companion coverage work targets the lowest-coverage backend app: `test_analyzer_app_coverage.py` adds tests for `signals.install_gremlin_on_creation` (chain assembly under `transaction.on_commit`), `signals.handle_analysis_completion` (with a documented mismatch where the handler checks the literal `"COMPLETE"` while `JobStatus.COMPLETED.value == "COMPLETED"`), `checks.check_unsynced_analyzers` (warning emitted / silent / import-error swallow), `startup.sync_analyzers_on_startup` (happy + exception path), `admin_views.AnalyzerSyncView` (GET helper + POST with and without `analyzer.add_analyzer` perm), `utils.get_gremlin_manifests` (HTTP 500, malformed manifest item, unknown gremlin id), `utils.auto_create_doc_analyzers` edge cases (no users, celery unavailable, update_existing=False), and the `sync_doc_analyzers` management command (`--dry-run` + normal run). `views.AnalysisCallbackView` and `views._create_analysis_notification` get their first happy-path tests (valid token + valid body → COMPLETED + `import_analysis` scheduled + ANALYSIS_COMPLETE notification; invalid body → FAILED + ANALYSIS_FAILED notification; create raising RuntimeError swallowed with warning).

- **Phase A review follow-ups on PR #1663** (`opencontractserver/shared/user_can_mixin.py`, `opencontractserver/shared/Managers.py`, `opencontractserver/shared/QuerySets.py`, `opencontractserver/utils/permissioning.py`, `opencontractserver/tests/permissioning/test_authorization_invariants_coverage.py`). Extracted the duplicated int/str → ``User`` resolution from ``AnnotationManager.user_can``, ``NoteManager.user_can``, ``RelationshipManager.user_can`` and ``_default_user_can`` into a single ``resolve_user_for_user_can`` helper in ``shared/user_can_mixin.py`` (Claude review #1 — ~40 lines of duplicated boilerplate removed). ``user_has_permission_for_obj`` shim now raises ``TypeError`` with an actionable message when the instance's ``_default_manager`` doesn't implement ``user_can`` (Claude review #2). Previously the call would surface as a confusing bare ``AttributeError`` deep inside a resolver during the incremental Phase A → Phase B migration of the ~170 legacy call sites. Removed the unexplained ``.order_by("created")`` from ``PermissionQuerySet.visible_to_user`` superuser branch (Claude review #7) — not every model that uses ``PermissionManager`` has a ``created`` column, so the new clause was a latent ``FieldError``. Added ``test_authorization_invariants_coverage.py`` (46 new test cases) covering the codecov-flagged patch lines and the Claude-flagged behavioural gaps: direct unit tests of the ``permission_cache_scope`` / ``cached_user_can`` / ``store_user_can`` machinery (cache went from 73.91% → 100% patch coverage); the access-widening case where a stranger with explicit doc+corpus READ now sees the relationship (Claude review #3); the ``created_by_extract`` privacy-recursion branch on annotations (Claude review #4 — the partner of the ``created_by_analysis`` test already in the file); the int/str user-id resolver paths for every per-model override; the anonymous note-visibility leaf branches under public/private corpus combinations; the ``PermissionQuerySet`` fallback body (superuser/anonymous/authenticated/guardian/LookupError); the ``UserFeedbackQuerySet`` guardian path; the shim's ``DoesNotExist`` deny and the ``TypeError`` guard for unmigrated managers.

- **Phase A of permission centralization: per-model `Manager.user_can` overrides + request-scoped permission cache** (`opencontractserver/shared/{Managers,QuerySets,permission_cache}.py`, `opencontractserver/conversations/models.py`, `opencontractserver/annotations/query_optimizer.py`, `opencontractserver/utils/permissioning.py`, `opencontractserver/tests/permissioning/test_authorization_invariants.py`). Issue #1655 — the follow-up to PR #1637 (Step 0 vertical slice). Every visibility-managed model now exposes a `Model.objects.user_can(user, instance, permission)` override aligned with its `visible_to_user` filter so the two surfaces answer the same question for the same `(user, instance)`: **`DocumentManager`** (thin delegate; default rules suffice), **`AnnotationManager`** (load-bearing order — superuser → structural-write-deny → privacy recursion via `Analysis.objects.user_can` / `Extract.objects.user_can` → MIN(doc, corpus) via `AnnotationQueryOptimizer._compute_effective_permissions`), **`RelationshipManager`** (structural-write-deny + MIN; deliberately preserves the legacy decision to NOT consult `created_by_analysis`/`created_by_extract`), **`NoteManager`** (creator OR MIN(doc, corpus), composed via `Document.objects.user_can` and `Corpus.objects.user_can`), **`UserFeedbackManager`** (adds the `commented_annotation.is_public` READ grant on top of the default rules), **`ConversationManager`** (READ uses `visible_to_user(...).filter(pk=).exists()` to preserve the CHAT/THREAD bifurcation; non-READ uses `_default_user_can` so is_public does NOT grant writes), **`ChatMessageManager`** (READ uses the same exists trick to preserve the moderator branch — corpus owner sees messages in threads on their corpus). New `opencontractserver/shared/permission_cache.py` provides a `ContextVar`-backed request-scoped read-only cache (`permission_cache_scope()`) keyed by `(user_id, app_label, model_name, pk, permission, include_group_permissions)`; dormant outside scope, wired into `_default_user_can` at the guardian-lookup boundary so any future request/Celery middleware can flip it on by entering the scope. Invariant test suite (`test_authorization_invariants.py`) expanded with 7 new `TransactionTestCase` classes (Document/Annotation/Relationship/Note/Conversation/ChatMessage/UserFeedback) plus a shared `_UserCanInvariantsMixin` that pins `Model.objects.user_can(u, obj, READ) == Model.objects.visible_to_user(u).filter(pk=obj.pk).exists()` across the fixture matrix, manager/instance-surface parity, and superuser bypass. Plus per-model invariants: is_public-grants-only-READ on Document, structural-write-deny on Annotation and Relationship, the new privacy-recursion bug-fix posture, MIN visibility for Notes, CHAT-no-context-inheritance and THREAD-inherits-corpus for Conversation, moderator-via-corpus-ownership for ChatMessage, and the `commented_annotation.is_public` READ grant (with no bleed into writes) for UserFeedback.

- **Materialised `OC_SUBTREE_GROUP` relationships at ingestion** (`opencontractserver/constants/annotations.py`, `opencontractserver/utils/subtree_groups.py`, `opencontractserver/pipeline/base/parser.py`, `opencontractserver/tests/test_subtree_group_materialization.py`). After a parser imports a document's structural annotations and explicit relationships, `BaseParser.save_parsed_data` now runs `build_subtree_groups_for_document` between the relationship-import step and the structural-set migration. The new utility walks the unified parent-child hierarchy (from `Annotation.parent` self-FKs plus any `OC_PARENT_CHILD_LABEL_NAME` structural Relationship rows) and creates one materialised `Relationship` row per non-leaf with the ancestor as the sole source annotation and the ancestor's full transitive descendant set as targets. Retrieval can then surface the larger "block" of an annotation hit with a single join instead of a recursive CTE per result. Guardrails: groups whose descendant set exceeds `SUBTREE_GROUP_MAX_DESCENDANTS` (default 500) are skipped with a warning; the walker prunes branches deeper than `SUBTREE_GROUP_MAX_DEPTH` (default 32) and detects cycles defensively via on-stack tracking. The built-in `OC_SUBTREE_GROUP` `AnnotationLabel` is resolved by filter-first / create-as-fallback (ignoring `creator` in the lookup) so the system-wide label never silently forks per-user in multi-user installations. Re-parse safe: prior `OC_SUBTREE_GROUP` rows are deleted from both `document`-scoped and `structural_set`-scoped tables, and freshly created rows are attached directly to the existing `StructuralAnnotationSet` when one is already linked to the document (avoiding orphaned `document`-scoped rows that would otherwise dangle because `_create_structural_annotation_set` short-circuits on a re-parse). Test coverage: 11 cases covering one-row-per-non-leaf, transitive content correctness, idempotent re-runs, structural-set migration through the parser, size cap, depth cap, cycle detection (with an `on_stack`-reachable cycle exercising the warning), empty document, parent-child relationship edges, label-non-proliferation, and the re-parse-after-structural-set path. **Post-review polish (closes #1662)**: hoisted the inline `_PRUNED_SAMPLE_CAP` magic number to `SUBTREE_GROUP_PRUNED_SAMPLE_CAP` in `opencontractserver/constants/annotations.py` (CLAUDE.md compliance); refactored the three `qs1 | qs2` union querysets in `subtree_groups.py` (annotations, parent-child relationships, idempotency delete) to single `Q`-OR filters so `.delete()` runs against an explicit WHERE clause rather than relying on Django queryset combining; made root traversal order deterministic (`sorted`) to mirror the existing `non_leaves` sort; clarified the comment above the per-row `Relationship.objects.create()` loop to note that `Relationship.save()` is overridden to call `clean()` (Django's default `Model.save()` does not, so `bulk_create` would silently bypass the document-XOR-structural_set constraint); added `test_depth_cap_warning_logged` mirroring `test_size_cap_warning_logged` to lock in the depth-cap warning contract.

- **OC_URL link annotations — clickable hyperlinks anchored to highlighted text** (`opencontractserver/annotations/models.py`, `opencontractserver/annotations/migrations/0072_annotation_link_url.py`, `opencontractserver/constants/annotations.py`, `config/graphql/{annotation_mutations,serializers,mutations}.py`, `frontend/src/assets/configurations/constants.ts`, `frontend/src/types/graphql-api.ts`, `frontend/src/components/annotator/types/annotations.ts`, `frontend/src/components/annotator/utils/urlAnnotation.ts` (new), `frontend/src/graphql/{queries,mutations}.ts`, `frontend/src/components/annotator/hooks/AnnotationHooks.tsx`, `frontend/src/components/annotator/display/components/{Selection,SelectionBoundary}.tsx`, `frontend/src/components/annotator/renderers/pdf/{PDF,PDFPage,SelectionLayer}.tsx`, `frontend/src/components/annotator/renderers/txt/TxtAnnotator.tsx`, `frontend/src/components/annotator/components/wrappers/TxtAnnotatorWrapper.tsx`, `frontend/src/components/annotator/components/modals/CreateUrlAnnotationModal.tsx` (new), `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`, `frontend/src/components/knowledge_base/document/document_kb/DocumentViewer.tsx`, `frontend/src/utils/transform.tsx`). Annotations carrying the new `OC_URL` label render as hyperlinks (underline + external-link icon, pointer cursor) and open their `link_url` on click in both the PDF viewer and the text/markdown viewer. The `Annotation.link_url` URL field (max 2048, nullable) carries the target; an "Add link…" item in the selection action menu prompts for a URL and calls a new `add_url_annotation` GraphQL mutation that auto-creates the `OC_URL` label per-corpus via `Corpus.ensure_label_and_labelset` (mirroring `OC_SECTION`). The pencil icon on an existing `OC_URL` annotation opens a URL-edit modal instead of the label modal. Held `Shift`/`Ctrl`/`Cmd` while clicking a link annotation falls back to the normal "toggle selection" behaviour so authors can still pick a link to delete or re-edit it. **Security**: `link_url` is validated both in `Annotation.save()` (always runs, regardless of `VALIDATE_ANNOTATION_JSON`) and in the GraphQL mutation/serializer layer via a shared `validate_link_url(...)` helper — only `http://`, `https://`, and site-relative `/...` paths are accepted, so `javascript:`, `data:`, and other dangerous schemes are rejected before persistence and never reach `window.open`. External targets open in a new tab with `noopener,noreferrer`; site-relative paths navigate in the current tab so the SPA router can resolve them.

### Security

- **Auth0 permissioning audit — privilege escalation, IDOR, and credential storage hardening** (`config/settings/base.py`, `config/graphql_auth0_auth/utils.py`, `config/graphql/annotation_mutations.py`, `config/graphql/extract_mutations.py`, `opencontractserver/analyzer/{models.py,views.py,migrations/0021_hash_analysis_callback_token.py}`, `opencontractserver/utils/analyzer.py`, `opencontractserver/users/{apps.py,checks.py,models.py}`, `opencontractserver/constants/auth.py`, `opencontractserver/tests/{test_analyzers.py,test_security_hardening.py}`, `docs/configuration/authentication.md`). Outcome of an end-to-end audit of every authorization pathway that touches Auth0. Verified findings only — agent-reported "CRITICAL" claims that turned out to be false positives (intentional IDOR-safe `SetCorpusVisibility`, `AnnotationImagesView`, moderation mutations, `conversation_queries.moderation_metrics`, etc.) were dropped after self-verification. Twelve real fixes shipped across three layers:

  - **F1 — Defense-in-depth allowlist for `is_superuser` JWT claim sync.** Added `AUTH0_SUPERUSER_SUB_ALLOWLIST` (env, default `[]`). `sync_admin_claims_from_payload` now refuses to flip `User.is_superuser` to `True` unless the user's Auth0 `sub` (Django `username`) is in the allowlist, regardless of what the verified token says. This blocks the structural privilege-escalation path where a misconfigured Auth0 tenant Action sources `is_superuser` from user-writable `user_metadata` instead of admin-only `app_metadata`. New `users.W001` system check warns at startup when `USE_AUTH0=True` but the allowlist is empty (the safe default that demotes existing superusers on next sync until operators populate the list).
  - **F2 — Tightened admin-claim cache window.** `ADMIN_CLAIMS_CACHE_TTL` reduced from `300s` to `30s` (`opencontractserver/constants/auth.py`). Bounds the privilege-retention gap when Auth0 demotes a user — was up to 5 minutes, now ≤30s on API requests. Admin login was already cache-bypassed and unchanged.
  - **F3 + F4 — `AddRelationship` IDOR oracle eliminated** (`config/graphql/annotation_mutations.py:466-560`). `Annotation.objects.filter(id__in=...)` replaced with `visible_to_user(user).filter(...)` plus a count comparison; `Corpus.objects.get(pk=...)` replaced with `visible_to_user(user).get(...)`. All not-found and no-permission branches now return a single `"Relationship target(s) not found or you do not have permission to create a relationship here."` message — no more annotation-ID echoing, no more `str(DoesNotExist)` leaks. Broad `except` no longer surfaces ORM text. **Follow-up:** the `document_id` argument is now visibility-checked too via `Document.objects.visible_to_user(user).filter(pk=document_pk).exists()`, closing the residual IDOR oracle where a caller with CREATE on a corpus could still land a relationship row on a `document_id` they could not see. Failure path collapses into the same not-found message. Coverage in `TestAddRelationshipDocumentIDOR` (`opencontractserver/tests/test_security_hardening.py`).
  - **F5 — `CreateMetadataColumn` corpus existence oracle eliminated** (`config/graphql/extract_mutations.py:170-280`). Same `visible_to_user`+unified-message pattern.
  - **F6 — `UpdateMetadataColumn` column existence oracle eliminated** (`config/graphql/extract_mutations.py:281-352`). `Column` inherits `BaseVisibilityManager` from `BaseOCModel`, so `Column.objects.visible_to_user(user).get(...)` already works.
  - **F7 — Datacell mutation broad-except message leaks** (`config/graphql/extract_mutations.py:51-138`). `Datacell.DoesNotExist` now caught explicitly with a generic message; broad `except` logs server-side and returns a fixed string instead of `f"...{e}"`.
  - **F9 — Bounded JWKS stale-cache fallback** (`config/graphql_auth0_auth/utils.py:_get_cached_jwks`). Added `_JWKS_STALE_GRACE = 3600s`. Stale JWKS continues to be served only within `_JWKS_CACHE_TTL + _JWKS_STALE_GRACE` of the last successful fetch; beyond that window, decode fails closed. Closes the indefinite-stale-key window where an Auth0 outage after key rotation would let the old (potentially compromised) key keep verifying tokens.
  - **F10 — Hash `Analysis.callback_token` at rest, plaintext only at submit time** (`opencontractserver/analyzer/{models.py,views.py,migrations/0021_hash_analysis_callback_token.py}`, `opencontractserver/utils/analyzer.py`). Replaced the `UUIDField` with `callback_token_hash` (`CharField(max_length=64)` storing SHA-256 hex). Helper methods `Analysis.rotate_callback_token() -> plaintext` and `Analysis.verify_callback_token(candidate) -> bool` (constant-time) live on the model. `submit_corpus_documents_to_analyzer` mints a fresh plaintext at submit time and sends it to the gremlin worker; the DB never stores the plaintext. The submit-time logging line is now redacted. Migration `0021_hash_analysis_callback_token` backfills `callback_token_hash` by hashing existing UUID `callback_token` values, so any in-flight analyzer that still holds a previously-issued plaintext continues to verify successfully. Disabled-test `__test_receive_callback_with_gremlin_results` and `test_correct_token_uuid_type_accepted` updated to use `rotate_callback_token`.
  - **F8 + F11 — Documentation hardening for AUTH0_CREATE_NEW_USERS and `User.email` non-uniqueness.** No code/schema change. `docs/configuration/authentication.md` now includes a Step 7 walkthrough for `AUTH0_SUPERUSER_SUB_ALLOWLIST`, env-var rows for the allowlist and `AUTH0_CREATE_NEW_USERS`, a `!!! danger` callout pinning admin claims to `app_metadata` (never `user_metadata`), and a note that `User.email` is informational only — identity is the Auth0 `sub` in `User.username`. Inline comment added at the email field declaration.

  **Verified false positives (NOT fixed — agents were wrong):** `SetCorpusVisibility` (`corpus_mutations.py:82-104`), `AnnotationImagesView`, `bulk_document_upload_status` cache-miss, GraphQL introspection in DEBUG, moderation mutations (`moderation_mutations.py:413/492/556`), `conversation_queries.moderation_metrics:553`, `UpdateCorpusMutation` preferred_embedder oracle, JIT email-collision claim. Each was re-read in source before being dismissed.

  **Operator action required before deploy:** populate `AUTH0_SUPERUSER_SUB_ALLOWLIST` with the Auth0 `sub` of every user who must retain Django `is_superuser`. Existing superusers whose subs are not in the allowlist will be demoted on their next claim sync (within ~30s of their next API call). See `docs/configuration/authentication.md` Step 7 and `users.W001`.

  **Review follow-ups** (`config/settings/base.py`, `config/settings/test.py`, `config/graphql_auth0_auth/utils.py`, `opencontractserver/analyzer/migrations/0021_hash_analysis_callback_token.py`, `opencontractserver/tests/{test_admin_auth.py,test_auth0_jwks_cache.py,test_security_hardening.py,test_metadata_columns_graphql.py}`):
  - Made `ADMIN_CLAIMS_CACHE_TTL` configurable via `AUTH0_ADMIN_CLAIMS_CACHE_TTL` so operators can tune the revocation-SLA vs. write-pressure trade-off without a code change.
  - Wired `AUTH0_CREATE_NEW_USERS` env var into `AUTH0_JWT` settings so `get_auth0_user_from_token` actually honours operator-disabled provisioning.
  - Migration `0021` reverse path uses `QuerySet.update()` instead of `model.save()` (no historical-model signals on downgrade) and `iterator(chunk_size=500)` to bound memory.
  - `_can_serve_stale` now asserts it is holding `_jwks_cache_lock` and carries a prominent docstring warning.
  - `SILENCED_SYSTEM_CHECKS = ["users.W001"]` added to test settings so empty-allowlist tests don't print noise.
  - New direct unit tests cover `TestAuth0SuperuserAllowlist` (allowlist gate), `TestAuth0SuperuserAllowlistSystemCheck` (`users.W001`), `TestAnalysisCallbackTokenHelpers` (rotate/verify edges), `TestCanServeStale` (stale-grace helper), and JWKS fail-closed paths beyond the stale-grace window. Updated the existing failing tests to populate `AUTH0_SUPERUSER_SUB_ALLOWLIST` for elevation paths and match the new IDOR-safe `"Corpus not found or you do not have permission"` message.

### Fixed

- **Import/export V2 follow-ups from PR #1635 review (closes #1649)** (`opencontractserver/constants/zip_export.py` (new), `opencontractserver/tasks/export_tasks_v2.py`, `opencontractserver/tasks/export_tasks.py`, `opencontractserver/tasks/import_tasks_v2.py`, `opencontractserver/tasks/fork_tasks.py`).

  - **Memory regression in V2 corpus export → fork OOM risk on large corpora** (`opencontractserver/tasks/export_tasks_v2.py`, `opencontractserver/tasks/export_tasks.py`, new `opencontractserver/constants/zip_export.py`). `build_corpus_v2_zip` materialised the entire export — including every active document's binary PDF content — into an `io.BytesIO` buffer before the caller (fork or `finalize_export`) consumed it. A 200 × 5 MB-PDF corpus produced a ~1 GB in-process allocation; on a worker with a 2 GB cap, fork would silently OOM. Replaced `io.BytesIO()` with a `tempfile.SpooledTemporaryFile` whose `max_size` is the new `EXPORT_SPOOL_MAX_SIZE_BYTES` constant (default 100 MB, settings-overridable). Small archives stay entirely in heap; large archives transparently roll over to a temp file the OS can page out, bounding peak heap to ~100 MB regardless of corpus size. Companion fix in `finalize_export`: switched the persistent write from `ContentFile(output_bytes.read())` (which re-buffered the *entire* archive in memory before writing) to `django.core.files.File(output_bytes, name=filename)`, so `default_storage.save` streams chunks straight off the spool. Parameter type widened from `io.BytesIO` to `IO[bytes]` to match. The Celery wrapper `package_corpus_export_v2` and the fork path now use the spooled buffer as a context manager so the on-disk temp file releases even if the import raises.
  - **Redundant exception summary in import V2 logging** (`opencontractserver/tasks/import_tasks_v2.py`). `logger.error("import_corpus_v2_from_bytes() - Exception: %s", e, exc_info=True)` duplicated the exception summary — `exc_info=True` already attaches the formatted traceback to the log record, so structured-log aggregators showed the exception text twice (once in the message body, once in the stack trace). Simplified to `logger.error("import_corpus_v2_from_bytes() failed", exc_info=True)` and added an inline comment explaining the convention so the duplication does not creep back.
  - **Partial-rollback contract documented in `_import_corpus` and `fork_corpus`** (`opencontractserver/tasks/import_tasks_v2.py`, `opencontractserver/tasks/fork_tasks.py`). `_import_corpus` opens nested `transaction.atomic()` blocks for ingestion sources, metadata schema, etc. Django promotes these to savepoints (not autonomous transactions), and the function's broad-except returns `None` after rolling back only the failing savepoint — earlier savepoints that already released remain pending against whatever atomic the caller wraps the call in. Added a docstring paragraph to `_import_corpus` spelling out the savepoint semantics and the caller's obligation to wrap in an outer atomic if it wants all-or-nothing semantics on failure. Added a matching block-level comment in `fork_corpus` at the `with transaction.atomic():` site explaining that the `raise RuntimeError(...)` on `imported_id is None` is what triggers the outer rollback, and that `transaction.on_commit` callbacks registered inside (the orphaned-blob GC) only fire on a successful outer commit so rollback is leak-safe.

- **Flaky `TestApprovalFlow` / `TestNestedApprovalGates` / `TestPydanticAIAgents*` / `TestRunAgentThreadActionAsync` / `TestLongConversationAPI` from stale `@override_settings(DATABASES=...)` decorator** (`opencontractserver/tests/test_tool_approval_gate.py:114`, `test_nested_approval_gates.py:132`, `test_pydantic_ai_agents.py:116,721`, `test_thread_corpus_actions.py:1190`, `test_long_conversation_api.py:35`). The four `test_tool_approval_gate.py` failures observed on the main CI/CD run (`test_event_serialisation`, `test_multiple_gates_interleaved`, `test_rejection_branch`, `test_single_gate_pause_resume`) — and intermittent failures across the four sibling files using the same harness — traced back to a class-level `@override_settings(DATABASES={"default": {"CONN_MAX_AGE": 0}})` decorator. The intent was "force fresh DB connections for async ORM," but the override fully *replaces* `settings.DATABASES["default"]` with a one-key dict missing `ENGINE` / `NAME` / `USER` / etc. Django itself flags `DATABASES` as a `COMPLEX_OVERRIDE_SETTINGS` (`django/test/signals.py`) and emits `UserWarning: Overriding setting DATABASES can lead to unexpected behavior.` for exactly this reason. Existing `BaseDatabaseWrapper` instances survive (they captured `settings_dict` at creation), but any *new* connection created while the override is active reads the broken settings. For `TransactionTestCase` + `@pytest.mark.serial` runs, `conftest.py:258-264` aggressively calls `db.connections.close_all()` in teardown, forcing reconnection during the override window for the next test; `TransactionTestCase` also flushes `django_content_type` + `auth_permission` between tests, so the next `setUp` racing with the stale ContentType cache hit a `duplicate key value violates unique constraint "auth_permission_content_type_id_codename_..."` (or `..."django_content_type_app_label_model_..."`) `IntegrityError`. Symptom on the failing-tests path was either `Document.DoesNotExist` (data not visible through the freshly-rebuilt connection) or the permission `IntegrityError` in `setUp`. `CONN_MAX_AGE=0` does nothing for tests anyway — it is only consulted by `close_old_connections`, wired exclusively to `request_started`/`request_finished` signals (`django/db/__init__.py:61-62`). Fix: dropped the decorator from all six sites (and the unused `override_settings` import where it became dead), and removed the now-misleading "@override_settings to prevent connection pooling issues" line from the `TestRunAgentThreadActionAsync` docstring in `test_thread_corpus_actions.py`. Local repro before fix: 2/8 failures on the file alone, 3/5 with all five files (one run hitting 34 failed tests). After fix: 5/5 single-file, 3/3 multi-file, plus a clean parallel `-n 4 --dist loadscope` run across all five files.

- **Auth0 admin login race condition — SDK loaded asynchronously while `initAuth0` ran synchronously** (`opencontractserver/templates/admin/auth0_login.html`, `opencontractserver/tests/test_admin_auth.py`). The Auth0 SPA SDK is loaded via `document.createElement('script')`, which is implicitly `async=true` per the HTML spec. The template had two separate inline `<script>` blocks: the first injected the SDK element with only an `error` listener, the second called `initAuth0()` synchronously. Because the SDK script was still fetching when `initAuth0` ran, `typeof auth0 === 'undefined'` was true and the misleading message `Failed to load Auth0 SDK - SRI check may have failed` was logged while the admin login UI showed "Auth0 unavailable (security check failed)" — even though the SRI hash matched the live CDN file and `cdn.jsdelivr.net` responded HTTP 200. Fix: merged the two scripts into one nonce'd block and wired `sdkScript.addEventListener('load', initAuth0)` (alongside the existing `error` listener) *before* `insertBefore`, so a cached SDK cannot fire `load` before the listener is attached. Removed the now-dead `typeof auth0 === 'undefined'` defensive check inside `initAuth0` (the `load` event guarantees the global is present) and the synchronous `initAuth0()` call that was the race trigger. Tightened the error message in `handleScriptError` from `"Failed to load Auth0 SDK - SRI check may have failed"` to `"Auth0 SDK failed to load (network or SRI check)"` since after the refactor the function only fires from the script element's `error` event, which triggers on any real load failure (network, CSP block, or SRI mismatch). Auth0 callback handling (`?code=…&state=…`), CSP nonce / integrity / crossOrigin attributes, and the rate-limit-aware password fallback form (separate script tag) are all preserved. New regression test `TestAdminLoginView.test_auth0_sdk_load_listener_wired` asserts that both `load` and `error` listeners attach before `insertBefore` and that no synchronous `initAuth0()` call remains in the rendered template, so the race cannot silently come back via a future refactor.

### Added

- **Sidecar + upversion intersection test** (`opencontractserver/tests/test_sidecar_import.py` — new `TestSidecarUpversioning` class). Pins the contract that re-importing a zip with a sidecar at an already-occupied corpus path (default pipeline route, i.e. ``skip_pipeline`` absent / falsy) (a) creates a new `Document` in the same `version_tree_id` with `parent` pointing at the prior version and `DocumentPath.version_number` incremented; (b) attaches the new sidecar's text annotations / doc-level labels to the **new** `Document`; (c) leaves the prior version's annotations attached to the now-non-current `Document` (no migration, no deletion); and (d) the current-path query (`DocumentPath.is_current=True, is_deleted=False`) resolves only to the new version's annotations. Sidecar title / description are intentionally not asserted here because they only land on the Document row via the `skip_pipeline=True` route (already covered by `test_skip_pipeline_creates_document_from_export_data`). Existing upversion tests in `test_zip_import_integration.py::TestDocumentUpversioning` covered the path/version-tree mechanics; this test pins the sidecar-payload-meets-collision case that they didn't exercise. Also factors out a shared `_SidecarImportTestMixin` to DRY the in-memory zip / `TemporaryFileHandle` helpers across the sidecar-import test classes in the file.

### Changed

- **Decompose `ChatTray.tsx` into thinner composer + sibling hooks** (`frontend/src/components/knowledge_base/document/right_tray/ChatTray.tsx`, plus new siblings `useChatStreamHandlers.ts`, `useChatAgentMessageHandler.ts`, `useChatSendHandlers.ts`, and `chatUtils.ts`). Next slice of the multi-PR decomposition tracked by #1446 (after #1459 / `ModernDocumentItem`, #1578 / `Corpuses`, #1586 / `ChatMessage`, #1580 / `DocumentKnowledgeBase` partial). `ChatTray.tsx` had grown to **1,815 lines**, but unlike prior decompositions it owned no inline styled-components or React sub-components — its weight was concentrated in stateful handler functions. Three hook extractions land it at **1,160 lines (-36%)**:

  - **`useChatStreamHandlers.ts`** _(new)_: Bundles the six chat-state mutators that ingest WebSocket frames and persist complete messages into `ChatSourceAtom` — `updateMessageApprovalStatus`, `appendStreamingTokenToChat`, `appendThoughtToMessage`, `finalizeStreamingResponse`, `handleCompleteMessage`, `mergeSourcesIntoMessage`. Each handler is wrapped in `useCallback` with stable setter deps; the bundle is `useMemo`-stabilized so callers can pass it as a single param without rebinding. The hook's param surface is deliberately kept free of `documentId` / `corpusId` so `CorpusChat.tsx` (which has the same five stream functions verbatim) can adopt it in a follow-up PR.
  - **`useChatAgentMessageHandler.ts`** _(new)_: Wraps the 188-line `handleAgentMessage` switch (10+ message types: ASYNC_START / ASYNC_CONTENT / ASYNC_THOUGHT / ASYNC_SOURCES / ASYNC_APPROVAL_NEEDED / ASYNC_APPROVAL_RESULT / ASYNC_RESUME / ASYNC_FINISH / ASYNC_ERROR / SYNC_CONTENT). Receives `pendingApprovalRef` (the existing stale-closure escape hatch from the WebSocket subscription) plus the stream-handlers bundle, and returns the stable `onMessage` callback consumed by `useWebSocketAuth`. Carries main's rich-mention `requesting_agent` plumbing through `ASYNC_APPROVAL_NEEDED` and the `persistence_failed` console warning on `ASYNC_FINISH` so sub-agent delegation behaviour from PR #1651 is preserved on this decomposition.
  - **`useChatSendHandlers.ts`** _(new)_: Bundles the three user-facing send actions — `sendMessageOverSocket` (debounced via `sendingLockRef`), `sendApprovalDecision`, and `sendTextImmediately` (programmatic send for the `initialMessage` prop). Takes `updateMessageApprovalStatus` as an input from the stream hook for optimistic approval UI.
  - **`chatUtils.ts`** _(extended)_: Adds `adjustTextareaHeight(textarea, maxHeight = 200)` — a pure DOM helper extracted from the composer. The composer wraps it in a `useCallback` that reads from `textareaRef.current`.
  - Behavior is preserved verbatim — same handler bodies, same `pendingApprovalRef` read pattern, same inline auto-scroll DOM side-effect on streaming token append, same empty-sources guard on `mergeSourcesIntoMessage`. Public API (`ChatTray` named export and the `WebSocketSources` / `MessageData` type re-exports) is unchanged; importers (`RightPanelContent.tsx`, `ChatTray.ct.tsx`, `ChatTrayTestWrapper.tsx`, `ContextMeter.ct.tsx`) need no changes. Rich-mention agent picker wiring (`useChatMentionPicker`, the `mentionPopover` portal, and the textarea's `handleMentionValueChange` integration) introduced upstream by PR #1651 is retained.
  - Render-body splitting is **deferred** to a follow-up PR: the natural sub-views (header / messages list / compaction banner / context meter / input footer / approval modal / conversation-list alternate branch) each share a large overlapping prop surface that has no atom-friendly seam without more preparatory work. Mixing render-body extraction with hook extraction in one slice would blow the diff and make review hard to bisect.

- **PII scan tool — review follow-ups from PR #1624 (closes #1638)** (`opencontractserver/llms/tools/core_tools/pii.py`, `opencontractserver/llms/tools/core_tools/_privacy_filter_client.py`, `opencontractserver/agents/{apps,checks}.py`, `opencontractserver/tests/test_pii_scan_tool.py`, `opencontractserver/tests/test_privacy_filter_client.py`, `docs/sample_env_files/backend/production/django.env`).

  - **`_persist_annotations_sync` switched from per-row `Annotation.save()` to `Annotation.objects.bulk_create(batch_size=200)`** (`pii.py`). A document with 200+ PII hits no longer pays 200+ round-trips inside the atomic block. The signal handler that normally queues `calculate_embedding_for_annotation_text` (`opencontractserver/annotations/signals.py:process_annot_on_create_atomic`) is bypassed by `bulk_create`, so we now register `transaction.on_commit` Celery enqueues directly inside the atomic block — mirroring the pattern in `annotations/models.py` for duplicated annotation sets — and keep the `embed-annot-{pk}` task-id-based dedup. PDF TOKEN_LABEL JSON is also explicitly `compact_annotation_json()`'d before `bulk_create` because `Annotation.save()`'s auto-compaction is skipped on the bulk path.

  - **Hoisted `plasmapdf.models.types` imports to module scope** (`pii.py`). `SpanAnnotation` and `TextSpan` previously imported inside an `if file_type == "application/pdf":` branch but referenced inside an inner loop branch — Python scoping made it work but static analyzers flagged both names as possibly-undefined.

  - **Dropped `CHUNK_SIZE` / `CHUNK_OVERLAP` aliases from `_privacy_filter_client.py`** (`_privacy_filter_client.py`, `test_privacy_filter_client.py`). The aliases looked like a public re-export of the canonical constants in `opencontractserver/constants/document_processing.py`; tests now import `PRIVACY_FILTER_CHUNK_SIZE` / `PRIVACY_FILTER_CHUNK_OVERLAP` directly from the constants module, and the client uses the long names internally.

  - **New `agents.W001` Django system check** (`opencontractserver/agents/checks.py`, `opencontractserver/agents/apps.py`). Surfaces the `PRIVACY_FILTER_URL set + PRIVACY_FILTER_API_KEY empty` misconfiguration on every `manage.py` invocation (in addition to the existing one-shot `logger.warning` emitted by `_privacy_filter_client.py` on first request). Production env template (`docs/sample_env_files/backend/production/django.env`) now has a `Privacy Filter` section that calls out the same risk explicitly.

  - **Regression test `test_persist_annotations_sync_duplicate_label_race`** (`test_pii_scan_tool.py`). Documents the accepted-duplicate behavior of `Corpus.ensure_label_and_labelset` under PostgreSQL READ COMMITTED — two concurrent scans on the same fresh corpus can both create labels with the same `(text, label_type)` because there is no DB-level uniqueness constraint. The test simulates the post-race outcome by patching `ensure_label_and_labelset` and verifies (a) both `_persist_annotations_sync` calls succeed, (b) two label rows exist, (c) each annotation points at its scan's own race-created label. If somebody later adds a `UniqueConstraint(fields=["text", "label_type"], ...)` on `AnnotationLabel`, this test fails and forces a deliberate revisit of the trade-off.

  - **`_persist_annotations_sync` defensive unknown-group filter pinned** (`opencontractserver/llms/tools/core_tools/pii.py`, `opencontractserver/tests/test_pii_scan_tool.py`). The set comprehension that builds `needed_groups` already filters detections by membership in `ENTITY_GROUP_LABELS`, and the per-detection loop also skips any group not in `ENTITY_GROUP_LABELS`. New `PersistAnnotationsUnknownGroupTests.test_unknown_entity_group_is_skipped_silently` pins both filters together: a batch mixing one known group with one unknown group yields exactly one persisted annotation and exactly one new label (for the known group). This guards against a future refactor that drops either filter and lets unknown groups leak into `ensure_label_and_labelset` (which would crash or, worse, succeed and create orphan labels with bogus colors / icons).

  - **`_persist_annotations_sync` lambda lifted to named factory for mypy** (`opencontractserver/llms/tools/core_tools/pii.py`). The inline `transaction.on_commit(lambda pk=ann.pk, cid=corpus_pk: ...)` registration broke `mypy` with `Cannot infer type of lambda`. Replaced with `_queue_embed(pk, cid) -> Callable[[], None]` so the return type is explicit and the closure binding stays correct. Behaviour is unchanged.

- **Corpus Home document-tree loads ~order-of-magnitude faster** (`config/graphql/filters.py`, `frontend/src/graphql/queries.ts`, `frontend/src/components/corpuses/DocumentTableOfContents.tsx`). The "Loading document structure…" wait on Corpus Home was driven by two over-fetched GraphQL queries. (1) `GET_DOCUMENT_RELATIONSHIPS` (capped at `first: 500`) selected the full `sourceDocument`/`targetDocument` objects (title, description, fileType, **icon**, slug, creator), plus `corpus { …creator }`, `creator { …username }`, `annotationLabel { id, text, color, icon }`, `data`, `created`, `modified`, and `myPermissions`. `DocumentTableOfContents.tsx:535-611` then consumed only four of those fields (`relationshipType`, `annotationLabel.text`, `sourceDocument.id`, `targetDocument.id`) — every `icon` selection still triggered `build_absolute_uri()` via `create_file_resolver` and the per-row document payloads were thrown away because `GET_CORPUS_DOCUMENTS_FOR_TOC` already provided the same fields. Worse, all relationships came back regardless of type/label and were filtered client-side to `relationshipType === "RELATIONSHIP"` AND `annotationLabel.text.toLowerCase() === "parent"`. (2) `GET_CORPUS_DOCUMENTS_FOR_TOC` selected `icon` and `creator { id, slug }`, neither of which the TOC renders (`renderNode` derives the on-screen icon from `fileType` via `getFileIcon`, and creator metadata is never displayed). Fix: a new `annotation_label_text` `iexact` filter on `DocumentRelationshipFilter` (paired with the existing `relationship_type` filter) so the parent-only restriction now runs on the server; a new ultra-lean `GET_CORPUS_DOCUMENT_TOC_EDGES` query that returns just `{ id, sourceDocument { id }, targetDocument { id } }` and lives alongside (not in place of) the original `GET_DOCUMENT_RELATIONSHIPS` because `CorpusDocumentRelationships.tsx:314` and `DocumentRelationshipModal.tsx:611` still need the rich payload; `GET_CORPUS_DOCUMENTS_FOR_TOC` slimmed to `{ id, title, description, slug, fileType }`. `RunCorpusActionModal.tsx` (the only other consumer of the doc query) reads only `node.id` / `node.title`, so the field removal is safe. End result: for a 76-document corpus, the relationships payload drops from "76+ source-and-target document blobs" to "76 ID-only edge rows," up to 152 `build_absolute_uri()` icon-resolver calls and 76 description blobs are eliminated, and any non-parent relationships (notes, future link types) never leave the server.

- **Fork ≡ export+import: V2 parity refactor + roundtrip-loss fixes** (`opencontractserver/tasks/fork_tasks.py`, `opencontractserver/tasks/export_tasks_v2.py`, `opencontractserver/utils/{export_v2,import_v2,etl}.py`, `opencontractserver/tests/{test_corpus_export_import_v2.py,test_corpus_forking.py,test_ingestion_source.py}`, and shared fixture helpers `opencontractserver/tests/{_corpus_fixture,_corpus_snapshot}.py`). Replaces the legacy bespoke `fork_corpus` machinery with a thin shell that drives the V2 export → V2 import pipeline so the two code paths can no longer drift. `build_corpus_v2_zip` is the pure builder that `package_corpus_export_v2` (Celery task) now wraps; `import_corpus_v2_from_bytes` is the in-process entry point that `fork_corpus` invokes after `build_corpus_v2_zip`. Restores fields that previously dropped on round-trip (manual-metadata `Fieldset` / `Column` / `Datacell` rows, `IngestionSource` rows, `DocumentPath` and `CorpusFolder` snapshots, structural-set membership) and adds a three-roundtrip invariant test (`TestV2ThreeRoundTripDataIntegrity`) plus error-handling coverage (`CorpusForkErrorHandlingTest`).
  - **Behavioural change:** `fork_corpus` no longer respects selective `doc_ids` / `annotation_ids` arguments. Any caller passing those now gets a *full* fork (with a `logger.warning`). No live caller in `opencontractserver/`, `config/`, or tests still passes selective args; legacy queued Celery tasks would still run safely (full fork instead of partial). A short note here flags the contract change for downstream forks of this repo.
  - **Import-side correctness:** `import_metadata_schema` now clears its `column_map` on rollback so callers can't accidentally re-link freshly imported rows to pks that no longer exist.
  - **Observability:** `package_metadata_schema` gains an explicit `-> MetadataSchemaExport | None` return annotation so the optional behaviour is part of the type contract.
  - **Storage GC for orphaned V2-import blobs** (`opencontractserver/tasks/fork_tasks.py`): when fork re-points a forked document's `pdf_file` / `pawls_parse_file` / `txt_extract_file` / `icon` / `md_summary_file` at the source corpus's storage, the freshly-written blob the V2 import just allocated is collected and deleted via `transaction.on_commit(default_storage.delete)`. Using `on_commit` (rather than synchronous `delete()` inside the atomic block) ensures the GC only fires when the fork commits — a rolled-back fork leaves the V2-import blobs intact because they are still the live references on whatever survives the rollback. Per-blob `try / except` in the callback makes the GC pass best-effort: a storage failure on one path logs at WARNING and continues with the rest instead of aborting and leaking the remainder. Two new tests in `CorpusForkOrphanedBlobGCTest` (`opencontractserver/tests/test_corpus_forking.py`) cover the happy path (orphan paths are deleted and never collide with live references on the forked corpus) and the failure-tolerant path (`default_storage.delete` raising on every path still attempts each one). Closes the silent storage leak flagged against the prior implementation.

- **Backend dependency audit — aggressive trim of unused / vestigial packages** (`requirements/base.txt`, `requirements/local.txt`, `requirements/production.txt`, `.pre-commit-config.yaml`, `compose/production/django/Dockerfile`, deleted `requirements/filetypes/docx.txt`, `requirements/ingestors/nlm_ingest.txt`, `requirements/processors/gliner.txt`, `.pylintrc`). Grep-audited every direct dependency across `opencontractserver/`, `config/`, scripts, compose, Dockerfiles, settings, and CI workflows.
  - **Removed from `base.txt`** (zero verifiable references in code): `pandas`, `python-slugify`, `django-model-utils`, `tiktoken`, `jsonschema`.
  - **Removed from `local.txt`**: `Werkzeug[watchdog]`, `ipdb`, `pytest-sugar`, `pylint-django`, `pylint-celery`.
  - **Removed from `production.txt`**: `django-anymail[mailgun]` (settings block in `config/settings/production.py` was fully commented out; a breadcrumb comment now points future operators at re-adding the package if they want to enable Mailgun).
  - **Deleted three orphan optional-feature requirement files**: `requirements/filetypes/docx.txt` (`mammoth` for an inactive docx pipeline), `requirements/ingestors/nlm_ingest.txt` (body was literally `# pass`), `requirements/processors/gliner.txt` (body was only commented placeholders).
  - **Deleted `.pylintrc`**: pylint is not invoked by pre-commit, CI, or any script.
  - **Deduplicated `django-extensions==4.1`**: was redundantly listed in both `base.txt` and `local.txt`; kept only in `local.txt` since it is only added to `INSTALLED_APPS` in `config/settings/local.py`.
  - **Moved `pytesseract` from `base.txt` to `local.txt`**: only used by `opencontractserver/tests/test_pdf_redaction.py` (which now also gains `@skipUnless(...)` guards on its two OCR-backed tests so they degrade gracefully when the binary is absent) to mirror docling-parser microservice OCR. Production OCR runs in the docling-parser container, so neither the package nor the `tesseract-ocr` system packages are needed in the production image (`tesseract-ocr` + `tesseract-ocr-eng` apt installs dropped from `compose/production/django/Dockerfile`).
  - **Pre-commit mypy hook kept in lockstep with `requirements/`**: the `additional_dependencies` list drops the same packages that `base.txt` / `local.txt` no longer install.
  - **Snyk pins** duplicated across all three requirements files are intentionally left in place per the existing comment in `production.txt` (Snyk does not understand `-r` inheritance).
  - **Net result**: ~10 direct deps removed from `base.txt`, ~5 from `local.txt`, 1 from `production.txt`, 4 files deleted, smaller production image (no Rust-built tiktoken wheel, no pandas, no tesseract apt packages).

### Added

- **Rich-mention agent delegation in chat** (`opencontractserver/llms/agents/mention_extractor.py`, `opencontractserver/llms/tools/delegation_tools.py`, `opencontractserver/llms/agents/unified_agent_conversation.py`, `opencontractserver/llms/agents/unified_agent_consumer.py`, `config/graphql/graphene_types.py`, `config/graphql/queries.py`, `frontend/src/components/chat/{AgentMentionPopover.tsx,RequestingAgentAttribution.tsx,agentChipStyles.ts,types.ts}`, `frontend/src/hooks/{useAgentMentionTrigger.ts,useChatMentionPicker.ts}`, `frontend/src/components/knowledge_base/document/right_tray/{ChatTray.tsx,ApprovalOverlay.tsx}`, `frontend/src/components/corpuses/CorpusChat.tsx`, `frontend/src/components/corpuses/corpus_chat/ApprovalModal.tsx`, `frontend/src/components/widgets/chat/{ChatMessage.tsx,ChatMessageTimeline.tsx,timelineEntryFactory.ts}`, `frontend/src/components/markdown/MarkdownMessageRenderer.tsx`, `frontend/src/assets/configurations/osLegalStyles.ts`). Users can `@`-mention an agent in `ChatTray` (document chat) and `CorpusChat` (corpus chat) to delegate that turn's work to a sub-agent via the conductor's tool surface. Per-turn allowlist (only @-mentioned agents become delegation tools, rebuilt by `UnifiedAgentConsumer` before each turn), scoped sub-agent context (no inherited conversation history; document/corpus tools attached based on the sub-agent's scope), and pinned-or-timeline attribution: `pin=true` creates a distinct `ChatMessage` row with `agentConfiguration` set so the sub-agent's reply gets its own bubble + attribution chip in the bubble header; `pin=false` keeps the sub-agent's work as a `tool_call`/`tool_result` pair inside the conductor's timeline with an `@<slug>` chip in place of the raw `delegate_to_<slug>` tool name. Approval bubbling: when a sub-agent's tool call requires confirmation, the `ASYNC_APPROVAL_NEEDED` frame carries `requesting_agent` so the approval modal attributes the request to `@<slug>` rather than the conductor. Backend pieces: shared mention extractor at `opencontractserver/llms/agents/mention_extractor.py` (single source of truth for the markdown URL grammar, used by `ChatMessageType.mentioned_resources` + the consumer's per-turn extraction), delegation tool factory + `StreamRelay` dataclass at `opencontractserver/llms/tools/delegation_tools.py`, `filter_by_scope` helper for resolving GLOBAL + corpus-scoped agent visibility, per-turn conductor rebuild in `UnifiedAgentConsumer.handle_query`. Frontend pieces: lightweight `@`-popover overlay (`useAgentMentionTrigger` hook + `AgentMentionPopover` component, debounced via `useUnifiedMentionSearch`) on the existing chat textarea, `MarkdownMessageRenderer` now wired into `ChatMessage` so inline `[@slug](/agents/slug)` markdown links render as styled chips, shared attribution chips on pinned sub-agent bubbles (`ChatMessage.styles.ts`) + timeline entries (`ChatMessageTimeline.styles.ts`) + approval modals (`RequestingAgentAttribution.tsx`). Pinned sub-agent rows expose `agentConfiguration` on `GET_CHAT_MESSAGES` so re-hydrated conversations restore the bubble + chip. Design spec: `docs/superpowers/specs/2026-05-13-rich-mention-agent-delegation-design.md`. Implementation plan: `docs/superpowers/plans/2026-05-13-rich-mention-agent-delegation.md`. Backend test coverage: `test_mention_extractor.py`, `test_delegation_tools.py`, `test_unified_agent_consumer_delegation.py`, `test_chat_message_mentioned_resources.py`, `test_mentions.py`. Frontend CT coverage: `AgentMentionPopover.ct.tsx` and new delegation scenarios in `ChatTray.ct.tsx` + `CorpusChat.ct.tsx`. Documentation screenshots: `chat--agent-mention--{popover-open,pinned-bubble,timeline-entry,approval-with-attribution}`.

  **Review follow-ups** (`opencontractserver/llms/tools/delegation_tools.py`, `opencontractserver/llms/agents/mention_extractor.py`, `config/websocket/consumers/unified_agent_conversation.py`, `frontend/src/components/chat/{AgentMentionPopover.tsx,RequestingAgentAttribution.tsx,agentChipStyles.ts}`, `frontend/src/components/widgets/chat/{ChatMessage.styles.ts,ChatMessageTimeline.styles.ts}`, `frontend/src/components/knowledge_base/document/right_tray/ChatTray.tsx`, `frontend/src/assets/configurations/osLegalStyles.ts`, `opencontractserver/tests/test_delegation_tools.py`, `frontend/tests/AgentMentionPopover.ct.tsx`):
  - `filter_by_scope` replaces an `assert document_id is not None` with an explicit `if document_id is None: return qs.filter(scope="GLOBAL")` so the path can't disappear under `python -O`.
  - `build_delegation_tool` / `_build_stream_relay_factory` tighten the `relay_factory` return type from `StreamRelay | None` to `StreamRelay` — the production factory always builds a relay, and the previous union widened the contract for no benefit. Test helper `_make_noop_relay` added so non-relay tests construct a real no-op relay instead of returning `None`.
  - `extract_mentions` swaps the legacy-pattern `any(m.type == ... for m in out)` dedupe scan (O(n²)) for a `(type, slug, corpus_slug)` set lookup (O(1) per match). `corpus_slug` is included in the key so a corpus-scoped document and a standalone document with the same slug stay distinct entries.
  - Agent attribution palette extracted to a single shared CSS fragment (`agentChipPaletteCss` in `frontend/src/components/chat/agentChipStyles.ts`, sourced from new `OS_LEGAL_COLORS.agentChipViolet` / `agentChipText` tokens). The three duplicated `#8b5cf6` / `#6366f1` / `#7c3aed` chip definitions in `ChatMessage.styles.ts`, `ChatMessageTimeline.styles.ts`, and `RequestingAgentAttribution.tsx` now compose this fragment, so the bubble-header, timeline, and approval-modal chips can never drift apart visually.
  - `AgentMentionPopover` gains keyboard navigation: ArrowDown / ArrowUp move the highlight with wrap-around, Enter selects the active option, the active row carries `aria-selected="true"`, and `scrollIntoView({ block: "nearest" })` keeps it visible inside the popover's scroll container. The handler is installed on `document` in the capture phase so the textarea (which retains focus) doesn't see Enter/Arrow keys while the picker is open. Three new CT cases in `AgentMentionPopover.ct.tsx` pin the wrap-around and Enter-select behaviour.
  - `ChatTray.tsx` inline `requesting_agent` cast replaced with `PendingApproval["requestingAgent"]` so the persisted-message and live-WebSocket frame shapes share one type definition.

  **Second round of review follow-ups** (`opencontractserver/llms/tools/delegation_tools.py`, `opencontractserver/llms/agents/mention_extractor.py`, `config/websocket/consumers/unified_agent_conversation.py`, `config/graphql/conversation_types.py`, `frontend/src/components/knowledge_base/document/right_tray/ChatTray.tsx`, `frontend/src/components/corpuses/CorpusChat.tsx`, `frontend/src/utils/agentMentionLink.ts`, `opencontractserver/tests/test_core_agents.py`):
  - **Dead `if relay is None` branches removed** in `delegation_tools._body`. The relay is non-Optional per the tightened factory contract; the previous defensive checks were unreachable. Each `if pin and relay is not None` collapses to `if pin`.
  - **Monkey-patched `_parent_message_id_box` eliminated.** `_build_stream_relay_factory` no longer stamps a private attribute onto its returned callable. `handle_query` now holds the shared one-slot box directly and passes it into `_stream_agent_response(..., parent_message_id_box=...)` as an explicit kwarg. Removes the `type: ignore` and the `getattr(relay_factory, "_parent_message_id_box", None)` lookup; the contract between the per-turn relay factory and the conductor stream consumer is now type-checked.
  - **`resolve_mentions_for_user` N+1 eliminated.** The resolver now does one batched `slug__in` / `id__in` query per mention type (corpus, document, annotation, agent) plus two batched `DocumentPath` queries (corpus-scope confirmation + best-effort context for standalone `@document` mentions) before the per-mention loop. The loop is now pure dict lookups against the pre-fetched maps — a 50-message conversation with 10 mentions per message drops from `O(n × m)` queries to a fixed `O(n_types)`.
  - **Explicit `_error` on `on_approval` auto-reject**: when the sub-agent fires its approval before the conductor has emitted its first event, the relay now returns `_error` in the decision payload so the delegation body's existing error guard surfaces it as a tool-call failure (`Sub-agent error: ...`) instead of feeding the conductor an empty string.
  - **Cross-user IDOR test for `_aget_conversation_visible_to_user`**: pins that user A receives `None` when probing a conversation owned by user B, plus owner/anonymous/bogus-id/bogus-user-id edges (5 new tests in `TestAgetConversationVisibleToUser`).
  - **`persistence_failed` flag surfaced via `console.warn`** in both `ChatTray.tsx` and `CorpusChat.tsx` ASYNC_FINISH handlers — when the backend reports that a pinned sub-agent ChatMessage couldn't be persisted, the warning makes the discrepancy visible in dev tools. The bubble still renders for the live session; a non-blocking toast is a follow-up.
  - **`filter_by_scope` docstring** now notes the sync-context requirement (DocumentPath lookup) and references the lone async caller that already wraps it correctly.
  - **`_LEGACY_CORPUS_RE` Python version guard**: top-of-module `sys.version_info < (3, 11)` check raises a loud RuntimeError instead of letting `re.compile` fail with a confusing `multiple repeat` error on older interpreters.
  - **`agentMentionLink.ts` stale comment fix**: lockstep reference points to `opencontractserver.llms.agents.mention_extractor._classify_url` (the real source) rather than the renamed `opencontractserver.utils.mentions`.
  - **Mechanical guard on `on_approval` `_pending_approvals` key collision** (`config/websocket/consumers/unified_agent_conversation.py`): the "one sub-agent per turn per `(parent_message_id, agent.pk)`" invariant is now enforced at runtime — a duplicate registration cancels the prior future and logs an error instead of silently leaking it.

- **`CorpusFolder.user_can` overridden to delegate to the parent corpus** (`opencontractserver/corpuses/models.py`, `opencontractserver/tests/permissioning/test_authorization_invariants.py`). Closes the footgun where `folder.user_can(user, perm)` would run against the folder row's (non-existent) guardian object-permission rows and silently return ``False`` for shared readers. Permissions for folders are inherited from the parent corpus, and every legacy call site in `DocumentFolderService` already routes through `folder.corpus.user_can(...)`; the override makes that delegation the only path so a future contributor can't get a wrong answer by accident. `CorpusFolderDelegatesUserCanToCorpusTestCase` (7 new tests) pins creator / shared-reader / stranger / anonymous / superuser semantics, the `is_public`-on-corpus toggle, and the full delegation matrix across every `PermissionTypes` member.

- **Centralized single-object authorization API: `Manager.user_can(user, instance, permission)` and `obj.user_can(user, permission)`** (`opencontractserver/utils/permissioning.py`, `opencontractserver/shared/{Managers,QuerySets,Models,user_can_mixin}.py`, `opencontractserver/corpuses/models.py`, `opencontractserver/tests/permissioning/test_authorization_invariants.py` (new)). Step 0 of the permission-centralization effort. Manager- and QuerySet-side `user_can` bodies share a new `UserCanMixin` (lives in `shared/user_can_mixin.py` to keep `utils.permissioning`'s eager-import chain out of the early startup graph). Instance-side `user_can(user, perm)` is provided by a sibling `InstanceUserCanMixin` applied to both `BaseOCModel` and the `TreeNode`-rooted `Corpus` / `CorpusFolder`, so every visibility-managed model exposes the same ergonomic surface (without it, `corpus.user_can(...)` calls in `folder_service.py` would raise `AttributeError` at runtime — the missing mixin on `Corpus`/`CorpusFolder` was the original CI failure mode for this PR). The codebase had two parallel authorization systems: `Manager.visible_to_user(user)` (filter side, well-centralized with per-model overrides for Annotation/Document/Conversation/etc.) and `user_has_permission_for_obj(user, instance, perm)` (single-object side, called from 104 non-test sites — mutations, optimizers, services, agent tools, tasks, imports). The two disagreed for the same `(user, instance)` because the function-side intentionally ignored creator status (see `permissioning.py:366-384`), so ~22 mutations and three service classes (`FolderService.check_corpus_*`, `DocumentQueryOptimizer._check_*_permission`, `user_can_modify_corpus`) repeated the compound `is_superuser or creator or has_perm` pattern as a workaround. This PR adds `Manager.user_can(...)` as the single source of truth that mirrors `visible_to_user` semantics: for READ, `Model.objects.user_can(u, obj, READ) == Model.objects.visible_to_user(u).filter(pk=obj.pk).exists()`; for non-READ, `user_can` is the only correct check (no queryset analogue). The default body lives in `_default_user_can` in `permissioning.py` and is shared by `BaseVisibilityManager.user_can` and `PermissionedTreeQuerySet.user_can`. `BaseOCModel.user_can(user, perm)` is a one-line ergonomic delegate so callers can write `corpus.user_can(user, PermissionTypes.UPDATE)`. New invariant test pins the read-equivalence, manager/instance-surface parity, superuser bypass, creator default access, the `is_public`-grants-only-READ asymmetry (a security-critical invariant the deleted FolderService methods enforced), and anonymous-only-reads-public for Corpus. Subsequent migration phases extend `user_can` overrides to Annotation/Relationship/Conversation/etc. and migrate the remaining 100+ call sites.

- **Authenticated MCP sessions and `create_thread_message` write tool** (`opencontractserver/mcp/server.py`, `opencontractserver/mcp/tools.py`, `opencontractserver/constants/mcp.py`, `opencontractserver/mcp/tests/test_mcp.py`).
  - MCP ASGI layer extracts `Authorization: Bearer <jwt>`, validates via `config.jwt_utils.get_user_from_jwt_token`, and stores the caller in a `ContextVar` (`_mcp_user`) that all tool handlers consume. Invalid/expired tokens → 401; absent tokens preserve anonymous semantics.
  - First write-capable MCP tool, `create_thread_message`, lets authenticated users post into any thread they can already see (`visible_to_user` on corpus, thread, parent message). Blank/oversized content raises `ValidationError`; the dispatcher surfaces both `PermissionDenied` and `ValidationError` as structured `{"error": ...}` tool results so the LLM can react instead of hitting a transport error.
  - Existing read tools (`list_documents`, `search_corpus`, `list_threads`, etc.) gain an optional `user` parameter and call `visible_to_user(user)` so authenticated callers see private resources they own / are shared on; anonymous callers still see only public, published resources.
  - Both dispatchers strip any client-supplied `user` argument before invoking the handler so a hostile client can't trigger a `TypeError: got multiple values for keyword argument 'user'` that would escape as a transport error.
  - Scoped dispatcher restructured to mirror the non-scoped one: corpus-visibility and unknown-tool checks moved outside the `try` block so each error path records telemetry exactly once (no double-records).
  - Tests cover the valid-JWT and invalid-JWT 401 paths, `ValidationError` → structured result on both dispatchers, the scoped write happy path with creator persistence, IDOR / anonymous rejection, the user-keyword-collision guard, and a new in-thread reply happy path that exercises a valid `parent_message_id`.
  - **Operator note — write access on public threads.** `create_thread_message` permission is `visible_to_user` (read) on the parent thread, not a separate write-specific gate. Threads attached to a public corpus are therefore writable by any authenticated token-holder — including tokens issued by the same Auth0 instance for users outside the deploying tenant. This is a deliberate policy: a public corpus is "open to discussion," not just "open to read." Operators who want read-only public discoverability should keep the corpus public but the thread private, or layer their own write gate at the proxy. The `test_create_thread_message_unrelated_user_can_post_to_public_thread` test in `opencontractserver/mcp/tests/test_mcp.py` pins this contract so a future visibility change cannot silently flip it.

- **MCP OAuth discovery endpoints for interactive sign-in** (`opencontractserver/mcp/server.py`, `opencontractserver/discovery/{views,urls}.py`, `opencontractserver/discovery/tests/test_discovery_views.py`, `opencontractserver/mcp/tests/test_mcp.py`). New `/.well-known/oauth-protected-resource` endpoint (RFC 9728) advertises the Auth0 tenant as the authorization server, and `/.well-known/mcp.json` now surfaces the metadata URL. The 401 response from the MCP ASGI layer emits `WWW-Authenticate: Bearer realm="opencontracts", resource_metadata="..."` so interactive MCP clients (Claude Desktop, Cursor) can discover Auth0 and bounce users through hosted login without preconfigured tokens. Auth0-mode deployments only; `USE_AUTH0=False` returns 404 from the protected-resource endpoint and omits the `resource_metadata` hint, with interactive sign-in unavailable until a local OAuth 2.1 authorization server is added.

- **Markdown-customizable user profile fields** (`opencontractserver/users/models.py`, `opencontractserver/users/migrations/0030_user_profile_about_markdown_user_profile_headline_and_more.py`, `config/graphql/{serializers,user_mutations,user_types}.py`, `frontend/src/components/modals/UserSettingsModal.tsx`, `frontend/src/views/UserProfile.tsx`, `frontend/src/graphql/{mutations,queries}.ts`). New User fields `profile_headline` (200 chars), `profile_about_markdown` (5,000 chars), and `profile_links_markdown` (5,000 chars) editable from the settings modal and rendered via `SafeMarkdown` on the profile view. Size limits are enforced server-side: `UpdateMe.mutate` delegates to `UserUpdateSerializer` (DRF `ModelSerializer`), which auto-applies each model field's `max_length` validator, so an oversized payload posted directly via GraphQL is rejected with a serializer error rather than silently persisted. The `max_length` on the two `TextField`s is **validator-only** (Postgres `TEXT` has no DB-level length cap) so any future code path that bypasses the serializer must validate length explicitly — documented inline at the model field declarations. Visibility piggybacks on the existing `User.objects.visible_to_user()` (private profiles stay hidden whole-record), and owners can still read their own fields on a private profile. `SafeMarkdown` now pins an explicit `urlTransform` allowlist (`https://`, `http://`, `mailto:`, `tel:`, relative / fragment) so `javascript:` / `data:` link targets are stripped even if `react-markdown`'s default ever loosens; protocol-relative URLs like `//phishing.example` (which browsers resolve to the page protocol) are explicitly rejected so a profile author can't disguise an external link as an in-app relative path. The `MarkdownTextarea` focus ring uses a 2px `accentAlpha(0.35)` `box-shadow` in addition to the border-color shift so it stays visible in high-contrast / forced-colors mode (WCAG 2.4.7). Backend test `test_owner_can_read_their_own_private_profile_markdown_fields` plus the `renders markdown headline, about, and links via SafeMarkdown` CT case in `user-profile.ct.tsx` cover markdown rendering plus the four URL-allowlist branches (`https://` pass, `javascript:` strip, protocol-relative strip, `data:` strip).

- **NavMenu "More" overflow keeps footer-essential links reachable on long-scroll views** (`frontend/src/components/layout/{NavOverflowMenu.tsx,overflowMenuItems.ts}` (new), `frontend/src/components/layout/{NavMenu,MobileNavMenu}.tsx`, `frontend/tests/NavMenu.ct.tsx`). Closes #1609. After the natural-flow refactor that restored scrolling on corpus Annotations / Analyses / Extracts tabs, the in-flow `<Footer>` is rendered at the end of the document — effectively unreachable on a corpus with thousands of annotations. Audited `Footer.tsx` and identified the small set of links that genuinely need to be reachable from any scroll position: Privacy Policy (`/privacy`), Terms of Service (`/terms_of_service`), and the GitHub repo link. Site Map (→ `/`) is redundant with the brand logo; Contact Us (→ `/contact`) was intentionally excluded because the route is **not** registered in `App.tsx` and would 404. Added a new `NavOverflowMenu` desktop component (kebab `MoreHorizontal` trigger in the `NavBar`'s `actions` slot, white surface dropdown with ARIA `role="menu"` + arrow-key navigation, ESC / outside-click close, focus restoration to the trigger per WCAG 2.4.3) and extended `MobileNavMenu` with a parallel "More" section in its sheet so the same links are one tap away on mobile. The in-flow `Footer` is unchanged on landing / corpus list / settings views — the overflow is purely additive. Eleven CT tests in `tests/NavMenu.ct.tsx` cover trigger presence, dropdown open, link contents, external-link `target`/`rel`, ARIA attributes, and ESC / outside-click close in both anonymous and authenticated states, on desktop and mobile.

### Added

- **`InlineChatBar` (CorpusHome landing + article-floating chat bars) now opens the agent @mention picker on `@`** (`frontend/src/components/corpuses/CorpusHero/InlineChatBar.tsx`, `frontend/src/components/corpuses/CorpusHome/CorpusLandingView.tsx`, `frontend/src/components/corpuses/CorpusHome.tsx`, `frontend/tests/CorpusHome.ct.tsx`). The bar accepts a new optional `corpusId` prop and, when provided, wires its existing `inputRef` through `useChatMentionPicker` — same hook as ChatTray and CorpusChat — so the rich-mention agent picker is reachable from the corpus landing chat input (`data-testid="corpus-home-landing-chat-input"`) and the article-floating controls (`data-testid="corpus-article-chat-input"`). Both `CorpusHome` and `CorpusLandingView` now plumb `corpus.id` to the bar. Pinned by `tests/CorpusHome.ct.tsx:typing @ in landing chat input opens the agent picker`, which mounts the full landing view, types a bare `@`, and asserts the popover surfaces with the mocked agent visible.

### Fixed

- **@mention picker rendered invisibly behind the DocumentKnowledgeBase fullscreen modal in ChatTray** (`frontend/src/hooks/useChatMentionPicker.tsx`, `frontend/src/assets/configurations/constants.ts`, `frontend/tests/ChatTray.ct.tsx`). The picker is portalled to `document.body`, which makes it a sibling — not a descendant — of the DKB's `.fullscreen-modal-overlay` (`Z_INDEX.APP_MODAL` = 3000). With the historical `zIndex: 1000` on the popover anchor, the modal overlay painted on top and the picker rendered correctly but invisibly behind it. The bug surfaced ONLY in document context (ChatTray) because CorpusChat is not mounted inside the DKB modal — that's why both surfaces shared the same hook yet behaved differently. Bumped the popover anchor to `Z_INDEX.APP_MODAL_CHILD` (3100), the existing semantic slot for popovers nested inside the DKB modal. Pinned by a regression assertion in `tests/ChatTray.ct.tsx:typing bare @ in ChatTray opens picker with full agent list` that reads the computed `z-index` off the anchor and asserts it is `> 3000`, so a future refactor that drops it back down breaks the test instead of the picker.

- **@mention agent picker never appeared on a bare `@` or 1-char fragment** (`frontend/src/components/threads/hooks/useUnifiedMentionSearch.ts`, `frontend/src/hooks/useChatMentionPicker.tsx`, `frontend/tests/ChatTray.ct.tsx`). `useUnifiedMentionSearch` gated all 5 mention-type queries (users / corpuses / documents / annotations / agents) behind `MENTION_SEARCH_MIN_CHARS` (=2), and `useChatMentionPicker` hid the popover until `agentItems.length > 0`. The combination meant typing `@` alone — or `@a` — produced no UI feedback at all, even though the trigger detector had opened. The agent list is intentionally tiny (a handful of templates plus a few per corpus), so the picker should appear the instant `@` is hit, the same way Slack / GitHub / Discord mention pickers do. Fix: (1) split out the agent search to fire on every debounced fragment regardless of length while leaving the other four open-ended resolvers gated by `minChars` (they're expensive `icontains` scans over open sets — keeping them gated preserves the performance reason the gate existed); (2) drop the `agentItems.length > 0` precondition from the popover render so `AgentMentionPopover`'s own "No matching agents." state is reachable from the live UI, not just from CT tests. Pinned by new `tests/ChatTray.ct.tsx:typing bare @ in ChatTray opens picker with full agent list (no minChars gate)`.

- **Migration-seeded agents had `slug=NULL` and crashed the @mention picker** (`opencontractserver/corpuses/template_seeds.py`, `opencontractserver/agents/models.py`, `opencontractserver/agents/migrations/0013_backfill_and_tighten_agent_slug.py`, `opencontractserver/tests/test_corpus_action_template.py`, `frontend/src/components/chat/AgentMentionPopover.tsx:103`). `create_default_action_templates` resolves the model via `apps.get_model("agents", "AgentConfiguration")` so the same function is reusable from both the data migration and the `seed_action_templates` management command. In migration context that registry returns a *historical* model class, which does NOT carry the live `save()` override that auto-generates a slug from the name — so the six template-owned agents created by migrations `0010` and `0011` (`Document Description Updater Agent`, `Corpus Description Updater Agent`, `Document Summary Generator Agent`, `Key Terms Annotator Agent`, `Document Notes Generator Agent`, `CAML Article Writer Agent`) landed in the DB with `slug=NULL`. `searchAgentsForMention` happily surfaced them, the frontend `AgentItem` typed `slug: string` lied about the data shape, and `AgentMentionPopover.matches` exploded with `can't access property "toLowerCase", a.slug is null` the moment the user typed `@`. Fix: (1) `template_seeds._build_unique_agent_slug` re-implements the `save()` slug algorithm and is now called explicitly when creating each `AgentConfiguration` so migration-context writes can't regress; (2) new migration `0013_backfill_and_tighten_agent_slug` first backfills any existing NULL slugs (mirroring the `0005` algorithm so prior deployments are healed in place) and then `AlterField`s the column to `null=False` — the DB is now the final guard, so any future seeder that forgets to set `slug=` fails loudly at write time instead of silently corrupting search results; (3) the model field comment now spells out why `blank=True` is kept (the live `save()` override fills the value before super-save) but `null=False` is non-negotiable; (4) `test_seeded_agent_configs_have_non_null_slugs` pins the seeder contract so a future seeder regression breaks the test instead of the picker.

### Changed

- **Deleted `FolderService.check_corpus_{read,write,delete}_permission` (3 sister wrappers, ~130 LOC) in favor of `corpus.user_can(user, perm)`** (`opencontractserver/corpuses/folder_service.py`, `opencontractserver/tests/test_document_folder_service.py`). Vertical slice for the centralization. The three wrappers each encoded the same `is_superuser or creator or guardian-perm` compound (the read variant additionally honored `is_public`; the write/delete variants explicitly rejected `is_public` to preserve the read/write asymmetry). All 27 internal callers in `folder_service.py` and 20 test sites in `test_document_folder_service.py` migrated to the new `corpus.user_can(user, PermissionTypes.{READ,UPDATE,DELETE})` API. The asymmetry is preserved by `_default_user_can`'s explicit "is_public branch only fires for READ" check and pinned by the new invariant test `test_is_public_grants_only_read_not_writes`.

- **Phase A: `PermissionQuerySet.visible_to_user` is now guardian-aware** (`opencontractserver/shared/QuerySets.py:193-247`). Previously the base `PermissionQuerySet` only filtered on `creator` and `is_public` and explicitly deferred guardian to per-model overrides (`DocumentQuerySet`, `AnnotationQuerySet`, `NoteQuerySet`). That gap created drift: a model whose default manager fell back to the raw `PermissionQuerySet` would have `visible_to_user(u).filter(pk=x).exists()` return `False` while `_default_user_can` (guardian-aware) returned `True` for an explicitly-shared object. Issue #1655 closes the drift by porting the same `apps.get_model(app_label, "{model}userobjectpermission")` lookup used by `BaseVisibilityManager.visible_to_user` into `PermissionQuerySet.visible_to_user`. Concrete subclasses (`DocumentQuerySet`, `AnnotationQuerySet`, `NoteQuerySet`) already override and are unaffected; the change kicks in only for any direct `PermissionManager` consumer.

- **Phase A: `user_has_permission_for_obj` is now a `DeprecationWarning`-emitting shim** (`opencontractserver/utils/permissioning.py:541-604`). The ~300 LOC body (annotation/relationship branches + standard codename intersection) was absorbed into per-model `Manager.user_can` overrides; the function now delegates to `type(instance)._default_manager.user_can(...)`. Every call emits `DeprecationWarning` with `stacklevel=2` so the originating call site shows up in the warning. The `include_group_permissions=False` default is preserved verbatim so existing callers keep getting the same answer when they omit the kwarg (note: `_default_user_can` defaults to `True` — the difference is intentional). Migration of the ~170 existing call sites is tracked in Phases B/C of issue #1655.

- **Phase A: Deleted `user_can_modify_corpus`** (`opencontractserver/utils/permissioning.py:815-879` removed; 4 production callsites migrated in `config/graphql/document_mutations.py:327` and `config/graphql/badge_mutations.py:92,212,329`; `opencontractserver/tests/test_user_can_modify_corpus.py` converted to assert the same matrix against the new API). The helper consolidated the inline `is_superuser or creator or guardian-perm` compound — exactly the cases that `corpus.user_can(user, PermissionTypes.UPDATE)` now handles via `_default_user_can`'s standard branch (superuser short-circuit + creator short-circuit + guardian codename intersection). One call site, one source of truth.

- **Cookie consent modal visual polish on mobile** (`frontend/src/components/cookies/CookieConsent.tsx`). The mobile presentation of the consent modal was cramped and felt off-brand compared with the rest of the OS Legal surfaces. Reworked mobile styling without touching desktop: `.oc-modal` now uses `display: flex; flex-direction: column` with a `flex-shrink: 0` header, a `flex: 1 1 auto; overflow-y: auto` body, and a `flex-shrink: 0` footer so the Accept button stays anchored above the safe-area inset with a soft top shadow instead of relying purely on `position: sticky`. The body's mobile background switches to `OS_LEGAL_COLORS.background` so the white data cards lift off the surface with their new `OS_LEGAL_SHADOWS.card` shadow. Data list items adopt `accentSurface` + `textPrimary` on mobile (replacing the heavier slate-chip look) to feel brand-cohesive. The demo banner gains a 3px warning-text left bar for clearer hierarchy; the header icon badge gains an inner accent ring; section labels, lead copy, and disclaimer text get tighter mobile sizing. The Accept & Continue footer button is 48px tall with `font-weight: 600` for a comfortable thumb target and visual weight.

### Fixed

- **Phase A: Annotation privacy recursion now honors creator status on the source `Analysis` / `Extract`** (`opencontractserver/shared/Managers.py` — new `AnnotationManager.user_can`; `opencontractserver/annotations/query_optimizer.py:99-149` — `_compute_effective_permissions` retargeted to `Document.objects.user_can` / `Corpus.objects.user_can`). Before Phase A, the recursive privacy check for an `Annotation` whose `created_by_analysis_id` (or `created_by_extract_id`) was set went through `user_has_permission_for_obj(user, source_analysis, permission)` (legacy `permissioning.py:612` / `:625`). That helper intentionally ignored creator status — its docstring at line 521 still warns "It does NOT consider: Creator status". So a user who owned both an Analysis AND a private annotation it created could be denied READ on their own annotation because the recursive call returned `False` for the creator with no explicit guardian grant. Phase A rewires both call sites to `Analysis.objects.user_can(...)` and `Extract.objects.user_can(...)`, which route through `_default_user_can` and DO honor creator (it's the canonical short-circuit at `permissioning.py:444-449`). The same recursion in `AnnotationQueryOptimizer._compute_effective_permissions` (which the new `AnnotationManager.user_can` calls into for the MIN(doc, corpus) computation) was updated for the same reason — keeping it on the deprecated helper would have fired `DeprecationWarning` on every annotation visibility check. Other sites that consumed `user_has_permission_for_obj` indirectly (mutations, optimizers, services) are migrated in Phases B/C. Pinned by `test_privacy_recursion_honors_creator_on_source_analysis` in `test_authorization_invariants.py`.

- **Orphan annotations/relationships visible in global search after a doc was soft-deleted from a corpus** (`opencontractserver/shared/QuerySets.py`, `opencontractserver/shared/Managers.py`, `opencontractserver/annotations/models.py`, `opencontractserver/documents/versioning.py`, `opencontractserver/tests/test_soft_delete_visibility.py`). When `RemoveDocumentsFromCorpus` soft-deleted a document, the resulting `DocumentPath(is_current=True, is_deleted=True)` tombstone hid the doc from `corpus.get_documents()` but left its annotations and relationships intact. Global annotation queries used `Annotation.objects.visible_to_user()` / `Relationship.objects.visible_to_user()` which had no DocumentPath awareness, so the rows surfaced in global searches pointing at documents the user could no longer navigate to ("annotations linked to unknown document"). Fix: (1) new shared helper `_exclude_soft_deleted_doc_orphans` in `QuerySets.py` excludes rows whose `(document_id, corpus_id)` pair has DocumentPath rows but none with `is_current=True, is_deleted=False` — same predicate used by `Corpus.get_documents()` and `AnnotationQueryOptimizer.get_corpus_annotations()`. Standalone docs (no paths at all) and structural rows (no document FK) are intentionally untouched so test fixtures and pre-corpus-isolation data keep working. (2) `AnnotationQuerySet.visible_to_user()` applies the helper before its existing privacy/visibility logic; superusers retain the full-bypass branch. (3) New `RelationshipManager` (subclass of `BaseVisibilityManager`) layered onto `Relationship.objects` applies the same filter on top of the inherited creator/public/explicit-permission base case. (4) Data is preserved — only display is hidden — so `RestoreDeletedDocument` (the trash-restore mutation) brings the annotations and relationships back into visibility without resurrection logic. (5) `permanently_delete_document` hardened to also delete corpus-scoped non-structural relationships explicitly tagged `document=doc` (previously only relationships referenced via the deleted annotations' source/target M2M were cleaned; orphan relationships with empty source/target survived). The structural-set GC continues to fire via the existing `_gc_orphan_structural_set` post_delete signal — its `Document.objects.filter(structural_annotation_set_id=set_id).exists()` check is exactly "any other corpus-isolated doc copy with the same content_hash". New `test_soft_delete_visibility.py` covers: visibility before/after soft-delete, non-owner visibility, superuser bypass, restore round-trip, standalone-doc regression guard, relationship cleanup of doc-tagged orphans on permanent delete, and the structural set lifecycle across two corpus-isolated copies (preserved when one copy remains, GC'd when the last copy is permanently deleted).

- **`isProfilePublic` toggle silently dropped from UpdateMe mutation** (`frontend/src/graphql/mutations.ts`). The Profile Visibility switch in `UserSettingsModal` looked like it worked locally (optimistic state flipped), but the field was missing from both `UpdateMeInputs` (TypeScript) and the `UPDATE_ME` mutation variable list, so the toggle never reached the server. The field is now part of the input type and the mutation variables, and a backend regression test (`test_update_me_persists_is_profile_public_toggle`) pins the round-trip.

- **PR #1600 review follow-ups (closes #1608)** (`opencontractserver/users/api/serializers.py`, `config/graphql/permissioning/permission_annotator/mixins.py`, `frontend/src/graphql/mutations.ts`, `opencontractserver/utils/export_v2.py`, `CHANGELOG.md`). Surgical fixes for the five items raised in #1608 against the slug-only privacy gate landed in PR #1600:

  - **`UserSerializer` ↔ `UserViewSet` `lookup_field` contract.** The DRF `UserSerializer` is currently unwired (no `UserViewSet` exists in the codebase), and the existing docstring already noted the constraint, but the warning was easy to miss. The class docstring is upgraded from `.. note::` to `.. warning::` with the exact viewset skeleton future contributors must use, and the `extra_kwargs[url][lookup_field]` line gains a paired inline comment. Greppable for `UserViewSet` so anyone wiring this serializer surfaces the privacy contract on the first search.
  - **`object_shared_with` GenericScalar rationale.** Added a block comment at the field declaration in `AnnotatePermissionsForReadMixin` explaining why the field is intentionally typed as `GenericScalar` rather than a named output type — external callers reading `objectSharedWith[].email` / `.username` will see those keys disappear at runtime rather than triggering a schema error, and the untyped scalar lets us evolve the inner shape without a schema-breaking rename.
  - **`isUsageCapped` nullable break in the hand-written login interface.** PR #1600 made `UserType.isUsageCapped` and `UserType.canImportCorpus` nullable to enforce the self-only gate, and the codegen'd `graphql-api.ts` picked up the change. The hand-written `LoginOutputs.tokenAuth.user.isUsageCapped` interface in `frontend/src/graphql/mutations.ts` still typed the field as a non-nullable `boolean`, so a future consumer of that exact field would receive `null` silently for any non-self code path. Switched to `Maybe<boolean>` (the schema contract) and documented why the value is always defined at this specific call site (login → self-view) so the type-narrowing makes sense to future readers.
  - **Frontend audit of `isUsageCapped` / `canImportCorpus` consumers (no further fixes needed).** Ran the grep from #1608 across `frontend/src`: the three production consumers (`Corpuses.tsx:944` `backendUser?.canImportCorpus`, `Corpuses.tsx:1414` `Boolean(backendUser?.canImportCorpus)`, `Documents.tsx:1315` `backend_user && !backend_user.isUsageCapped`) all use null-safe access (optional chaining, explicit `Boolean()` coercion, truthy short-circuit). No non-null assertions (`!`) exist on either field in the repo. Verified.
  - **`export_v2.py` PII leak — `# TODO(#1608)` markers added.** PR #1600 deliberately left the V2 corpus export format unchanged because rewriting `author_email` / `creator_email` to slug-based keys is a breaking change to the export schema. Added `# TODO(#1608)` comments at the four leak sites (one in `package_md_description_revisions` covering revision authors; one umbrella TODO in `package_conversations` covering the conversation / message / vote loops) and updated the existing CHANGELOG bullet under PR #1600's "Out of scope" line to reference #1608 explicitly so the gap stays linked when the next export-version bump happens.

- **Corpus icons missing on the landing "Featured Collections" list (anonymous + authenticated)** (`frontend/src/components/landing/FeaturedCollections.tsx`). The `GET_DISCOVERY_DATA` query selects `corpus.icon` and the backend `resolve_icon` returns an unauthenticated absolute media URL (no permission gate — icon visibility is implicit from the corpus being returned by `Corpus.objects.visible_to_user`, which already includes public corpora for anonymous users), but the `CollectionCard` render call in `FeaturedCollections.tsx` never forwarded `image`/`imageAlt` props. The result: every featured collection rendered the type-based placeholder glyph instead of its uploaded icon, so anonymous landing-page visitors saw the same generic icon for all collections even though the icon URL was sitting in the GraphQL response. Now passes `image={corpus.icon || undefined}` and `imageAlt={corpus.title || "Corpus icon"}` — matching how `CorpusListView.tsx:620-621` already wires the same `@os-legal/ui` `CollectionCard`.

- **Mobile layout overflow across CAML editor, DocumentKnowledgeBase, floating annotation/document controls, and corpus TOC** (`frontend/src/components/corpuses/CamlArticleEditor.tsx`, `frontend/src/components/corpuses/caml/CamlArticleFrame.tsx`, `frontend/src/components/corpuses/CorpusHome.tsx`, `frontend/src/components/corpuses/DocumentTableOfContents.tsx`, `frontend/src/components/knowledge_base/document/{FloatingDocumentControls,LayoutComponents,NewNoteModal,NoteEditor}.tsx`, `frontend/src/components/knowledge_base/document/document_kb/{DocumentModals,HeaderBar,styles}.{tsx,ts}`, `frontend/src/components/knowledge_base/document/styled/HeaderAndLayout.tsx`, `frontend/src/components/annotator/labels/EnhancedLabelSelector.tsx`, `frontend/src/components/layout/{CardLayout,MobileNavMenu}.tsx`, `frontend/src/styles/appShellLayout.ts`, `frontend/src/utils/{layout,react}.ts`, `frontend/src/index.css`). Several mobile-only layout regressions are addressed in one sweep: (1) the DocumentKnowledgeBase fullscreen modal now propagates `--oc-dkb-visible-viewport-height` / `--oc-dkb-visible-viewport-offset-top` so the floating controls stay above the on-screen keyboard; (2) the CAML article editor fullscreen modal gets a class-targeted `createGlobalStyle` and a new `CamlArticleFrame` viewport guard to contain the third-party CAML library's content; (3) mobile annotation tools and the document FAB are rendered via `createPortal` to escape stacking contexts; (4) all stacking-order constants (mobile annotation tools 3048, floating controls 3050, annotation label modal 3100, app modal 3000/3100) now live in `Z_INDEX` in `frontend/src/assets/configurations/constants.ts` per the CLAUDE.md "no magic numbers" rule, instead of being defined file-local; (5) `FullScreenModal` uses a module-level open-counter so nested fullscreen modals don't release the body scroll lock prematurely; (6) `HeaderBar` is refactored from inline-style soup into named styled components with an icon-only mobile pattern; (7) `DocumentTableOfContents` flex/height contract is reworked so a single-document corpus renders correctly inside the new modal viewport (intentional UX trade-off: the prior `isSingleTopLevelLeaf` shortcut that auto-skipped the document header is removed in favor of a uniform layout — users now expand the document node). New CT regression tests at iPhone-13-sized viewports (390x844) pin the worst regressions. `hasRenderableNode` and `visualViewportAwareBottom` utility helpers live in `frontend/src/utils/{react,layout}.ts` with unit tests.

- **Mobile-nav follow-up: dialog/landmark semantic conflict, missing type imports, and iOS scroll-lock bleed-through** (`frontend/src/components/layout/MobileNavMenu.tsx`, `frontend/src/utils/initials.ts`, `frontend/tests/NavMenu.ct.tsx`). Closes #1610 — surgical follow-up to PR #1606. Code-review feedback on the new floating-sheet mobile nav surfaced four real defects (#1, #6, #7, plus three latent code-clarity issues #2/#3/#4) and confirmed one non-issue (#5):

  1. **`role="dialog"` on a `<nav>` erased the navigation landmark.** `Sheet` was `styled(motion.nav)` _and_ carried `role="dialog"`, which per ARIA overrides the implicit `<nav>` landmark — assistive tech announced it as a dialog but the landmark vanished from landmark navigation. Split the two roles: `Sheet` is now `styled(motion.div)` with `role="dialog"` + `aria-label="Navigation menu"`, and the actual navigation landmark moves to a new inner `SheetNav` (`styled.nav`) with `aria-label="Site navigation"` wrapping only the browse items + Account actions (the AuthFooter stays outside the landmark since it's auth state, not navigation).
  2. **`React.ReactNode` / `React.FC` referenced an unimported namespace.** With the new JSX transform (`"jsx": "react-jsx"`) the file doesn't import `React`, so `React.*` type references relied on the implicit global — fragile and confusing. Switched to `import type { FC, ReactNode } from "react"` and use `FC<…>` / `ReactNode` directly.
  3. **iOS Safari rubber-band scroll bled through the modal.** `document.body.style.overflow = "hidden"` doesn't stop overscroll on iOS — only `document.documentElement.style.overflow` does. The component now locks/restores both `<body>` _and_ `<html>` overflow on open/close, and both the unmount cleanup and `onExitComplete` restore the saved values.
  4. **Latent: toggle outside focus trap, `aria-controls` on portalled target, `initialsFor` whitespace-only word boundary.** Added inline comments documenting why each is intentional / acceptable rather than changing behavior. The toggle's z-index 1100 placement above the sheet is required so it stays tap-targetable while the dialog is open; the trap's "focus leaked outside" branch already pulls focus back. `aria-controls` resolves correctly once the portalled sheet mounts and screen readers tolerate the dangling reference when closed, so the attribute stays static. `initialsFor` is tuned for Latin/CJK whitespace-separated names (Auth0 display names, local usernames) and degrades gracefully for Arabic/Thai/etc. — switch to `Intl.Segmenter` if richer handling is ever needed.
  5. **Tests updated to the new DOM structure.** `NavMenu.ct.tsx` previously located the sheet via `[aria-label="Site navigation"]` — that label now lives on the inner `<nav>`, so 15 sheet-locator sites switched to the stable `#mobile-nav-sheet` id (already wired up via `aria-controls`). Updated the aria-modal/aria-label test to assert the new `"Navigation menu"` value on the dialog, and added a regression `should expose the navigation landmark as a separate <nav> inside the dialog` that pins the split: the dialog is a `<div>`, the inner `<nav aria-label="Site navigation">` is a sibling landmark inside it, and the nav landmark wraps the browse items. All 33 NavMenu CT tests pass.

- **Dark mode contrast in CAML article renderer** (`frontend/package.json`). Upgraded `@os-legal/caml` and `@os-legal/caml-react` from 0.1.0 to 0.1.2 to pull in a rendering fix that restores text/background contrast in dark-mode CAML article views. Also updated the `resolutions` entry so both the top-level app and `caml-react`'s own peer resolve to the same 0.1.2 version (previously the stale `^0.1.0` resolution caused `caml-react` to bundle caml 0.1.0 internally despite the dependency version bump).

- **OAuth IDs leaked publicly on leaderboard and user profiles** (`opencontractserver/users/models.py`, `config/graphql_auth0_auth/utils.py`, `config/graphql/search_queries.py`, new migration `0029_backfill_oauth_slugs.py`, new management command `regenerate_user_slugs`). When social/OAuth users first logged in, `User.save()` generated their URL slug from `self.username` — which for Auth0 users is the raw provider sub like `google-oauth2|114688257717759010643`. `sanitize_slug()` strips the `|` separator but keeps everything else, so slugs like `google-oauth2114688257717759010643` were committed to the DB. These slugs then surfaced verbatim on the community leaderboard (via `entry.user.slug` → `getCreatorDisplay()`) and as navigation targets. Secondary issue: `resolve_search_users_for_mention` searched the `username` and `email` DB columns, allowing any authenticated user to confirm whether a specific OAuth sub or email address maps to an existing account (even though those fields return `null` in the response). Three-part fix: (1) `User.save()` now generates handle **before** slug so the handle is available as the slug base; for social/OAuth users (`"|" in username` or `is_social_user=True`), `handle` is used as `base_value` instead of `username`. (2) `configure_user()` in the Auth0 middleware now sets `is_social_user=True` so newly created Auth0 users are correctly flagged. (3) `resolve_search_users_for_mention` now searches only `slug__icontains` and `handle__icontains`. Migration `0029` backfills all existing users whose slug equals `sanitize_slug(username)` — i.e. was OAuth-derived — replacing it with a handle-based slug. The `regenerate_user_slugs` management command re-runs the same repair on demand.

- **Corpus tabs (Annotations / Analyses / Extracts) lost scrolling** (`frontend/src/components/layout/CardLayout.tsx`, `frontend/src/views/Corpuses.styles.ts`, `frontend/src/views/Corpuses.tsx`, `frontend/src/components/annotations/{AnnotationsPanel,CorpusAnnotationCards}.tsx`, `frontend/src/components/extracts/{ExtractCards,CorpusExtractCards}.tsx`, `frontend/src/components/analyses/{AnalysesCards,CorpusAnalysesCards}.tsx`). The corpus view chain was a viewport-stuck flex tower: every layer (`CardContainer`, `ScrollableSegment`, `CorpusViewContainer`, `MainContentArea`, the per-tab wrapper divs, plus each panel's own `Container` / `AnnotationsListContainer`) declared `height: 100%; max-height: 100vh; overflow: hidden`, with the inner panel using `flex: 1; overflow-y: auto` to scroll. After the SPA-shell refactor (#1561) made the outer wrapper use `min-height: 100vh` instead of `height: 100vh`, none of those `height: 100%` declarations could resolve to a definite value (per CSS spec percentage heights compute to `auto` when the containing block height is not specified explicitly), so the inner `overflow-y: auto` containers silently grew to their content height instead of scrolling. Three wrappers (`CorpusAnnotationCards`, `CorpusAnalysesCards`, `CorpusExtractCards`) were also passing `style={{ minHeight: "70vh", overflowY: "unset" }}` to their panels — leftover from a much earlier outer-scroll era — actively cancelling whatever inner scroll did try to engage. Replace the whole tower with natural-flow: drop the `height: 100%; max-height; overflow: hidden` clamps from `CardContainer`, `ScrollableSegment`, `CorpusViewContainer`, `MainContentArea`, the per-tab wrapper divs, and the panels' own `Container` / `AnnotationsListContainer` / `ExtractCards Container` / `AnalysesCards` div; drop the legacy inline `style` overrides on the three corpus wrappers. Cards stack, the document scrolls naturally, and the App-shell's `min-height: 100vh` outer keeps the sticky-footer behaviour intact. `CorpusViewContainer` keeps a `min-height: 100dvh` so the corpus view fills at least the viewport when content is short.

- **Cookie consent modal scroll + off-screen on mobile** (`frontend/src/components/cookies/CookieConsent.tsx`). `.oc-modal` used `max-height: 100vh` on mobile, which doesn't shrink with browser chrome (URL bar / toolbars), pushing the bottom of the modal — including the Accept button — below the visible viewport. The body also redefined `max-height: 70vh` / `calc(100vh - 180px)` and `overflow-y: auto`, fighting the base `@os-legal/ui` `.oc-modal-body { flex: 1; min-height: 0; overflow-y: auto }` flex setup. Switch the modal's mobile cap to `100dvh`, drop the redundant body `max-height` overrides (let base flex handle it), and zero out the overlay's mobile `padding` so the fullscreen mobile modal isn't inset by 16px on each side.

- **Extracts view (`/extracts`): broken pagination + per-row N+1 + auth-change refetch storm** (`frontend/src/views/Extracts.tsx`, `frontend/src/graphql/queries.ts`, `config/graphql/extract_types.py`, `frontend/src/components/extracts/ExtractListCard.tsx`). Audit follow-up to the Documents view perf work (PRs #1553, #1556, #1554, #1566). Three cooperating issues:

  1. **Broken pagination.** `GET_EXTRACTS` declared no `$cursor` / `$limit` parameters and did not pass `first` / `after` to the `extracts(...)` connection arguments, so the server quietly clamped every request to `max_limit=15` and `fetchMore`'s cursor variable was sent over the wire but never honoured. Infinite scroll never advanced beyond the first 15 rows.
  2. **Per-row N+1 in `resolve_full_document_list`.** The list query selected `fullDocumentList { id }` purely to read `.length` on the frontend, which on the backend iterates `self.documents.all()` and calls `user_has_permission_for_obj` for every document on every extract — 15 extracts × N documents permission checks per page. Fieldset's `fullColumnList { id }` similarly shipped full Column rows when only a count was needed.
  3. **`useEffect(refetch on currentUser)` storm.** Every `userObj` reactive-var settle fired a refetch on top of the implicit `useQuery` mount fetch — the same shape Documents.tsx removed in PR #1553.
     Fix: new slim `GET_EXTRACTS_FOR_LIST` query (`frontend/src/graphql/queries.ts`) wires `first` / `after` properly and selects only fields the list view actually renders. `ExtractType` gains a `documentCount` resolver (and `FieldsetType` a `columnCount` resolver) that read from the existing `prefetch_related("documents", "fieldset__columns")` cache populated by `ExtractQueryOptimizer.get_visible_extracts` — no extra COUNT queries per row, no per-doc permission fan-out. `ExtractListCard.formatStats` prefers the new count fields and falls back to `fullDocumentList?.length` so consumers still on the heavy `GET_EXTRACTS` (`ExtractItem`, `CorpusExtractCards`, `CamlArticleEditor`, `CreateExtractModal`) keep working unchanged. Variables are `useMemo`'d on the search-term primitive, the auth-change effect is removed, `fetchMore` passes both `limit` and `cursor`, and `EXTRACTS_PAGE_SIZE = 20` makes first paint pay for one focused page instead of the previous max_limit. Test wrapper (`frontend/tests/ExtractsTestWrapper.tsx`) projects fixtures to the slim shape so the existing CT suite runs unchanged. Backend regression `test_extracts_list_count_fields_match_lists` (`opencontractserver/tests/test_extract_queries.py`) pins `documentCount === len(fullDocumentList)` and `columnCount === len(fullColumnList)` so the slim/legacy queries can never drift. Frontend regression `frontend/src/views/Extracts.refetch-shape.test.ts` pins the three fixes at the source level (Apollo's request dedup makes the storm invisible to MockedProvider).

- **Annotations view (`/annotations`): refetch storm + non-memoised query variables** (`frontend/src/views/Annotations.tsx`). The view had two `useEffect` blocks that called `refetch_annotations()`/`refetch_corpus()` on filter and corpus changes, doubling every refetch on top of Apollo's implicit one (the variables already drove the refetch). On top of that, `annotation_variables` was a fresh `let`-bound object literal each render, forcing Apollo to deep-compare the variables on every parent re-render before deciding _not_ to refetch. Fix: (a) memoise `annotation_variables` on the underlying primitives (search term, label/labelset/corpus filters, structural-annotation toggle); (b) drop both filter-change and `opened_corpus` refetch effects — `useQuery` already refetches when its variables change, and `AuthGate.resetOnAuthChange` already clears the cache on login/logout, so the explicit auth-token refetch is also redundant; (c) remove the now-unused `refetch_annotations` / `refetch_corpus` destructures and the `auth_token` import. New regression `frontend/src/views/Annotations.refetch-shape.test.ts` pins the structural fix.

- **Strip per-request `.count()` debug logs from `resolve_annotations`** (`config/graphql/annotation_queries.py`). The resolver had seven `logger.info` statements that embedded `{queryset.count()}` in their format strings — every annotations request issued ~5 extra `SELECT COUNT(*)` queries (pre-corpus filter, post-corpus filter, pre-label-type filter, post-label-type filter, pre-analysis-isnull filter, post-analysis-isnull filter, plus the `visible_to_user` fallback log) regardless of log level, since f-string interpolation runs unconditionally. The chatty filter-application log statements (`"Filtering by labelset_id: ..."`) were also removed — they did not embed counts but added noise to every request without helping debug a real production issue. The `structural_set__documents` prefetch is intentionally retained because `AnnotationType.resolve_document` walks it for structural annotations.

- **Anonymous users could see public structural annotations but not their attached images** (`opencontractserver/annotations/views.py`, `opencontractserver/llms/tools/image_tools.py:472`, `opencontractserver/tests/test_annotation_images_api.py`, `docs/permissioning/consolidated_permissioning_guide.md`). `AnnotationImagesView` (`/api/annotations/<id>/images/`) was hardcoded to `IsAuthenticated`, so the frontend hook silently rendered "No thumbnail" for anonymous viewers even when the annotation was returned by the GraphQL feed under public-read rules. The endpoint now uses `AllowAny`; `get_annotation_images_with_permission` delegates directly to `AnnotationQuerySet.visible_to_user` so image visibility is **identical** to annotation visibility (no forked rule set to drift). Throttling is split into authenticated (`AnnotationImagesThrottle`) and anonymous (`AnnotationImagesAnonThrottle`) buckets, each gated on the matching user type so anonymous requests no longer consume two cache slots in lockstep. IDOR protection unchanged — empty 200 for missing/unauthorized.

### Changed

- **Redesign mobile navigation as a floating sheet that overlays content** (`frontend/src/components/layout/MobileNavMenu.tsx` (new), `frontend/src/components/layout/NavMenu.tsx`, `frontend/tests/NavMenu.ct.tsx`). The previous mobile menu used `@os-legal/ui`'s built-in inline drawer (`.oc-navbar__mobile-menu`), which expanded as a full-width opaque dark panel that pushed page content downward and clashed visually with the os-legal "transparent infrastructure" aesthetic.

  - At `< 1100px`, `NavMenu` now renders a custom `MobileNavMenu` (compact dark header + floating, rounded white sheet) instead of the `@os-legal/ui` `NavBar`. Desktop (≥ 1100px) is unchanged.
  - The new sheet is portalled to `document.body`, sits 8px below the header with a 12px side gutter, has a 16px border-radius, soft layered shadow, and a `backdrop-filter: blur(8px)` semi-transparent backdrop dimming the page beneath. Spring-animated entry/exit via `framer-motion` (already a dependency).
  - Active nav items use the design-system teal (`OS_LEGAL_COLORS.accent` on `accentSurface`) with a 3px left bar, matching the hover/selection language used elsewhere. Auth footer renders a teal "Sign in" CTA when signed out and a user chip with avatar initials + "Signed in" label when authenticated.
  - Closes on backdrop tap, ESC key, route change, and any nav-item / user-action selection. Locks `body` scroll while open and exposes accessible `aria-modal="true"`, `aria-label="Site navigation"`, and `aria-expanded` semantics.
  - Updated the two responsive tests in `frontend/tests/NavMenu.ct.tsx` to target the new component via stable `aria-label` selectors (`"Open navigation"`, `"Close navigation"`, `"Site navigation"`) instead of the now-removed `.oc-navbar__mobile-toggle` / `.oc-navbar__mobile-link` library classes. Test intent is unchanged; only selectors moved to the new implementation.
  - Review follow-ups: `handleLogin` wrapped in `useCallback` for consistency with `runAndClose`; `logoNode`, `mobileNavItems`, and `mobileUserActions` memoized in `NavMenu`; section labels (`"Browse"` / `"Account"`) lifted to named constants; the Sheet's programmatic-focus fallback suppresses its `:focus-visible` outline; an unmount-cleanup `useEffect` restores `body.style.overflow` if the component is torn down mid-exit-animation (so `AnimatePresence#onExitComplete` not firing can't leave the page scroll-locked). Eleven additional CT tests cover the Tab focus trap (forward / backward / focus-leaked-outside), the authenticated user-actions section, `runAndClose` for both nav items and user actions, the signed-out `handleLogin` path, and the toggle / sheet ARIA semantics.

- **Tokenize hardcoded navigation colors and tighten test-wrapper / focus-effect conventions in the Corpuses split** (`frontend/src/views/Corpuses.styles.ts`, `frontend/src/views/CorpusQueryView.tsx`, `frontend/tests/CorpusQueryViewTestWrapper.tsx`, `frontend/src/assets/configurations/osLegalStyles.ts`). Follow-up to PR #1578 review feedback:
  - Added `OS_LEGAL_COLORS.navBlue` (`#4a90e2`) and `navIndigo` (`#6366f1`) tokens plus matching `navBlueAlpha` / `navIndigoAlpha` rgba helpers. These are intentionally distinct from `primaryBlue` (`#3b82f6`) — they capture the slightly-different navigation/sidebar blue that recurs across legacy chat, search, and note components, so multiple files can converge on a single named source. Cross-file harmonization with `primaryBlue` is tracked in #1446.
  - Replaced ten hardcoded `rgba(74, 144, 226, ...)` / `rgba(99, 102, 241, ...)` / `#4a90e2` / `#6366f1` literals across `NavigationItem`, `NavigationToggle`, `NavigationItems` (scrollbar focus ring), and `BackNavButton` with the new tokens / helpers — visual fidelity preserved exactly.
  - Replaced `React.useState(() => { showQueryViewState(initialQueryViewState); return null; })` in `CorpusQueryViewTestWrapper.tsx` with a plain `useEffect` keyed on `[initialQueryViewState]`. The `useState` initializer pattern was an unconventional way to fire a side effect during render; `useEffect` clearly signals "side effect on mount" and is lint-clean. All 7 `CorpusQueryViewCoverage.ct.tsx` tests continue to pass.
  - Annotated the `eslint-disable react-hooks/exhaustive-deps` on the mount-only focus effect in `CorpusQueryView.tsx` with a one-line WHY: `isDesktop` is read once to avoid re-firing on viewport resize (which would pop the mobile keyboard mid-session).

### Added

- **Agent tools for step-by-step CAML article citation review**
  (`opencontractserver/llms/tools/core_tools/caml_article.py`,
  registered in `opencontractserver/llms/tools/tool_registry.py`). Three new
  async tools compose into a "find a candidate → verify the match → ask the
  user → replace one location" loop the agent can run against the corpus's
  `Readme.CAML` article:

  - `read_corpus_caml_article` (read-only, `requires_corpus`) loads the
    article via `Document.objects.visible_to_user(user)` and returns it as
    blank-line-delimited blocks with each block's existing `{{@cite ...}}`
    directives plus a `needs_citation_candidate` heuristic that flags prose
    blocks lacking a citation. Headings, lists, code fences, blockquotes,
    table rows, and `[component:…]` markers are excluded so the candidate
    list stays focused on text the user would actually want to cite.
  - `propose_caml_citation_match` (read-only, `requires_corpus`) runs the
    same `CoreAnnotationVectorStore.async_search` path the renderer's
    `useCiteHandler` uses, scoped to the invoking user's `user_id` so
    `Annotation` visibility is honoured. Returns ranked
    `{annotation_id, raw_text, label_text, label_color, document_id,
document_title, corpus_id, page, similarity_score}` candidates capped at
    25 entries.
  - `apply_caml_article_edit` (`requires_approval=True`,
    `requires_write_permission=True`, `requires_corpus`) replaces a single
    occurrence of `target_text` inside `Readme.CAML` with `replacement_text`.
    Fails closed if the substring matches zero or more than one location, and
    layers a defense-in-depth UPDATE check (`is_superuser` /
    `creator_id == user.id` / `user_has_permission_for_obj(... UPDATE)`) on
    top of the wrapper's READ check on `deps.corpus_id`. The edit rewrites
    `Document.txt_extract_file` in place via `FieldFile.save`, so the
    existing Document row is preserved and frontend deep-links to
    `Readme.CAML` continue to resolve. The `rationale` argument is surfaced
    in the approval modal so the user can see why each replacement was
    proposed.
  - Agent flow is one approval prompt per replacement, matching the issue
    request to "step through changes with the user — one citation at a
    time". The directive registry on the frontend is already extensible
    (`frontend/src/components/corpuses/caml/directiveRegistry.ts`), so the
    inserted `{{@cite scope}}` markers are resolved at render time by the
    existing `useCiteHandler` semantic search; no frontend changes needed.
  - Tests in `opencontractserver/tests/test_caml_review_tools.py` cover
    block parsing, candidate heuristics, IDOR-safe error messages, the
    single-occurrence rule, the creator/superuser/UPDATE check matrix, and
    registry resolution including the `requires_approval` /
    `requires_write_permission` / `requires_corpus` flags.
  - Each `apply_caml_article_edit` call rotates `Document.txt_extract_file`
    to a fresh storage blob and queues an `on_commit` cleanup that deletes
    the previous blob, so long-lived corpora don't accumulate orphan files
    on every citation edit. The UPDATE permission check runs **inside** the
    `select_for_update` transaction so a permission revocation between the
    initial load and the locked write cannot slip through. The prose
    heuristic also rejects `***` / `* * *` thematic breaks (previously only
    `-`/`_`/`=` runs were excluded). `CAML_CITATION_MAX_CANDIDATES` and
    `CAML_EDIT_PREVIEW_RADIUS_CHARS` moved to
    `opencontractserver/constants/document_processing.py`.
  - `propose_caml_citation_match` now wraps **both** `CoreAnnotationVectorStore`
    construction and the `async_search` call in the same try/except, so a
    corpus that has no `preferred_embedder` and no `PipelineSettings` default
    surfaces the same friendly "Semantic search failed" `ValueError` the
    agent already knows how to recover from instead of leaking the raw
    `get_embedder() resolved no embedder_path` error from `__init__`. Pinned
    by a new `test_constructor_failure_surfaces_as_value_error` test.
  - `ProposeCamlCitationMatchTests` now patches `__init__` alongside
    `async_search` on `CoreAnnotationVectorStore`. The real constructor calls
    `get_embedder()` against the migration-seeded `PipelineSettings`
    singleton, but `TransactionTestCase` truncates that row between tests
    (`serialized_rollback=False` default), so any other class running first
    on the same pytest-xdist `--dist loadscope` worker would leave us with no
    default embedder and the constructor would raise before the patched
    `async_search` could execute. Coverage of `caml_article.py` is now 100%
    via additional tests for `_safe_delete_storage_path`, `_read_caml_content`
    on a falsy `txt_extract_file`, the `User.DoesNotExist` branches in
    `_read_corpus_caml_article` / `_apply_caml_article_edit` /
    `_assert_corpus_visible_to_user`, the empty-`target_text` guard, and
    component-marker / numbered-list rejection in `_looks_like_prose`.

- **PII scan & auto-annotate agent tool** (`scan_and_annotate_pii` / `ascan_and_annotate_pii`)
  - Scans documents via the new `privacy_filter` Docker service
    ([Open-Source-Legal/privacy-filter](https://github.com/Open-Source-Legal/privacy-filter))
    for 8 PII categories (email, phone, person name, address, account number, URL,
    date, secret).
  - Creates one labeled annotation per detection — TOKEN_LABEL via PlasmaPDF for PDFs,
    SPAN_LABEL for plain text.
  - Agent knobs: `min_score` (default 0.5), `entity_groups` allowlist, `dry_run` preview
    mode, `start_char` / `end_char` to scope the scan to a slice.
  - Auto-creates 8 colored/iconed labels via `Corpus.ensure_label_and_labelset`.
  - Files: `opencontractserver/llms/tools/core_tools/pii.py`,
    `opencontractserver/llms/tools/core_tools/_privacy_filter_client.py`,
    `opencontractserver/llms/tools/tool_registry.py`.
  - New `privacy_filter` service in `local.yml` (with `required: false` so devs without
    the image pulled can still bring up the stack) and `production.yml` (with mandatory
    `PRIVACY_FILTER_API_KEY` env var via `${...:?}` syntax).
  - Settings: `PRIVACY_FILTER_URL`, `PRIVACY_FILTER_API_KEY`,
    `PRIVACY_FILTER_TIMEOUT_SECONDS` in `config/settings/base.py`.
  - 18 unit tests (12 tool + 6 client); the client is mocked in tool tests so CI does
    not need the privacy-filter container.

### Changed

- **Adaptive `load_document_text` tool sizes its slice to the agent's remaining context budget** (`opencontractserver/llms/tools/pydantic_ai_tools.py`, `opencontractserver/llms/agents/pydantic_ai_agents.py`, `opencontractserver/llms/tools/tool_registry.py`). Whole-document tasks like summarisation used to fan out into hundreds of `load_document_text` calls because the tool description hard-coded a 5K-50K char chunk recommendation regardless of how much room the model actually had. The agent now snapshots its per-turn budget onto `PydanticAIDependencies` and the document-agent factory swaps the registry-built tool for a budget-aware variant.
  - `PydanticAIDependencies` gains four new fields populated at agent construction and refreshed each turn: `model_name`, `context_window_tokens`, `estimated_used_tokens`, and `compaction_threshold_ratio`. Two helpers — `remaining_tokens_until_compaction()` and `recommended_chunk_chars(reserve_ratio=0.25)` — derive the budget the next read should target. The default `reserve_ratio` of 0.25 leaves 25% of the remaining tokens for the assistant's own response and the ~10% over-estimate baked into `estimate_token_count`.
  - `PydanticAICoreAgent._refresh_context_budget()` is invoked after every `_get_message_history()` call (`_chat_raw`, `_stream_core`, the structured run path, and `resume_with_approval`), so the snapshot reflects the post-compaction state entering each turn. Tools read the budget through closure-by-reference on the shared `agent_deps` instance — Pydantic v2 BaseModels mutate cleanly and the agent factory now constructs `agent_deps_instance` _before_ the custom tool closures so they see live values at runtime.
  - In the document-agent factory, `load_document_text` is no longer auto-built from the registry; a custom adaptive variant takes its place. When `end` is omitted the tool defaults to `min(total_chars, start + recommended_chunk_chars())` (with a `MIN_IMPLICIT_DOCUMENT_CHUNK_CHARS` floor — 5,000 chars — for starved budgets). The return value is now a structured dict — `{text, total_chars, returned_range, chars_remaining, suggested_next_start, context_budget_chars, budget_was_applied}` — which both reports the budget back to the LLM for planning and bypasses the wrapper's string-only 50K truncation so large slices reach the model intact when the budget allows. `budget_was_applied` is `True` iff the slice size was driven by the budget heuristic (`end` was omitted) so the LLM can tell whether `context_budget_chars` shaped the response or merely advertises the next implicit-call budget.
  - New `get_remaining_context_budget` tool (registered under `ToolCategory.DOCUMENT`, custom-built in the document-agent factory) lets the agent inspect its budget without performing a load — useful for multi-step planning before whole-document reads.
  - Both `PydanticAIDocumentAgent` and `PydanticAICorpusAgent` factories now thread `model_name`, `context_window_tokens`, and `compaction_threshold_ratio` into their `PydanticAIDependencies` instance so future budget-sensitive tools on either agent inherit the snapshot. The corpus agent does not currently expose `load_document_text` (the tool is per-document), but the wiring is consistent so the ground is prepared.
  - Tool registry description for `load_document_text` updated to advertise the adaptive defaulting, and the tool-registry NOTE listing runtime-context-only tools now includes `get_remaining_context_budget`.
  - Multi-call drift fix: `PydanticAIDependencies.turn_implicit_doc_text_chars` accumulates the implicit-chunk bytes already returned within a single turn and is folded into the budget so successive `end`-less reads back off proportionally; `_refresh_context_budget` resets it at the start of every turn. When `_stream_core` is called with an explicit `message_history` (the `resume_with_approval` path), a fresh `_HistoryResult` is now built from that explicit list via `_history_result_from_messages` and run through the same `_refresh_context_budget` codepath — so `estimated_used_tokens`, `context_window_tokens`, and `compaction_threshold_ratio` all refresh, not just the per-turn tally.
  - `_refresh_context_budget` now delegates to a `@staticmethod _apply_context_budget(deps, config, history_result)` helper, and `_history_result_from_messages` is also a `@staticmethod` taking an explicit `config`. The transformations are unit-testable without bypassing `__init__`, and the indirection lets the explicit-history and DB-history paths share one codepath without leaking class-internal state.
  - Cache hygiene: `core_tools.text_extracts` exposes `get_cached_txt_extract_length(document_id)` and `_DOC_TXT_CACHE` is no longer re-exported from `core_tools/__init__.py` — consumers must use the accessor. The cache backend can later evolve (async-safe structure, Redis) without rippling across consumers.
  - The closure body of the adaptive tool was extracted into a module-level `_make_load_document_text_tool(agent_deps, doc_id)` builder so it can be exercised end-to-end in unit tests instead of requiring a full `DocumentAgentContext`.
  - Regression coverage in `opencontractserver/tests/test_context_guardrails.py` pins the budget arithmetic — `TestContextBudgetSnapshot` covers positive remaining tokens, zero-floor when usage exceeds the threshold, default 25% reserve, and explicit `reserve_ratio` overrides; `TestLoadDocumentTextDriftMath` pins the multi-call drift behaviour (second implicit call backs off, drift floors at `MIN_IMPLICIT_DOCUMENT_CHUNK_CHARS`, `_refresh_context_budget` reset returns to the full budget); `TestRefreshContextBudgetFallback` pins the fallback semantics for zero vs non-zero `_HistoryResult.context_window`; `TestLoadDocumentTextClosureIntegration` drives the actual closure with a stubbed cache and pins the dict-shape contract (including `budget_was_applied`) plus the in-turn tally mutation through the closure reference.
  - Defensive-zero fallback: `_refresh_context_budget` no longer uses `or` to choose between `_HistoryResult.context_window` and `get_context_window_for_model(...)` — `or` would conflate "missing" with a legitimate zero, so the snapshot now uses an explicit `> 0` check instead.
  - Observability: `load_document_text` emits `logger.warning` when `recommended_chunk_chars()` exceeds `LARGE_IMPLICIT_CHUNK_WARN_RATIO` (50%) of the model's full context window in characters. The threshold is _relative_ so 1M-token models (Gemini 1.5, Claude Opus large-context) don't spam the warning on legitimately large recommendations — only real `CHARS_PER_TOKEN_ESTIMATE` drift trips it. This is a cheap production signal when the heuristic drifts (e.g. for multilingual or code-heavy documents whose chars-per-token ratio differs from the `CHARS_PER_TOKEN_ESTIMATE` default).
  - Breaking-shape note for downstream callers: `load_document_text` now returns `dict[str, Any]` rather than `str`. Python code that called the tool programmatically and unwrapped a string return will break at runtime. The change is intentional — the wrapper's 50K-char string truncation is bypassed so large slices reach the model intact, and the dict carries the budget-planning fields the LLM uses to decide whether to chunk further.
  - Review round 5 follow-up: (1) empty-document edge case — `_make_load_document_text_tool` previously detected "never cached" via `cached_len == 0`, which collided with "loaded and empty" and forced a redundant cache-prime fetch on every call for a 0-byte document. `core_tools.text_extracts` now exports a `is_txt_extract_cached(document_id)` membership predicate and the closure uses that instead; the prime now fires exactly once for an empty document, not once per call. (2) `PydanticAIDependencies.recommended_chunk_chars` now emits a `logger.warning` when `reserve_ratio` falls outside `[0.0, 1.0]` — the value is still clamped (preserving back-compat) but the warning makes "unexplained 0-char budget" diagnosable instead of silent. (3) Tool-registry description for `load_document_text` no longer mentions the document-agent's adaptive override (which is implementation-noise the LLM doesn't need to see); the explanation moved to a `# NOTE` code comment. (4) The `get_remaining_context_budget` registry description was tightened to enumerate all six returned fields (it previously omitted `compaction_threshold_ratio`, drift-prone if the in-agent helper grows). (5) Regression coverage in `test_context_guardrails.py`: `TestIsTxtExtractCached` pins the membership-vs-length distinction; `TestLoadDocumentTextClosureIntegration.test_empty_document_does_not_repopulate_cache` asserts the empty-document prime fires exactly once across two implicit calls; `TestContextBudgetSnapshot.test_recommended_chunk_chars_reserve_above_one_clamps_and_warns` / `..._below_zero_...` pin the warning + clamp behaviour.
  - Review round 6 follow-up (issue #1611): (1) `get_document_text_length_tool` (in `pydantic_ai_agents.py`) had the same "empty document re-loads on every call" pitfall round 5 fixed in `_make_load_document_text_tool` but missed at this sibling site — the guard was `cached_len > 0`, which is false for a legitimately-empty document so a second full-document load fired on every call. Switched to the `is_txt_extract_cached` membership predicate so the prime is consulted once and the second load only fires on the genuine cache-miss fallback. (2) `_history_result_from_messages._part_text` used `getattr(part, "content", None) or getattr(part, "args", None) or ""` — the `or` chain conflates an empty string (e.g. a `ToolReturnPart` with empty content) with "missing" and falls through to `args`, double-counting tokens in the budget estimate. Replaced with an explicit `is None` check matching the defensive-zero pattern in `_refresh_context_budget`. (3) `TestLoadDocumentTextClosureIntegration._patches()` now also patches `is_txt_extract_cached` to return `True`; without this the real predicate returned `False` for the fake-loaded `DOC_ID` and the prime path was being silently exercised on every call in three of the four tests, so the warm-cache code path the tests claimed to exercise wasn't actually covered. The fourth test (`test_empty_document_does_not_repopulate_cache`) still leaves the predicate unpatched on purpose to drive the membership-vs-length distinction end-to-end. (4) Documented at the `PydanticAIDependencies.turn_implicit_doc_text_chars` field that the in-turn drift correction is _partial_ — only implicit `load_document_text` calls feed the tally; other heavy tool outputs (vector-search results, annotation lists) consume context but are not tracked. Future maintainers adding a budget-aware tool now have a pointer to the design constraint at the source of truth.

### Fixed

- **`get_embedder` redundant tuple cast** (`opencontractserver/utils/embeddings.py:132`). The follow-up patch in commit 85496ad re-introduced a `cast(tuple[Optional[type[BaseEmbedder]], Optional[str]], ...)` on the return statement that mypy now flags as redundant -- the function's declared return type already matches what mypy infers, so the cast trips `warn_redundant_casts = True`. This was the only mypy error on the branch and was blocking pre-commit (and therefore Backend CI's lint step) for any new commit. Reverted to the bare `return embedder_class, embedder_path` from PR #1545, which both satisfies mypy and matches the function's declared return type.
- **Reddit-style auto-assigned user handles** (issue #1574). Auth0 social-login users without a populated `name`/`given_name`/`first_name` previously rendered as the redacted `user_<id>` fallback in the GraphQL `displayName` resolver chain. New `User.handle` field (`opencontractserver/users/models.py`) is auto-assigned on `save()` from a curated `adjective × noun` namespace (`opencontractserver/users/handle_wordlists.py`, ~56k base combos) via `opencontractserver/users/handle_generator.py`. Handles surface as a higher-priority branch in `UserType.displayName` (`config/graphql/user_types.py`) — chain order: `name` → `given_name + family_name` → `first_name + last_name` → `handle` → `username` (only when not a `provider|sub` Auth0 identifier) → redacted `user_<id>` fallback. Schema + data migrations (`opencontractserver/users/migrations/0027_user_handle.py`, `0028_backfill_user_handles.py`) backfill existing users in one pass; the same generator powers the `regenerate_user_handles` management command for re-running with `--reroll-suffixed` (after the word list grows) or `--reroll-all`. User-facing handle editing is intentionally out of scope; the resolver still degrades to the redacted fallback for any user the migration hasn't reached. Coverage in `opencontractserver/tests/test_user_handle.py` pins generator uniqueness, no-PII-leakage, fixed-seed determinism, suffix promotion under saturation, the resolver chain priority, and the management command's three modes.
  - The django-guardian Anonymous system user is excluded from auto-assignment, the migration backfill, and the `regenerate_user_handles` queryset — it never surfaces to other users and so never needs a handle. Without this exclusion the post-migrate signal that creates Anonymous (after `0028_backfill_user_handles` runs) could leave it with `handle=NULL`, which made `regenerate_user_handles` see two candidates instead of one in a freshly-bootstrapped DB.
  - `User.save()` no longer string-parses the `IntegrityError` message to detect handle-uniqueness collisions; instead it queries for an existing row holding the chosen handle and only re-rolls when that hit confirms the conflict. The previous `"handle" in str(exc)` heuristic was Postgres-specific and would silently swallow non-handle integrity errors if the constraint were ever renamed.
  - Curated word list cleanup: `flora` and `ladybug` moved from `ADJECTIVES` to `NOUNS` so `flying ladybug` no longer surfaces as a noun-noun pairing in generated handles.
  - `regenerate_user_handles --reroll-suffixed`: keep the user's own row in `scope_qs` (rather than excluding by pk) so its current DB handle blocks `generate_handle` from re-selecting the same value. Setting `user.handle = None` in Python alone does not reach the DB, so excluding the pk could let `cleverFox42` round-trip through the generator and back to itself, defeating the flag's purpose. Same change applies to `--reroll-all`. Regression coverage in `test_user_handle.py::RegenerateUserHandlesCommandTests::test_reroll_suffixed_targets_only_numeric_tails` (now reliably re-rolls under the fix).
  - Anonymous-user exclusion is now pinned by tests in `test_user_handle.py::UserModelHandleAutoAssignTests` (one test for `User.save()`, one for `regenerate_user_handles --reroll-all`) so the four exclusion sites (model save, schema migration, data migration, management command) cannot drift apart silently.
  - Architecture doc at `docs/architecture/user_handles.md` (wired into `mkdocs.yml`) documents the resolution chain, generator semantics, migrations, management command modes, and tunables.
  - Post-merge cleanup pass on the handles PR: (1) `regenerate_user_handles` now raises `CommandError` when `--reroll-all` and `--reroll-suffixed` are passed together (previously the narrower flag was silently dropped); (2) the `\d+$` "is suffixed" regex is now sourced from a single `_SUFFIXED_RE` constant used by both the queryset filter and the compiled `_SUFFIXED_PATTERN` (eliminates a drift trap); (3) removed the unused `handle_field` parameter from `generate_handle` per YAGNI — every caller used the default; (4) added explanatory comment in `User.save()` that `self.pk` stays `None` across failed INSERT retries (so `scope_qs.exclude(pk=self.pk) if self.pk else scope_qs` is correct without a committed-row guard); (5) trimmed `"ox"` from `NOUNS` so the wordlist actually honours its documented 3-12 character bound; (6) tightened the `test_wordlists_provide_a_meaningful_namespace` floor from 10k to 40k and added `test_wordlists_respect_length_bounds` plus `test_reroll_all_and_reroll_suffixed_are_mutually_exclusive`.

### Changed

- **`DocumentKnowledgeBase` decomposed into focused modules** (`frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`, `frontend/src/components/knowledge_base/document/document_kb/{SidebarTabs,DocumentViewer,HeaderBar,helpers,useResizeHandle,useDocumentMarkdown,useDocumentLoader,useStructuralAnnotations,useContainerWidth}.{ts,tsx}`). Extracts sidebar tabs (desktop + mobile), the central document viewer (PDF / TXT / DOCX dispatch), the resize handle hook, the markdown summary fetcher, the structural-annotations loader, the container-width tracker, and the entire data-loading orchestration into dedicated modules. Behavior preserved (no UI/UX change), but the main file shrinks ~60% and each extracted module is independently testable. Tightens prop types — `DocumentViewer.createAnnotationHandler` is now `(annotation: ServerTokenAnnotation) => Promise<void>` instead of `any`. PDF.js types (`PDFDocumentProxy`, `PDFPageProxy`, `PageTokens`) replace three `any`s in `useDocumentLoader.buildPdfPages`. `processAnnotationsData` collapses two `setPdfAnnotations` calls into one to avoid an intermediate render in Apollo's `onCompleted` (outside React's automatic batching). `useContainerWidth` now creates/disposes its `ResizeObserver` inside the ref callback, so the observer correctly reattaches on conditional re-mounts. The internal `setViewState` setter is no longer exported from `useDocumentLoader` — callers should not poke at the hook's internal state. Discussions tab in `SidebarTabs` switches to `MessagesSquare` to disambiguate from the Chat tab's `MessageSquare`. Header date no longer flashes today's date when `metadata.created` is null — renders an em-dash placeholder instead. Follow-up review remediation: `SelectionLayer.finalizeSelection` no longer mutates `finalizingRef.current` from inside a `setMultiSelections` updater (the new line was paired with pre-existing side effects in the same updater — both updater functions are now pure, with the side effects sequenced after); the text-document branch of `DocumentViewer` now uses `id="text-container"` instead of inheriting the misleading `pdf-container` id from the pre-refactor monolith; `MENU_INTERACTION_COOLDOWN` is hoisted into `frontend/src/assets/configurations/constants.ts` as `SELECTION_MENU_COOLDOWN_MS`; `PANEL_WIDTH_FULL_PCT` gains a comment clarifying that the "full" preset deliberately stops at 90 %; `DesktopSidebarTabsProps` drops the unused `showRightPanel` from its prop bag via `Omit<CommonProps, "showRightPanel">`; and `useDocumentLoader.ts` gains a six-test smoke suite covering the auth-gated skip path, the document-only TXT happy path, the null-document ERROR path, the `threadCount` plumbing, the corpus-context skip, and the annotations-only refetch skip.

### Fixed

- **PDF text selection: action menu silently dropped when mouseup landed on overlapping fixed-position UI** (`frontend/src/components/annotator/renderers/pdf/SelectionLayer.tsx`). The selection finalisation handler was bound to React's `onMouseUp` on the SelectionLayer element itself, so a drag that released over an overlapping `position: fixed` sibling — most notably the right-edge `SidebarTab` buttons whose z-index sits above the selection layer — fired mouseup on the sibling instead. The selection state was left mid-flight: `localPageSelection` set, no `setShowActionMenu(true)`, and no `pendingSelections` flush. Surfaced by `tests/DocumentRenderingCornerCases.ct.tsx` "selection menu off-screen on mobile" once the `useContainerWidth` extraction below started reporting the post-modal-animation container width (causing the test's calculated drag end coordinates to land exactly on a sidebar tab); was a latent UX bug for mobile users selecting near the viewport's right edge in production. Fix: extract a `finalizeSelection(clientX, clientY, shiftKey)` helper and attach a document-level `mouseup` (and `mousemove` for off-element drift) listener while `localPageSelection` is non-null. The listeners detach as soon as the selection finalises or the component unmounts. Both the React-bound `onMouseUp` and the document fallback call the same helper, so single-page release behaviour is preserved exactly.

### Breaking Changes

- **`UserType.canImportCorpus` is now nullable** (`Boolean` instead of `Boolean!`). Cross-user / anonymous viewers receive `null`; only self-views still get a concrete `true` / `false`. Hand-written client queries that destructured the field without a default will start receiving `null`. TypeScript codegen picks up the change automatically; check unstructured consumers. Driven by the slug-only privacy policy below.
- **`objectSharedWith` payload shape change**: collaborator entries on shared corpuses / documents / labelsets no longer expose `email` or `username` — only `id`, `slug`, and the permissions list. Because `objectSharedWith` is typed as `GenericScalar` (no GraphQL schema enforcement of inner shape), external clients reading `objectSharedWith[].email` / `.username` will see those keys silently disappear at runtime instead of getting a schema error. Migrate consumers to `objectSharedWith[].slug`. Driven by the slug-only privacy policy below.

### Security

- **Slug-only user privacy policy across the GraphQL surface** (`config/graphql/user_types.py`, `config/graphql/permissioning/permission_annotator/mixins.py`, `config/graphql/og_metadata_queries.py`, `config/graphql/og_metadata_types.py`, `config/graphql/worker_queries.py`, `opencontractserver/users/models.py`, `opencontractserver/users/api/serializers.py`, `opencontractserver/llms/tools/core_tools/links.py`). Hardens user-PII exposure so non-self viewers see only the public `slug`; everyone else (including superusers via the public API) gets `null` for `email`, `username`, `name`, `firstName`, `lastName`, `givenName`, `familyName`, `phone`, `emailVerified`, and `isSocialUser`. PII access is reserved for Django admin (server-side, gated), not the GraphQL layer. Specifically:

  - `UserType` declares each PII field as a custom `graphene.String`/`Boolean` so the resolver gate runs even though `DjangoObjectType` would otherwise auto-resolve from the model. A shared `_is_self_view(user_obj, info)` helper enforces the contract uniformly; `Meta.exclude` adds a defensive layer that hides `password`, `last_ip`, `last_login`, `last_synced`, `synced`, `auth0_Id`, and `first_signed_in` from every viewer (including the user themselves — those have no client use case and were leaking operational metadata).
  - `displayName` resolver: non-self viewers always receive the user's `slug` (or a stable `user_<pk-suffix>` redaction when slug is unset). Self-views still walk the rich profile-name fallback so settings UIs can still greet users by name; the OAuth `provider|sub` redaction for social-login users still applies. The `email` resolver lost its superuser exception — admin tooling continues to work because the Django admin UI is the official PII path.
  - `User.__str__` no longer embeds the email (was `f"{self.username}: {self.email}"`, now `self.slug or self.username`) — closes a small log/admin leakage surface.
  - `AnnotatePermissionsForReadMixin.resolve_object_shared_with` (`config/graphql/permissioning/permission_annotator/mixins.py`) previously emitted `email` and `username` for every collaborator on a shared corpus / document / labelset to every viewer of that object. The payload now carries only `id`, `slug`, and the permissions list — every PII consumer in the existing client trees migrates to `slug`. **Breaking change for external consumers**: this field is typed as `GenericScalar` (no schema enforcement of payload shape), so external GraphQL clients that read `objectSharedWith[].email` / `.username` will see those keys disappear at runtime rather than getting a schema error. External integrations should migrate to `objectSharedWith[].slug`.
  - `og_metadata_queries.py` (corpus / document / document-in-corpus / thread / extract OG cards) and `worker_queries.py` (worker-account creator) switched `creator.username` → `creator.slug` so social-media link previews and worker-account dropdowns never surface the raw OAuth `sub`. OG `creator_name` field descriptions updated from "Username of …" to "Public slug of …".
  - `opencontractserver/llms/tools/core_tools/links.py` (Markdown link generator that the agent toolchain uses to surface annotation/corpus/document URLs to the LLM) was reading `creator.username` to populate the path's `userSlug` segment — wrong for OAuth users since the URL routing layer expects the case-sensitive slug. Now reads `creator.slug` everywhere; the variable was already named `user_slug`.
  - `opencontractserver/users/api/serializers.py` (`UserSerializer`) — currently unwired-but-defined DRF serializer reduced from `["username", "name", "url"]` to `["slug", "url"]`, with `lookup_field` switched to `slug` so any future view that wires it up follows the same contract by default.
  - Frontend leaderboards (`frontend/src/components/community/Leaderboard.tsx:582`, `frontend/src/components/landing/CompactLeaderboard.tsx:364,370`) now render `entry.user.slug` / `contributor.slug` directly instead of `displayName`. The backend already redacts `displayName` to slug for non-self viewers, but reading the slug field explicitly removes any chance of a stale Apollo cache leaking the self-view rendering when a logged-in user looks at their own row, and makes the privacy contract visible in the JSX.
  - Coverage: 12 new tests in `opencontractserver/tests/test_user_privacy.py` (self-view PII visibility, cross-user redaction, superuser redaction, anonymous redaction, redacted-handle fallback, `__str__` privacy, `object_shared_with` slug-only payload). Existing `test_user_display_name.py::EmailResolverTestCase` updated to assert that superusers no longer see other users' emails through the GraphQL API; new `CrossUserDisplayNameTestCase` pins the slug-fallback contract for `displayName`. `test_og_metadata_queries.py` `creatorName` assertions updated to read `self.user.slug` (underscores in usernames normalise to hyphens via the slug sanitizer). Follow-up review remediation: 4 additional `test_user_privacy.py` cases pin the self-only gate on `canImportCorpus` (cross-user → null, anonymous → null, self-view → boolean) and the `is_active=False` gate on `_is_self_view` (a deactivated user with a still-live session must not pass the gate, since `AbstractBaseUser.is_authenticated` is statically `True`); `_is_self_view` now also checks `is_active`. The deferred `from django.conf import settings` inside `resolve_can_import_corpus` was hoisted to module scope. Frontend `getCreatorDisplay` (`frontend/src/utils/userDisplay.ts`) now decodes the Relay global ID and uses the last 6 chars of the numeric pk so its no-slug fallback matches the backend `_redacted_handle` output (`user_<pk-suffix>`) instead of producing a different string for the same user — covered by 7 new `userDisplay.test.ts` cases.
  - **Out of scope for this PR / tracked in #1608**: V2 corpus export (`opencontractserver/utils/export_v2.py`) still writes `author_email` / `creator_email` into the export ZIP — this is a deliberate data-export feature owned by the user doing the export, but does cross the same privacy boundary for collaborator emails on a shared corpus. Switching to slug-based author resolution would be a breaking change to the export format and is left for a follow-up (`# TODO(#1608)` markers added at the four leak sites in `export_v2.py`).
  - **Schema-type break**: `UserType.canImportCorpus` is now `Boolean` (nullable) instead of `Boolean!`. Cross-user / anonymous viewers receive `null`; only self-views still get a concrete `true` / `false`. Hand-written client queries that trusted the non-null contract (e.g. destructured the field without a default) will start receiving `null`. TypeScript codegen picks up the change automatically; check unstructured consumers.

- **Close corpus-less `ModerationAction` auth gap in `resolve_moderation_action`** (`config/graphql/conversation_queries.py`, closes #1594). `resolve_moderation_action` previously short-circuited to `return action` whenever `conversation.chat_with_corpus` was `None`, leaking document-only and orphaned moderation actions to any authenticated user via direct ID lookup. The resolver now routes every case through `Conversation.can_moderate` (the same canonical authorization helper used by the moderation mutations), which accepts: superuser; conversation creator; corpus owner/designated moderator (when `chat_with_corpus` is set); document owner (when `chat_with_document` is set). Actions whose `conversation` FK is `NULL` (a rare historical edge — the column is nullable but every real action has one) fail closed for non-superusers. New regression coverage in `opencontractserver/tests/test_moderation.py` pins all four allowed paths plus the previously-leaking unauthorized-user case for both document-only and corpus-less conversations.
- **Bump `postcss` transitive dependency to `^8.5.10` to resolve GHSA-disclosed XSS** (`frontend/package.json`, `frontend/yarn.lock`, dependabot alert #87). `styled-components@6.1.19` pinned `postcss@8.4.49` exactly, and `tailwindcss`/`tinyglobby` resolved a separate `^8.5.6` line to `8.5.8`; both fall under the moderate-severity advisory ("PostCSS has XSS via Unescaped `</style>` in its CSS Stringify Output", patched in `8.5.10`). Added a `"postcss": "^8.5.10"` entry to the `resolutions` block so both spec ranges collapse to a single patched `8.5.14` lockfile entry. PostCSS runs in build-time tooling (Vite, tailwindcss), so the practical exposure was bounded, but the alert is closed regardless.
- **Stop echoing `str(exception)` from document-import REST views** (`opencontractserver/document_imports/services.py`, `opencontractserver/document_imports/views.py`, CodeQL `py/stack-trace-exposure` alerts #74 and #75). The `DocumentImportView.post` and `DocumentsZipImportView.post` 403 paths returned `{"error": str(e)}` for any `PermissionError` raised by the service layer, which CodeQL flags as exception-data flow into an HTTP response. The two service-layer raises were intentionally human-readable (usage-cap and bulk-upload-denied messages), but the data-flow link itself is the issue. Fix: introduce `DocumentImportPermissionError(PermissionError)` carrying a stable `code` (`USAGE_CAP`, `BULK_UPLOAD_DENIED`); both raises switch to the new class. The REST views catch the subclass and build the response body from a view-local `_public_permission_message(code)` literal-string lookup — exception state never reaches the response. The exception still inherits from `PermissionError` and `str(e)` returns the same human-readable message, so the GraphQL upload mutation (`config/graphql/document_mutations.py:207`), which lets `PermissionError` propagate as a Graphene error, and the assertion in `opencontractserver/tests/test_usage_caps.py` that pins the user-visible cap message keep working unchanged.

### Changed

- **Removed legacy `@testing-library/react-hooks` from frontend test infrastructure** (closes #1576). The package still calls the React 17-era `ReactDOM.render` API, so every hook test that imported from it printed a "ReactDOM.render is no longer supported in React 18" warning. The repo already had a thin React-18-native replacement at `frontend/src/test-utils/renderHook.tsx`, but 16 test files still imported from the legacy package. Migration:
  - Extended `frontend/src/test-utils/renderHook.tsx` with `RenderHookOptions<TProps>` (the legacy `{ wrapper, initialProps }` shape), plus `waitForNextUpdate` and `waitFor` on the hook return so it is a drop-in for every call site we use. The new helpers wrap each polling iteration in `act` so React 18 flushes pending state updates (Apollo cache writes, `fetch().then(setState)` resolutions, reactive-var subscribers) between polls — a single act boundary across the whole loop would defer the flush until the loop exits, leaving `renderCount` and `result.current` perpetually stale.
  - The standalone `waitFor` export matches `@testing-library/react-hooks` v8 semantics: a callback returning `undefined` (e.g. `() => expect(...)`) succeeds when it stops throwing; a callback returning a boolean succeeds on truthy. This lets `AnalysisHooks.sync.test.tsx` keep its `expect(...)`-inside-waitFor pattern after swapping `import { waitFor } from "@testing-library/react"` for `import { waitFor } from "../../../../test-utils/renderHook"`. The RTL version polls without `act`, so under `IS_REACT_ACT_ENVIRONMENT = true` (set in `setupTests.ts`) async hook updates never become visible to the probe and the predicate never sees the post-resolve state.
  - Updated the 16 test files (`useAnnotationImages`, `AnalysisHooks.sync`, `useClearTextBlockOnInteraction`, `useTextSearch`, `AnnotationHooks`, `useVisibleAnnotations`, `useUploadMutations`, `useMessageBadges`, `useAgentChat`, `useCacheManager`, `useFeatureAvailability`, `useBadgeCelebration`, `useAuthReady`, `useNetworkStatus`, `useCorpusMdDescription`, `useRefetchOnAuthChange`) to import from `../../test-utils/renderHook` (or the equivalent relative path).
  - Updated `useWebSocketAuth.test.ts:167` to pass `{ initialProps: { enabled: false } }` instead of the bare `{ enabled: false }` second arg now that the helper accepts an options object instead of a raw `initialProps` value.
  - Removed `@testing-library/react-hooks` from `frontend/package.json` `devDependencies` and pruned its entry from `yarn.lock`. The transitive `react-error-boundary` lockfile entry will be GC'd on the next `yarn install`.

### Fixed

- **Infinite navigation loop on `/e/:user/:extractId` extract route** (`frontend/src/App.tsx`, `frontend/src/components/routes/ExtractLandingRoute.tsx` removed). Visiting any extract by its canonical `/e/<creator>/<extractId>` URL tripped the navigation circuit breaker with `"/e/<...>" appeared 3 times`. Two pieces of routing code were redirecting in opposite directions and fighting each other:

  1. `ExtractLandingRoute.tsx:32-36` (mounted on `/e/:userIdent/:extractIdent`) redirected to `/extracts/<extract.id>` as soon as `openedExtract` was populated.
  2. `CentralRouteManager` Phase 3 then read `buildCanonicalPath(...)` (`frontend/src/utils/navigationUtils.ts:213`), which returns `/e/<creator.slug>/<extract.id>` as the canonical path for an extract, and redirected `/extracts/<id>` back to `/e/<...>`. Loop.
     Fix: route `/e/:userIdent/:extractIdent` directly to `ExtractDetailRoute` (whose JSDoc and the routing-guide table at `docs/frontend/routing_system.md:178-179` already declared this as the intended setup), and delete the dead `ExtractLandingRoute.tsx` + its test. `/extracts/<id>` still works — Phase 3 redirects it once to the canonical `/e/<creator>/<id>` and stops, since the next render is already on the canonical path. Updated `docs/frontend/routing_system.md` to reflect the removed component and the actual extract-routing flow.

- **Document deletion follow-up hardening** (issue #1572). Six surgical fixes to the orphan-blob reclaim path landed in PR #1487 / issue #1492:

  - `Document.blob_field_names()` (`opencontractserver/documents/models.py:365-389`) now returns an immutable `tuple[str, ...]` cached on the class. The previous implementation returned the same shared `list` on every call, so a misbehaving caller doing `names.append(...)` would corrupt the cache for every subsequent caller in the process. Latent footgun, no current caller mutated the result.
  - `_schedule_blob_cleanup_post_delete` (`opencontractserver/documents/signals.py:298-343`) now registers AT MOST ONE `transaction.on_commit` flush callback per atomic context, instead of one per `post_delete` signal. The flush callback was promoted to module scope and tagged with a sentinel attribute (`_FLUSH_CALLBACK_MARKER`) so we can detect prior registrations in `connection.run_on_commit` without comparing closure identity. `QuerySet.delete()` of N rows now costs O(1) on_commit registrations instead of O(N). Rollback safety preserved: Django drops our callback from the queue on savepoint/transaction rollback, so the next `post_delete` in a fresh transaction re-registers cleanly. The orphan re-check inside `cleanup_orphaned_document_blobs_task` continues to backstop the cross-transaction accumulator bleed for rolled-back paths.
  - `cleanup_orphaned_document_blobs_task` (`opencontractserver/tasks/cleanup_tasks.py:25-131`) now calls `default_storage.delete()` unconditionally instead of probing with `default_storage.exists()` first, closing a TOCTOU window where a concurrent process could remove the blob between the check and the delete and turn a benign race into an alarming ERROR log. `FileNotFoundError` is now caught at DEBUG level for backends that raise on missing paths; `FileSystemStorage` and `S3Boto3Storage` are idempotent on missing paths so the deleted-count semantics shift slightly (a successful delete of an already-absent path counts as 1).
  - `cleanup_orphaned_document_blobs_task` gained `autoretry_for=(Exception,), max_retries=3, default_retry_delay=60`. The task is idempotent (the orphan re-check absorbs whatever happened on a previous attempt), so transient failures (worker crash mid-execution, transient DB / storage error during the orphan check) now retry cleanly instead of silently losing the remaining paths.
  - The `__in` lookup in the orphan check (`opencontractserver/tasks/cleanup_tasks.py:90-97`) now passes `sorted(candidate_paths)` (a `list`) instead of a `set`, eliminating non-deterministic query plans on backends sensitive to set input ordering.
  - The undocumented "store an arbitrary attribute on `django.db.connections[alias]`" pattern used by the per-DB-connection accumulator in `opencontractserver/documents/signals.py:175-187` is now flagged in a comment with a documented escape hatch (switch to `WeakValueDictionary` keyed by `id(connection)` or `threading.local`) should the assumption ever break.
  - Regression coverage: new `test_bulk_delete_registers_single_on_commit_callback` in `opencontractserver/tests/test_blob_retention.py` deletes 20 rows inside a single atomic block and asserts that exactly one blob-cleanup callback (filtered by the sentinel marker) is appended to `connection.run_on_commit`. The pre-fix behavior would have been 20.

- **Conflicting migrations on `annotations` app block all `migrate` runs on `main`** (`opencontractserver/annotations/migrations/0071_grounding_annotation_unique_constraints.py`, `opencontractserver/tests/test_extraction_grounding.py`). PR #1418 (`Back grounding get_or_create with DB-level uniqueness`) merged after PR #1503 (`Add default LabelSet seeding`), but its migration branched off the older parent `0068_enforce_embedder_path_not_null` and was named `0069_grounding_annotation_unique_constraints`. That created two leaf nodes — `0069_grounding_annotation_unique_constraints` and `0070_seed_default_labelset` (already applied in production on top of `0069_labelset_is_default_and_seed_default`) — and Django bailed with `CommandError: Conflicting migrations detected; multiple leaf nodes …`, breaking Backend CI and Frontend E2E (the latter shells out to `migrate` during test setup). Resolution: renumber the unmerged grounding migration in place — `0069_grounding_annotation_unique_constraints` → `0071_grounding_annotation_unique_constraints` — and re-parent it on `0070_seed_default_labelset`. This is safe because PR #1418 had only just merged (35 minutes before this fix) and CI had been red the whole time, so the migration had not been recorded in any environment's `django_migrations` table. The renumber yields a single linear leaf with no merge migration carrying forward. Test `TestMigration0069M2MDedup`, which loads the migration file by name to exercise its `_repoint_cross_references` helper directly, was renamed and updated to reference the new filename.

### Changed

- **Shared view layout primitives** (`frontend/src/components/layout/PageLayout.tsx`, `frontend/src/components/layout/DiscoveryViewLayout.ts`). Eight top-level views in `frontend/src/views/` had been hand-rolling near-identical page chrome (`PageContainer`, `ContentContainer`, `HeroTitle`, `HeroSubtitle`, `StatsContainer`, `SectionHeader`, `SectionTitle`, `EmptyStateWrapper`); two more (`GlobalDiscussions`, `DiscoverSearchResults`) duplicated the discussion-style page primitives. Pure refactor with no functional or layout changes — net **-321 lines** (583 deleted from views, 262 added across two new primitive modules). `SectionHeader` exposes `$gap` and `$wrap` transient props so DiscoveryLanding's section rows preserve their pre-refactor single-line layout (no gap, no wrap), while the four list views keep their canonical `gap: 16px; flex-wrap: wrap` defaults.

- **Consolidated inline corpus moderation checks onto `Corpus.user_can_moderate`** (issue #1450, cluster 3). `config/graphql/conversation_queries.py:467-505` (`resolve_moderation_action`) and `:514-550` (`resolve_moderation_metrics`) each repeated the same hand-rolled `if not user.is_superuser: is_owner = corpus.creator == user; is_moderator = corpus.moderators.filter(user=user).exists(); if not is_owner and not is_moderator: return None` block — the exact "drift" pattern the issue calls out. Extracted into a single `Corpus.user_can_moderate(user)` instance method on `opencontractserver/corpuses/models.py:670` that owns the canonical superuser/creator/designated-moderator decision and short-circuits anonymous / `None` users. Behavior is preserved verbatim — the helper intentionally counts ANY `CorpusModerator` row (regardless of the `permissions` list), matching the previous queries; the divergence with the stricter `Conversation.can_moderate()` (which additionally requires `bool(moderator.permissions)`) is documented in the helper's docstring and tracked separately so this consolidation PR doesn't double as a behavior-change PR. Regression coverage in `opencontractserver/tests/test_corpus_user_can_moderate.py`: superuser, creator, designated-moderator-with-permissions, designated-moderator-with-empty-permissions, outsider, anonymous, `None`, and a cross-corpus leak guard.

- **Cookie consent dialog redesign** (`frontend/src/components/cookies/CookieConsent.tsx`). Two-column data grid for the "Data we collect" and "Data you agree to share" sections, semantic HTML (`<section>`, `<aside>`, `<ul>`), responsive collapse to a single column below 720px, and an icon badge in the header. Adds tokens to `frontend/src/assets/configurations/osLegalStyles.ts` (`OS_LEGAL_SPACING.modalMaxWidth`, `modalSideGutter`, `borderRadiusListItem`, `borderAccentWidth`, `iconBadgeDesktop`, `iconBadgeMobile`; `OS_LEGAL_SHADOWS.modalOverlay`) and `COOKIE_CONSENT_GRID_BREAKPOINT` in `frontend/src/assets/configurations/constants.ts` so the modal no longer relies on inline magic numbers. Disclaimer text is held at 0.75rem (12px) to meet the WCAG-recommended legal-text minimum, and `DataListItem` text colour was raised to `textSecondary` for AA contrast on `surfaceLight`.

### Fixed

- **Leaderboard leaked raw OAuth `provider|sub` IDs into the USER column + polled aggressively** (issue #1557). Two related defects on the Leaderboard / Top Contributors surfaces.

  - **A. Raw OAuth IDs in USER column.** `frontend/src/components/community/Leaderboard.tsx:570` rendered `entry.user.username` directly, and `frontend/src/components/landing/CompactLeaderboard.tsx:398` rendered `contributor.username`. For social-login users, `User.username` is the Auth0 `sub` claim (e.g. `google-oauth2|114688257717759010643`) — see `config/graphql_auth0_auth/utils.py:291` (`jwt_get_username_from_payload_handler`) and `:227` (`get_or_create(username=remote_username)`). Fix: added a `displayName` resolver to `UserType` (`config/graphql/user_types.py`) that resolves in priority `name` → `given_name` + `family_name` → `first_name` + `last_name` → `username` → redacted `user_<last 6 chars of sub>` fallback. The redaction branch fires only when `is_social_user=True` (not just when `|` appears in the username), because the project's `UserUnicodeUsernameValidator` (`opencontractserver/users/validators.py`) explicitly allows `|` in locally-chosen usernames — so a local user named e.g. `alice|admin` is legitimate and must not be redacted. The redacted suffix is taken from after the last `|`, so short subs like `auth0|abcde` redact to `user_abcde` with no `|` leak; suffix length lives in `OAUTH_SUB_DISPLAY_SUFFIX_LENGTH` in `opencontractserver/constants/auth.py`. The raw `sub` is never returned. `GET_LEADERBOARD` (`frontend/src/graphql/queries/leaderboard/queries.ts`) and `GET_DISCOVERY_DATA` (`frontend/src/graphql/landing-queries.ts`) now select `displayName` instead of `username`; both leaderboard components and their TS types (`frontend/src/types/leaderboard.ts`, `LeaderboardEntry` in `landing-queries.ts`) read `displayName`. Backend coverage in `opencontractserver/tests/test_user_display_name.py` pins down the priority order, the social-user gate, and the redaction guarantees so a regression cannot quietly re-expose the raw `sub` or falsely redact a local username.
  - **B. Over-aggressive polling.** `Leaderboard.tsx` had `pollInterval: 60000` on `GET_LEADERBOARD` and `pollInterval: 120000` on `GET_COMMUNITY_STATS` — the queries refetched on a fixed clock regardless of tab visibility, re-rendering rows and resetting in-flight UI state (rank chip animations, hover state). Compounded by the fact that `metric` / `scope` / `limit` are query variables, so every filter change _also_ issued a fresh request on top of the cadence. Fix: dropped both `pollInterval`s; queries now use `fetchPolicy: "cache-and-network"` + `nextFetchPolicy: "cache-first"` + `notifyOnNetworkStatusChange: false` so the user sees instant cached data while a single background refresh runs on mount or filter change. A `visibilitychange` listener performs one refetch when the tab returns to the foreground — hidden tabs no longer hit the network at all.
  - **C. PII-gated `email` resolver.** Even with the leaderboard query no longer selecting `email`, `UserType` is a `DjangoObjectType` so the field would auto-resolve to any client that selected it on a public `user` subtree. Added a `resolve_email` method on `UserType` (`config/graphql/user_types.py`) that returns `None` unless the requester is authenticated AND (viewing themselves OR a superuser). The `me` query and admin tooling continue to see the real address. Coverage: `EmailResolverTestCase` in `opencontractserver/tests/test_user_display_name.py` pins the self-only / superuser / cross-user / anonymous / blank-email / context-missing matrix. Frontend follow-through: `email` is dropped from the `LeaderboardUser` GraphQL selection set, the TypeScript `LeaderboardUser` interface, and all leaderboard test mocks (`Leaderboard.ct.tsx`, `LeaderboardTestWrapper.tsx`).
  - **D. Avatar initials regression on leading-whitespace single-token names.** `getLeaderboardInitials` (`frontend/src/utils/leaderboardAvatar.ts`) used `name.substring(0, 2)` on the un-trimmed input in the single-token branch, so `"  alice"` rendered as `"  "` instead of `"AL"`. Fixed by using the trimmed token; whitespace-only names now return `"?"` instead of falling through to silent whitespace. Test coverage in `frontend/src/utils/__tests__/leaderboardAvatar.test.ts`.
  - **E. `useTabVisibilityRefresh` no longer silently swallows errors in production.** Promise rejections and synchronous throws from a refresh callback (e.g. Apollo query torn down between visibility transitions, network outage) were only logged in dev. Switched to unconditional `console.error` so consistently failing refreshes surface in production browser consoles / Sentry. Test updates in `frontend/src/hooks/__tests__/useTabVisibilityRefresh.test.ts`.

- **SPA layout: redundant 100vh on inner wrapper exposed gradient strip before footer** (`frontend/src/App.tsx`, issue #1558). The shell stacked three full-viewport-height constraints — `minHeight: 100vh` on the outer wrapper, a second wrapper with `minHeight: 100vh` + `overflow: hidden`, and a third with `minHeight: 100vh` + `maxHeight: 100vh` + `height: 100vh` + `overflow: hidden` — clamping the routed content area to exactly the viewport. On short pages (e.g. discovery landing with the empty-state CompactLeaderboard) the `CallToAction` panel that follows in the natural document flow was clipped to a thin band rather than shown in full, producing a "blue-to-purple gradient strip" between a flat light-gray gap and the dark footer. The footer wrapper compensated with a `marginTop: -1.5rem` hack to hide the resulting visual seam. Fix: collapse the layout to one `minHeight: 100vh` flex column on the outer wrapper, a single inner flex shell with `flex: 1, minHeight: 0`, and `flex: 1` on the `AppContainer` main slot so it grows to push the footer down naturally. Drop `overflow: hidden` from inner wrappers and the `-1.5rem` margin compensation. Pages that scroll internally (DiscoveryLanding, Leaderboard, Documents, Corpuses, etc.) keep working because they own their own `height: 100%; overflow-y: auto` containers; pages without internal scroll containers (PrivacyPolicy, TermsOfService) now scroll the body instead of being clipped. Also drops `justifyContent: center` from the outer wrapper, which would have negatively offset overflow content if any page exceeded 100vh under the new flex flow. Also drops the carried-over `minWidth: 100vw` from `APP_CONTAINER_STYLE` — `100vw` includes the scrollbar width but the parent flex container does not, so under the new flow (no `overflow: hidden` to mask it) it would have forced a horizontal scrollbar on systems with non-overlay scrollbars whenever vertical scroll was active; `width: 100%` already fills the parent. Regression coverage: layout invariants are pinned in `frontend/src/styles/__tests__/appShellLayout.test.ts` (style constants) and `frontend/src/components/layout/__tests__/AppShell.test.tsx` (rendered DOM); the modal stack that hangs off the shell — `SelectAnalyzerOrFieldsetModal`, `DocumentUploadModal`, the EDIT/VIEW `CRUDModal` pair — was also extracted into `frontend/src/components/layout/AppDocumentModals.tsx` so the conditional/ternary branches that decide which modal renders can be unit-tested without standing up the routing tree (`frontend/src/components/layout/__tests__/AppDocumentModals.test.tsx`).

- **`user_has_permission_for_obj` per-row N+1 on Document GraphQL resolvers** (`opencontractserver/utils/permissioning.py`, `opencontractserver/shared/Managers.py`, `config/graphql/permissioning/permission_annotator/mixins.py`, issue #1555). `DocumentType.canViewHistory` (`config/graphql/document_types.py:879-897`) and `DocumentType.canRetry` (`config/graphql/document_types.py:927-957`) — both selected by the shared `GET_DOCUMENTS` query — were calling `user_has_permission_for_obj` per row, and that helper's underlying `get_users_permissions_for_obj` issued `getattr(instance, "<model>userobjectpermission_set").filter(user_id=user.id)` and `…groupobjectpermission_set.filter(group_id__in=…)` on each call. `.filter(...)` on a related manager bypasses any default `prefetch_related` cache, so the prefetches set up by `_apply_document_prefetches` (`_prefetched_user_perms`, `documentgroupobjectpermission_set__permission/__group`) were silently ignored — every paginated `documents` page was charged 2-3 extra DB round trips per row. The same bypass affected `resolve_my_permissions` for group permissions. Fix:

  - `_apply_document_prefetches` now writes user-id-suffixed attributes — `_prefetched_user_perms_uid_<user.id>` and `_prefetched_user_group_perms_uid_<user.id>` — both pre-joined to `permission` via `select_related`. The group-perm prefetch is now filtered to the user's own group ids at the SQL layer (replacing the unfiltered `documentgroupobjectpermission_set__permission/__group` lookups), so the fetched row count is bounded by the user's group membership rather than the total number of groups with perms on the document.
  - `get_users_permissions_for_obj` now consumes those prefetched lists directly (reading `perm.permission.codename` from the in-memory rows) when the user-id-suffixed attribute is present. The user-id suffix is what makes the cache safe to consume: a lookup for a different user finds no attribute and cleanly falls through to the legacy guardian path.
  - `resolve_my_permissions` updated to read from the same attributes — eliminates the second N+1 it was paying via its own `…groupobjectpermission_set.filter(group_id__in=…)` call.
  - Regression coverage in `opencontractserver/tests/permissioning/test_permissioned_querysets.py::UserHasPermissionForObjPrefetchFastPathTest`: pins zero queries across 5 documents for both direct and group-permission lookups, verifies the fallback path still works for instances loaded without the prefetch, and verifies a different-user lookup correctly falls through (the suffix makes the cache safe under multi-user lookups).

- **Loading-overlay UX regression carried over to seven other infinite-scroll lists** (follow-up to PR #1565 / issue #1559). PR #1565 fixed the `LoadingOverlay`-during-`fetchMore` antipattern in `frontend/src/views/Documents.tsx` only, but seven sibling infinite-scroll views/components copied the same shape and exhibited the same bug: a full-cover `LoadingOverlay` (most of them `inverted`, painting a near-black band) was shown for the entire `loading === true` window, including `fetchMore` requests, dimming or covering already-rendered rows mid-scroll. Fixed in: `frontend/src/views/Extracts.tsx`, `frontend/src/views/LabelSets.tsx`, `frontend/src/components/exports/ExportList.tsx`, `frontend/src/components/extracts/ExtractCards.tsx`, `frontend/src/components/corpuses/CorpusListView.tsx`, `frontend/src/components/analyses/AnalysesCards.tsx`, `frontend/src/components/documents/DocumentCards.tsx`. Each call site now scopes `LoadingOverlay.active` to the initial load only (`loading && filteredItems.length === 0`, or for `DocumentCards` the existing `showEmptyState` gate which already encodes "no items and no folders") and drops the `inverted` flag so the dark variant is reserved for genuinely blocking states. To DRY the spinner that replaces the overlay during `fetchMore`, the `FetchMoreFooter` styled-component originally inlined in `Documents.tsx` was extracted to `frontend/src/components/widgets/infinite_scroll/FetchMoreFooter.tsx` (a small, accessible `role="status" aria-live="polite"` strip with a Lucide `Loader2` icon and an entity-specific message like "Loading more extracts…"), and Documents.tsx was migrated to consume the shared component while preserving its existing `data-testid="documents-fetch-more-spinner"`. Each new call site gets a parallel `data-testid` (`extracts-fetch-more-spinner`, `labelsets-fetch-more-spinner`, `exports-fetch-more-spinner`, `extract-cards-fetch-more-spinner`, `corpuses-fetch-more-spinner`, `analyses-fetch-more-spinner`, `document-cards-fetch-more-spinner`) so future regression tests have a stable hook. The shared `FetchMoreOnVisible` stale-closure / `rootMargin` fixes from PR #1565 already covered every consumer because the bug was in the shared component itself; this follow-up only covers the _overlay_ half of the regression for the consumers PR #1565 didn't touch.

- **Documents view (`/documents`) infinite scroll never fires; dim overlay obscures grid during load** (issue #1559). Two cooperating bugs prevented the second-and-later pages from loading and made the page look broken on the first paint:

  1. `frontend/src/views/Documents.tsx` rendered an `inverted` full-cover `LoadingOverlay` over the grid container for the entire `documents_loading === true` window — including refetches and `fetchMore` — which painted a near-black band over already-rendered rows. Replaced with: (a) an unobtrusive non-inverted overlay scoped to the _initial_ load only (`documents_loading && filteredDocuments.length === 0`), and (b) a small footer-pinned spinner ("Loading more documents…", `data-testid="documents-fetch-more-spinner"`, `role="status"` for a11y) that renders below the grid only while a `fetchMore` request is in flight and there is still a next page. The dark/inverted variant is now reserved for genuinely blocking states.
  2. `frontend/src/components/widgets/infinite_scroll/FetchMoreOnVisible.tsx` had two compounding issues: its `useEffect` deps were `[entry, vertical, inView]` and did **not** include the `fetchNextPage` / `fetchPreviousPage` callbacks, so once Apollo's `loading` flag toggled true→false during the first `fetchMore`, the parent re-created its `useCallback` handler and the observer's effect kept calling the stale closure (which captured `loading=true` or a now-replaced Apollo handle) — silently dropping every subsequent fetch. Additionally, the `useInView` config had no `rootMargin`, so the sentinel only fired when the user reached the absolute bottom. Fix: hold the latest callbacks in `useRef` slots that are refreshed every render and dereferenced inside the effect, so observer events always invoke the current closure regardless of how many times the parent's handler identity has churned mid-flight; and default `rootMargin` to `"200px 0px"` so the sentinel triggers ~one viewport's worth before the absolute bottom. The new `rootMargin` prop is opt-in for callers that need a different value; existing call sites (`AnnotationList`, `ExportList`, `ExtractCards`, `SearchResults`, `UnifiedContentFeed`, `ConversationListView` ×2, `CorpusListView`, `AnnotationsPanel`, `ThreadList`, `AnalysesCards`, `DocumentMetadataGrid`, `DocumentCards`, `Extracts`) inherit the safer default with no API change.
  3. Regression coverage: new Playwright CT in `frontend/tests/Documents.ct.tsx` ("Documents View - Infinite Scroll (issue #1559)") mounts the view with a 20-doc first page (`hasNextPage: true`, `endCursor: "cursor-page-1-end"`) and a single distinctively-named second-page mock keyed on `{ limit: 20, cursor: "cursor-page-1-end" }`, scrolls the `.FetchMoreOnVisible` sentinel into view, and asserts the second-page document title appears. The test uses an inline `InMemoryCache` with the same `relayStylePagination` keyArgs as `frontend/src/graphql/cache.ts` so `fetchMore` merges into the existing connection. On the pre-fix code the second mock is never consumed (stale `handleFetchMore` closure no-ops on the captured `documents_loading=true`), so the assertion times out.

- **`DocumentType.versionCount` per-row N+1** (`config/graphql/document_types.py:632`, `opencontractserver/documents/query_optimizer.py`, closes #1554). `resolve_version_count` was issuing one `COUNT(*)` query per document row (`Document.objects.filter(version_tree_id=self.version_tree_id).count()`), so any GraphQL query that selected `versionCount` on a paginated documents connection paid up to 100 extra COUNT queries per page. The shared `GET_DOCUMENTS` query selects this field, so every active consumer (`CorpusDocumentCards`, `SelectDocumentsModal`, `DocumentRelationshipModal`, corpus tabs, metadata workflow) was hit. Fix mirrors the request-level cache pattern already established for `doc_relationship_count` in the same file (`document_types.py:395-432`):

  - New `DocumentVersionQueryOptimizer.get_version_counts_by_tree` (`opencontractserver/documents/query_optimizer.py`) computes `{version_tree_id: count}` for every visible tree in **one** aggregated SQL query and caches the result on `info.context`. Subsequent per-row resolvers in the same GraphQL request collapse to dict lookups.
  - The aggregation is scoped to `Document.objects.visible_to_user(user)` so the badge cannot leak the existence of versions hidden from the requesting user — the previous unscoped `.count()` would have surfaced private versions in the visible count.
  - Resolver returns `1` (not `0`) when the tree is absent from the cache: the resolver is only reachable on a document the user can already see, so its tree always has at least one visible version.
  - Out of scope (tracked separately): `canViewHistory` / `canRetry` per-row permission checks.
  - Regression coverage: `opencontractserver/tests/test_document_version_count_optimizer.py` covers the normal multi-tree case, partial-visibility (count must collapse to only the visible subset), outsider (no counts surface), superuser, request-level cache memoisation, and an end-to-end integration test that resolves `versionCount` on 15 documents in a shared request and asserts no `version_tree_id` SQL runs after the first batched aggregation.

- **Browse Annotations (`/annotations`): tall dark loading band + image cards rendering "Failed" + misleading Doc Labels stat** (issue #1560).

  - **Loading band (issue A).** `frontend/src/components/annotations/AnnotationsPanel.tsx` previously rendered a `LoadingOverlay` with `inverted` (rgba(0,0,0,0.85)) anchored by `position: absolute; inset: 0` over a flex container that had no positive `min-height` while `items` was empty — producing a tall, near-black band against the otherwise light page. The overlay is replaced with shimmer-animated card-shaped skeletons (`AnnotationCardSkeleton`) that match the eventual `ModernAnnotationCard` two-up grid layout (label row, label color, body, footer). Skeletons participate in normal flow, so the grid has predictable height while data loads. The now-unused `loadingMessage` prop is dropped from `AnnotationsPanelProps` and from both call sites (`Annotations.tsx`, `CorpusAnnotationCards.tsx`).
  - **"Failed" image cards (issue B).** `frontend/src/components/annotator/hooks/useAnnotationImages.tsx` flagged any non-200 REST response as `error`, which the preview rendered as a dashed-border "Failed" placeholder on every image annotation when the JWT wasn't yet in the `authToken` reactive var (the `IsAuthenticated`-gated `/api/annotations/{id}/images/` endpoint then returns 401). The hook now also tracks `hasFetchedEmpty` and treats 401/403/404/429 the same as a successful empty payload — the backend's IDOR-safe contract already returns `{images: []}` for missing/unauthorized/throttled, so we surface the same gap as a neutral placeholder rather than an alarming failure. `AnnotationImagePreview` accepts the new prop and renders a `NoThumbnailState` (image icon + "No thumbnail" + dashed border + "No thumbnail generated for this annotation" tooltip) for the empty-success case while keeping the genuine `error` branch for 5xx/network/parse failures. Both consumers — `ModernAnnotationCard` (Browse Annotations cards) and `HighlightItem` (sidebar) — wire `hasFetchedEmpty` through. Cache-poisoning fix: 401/403/429 are explicitly **not** memoized in the module-level `imageCache`, so a render that lands before the JWT resolves doesn't lock the annotation into "no thumbnail" forever — only 404 (permanently no thumbnail row) and 200-empty are cached. `AnnotationImagePreview` short-circuits to `null` while `images === null` so the placeholder doesn't flash before the loading spinner appears. New regression tests in `frontend/src/components/annotator/hooks/__tests__/useAnnotationImages.test.tsx` pin the matrix: no fetch when not IMAGE, empty 200 → `hasFetchedEmpty`, 401/404 → `hasFetchedEmpty`, 500 → `error`, populated 200 → images, plus the cache-policy guards (401 → re-fetches on next mount, 404 → second mount skips network). The `HighlightItem.scroll` test mock and the `AnnotationsPanel` Loading State test were updated to match the new contract.
  - **Misleading Doc Labels stat (issue C).** `frontend/src/views/Annotations.tsx:455` computed `docLabels`/`textLabels`/`humanAnnotated` from the first 20 loaded annotations while `Total` correctly used the server-side `totalCount`. With ~30k annotations the breakdown tiles read "0" and looked authoritative. The `stats` memo now also exposes `loadedCount` and an `isPartial` flag; when partial, the breakdown labels gain an "(in view)" suffix and the tile gets a tooltip ("Counted from X of Y loaded so far. Scroll to load more."). The Total tile is unchanged — it was already accurate.

- **Documents view (`/documents`) refetch storm + over-fetched query** (`frontend/src/views/Documents.tsx`, `frontend/src/graphql/queries.ts`, `frontend/src/services/cacheManager.ts`, `frontend/src/components/widgets/modals/UploadModal/hooks/useUploadMutations.ts`). The top-level Documents view was issuing ~7 `GET_DOCUMENTS` requests on every visit — one from `useQuery` plus six redundant `useEffect(() => refetchDocuments())` hooks watching `current_user`, `location`, `document_search_term`, `filtered_to_label_id`, `filtered_to_labelset_id`, `filtered_to_corpus`, all of which fired on mount. `nextFetchPolicy: "network-only"` forced every refetch to skip the cache. The query itself selected ~22 fields per row when the UI only renders 7 — including the per-row N+1 `versionCount` (`Document.objects.filter(version_tree_id=…).count()` per row), per-row `canViewHistory` / `canRetry` `user_has_permission_for_obj` checks, four file-URL signing fields the list never displays, and the `doc_label_annotations` connection. With `RELAY_CONNECTION_MAX_LIMIT = 100` and no explicit `first:` argument, the first paint was charged for 100 fully-resolved DocumentType rows. Fix:
  - New focused `GET_DOCUMENTS_FOR_LIST` query selects only what the list view renders (`id slug title fileType backendLock pageCount icon created creator { id slug email }`). The shared `GET_DOCUMENTS` is preserved for its other consumers (`CorpusDocumentCards`, `SelectDocumentsModal`, `DocumentRelationshipModal`, corpus tabs, metadata workflow tests).
  - Six redundant `useEffect(refetchDocuments)` hooks deleted. Apollo's `useQuery` already refetches on `variables` change (deep-compared), so the reactive-var values that _should_ drive a refetch are now memoized into `documentVariables` and the ones that shouldn't (`location` re-renders, `current_user` settling, `filtered_to_labelset_id`) no longer trigger one. `filtered_to_labelset_id` is intentionally not a query variable — it only scopes the label-picker dropdown.
  - `nextFetchPolicy: "network-only"` removed; default cache-first behavior restored.
  - Initial query now passes explicit `first: DOCUMENTS_PAGE_SIZE` (20) so first paint matches the paged-load size.
  - `document_items` memoized on the stable Apollo `edges` reference (was being rebuilt via `.map().filter()` on every render, churning `filteredDocuments` / `stats` / `statusFilterItems` deps — same shape of bug fixed in PR #1517 for `CorpusDocumentCards`).
  - Latent bug: query used to ask for `creator { id slug }` while the UI called `getInitials(doc.creator?.email)`; `email` was never selected and avatar initials always defaulted. Fixed in the new query's selection set.
  - `useUploadMutations` and `cacheManager.invalidateRelatedQueries` updated to include `GET_DOCUMENTS_FOR_LIST` alongside `GET_DOCUMENTS` in their refetch lists, so an upload from the Documents page still refreshes the list. Apollo's `refetchQueries` ignores entries with no active observable, so listing both is a safe union.
  - Deleted unused `frontend/tests/DocumentsTestWrapper.tsx` (no consumers in the codebase).
  - Regression coverage: new Vitest source-level test (`frontend/src/views/Documents.refetch-shape.test.ts`) pins the four structural fixes (no `network-only`, no `useEffect(refetchDocuments)`, slim query in use, explicit page-size constant). Behavioral counter-style tests cannot catch the storm because Apollo's built-in query deduplication merges concurrent in-flight requests with the same key into a single network call before they reach `MockLink`.

### Changed

- **Typing: graduated `opencontractserver.llms.*` (non-tools) modules out of the mypy baseline** (issue #1484, follow-up to #1468). Removed the `[mypy-…]` `ignore_errors` blocks for `llms.agents.agent_factory`, `llms.agents.core_agents`, `llms.agents.pydantic_ai_agents`, `llms.agents.timeline_stream_mixin`, `llms.api`, `llms.client`, `llms.vector_stores.core_conversation_vector_stores`, `llms.vector_stores.core_vector_stores`, and `llms.vector_stores.pydantic_ai_vector_stores`. Fixes (~125 errors) included: replacing `callable`/`any`/`dict[str, any]` with `typing.Callable[..., Any]` / `typing.Any`; adding explicit `Optional[...]` to satisfy `no_implicit_optional`; tightening `_resolve_framework`/`_resolve_tools` helpers in `llms.api`; typing `Annotation`/`ChatMessage` querysets as their custom `AnnotationQuerySet`/`ChatMessageQuerySet` so the `search_by_embedding` extension method resolves; switching `CorpusAgentContext.documents` to a non-Optional `list[Document]` with `default_factory=list`; aligning `CoreAgent.stream` / `CoreAgent.resume_with_approval` Protocol signatures with their PydanticAI implementations; and pruning unused `# type: ignore` comments. Pre-existing errors outside `llms.*` are untouched.

- **`opencontractserver.tasks.*` graduated out of the mypy baseline** (issue #1482, refs #1447). All sixteen task modules (`agent_tasks`, `analyzer_tasks`, `badge_tasks`, `corpus_tasks`, `data_extract_tasks`, `doc_analysis_tasks`, `doc_tasks`, `embeddings_task`, `export_tasks`, `export_tasks_v2`, `extract_orchestrator_tasks`, `fork_tasks`, `import_tasks`, `import_tasks_v2`, `lookup_tasks`, `memory_tasks`) now type-check cleanly. Their `[mypy-…]` blocks are removed from `mypy.ini` and the corresponding 227 lines are pruned from `docs/typing/mypy_baseline.txt`. Additional tightening: `LabelLookupPythonType.{text_labels,doc_labels}` narrowed from `dict[str | int, …]` to `dict[str, …]` (the producer in `utils.etl.build_label_lookups` already emits string keys), and the `corpus_id` parameter on `build_label_lookups` was corrected to `int`. `utils.importing.{load_or_create_labels, import_doc_annotations}` accept `Mapping` instead of `dict` so callers can pass `OpenContractDocExport` / `AnnotationLabelPythonType` TypedDicts without `cast()`.

- **Typing: graduate `opencontractserver.conversations.models` from mypy baseline** (refs Issue #1447, closes #1478): removed the last remaining `[mypy-opencontractserver.conversations.models]` `ignore_errors` block from `mypy.ini` and pruned 29 baseline error lines from `docs/typing/mypy_baseline.txt`. Per-file fixes in `opencontractserver/conversations/models.py`:

  - Replaced the `Optional["AbstractBaseUser"]` parameter on `ConversationQuerySet.visible_to_user` and `ChatMessageQuerySet.visible_to_user` with `"UserModel | AnonymousUser | None"` (forward-referenced via `TYPE_CHECKING` import of `opencontractserver.users.models.User as UserModel`). Cleared the 2× `AnonymousUser → AbstractBaseUser | None` assignment errors at the `if user is None: user = AnonymousUser()` reassignment, the 6× `[union-attr]` errors on `.is_superuser` / `.is_anonymous` / `.id`, and the `Incompatible type for lookup 'creator'` error on `Document.objects.filter(creator=user)` (after narrowing past the `is_anonymous` early-return, mypy resolves user to `User`).
  - Aligned `SoftDeleteManager.visible_to_user`, `ConversationManager.visible_to_user`, and `ChatMessageManager.visible_to_user` signatures with `BaseVisibilityManager.visible_to_user` (added `lightweight: bool = False` and `with_doc_label_annotations: bool = False` kwargs forwarded to `super()` for parity), eliminating the 3× `Signature of "visible_to_user" incompatible with supertype` `[override]` errors without resorting to `# type: ignore[override]`.
  - Stopped reassigning `base_qs` after `.annotate(...)` in `ConversationQuerySet.search_by_embedding` and `ChatMessageQuerySet.search_by_embedding`; the reassignment surfaced a `ConversationQuerySet[Model, Model]` vs `ConversationQuerySet[_Model, _Row]` (and the `ChatMessageQuerySet` equivalent) generic mismatch from django-stubs because `.annotate` widens the row TypeVar. The function now uses a separate `annotated_qs` local for the post-annotate pipeline.
  - Scoped `# type: ignore[misc]` on the `objects = ConversationManager()` and `objects = ChatMessageManager()` class-variable overrides (django-stubs flags re-declaring `BaseOCModel.objects`; the manager substitution is the documented mechanism for plugging in custom QuerySets).
  - Replaced narrowing `assert self.message is not None` in `ModerationAction.__str__` with a defensive `if/else` guard so the fallback path is exercised under `python -O` and covered by tests.
  - Lifted `from django.contrib.auth.models import AnonymousUser` from three local function imports to a module-level import so the type annotation does not need a `TYPE_CHECKING` indirection.

  Verified locally with `mypy --config-file mypy.ini opencontractserver config` reporting "Success: no issues found in 1032 source files".

- **Typing: graduated 11 `opencontractserver.utils.*` modules out of the mypy baseline** (chunk 2 of issue #1481, follow-up to #1331/#1447). Modules graduated and `ignore_errors` removed from `mypy.ini`: `export_v2`, `files`, `import_v2`, `importing`, `logging`, `mention_parser`, `multimodal_embeddings`, `packaging`, `pdf_token_extraction`, `sharing`, `storages`. 54 baseline error entries pruned from `docs/typing/mypy_baseline.txt`. Highlights:

  - V2 export builders now emit narrowly typed `TypedDict` payloads (`StructuralAnnotationSetExport`, `CorpusFolderExport`, `OpenContractsRelationshipPythonType`, `DescriptionRevisionExport`).
  - `packaging.unpack_corpus_from_export` discriminates V1 vs V2 input via the `OpenContractCorpusV2Type` overlay; the V1/V2 detection key is the presence of `post_processors` (legacy V1 exports never carry it).
  - `multimodal_embeddings` propagates `PawlsTokenPythonType` through the image-token pipeline.
  - `import_v2` and `packaging` use `TYPE_CHECKING` `User` aliases so `User = get_user_model()` is no longer used as a runtime annotation.
  - `sharing.make_analysis_public` now guards against analyses with no `analyzed_corpus` and returns a graceful error response instead of crashing on `AttributeError`.
  - `files.split_pdf_into_images` preserves its `Literal["PNG", "JPEG"]` parameter narrowing across the case-normalisation step.
  - **Latent bug fix uncovered by typing**: `packaging.unpack_label_set_from_export` and `unpack_corpus_from_export` had inverted `isinstance(user, str)` checks on parameters typed `int | UserModel`; the `int` branch was unreachable. Pinned by `opencontractserver/tests/test_packaging_typing_bug_fixes.py`.
  - Pre-commit `mypy` continues to run clean on the full project surface (`opencontractserver` + `config`).

- **typing: graduate `opencontractserver.mcp.server` and `opencontractserver.mcp.tools` from the mypy baseline** (issue #1486, sub-issue of #1447). Both modules pass `mypy` cleanly and their `[mypy-...]` blocks are removed from `mypy.ini`; the corresponding 7 entries are pruned from `docs/typing/mypy_baseline.txt`. Real-typing changes (no `# type: ignore` shortcuts where a proper annotation would do):

  - `opencontractserver/mcp/server.py` — `resources` now annotated as `list[Resource]`; resource `uri` strings wrapped in `pydantic.AnyUrl(...)` to match `Resource.uri`'s `Annotated[AnyUrl, ...]` field; `MCPLifespanManager._run_context` and `ScopedMCPLifespanManager._run_context` typed as `AbstractAsyncContextManager[None] | None` instead of bare `None`, fixing the `__aenter__` / assignment errors; ASGI `app(scope, receive, send)` and the inner `_handle_mcp_request` now use a shared `MutableMapping[str, Any]`-based `ASGIScope`/`ASGISend`/`ASGIReceive` alias compatible with Starlette and `StreamableHTTPSessionManager.handle_request`; `handle_sse_connection` annotated as `(request: Request) -> Response`; `__init__` and `main` get explicit `-> None` returns.
  - `opencontractserver/mcp/tools.py` — single `# type: ignore[attr-defined]` on `.search_by_embedding(...)` (mirrors the existing pattern in `opencontractserver/discovery/views.py`); django-stubs cannot model the custom `DocumentQuerySet.search_by_embedding` through the `from_queryset(...)` manager. `_text_search_fallback` parameters typed (`corpus: Corpus`, `user: User | AnonymousUser`).
  - Cross-module ripples to keep ASGI types honest (helper signatures previously used `dict[str, Any]`, which is invariant in `Callable` argument position and would have rejected our `MutableMapping`-based `send`/`receive`): `config/ratelimit/keys.py:139` (`get_client_ip_from_scope`), `config/ratelimit/decorators.py:371` (`check_mcp_rate_limit`), and `opencontractserver/mcp/telemetry.py:320` (`get_user_agent_from_scope`) widened to `MutableMapping[str, Any]`. `opencontractserver/corpuses/folder_service.py` — `check_corpus_read_permission` and `get_corpus_documents` widened from `user: User` to `user: User | AnonymousUser`. Both methods already handled the anonymous case at runtime via `user.is_anonymous`/`user.is_superuser` branches; the type widening just makes the public-corpus MCP entry points type-check without casts.

  Per the issue's recommendation, `opencontractserver/mcp/tests/test_mcp` (the 319-error worst-offender file) is intentionally **not** included here — it's tracked for a dedicated follow-up PR so the runtime-code graduation lands first and clean.

- **Docker image hardening follow-ups to PR #1500** (issue #1505). Six review observations on the slim-production-image PR are addressed without altering the deployed runtime semantics:

  - **Python pinned to `3.11.15-slim-bookworm`** in both `compose/production/django/Dockerfile` and `compose/local/django/Dockerfile` (was the floating `3.11-slim-bookworm` alias). Rebuilds on different dates can no longer silently pick up a new CPython point release that changes runtime behaviour. Bumps are now an intentional edit, not a side effect of the calendar.
  - **spaCy model install switched to the pinned-URL `wget` pattern in production** (`compose/production/django/Dockerfile`). The runtime stage previously called `python -m spacy download en_core_web_sm` / `en_core_web_lg`, which resolves the wheel URL at build time and could silently drift to a different patch. The production build stage now downloads the same explicit GitHub release URLs already used in `compose/local/django/Dockerfile` into `/usr/src/app/wheels`, with the same retry semantics; the runtime stage installs them via the existing `pip install --no-index --find-links=/wheels/` path and runs a `spacy.load(...)` smoke test. `wget` is added to the build stage only — it never reaches the runtime image.
  - **Release workflow split for the Django image** (`.github/workflows/docker-build-release.yml`). The previous step combined `load: true` (so the size-budget gate could `docker image inspect` the result) with `push: true`. That combination only works on single-platform builds, is not well-documented across Buildx versions, and some runner/daemon configurations reject it. The Django image is now built with `load: true, push: false`, gated on the 1.5 GiB size budget, then pushed via `docker push <tag>` per tag — the daemon already has the layers from the load step, so no rebuild occurs. Non-Django images keep the original combined `push: true, load: false` path.
  - **`.dockerignore` documents why `locale/` is safely excluded** — the tree contains no `.po`/`.mo` files (only a placeholder `README.rst`) and no `compose/*/start*` script invokes `manage.py compilemessages`. The runtime image still installs `gettext` so future i18n work has a build hook ready; if `.po` files are added, drop the `locale/` line so the build context picks them up.
  - **`brotli>=1.1.0` floor rationale documented** in `requirements/base.txt`. The constraint was not chosen incidentally — 1.1.0 is the current major-version line under active upstream maintenance. The underlying `_brotli` C-extension API that `httpx` looks up has been stable since 1.0.x, so a future transitive constraint can lower the floor safely if needed.
  - **`ffmpeg` removal verified** by an exhaustive `grep` across `opencontractserver/`, `compose/`, `config/`, `frontend/`, and the workflow tree: no Python `import`, `subprocess`, or shell invocation of `ffmpeg` exists. The PR #1500 removal of the apt package is safe; this entry records the verification trail rather than re-removing anything.
  - **SHA-256 checksums added for spaCy wheel downloads** in both `compose/production/django/Dockerfile` and `compose/local/django/Dockerfile`. Pinning to an explicit release URL is good but without a hash check the build is still vulnerable to a GitHub release being retroactively modified or a CDN MITM. Each `wget` step is now followed by a `sha256sum -c` verification that fails the build immediately if the fetched wheel doesn't match the expected digest.
  - **Size-budget gate confirmed intact** after the `id: build` step-ID removal. The gate at `.github/workflows/docker-build-release.yml:93` uses `${{ steps.meta.outputs.tags }}` (not the removed `steps.build.outputs.imageid`) to identify the locally-loaded image for `docker image inspect`, so the split build path works correctly.

- **Graduated remaining `config.graphql.*` modules from the mypy baseline** (issue #1485, sub-issue of #1447). Removed the per-module `ignore_errors` blocks for the final 14 modules (`annotation_mutations`, `annotation_queries`, `base`, `conversation_queries`, `corpus_mutations`, `document_mutations`, `document_types`, `extract_mutations`, `extract_queries`, `label_mutations`, `permissioning.permission_annotator.{middleware,mixins,utils}`, `social_queries`) and pruned 174 corresponding lines from `docs/typing/mypy_baseline.txt`. The entire `config.graphql.*` surface is now type-checked. Notable code changes made in service of typing:

  - `AnnotatePermissionsForReadMixin` declares `_meta` / `is_public` under `TYPE_CHECKING` so the mixin no longer trips `attr-defined` errors when combined with a `DjangoObjectType` (`config/graphql/permissioning/permission_annotator/mixins.py`).
  - All `from_global_id(kwargs.get("id", None))[1]` Relay node resolver patterns rewritten to `from_global_id(kwargs["id"])[1]` (the `id` arg is required for relay, so the `.get(..., None)` was dead defensiveness that produced an `Any | None` mypy can't accept).
  - Optimizer call-sites (`AnnotationQueryOptimizer.get_document_annotations`, `RelationshipQueryOptimizer.*`, `BadgeQueryOptimizer.check_user_badge_visibility`) now coerce the `from_global_id` string pk to `int` instead of relying on Django's lenient `pk` lookup.
  - PEP 484 implicit-Optional defaults converted to explicit `T | None` in `corpus_mutations.CreateCorpusAction.mutate`, `corpus_mutations.UpdateCorpusAction.mutate`, and `document_mutations.StartCorpusExport.mutate`.
  - Removed dead `language_model_id` parameter and assignment in `UpdateColumnMutation` — the `language_model` field was deleted from `Column` in migration `0012_auto_20240622_2225` and the parameter was unreachable from the mutation's `Arguments`.
  - `timezone.timedelta(...)` (django-stubs no longer re-exports `timedelta`) replaced with a direct `from datetime import timedelta` in `conversation_queries.resolve_corpus_moderation_stats`.

- **Type checking: graduated `opencontractserver.discovery.tests.test_discovery_views`** (`opencontractserver/discovery/tests/test_discovery_views.py`, closes #1479, refs #1447). Removed the module's `[mypy-…]` `ignore_errors` block and pruned the corresponding 44 baseline entries from `docs/typing/mypy_baseline.txt`. The 49 errors mypy reported were all of the same shape — class-level attributes assigned inside `setUpTestData(cls)` that mypy doesn't follow because the assignments live in a `@classmethod` body. Fix uses Option 1 from the issue: declare the attributes as class-level annotations (`owner: User`, `public_corpus: Corpus`, etc.) on each affected test class (`LlmsTxtTest`, `LlmsFullTxtTest`, `SitemapXmlTest`, `WellKnownMcpTest`, `SearchApiTest`) so mypy sees the names without changing runtime behavior. Replaced `User = get_user_model()` with a direct `from opencontractserver.users.models import User` so the annotations resolve to the real model class (the dynamic `get_user_model()` callable returns `type[AbstractBaseUser]`, which mypy cannot use as a type).

- **Consolidated corpus-modify permission checks behind `user_can_modify_corpus`** (`opencontractserver/utils/permissioning.py`, refs #1450). Four GraphQL mutations were repeating the same inline `corpus.creator == user or user_has_permission_for_obj(user, corpus, UPDATE, include_group_permissions=True)` check (`config/graphql/badge_mutations.py:91, 211, 328` and `config/graphql/document_mutations.py:416`). The pattern is the cluster-2 anti-pattern flagged in the issue: as the permission model evolves (sharing, group inheritance, MIN(doc, corpus)) inline copies drift apart. Replaced each with `user_can_modify_corpus(user, corpus)`, which encodes the canonical "is_superuser OR creator OR explicit guardian UPDATE" semantics in one place. Behaviorally identical: superuser was already admitted via `user_has_permission_for_obj` (returns True for superusers on guardian-enabled models); the helper just makes the short-circuit explicit. Anonymous and `None` callers correctly return `False`. New tests in `opencontractserver/tests/test_user_can_modify_corpus.py` pin the matrix (superuser, creator, user-level UPDATE, group-level UPDATE, group-perm-disabled, outsider, anonymous, `None`, id-by-int, id-by-str). Out of scope for this consolidation PR (flagged for separate review per the "consolidation PRs should not also be behavior-change PRs" rule): `corpus_mutations.py` `UpdateCorpusDescription`/`ReEmbedCorpus` (intentionally creator-only per docstring), `corpus_mutations.py` `CreateCorpusAction`/`AddTemplateToCorpus` (use `PermissionTypes.CRUD` not UPDATE — switching widens access), `worker_mutations.py:154` (no group-perm fallback today). Also removed two stale `resolve_oc_model_queryset` references from `opencontractserver/shared/Managers.py:139` and the docstring of `opencontractserver/tests/test_visibility_managers.py` — that helper has no live callers and visibility lives in the manager method now.

- **Graduated `opencontractserver.pipeline.*` from the mypy baseline** (issue #1483, refs #1447, #1331). All fourteen pipeline modules — the `base` package (`base_component`, `chunked_parser`, `embedder`, `parser`, `post_processor`), the embedders (`multimodal_microservice`, `openai_embedder`, `sent_transformer_microservice`), the parsers (`docling_parser_rest`, `docxodus_parser`, `llamaparse_parser`, `oc_markdown_parser`, `oc_text_parser`), `post_processors.pdf_redactor`, plus `pipeline.registry` and `pipeline.utils` — are now type-checked from scratch (103 baseline entries removed from `docs/typing/mypy_baseline.txt`). Notable changes that surfaced from the graduation: shared component metadata (`title`, `description`, `author`, `dependencies`, `input_schema`) is now declared on `PipelineComponentBase` so `type[PipelineComponentBase]` lookups in `pipeline.utils.get_metadata_for_component` resolve attributes through the base class instead of bare `type`; `BasePostProcessor` now declares `supported_file_types: list[FileTypeEnum]` (previously accessed but never declared on the base); `BaseMultimodalMicroserviceEmbedder.url_setting_name` is now a typed class attribute (referenced in error messages, never declared); `pipeline.utils.run_post_processors` validates that the class loaded by `get_component_by_name` is actually a `BasePostProcessor` subclass before calling `process_export`. Image-token construction in `docling_parser_rest` and `llamaparse_parser` only sets optional `PawlsTokenPythonType` fields when present, eliminating spurious `None` writes against `NotRequired`-but-non-`None` slots. `llamaparse_parser._build_image_token` extracted as a shared helper. No runtime behaviour change; all interactions with broader union types are preserved through targeted `cast()` calls at the boundary.
- **Type-checking: graduated `opencontractserver.utils.*` chunk 1 from mypy baseline** (issue #1480, sub-issue of #1447). Removed `[mypy-…]` ignore blocks for `utils.analysis`, `utils.analyzer`, `utils.cleanup`, `utils.corpus_forking`, `utils.embeddings`, and `utils.etl` and fixed all surfaced errors. Notable production fixes uncovered by typing:
  - `opencontractserver/utils/cleanup.py:34` — `delete_analysis_and_annotations` was calling `Analysis.annotations.filter(...)` on the class-level reverse descriptor (would fail at runtime with `AttributeError`); now correctly uses `analysis.annotations.all().delete()`.
  - `opencontractserver/utils/analyzer.py:204` — `run_analysis` now raises `ValueError` early when `analyzer.host_gremlin` is `None` instead of crashing on `gremlin.url` later.
  - `opencontractserver/utils/analyzer.py:272` — Submission-response logging used `f"{submission_response.content}"` which produced `b'…'` rather than the response body; switched to `!r` for safe bytes formatting.
  - `opencontractserver/utils/analyzer.py:install_analyzers` and `opencontractserver/tasks/analyzer_tasks.py:install_analyzer_task` — return type corrected from `list[int]` to `list[str]` (Analyzer PKs are CharField).
  - `opencontractserver/utils/embeddings.py` — `get_embedder` / `aget_embedder` parameter and return types now reflect that any of `corpus_id`, `mimetype_or_enum`, `embedder_class`, and `embedder_path` may be `None`. Removed an outdated `# type: ignore[attr-defined]` on `embed_text`.
  - `opencontractserver/utils/etl.py:404` — Renamed inner PDF-burn-in loop variable from `page` to `pdf_page` so it no longer collides with the `PageAnnotationData` `page` from `iter_page_annotations`. Color tuple is now an explicit `tuple[float, float, float]` instead of variable-length.
  - Pruned 52 corresponding lines from `docs/typing/mypy_baseline.txt`.

### Fixed

- **typing: graduate `opencontractserver.shared.{Managers,decorators}` from mypy baseline** (refs #1447, closes #1477). Drains the last two `[mypy-opencontractserver.shared.*]` ignore-blocks from `mypy.ini` (37 baseline lines pruned: 6813 → 6776 in `docs/typing/mypy_baseline.txt`). Per-file fixes:

  - `opencontractserver/shared/Managers.py`: drop the `if TYPE_CHECKING / else AbstractUser as User` aliasing block (mypy treated `User` as a value, not a type) — replace with explicit `AbstractBaseUser` annotations on `EmbeddingManager.store_embedding(creator=...)` and `UserFeedbackManager.by_creator(creator=...)`, and `Any` for the duck-typed `_apply_document_prefetches(user=...)` argument; cast `self.model` to `Any` once in `BaseVisibilityManager.visible_to_user` so downstream `model_cls.objects.*` and `cast(Any, self.model).DoesNotExist` (in `UserFeedbackManager.get_or_none`) typecheck without `_default_manager` semantic drift; coalesce `self.model._meta.model_name or ""` so the `.upper()` calls don't union-attr against `None`; align `PermissionManager.visible_to_user` and `UserFeedbackManager.visible_to_user` with `BaseVisibilityManager.visible_to_user(user, lightweight, with_doc_label_annotations)` (kwargs accepted for parity, no behaviour change); silence `[misc]` on the `PermissionManager.from_queryset(...)` dynamic base for `AnnotationManager` / `NoteManager`. Removes three unused `for_user(...)` delegation methods on `PermissionManager`, `AnnotationManager`, `NoteManager` that called `self.get_queryset().for_user(...)` against querysets that never implemented it — would have raised `AttributeError` at runtime. No callers in `opencontractserver/`, `config/`, or tests.
  - `opencontractserver/shared/decorators.py`: type `pdf_data_layer` and `pdf_text_extract` so the PDF / TXT branches of both `doc_analyzer_task` and `async_doc_analyzer_task` typecheck. `pdf_text_extract` is annotated `Optional[str]` and asserted non-`None` in the TXT branch (only reachable when `txt_extract_file` is non-empty); `pdf_data_layer` is annotated `Any` to absorb the `[] | PdfDataLayer` union without forcing `plasmapdf` stubs. Three inner helpers (`get_analysis_creator`, `get_analysis_analyzer`, `celery_task_with_async_to_sync.wrapper`) gain `-> Any` return annotations. Final return-annotation coverage on the graduated modules: 100% on `Managers.py` and 100% on `decorators.py`.

- **Document deletes now reclaim orphaned file blobs from storage** (`opencontractserver/documents/signals.py:170-300`, `opencontractserver/tasks/cleanup_tasks.py:25-110`, `opencontractserver/shared/Managers.py:413-470`, issue #1492). Previously, deleting a `Document` row left its `pdf_file`, `txt_extract_file`, `pawls_parse_file`, `icon`, and `md_summary_file` blobs in storage forever, so storage cost grew unbounded. Two new signal handlers on `Document` capture every populated FileField path in `pre_delete` and schedule a single Celery task per transaction (via `transaction.on_commit`) that removes paths whose live reference count hit zero. The task re-checks orphan status at execution time across every FileField, so corpus-isolated copies (issue #1464) and races with concurrent uploads are handled correctly. Bulk `QuerySet.delete()` is covered by the same signals — once listeners are connected Django disables its fast-delete path and dispatches per-instance signals. New manager method `Document.objects.unique_blob_paths_for_many(qs_or_pks)` gives callers a 2-queries-per-FileField primitive for queryset-scale orphan checks (vs N+1 per row in `unique_blob_paths`). Transaction safety: `on_commit` callbacks never fire on rollback, so a failed delete never reaches the storage layer. Regression tests in `opencontractserver/tests/test_blob_retention.py` cover per-row delete, queryset bulk delete (one-of-pair vs both-of-pair), `permanently_delete_document` (Rule Q1 cascade), `DocumentPath` PROTECT semantics, transaction rollback, sparse FileField sets, and direct task contract. Coverage spans every FileField on Document so adding a new file field is automatically protected.

- **STRIDE P0 follow-up review fixes** (issue #1466, follow-up to PR #1463).

  - `_add_doc_type_annotation` test helper in
    `opencontractserver/tests/test_add_annotation_idor.py:157` now passes the
    `user` argument it accepts into `_MutationContext` instead of the
    hard-coded `self.attacker`. The previous version made the helper appear
    parameterised but always executed as the attacker, so any future caller
    that passed `self.victim` would have silently re-run the attacker case
    and could miss a regression.
  - `_resolve_annotation_parents` in
    `config/graphql/annotation_mutations.py:218` now imports `DocumentPath`
    at module level alongside `Document` (no circular dependency exists),
    removing the unexplained lazy import.
  - `resolve_bulk_document_upload_status` in
    `config/graphql/document_queries.py:215` now constructs the opaque
    "not found" `BulkDocumentUploadStatusType` only inside the rejection
    branch instead of allocating it unconditionally on every status query.
    (Code-quality cleanup, not a security fix — the previous code
    allocated the response eagerly but only ever returned it on the
    rejection path, so externally-observable behaviour is unchanged.)
  - `BULK_UPLOAD_OWNER_CACHE_TTL_SECONDS` in
    `opencontractserver/constants/zip_import.py:60` carries a comment
    explaining why the 24-hour default is intentionally generous (a large
    zip can take many hours and the user must remain able to poll progress
    for the full lifetime of the job).
  - `ingest_doc` in `opencontractserver/tasks/doc_tasks.py:350` carries a
    comment explaining why the worker-side check uses `PermissionTypes.READ`
    while the parallel `retry_document_processing` path uses
    `PermissionTypes.UPDATE`.
  - `retry_document_processing` in
    `opencontractserver/tasks/doc_tasks.py:846` now emits a
    `[SECURITY]`-tagged log line on the `User.DoesNotExist` early-exit and
    returns the more accurate `"Invalid user for retry"` message instead
    of the misleading `"Document not found"`. Audit symmetry with the
    `ingest_doc` path is now preserved.
  - `Corpus._propagate_public_status_to_documents` in
    `opencontractserver/corpuses/models.py:522` moves the
    `cross_owner_blocked_ids` query inside the `transaction.atomic()`
    block, closing the narrow race where a document added to a new
    cross-owner private corpus between the membership check and the
    update could be incorrectly publicized. The same method now passes
    `batch_size=500` to `Notification.objects.bulk_create` so a corpus
    with thousands of cross-owner documents does not blow up a single
    SQL statement.

- **Bulk and single document import no longer crash with Apollo "Payload allocation size overflow"** before the network request fires. The previous transport base64-encoded the entire file (PDF or zip) into a single GraphQL `String` variable; Apollo then `JSON.stringify`'d the request body, which V8 cannot allocate as a contiguous string for large files (~33% inflation pushes a moderately sized zip past the V8 string limit / heap fragmentation ceiling). The frontend now streams uploads as `multipart/form-data` via `fetch` to two new REST endpoints, so the binary bytes never become a JS string. Files affected:

  - `frontend/src/components/widgets/modals/UploadModal/hooks/useUploadMutations.ts` — replaces the `UPLOAD_DOCUMENT` and `UPLOAD_DOCUMENTS_ZIP` Apollo mutations with `fetch` calls.
  - `frontend/src/utils/importHttp.ts` (new) — multipart helpers (`importDocumentMultipart`, `importDocumentsZipMultipart`) that attach the JWT bearer token and parse REST error bodies into structured results.
  - `frontend/src/utils/files.ts` — `toBase64` removed (last call site was the upload hook).

- **CorpusChat source citation deep-link: silent no-ops + an annotation-id fallback** (`frontend/src/views/Corpuses.tsx`, `frontend/src/components/corpuses/CorpusChat.tsx`). When asking questions of a corpus, clicking a source citation could silently do nothing. `handleSourceNavigate` had three guarded silent-return paths (no `document_id`, missing corpus slugs, empty `textBlock`); each one now emits a console.warn naming the failing condition so the cause is observable in DevTools. CorpusChat's source `onClick` also logs every click reaching the handler so we can confirm the click chain is intact. The third silent-return (text-block construction failed — typically a PDF source whose multipage `annotation_json` has page keys but empty `tokensJsons` arrays, leaving `encodePdfBlock` to return `""`) now falls back to a `?ann=<global-annotation-id>` deep-link when the source has a real (positive) `annotation_id`. As a last resort, navigates to the document with no highlight rather than no-op. Follows the routing-system contract: navigation goes through `navigate(url)` with Relay/global IDs in the path so CentralRouteManager Phase 1 can canonicalize, and `?ann=`/`?tb=` are picked up by Phase 2 — no direct reactive-var sets.

- **Drop the standalone "AI Assistant is thinking…" pill in favor of an animated ticker icon** (`frontend/src/components/widgets/chat/ChatMessage.tsx`, `frontend/src/components/corpuses/CorpusChat.tsx`, `frontend/src/components/knowledge_base/document/right_tray/ChatTray.tsx`). The old pill (pulse-dots + spinner + "AI Assistant is thinking…") rendered as a separate banner under the messages and duplicated the per-message in-flight signal. Replaced with: (a) a soft-pulse + glow halo on the `StreamingThoughtTicker` icon plus a tiny breathing dot at the right edge — the icon itself is the activity cue, so a tool_call still reads as a wrench, a thought still reads as a lightning bolt; (b) a generic "Thinking" placeholder ticker rendered when the assistant is mid-stream but no timeline entry has arrived yet, replacing the per-message `ProcessingIndicator`; (c) a standalone warm-up ticker rendered between user-send and the first assistant message in both `CorpusChat` and `ChatTray`, so the in-flight cue is universal across corpus-chat and document-chat. The mid-stream rendering rule was also tightened: the ticker only shows when the assistant is streaming AND no content has arrived yet (as soon as text tokens stream in, the bubble takes over and accumulates the response). The bordered `TimelinePreview` panel is suppressed entirely mid-stream and only appears post-completion as a collapsible summary inside the bubble. Removed the dead `ProcessingIndicator` component + its testid `processing-indicator`. Tests updated: rewrote `chat-message-processing-indicator.ct.tsx` to assert the ticker contract; updated the user-message check in `ChatMessageCoverage.ct.tsx`; new regression test `the legacy 'AI Assistant is thinking…' pill is gone after a send` in `CorpusChat.ct.tsx`.

- **Streaming chat timeline replaced with a single-line "now-thinking" ticker** (`frontend/src/components/widgets/chat/ChatMessage.tsx`). The bordered, scrollable `TimelinePreview` panel rendered while the assistant was streaming pushed the chat input off-screen on multi-step responses and shifted with every new step. New `StreamingThoughtTicker` shows only the latest step (icon + truncated title) on a single ~24px line and fades the previous step out as a new one arrives (subtle 250ms y-translate + opacity transition via `AnimatePresence exitBeforeEnter`). The full bordered timeline is no longer rendered mid-stream — it returns automatically once the message finalizes via the existing collapsible Timeline inside the message bubble (unchanged). Cleaned up the now-unused `expandLatestOnly` prop on `TimelinePreview` and the dormant `compact` mode added in the previous iteration; `TimelineContent` `max-height` reduction 400 → 220 px is kept so the post-completion expanded view stays bounded for long step lists. Updated the existing "should show timeline instead of processing indicator when timeline arrives first" test to assert the ticker, not the panel; new regression tests `streaming renders the latest-step ticker, not the bordered timeline panel` and `once the message is complete, the ticker disappears and the collapsible Timeline returns` pin both halves of the streaming → finalized handoff.

- **CorpusChat: duplicate Back button on the conversation-list views** (`frontend/src/components/corpuses/CorpusChat.tsx`, `frontend/src/views/Corpuses.tsx`). Same class of bug as the conversation-view header but on the list-view side: the `ConversationListView` filter-bar always rendered its own `←` Back next to "+ New Chat" whenever an `onBack` callback was passed, while the parent (`renderNavigationHeader` for the "Conversation History" / chatExpanded list views, and `TabNavigationHeader` on the corpus "Chats" tab) was simultaneously rendering its own outer Back. Fix: new `hideListBackButton` prop on `CorpusChat` forwards `onBack` to the list view as `undefined` when the parent already provides Back navigation. Wired up at all three call sites — chatExpanded path and VIEW-mode path use `hideListBackButton={isDesktop}` (mobile keeps the inner Back since the outer header is desktop-only), and the chats tab uses `hideListBackButton` unconditionally because its `TabNavigationHeader` renders on every screen size. New regression test: `hideListBackButton hides the filter-bar Back in the conversation list`.

- **CorpusChat: duplicate stacked Back buttons + sluggish chat input** (`frontend/src/views/Corpuses.tsx`, `frontend/src/components/corpuses/CorpusChat.tsx`, `frontend/src/components/corpuses/corpus_chat/styles.ts`). Two issues in the corpus chat surface:

  - The dashboard "search → chat" flow rendered the parent `Back / Chat` navigation header on top of `CorpusChat`'s own `← / Conversation` header, leaving two visually-stacked back buttons inside a conversation. The parent's header is now suppressed whenever the inner CorpusChat is in a conversation (mirroring the pattern already in use for the corpus "Chats" tab), so a single Back affordance is on screen at any time. Clicking it returns to the chat list, the parent header reappears, and clicking again returns to the search view — a single button that the user can click twice. New `chatExpandedInConversation` mirror state in `CorpusQueryView` is pre-set when entering chat to avoid a one-frame flash of the outer header.
  - The chat textarea started at ~2 visible rows (~73px) and never auto-grew — typing past the visible area scrolled inside the static box. The textarea now starts at one row (40–48px), auto-resizes to its `scrollHeight` up to a 140px content-box cap, then becomes scrollable. Container padding was tightened (1.25rem 1.5rem → 0.625rem 0.875rem) and the send button was resized to 40×40 (38×38 mobile) so the input area is no longer a giant white rectangle when empty. Shift+Enter inserts a newline, bare Enter sends. Mirrors the existing pattern in `ChatTray`.
  - New regression tests in `frontend/tests/CorpusChat.ct.tsx`: "conversation view shows exactly one Back button (not two stacked)", "onViewModeChange notifies parent when entering / leaving conversation", and "chat input starts compact and grows with multi-line content". New auto screenshots `corpus--chat--input-compact` and `corpus--chat--input-expanded`.

- **`GetDocumentKnowledgeAndAnnotations` no longer eagerly loads structural annotations & relationships** (`frontend/src/graphql/queries.ts`, `config/graphql/document_types.py`, `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`, `frontend/src/components/annotator/context/AnnotationAtoms.tsx`). The initial document-open query was returning every structural annotation and structural relationship for the document — for documents with thousands of structural items, this meant ~30s to first render even though the UI hides them by default. The frontend already had a separate lazy `GET_DOCUMENT_STRUCTURAL_ANNOTATIONS` query that fires when the user toggles structural visibility, but the main query never passed `isStructural: false`, so the work was being done twice. Fix:

  - The main query now passes `isStructural: false` to both `allAnnotations` and `allRelationships` (and the same for `GET_DOCUMENT_ANNOTATIONS_ONLY` used during analysis switching).
  - `Document.allRelationships` GraphQL field gains an `isStructural: Boolean` argument (mirroring `allAnnotations`); the underlying `RelationshipQueryOptimizer.get_document_relationships` already supported the filter.
  - New `Document.allStructuralRelationships` field — the relationship analogue of `allStructuralAnnotations` — and the lazy `GET_DOCUMENT_STRUCTURAL_ANNOTATIONS` query now fetches both in a single round-trip.
  - New `structuralRelationshipsAtom` (paralleling `structuralAnnotationsAtom`) plus a derived `allRelationsAtom` and `useAllRelations` hook for consumers that need the full set (`useVisibleAnnotations`, `AnnotationList`, `UnifiedContentFeed`). User-editable relation CRUD continues to read/write `pdfAnnotationsAtom.relations` only.
  - Backend coverage: new tests in `test_structural_annotations_graphql_backwards_compat.py` for `allRelationships(isStructural: false|true)` and `allStructuralRelationships` (pre- and post-migration).

- **Bound peak memory in PDF image extraction step** (`opencontractserver/utils/pdf_token_extraction.py`, issue #1498). The `extract_images_from_pdf` step run after document reassembly in the Celery `ingest_doc` pipeline previously rasterised each page once _per embedded image_ via `pdf2image.convert_from_bytes` and never explicitly closed PIL Image / BytesIO buffers, so peak RSS scaled linearly with `page_count * images_per_page`. A 127-page PDF would OOM Celery workers (cgroup SIGKILL, exit 137) that comfortably handled 30-page documents. Fix: pages are now processed one at a time; rasterisation is cached _per page_ via the new `_render_pdf_page` / `_crop_region_from_rendered_page` split, so a page with N images is rendered exactly once; PIL Image and BytesIO buffers are closed in inner `finally` blocks; pdfplumber's per-page parse cache is flushed via `page.flush_cache()`; and `gc.collect()` runs every `IMAGE_EXTRACTION_GC_INTERVAL_PAGES` pages (default 1) to reclaim Poppler subprocess buffers. Peak working set per call is now `len(pdf_bytes) + 1 rendered page + 1 decoded image bytes` — independent of total page count. Two new env vars: `IMAGE_EXTRACTION_DPI` (default 150, page-render RSS scales as ~DPI²) and `IMAGE_EXTRACTION_GC_INTERVAL_PAGES` (default 1, set to 0 to disable explicit collection). Regression tests in `opencontractserver/tests/test_pdf_token_extraction.py::TestExtractImagesMemoryBound` pin: render-once-per-page contract, rendered-page closure, pdfplumber cache flush, gc.collect cadence, eager per-image buffer close, and synthetic 100-page scaling.

- **Doc-blob deletion guard follow-up** (issue #1496). Tightened the safety net introduced in #1464:

  - `Document.safe_delete_field_blob` (`opencontractserver/documents/models.py:415-421`) now catches the specific `django.core.exceptions.FieldDoesNotExist` raised by `Options.get_field()` instead of bare `Exception`, so unrelated runtime errors (e.g. `AttributeError`, ORM internals) are no longer re-raised as misleading `ValueError`s.
  - The docstring now calls out the inherent TOCTOU window between `unique_blob_paths` and the storage delete, and tells callers in high-concurrency paths that a row-level lock on the single mutated document is not sufficient — sibling rows must be locked too.
  - `opencontractserver/shared/Managers.py` gains `from __future__ import annotations` so the PEP 585 `set[str]` return annotation on `DocumentManager.unique_blob_paths` evaluates lazily and remains importable on Python 3.8.
  - `unique_blob_paths(self, doc: "Document")` replaces the looser `doc: Any` so type checkers can flag invalid callers; the `Document` type is imported under `TYPE_CHECKING` to avoid a circular import.
  - Added a comment on the subTest loop in `test_shared_blob_is_preserved_for_sibling` (`opencontractserver/tests/test_blob_retention.py`) explaining why the `refresh_from_db()` calls at the top of each iteration are required for isolation as the loop progressively clears fields on `copy`.

- **`PipelineSettings.get_parser_kwargs` now merges encrypted secrets** (`opencontractserver/documents/models.py:1260`, issue #1515). The runtime path that resolves parser configuration in `opencontractserver/tasks/doc_tasks.py:424` previously returned only `parser_kwargs[parser_class_path]` and silently dropped any secret stored under `encrypted_secrets[parser_class_path]`. Parsers like `LlamaParseParser` whose API key lives only in `encrypted_secrets` (the documented secure location) were invoked with an empty `api_key=""` placeholder and failed with `"LlamaParse API key not configured"`. Fix: `get_parser_kwargs` now overlays decrypted `encrypted_secrets[parser_class_path]` on top of `parser_kwargs[parser_class_path]`, with secrets winning on key collision. Operators may keep `parser_kwargs.api_key = ""` as a schema marker without clobbering the real secret. The merged dict is built fresh on every call so a long-lived `PipelineSettings` reference does not retain decrypted secrets in memory. Regression tests in `opencontractserver/tests/test_pipeline_settings.py` cover merge, secret-wins, secret-only, no-mutation, and log-redaction.

- **`PipelineSettings.get_component_settings` now merges encrypted secrets** (`opencontractserver/documents/models.py:1297`, follow-up to #1515). The model-level component settings getter previously returned only the plaintext `component_settings[component_class_path]` entry, which would silently drop encrypted values for any direct caller that bypassed `get_full_component_settings`. The method now overlays decrypted `encrypted_secrets[component_class_path]` on top of the plaintext settings using the same precedence as `get_parser_kwargs` (secrets win on key collision; empty placeholders allowed; merged dict built per call so secrets are not retained on the model). New regression test `test_get_component_settings_merges_encrypted_secrets`.

### Added

- **Multipart REST import endpoints under `/api/imports/`** (`opencontractserver/document_imports/`, new app):

  - `POST /api/imports/documents/` — single-document import (`DocumentImportView`).
  - `POST /api/imports/documents-zip/` — bulk-zip import (`DocumentsZipImportView`); stages the archive in a `TemporaryFileHandle` and queues `process_documents_zip` (`opencontractserver/tasks/import_tasks.py`) exactly as the GraphQL path did.
  - `opencontractserver/document_imports/services.py` exposes `import_document_for_user` and `import_documents_zip_for_user`. Both `UploadDocument` / `UploadDocumentsZip` GraphQL mutations now delegate to these services so the two transports validate identically (visibility + EDIT permission with unified IDOR-safe error message; usage-cap enforcement; folder-in-corpus check; allowed-MIME enforcement; bulk-job owner cache binding).
  - New setting `MAX_DOCUMENT_IMPORT_SIZE_BYTES` (env-overridable, defaults to `MAX_FILE_UPLOAD_SIZE_BYTES`) caps multipart import size at the view layer with an explicit `413` response. Set to `0` to disable.
  - Auth: bearer JWT only. Both views pin `authentication_classes = [GraphQLJWTAuthentication]` rather than inheriting the DRF default (`JWT + Session + Token`), which narrows the threat model: browsers do not auto-attach `Authorization: Bearer`, so the endpoints are CSRF-safe by construction and any future change to `DEFAULT_AUTHENTICATION_CLASSES` cannot silently widen the surface here. Session callers (e.g. Django admin) must keep using GraphQL.
  - Per-endpoint throttle: both views apply `DocumentImportThrottle` (scope `document_imports`, default `120/hour` via `REST_FRAMEWORK["DEFAULT_THROTTLE_RATES"]`). The global `user` rate (`1000/hour`) is far too permissive for an endpoint where a single request can be hundreds of MB; `120/hour` is a workable default and operators can tune via the standard DRF setting.
  - Tests: `opencontractserver/tests/test_document_imports_rest.py` covers both views (auth, owned/inaccessible/read-only corpus IDOR uniformity, folder-in-corpus check, unsupported MIME, oversize 413, usage-cap, custom-meta JSON pass-through, owner-cache binding) plus direct service tests for the bytes/UploadedFile branches. Three new security-focused classes round out the suite: `DocumentImportRealJWTAuthTests` exercises the actual JWT validation path (valid bearer → 201, missing/tampered/expired → 401/403; the previous tests used `force_authenticate` which bypasses auth classes), `DocumentsZipNonZipRejectionTests` verifies `_peek_zip_magic` end-to-end against a PDF-as-zip and random bytes, and `DocumentImportFolderIDORTests` confirms a folder ID belonging to another user's corpus is rejected without document creation in both attacker shapes (own corpus + foreign folder, foreign corpus + foreign folder). `frontend/src/components/widgets/modals/UploadModal/hooks/__tests__/importHttp.test.ts` (8 cases) verifies the request body is `FormData` — not JSON — with correct URL, bearer auth header, blank-string omission, and structured error parsing. `useUploadMutations.test.tsx` (8 cases) covers sequential upload semantics, status-callback ordering, partial-failure resilience, and refetch+`onComplete` exactly-once.

- **Discover search crash on null annotation/note documents** (`frontend/src/views/DiscoverSearchResults.tsx:467-469, 600-603` and `frontend/src/utils/navigationUtils.ts:326-342, 358-400`). Running a search on the Discover view could throw `TypeError: can't access property "slug", e is null` when the backend returned an annotation or note edge whose `document` field was null (e.g., document filtered out by visibility or recently soft-deleted). `getDocumentUrl` accessed `document.slug` directly, and the section render lists never filtered for the missing relation. Fix: filter edges with `node.document == null` out of the annotation and note lists, and widen `getDocumentUrl` / `getCorpusUrl` to accept nullable inputs and fall through to the existing `"#"` no-op return path instead of throwing.

### Changed

- **Reduced chatty permissioning INFO logs** (`opencontractserver/utils/permissioning.py:365-373`, issue #1436). `user_has_permission_for_obj` previously emitted two `INFO` log lines on every call ("Starting check for ..." and "App name: ..."), which fired on essentially every authenticated request and dominated Django pod logs in production. The two lines are now collapsed into a single `logger.debug(...)` call using lazy `%`-formatting, so the strings are not built unless DEBUG logging is enabled. Permission denial logs (structural-write denial, analysis/extract privacy denials) remain at `INFO`, and all `WARNING`/`ERROR` paths are unchanged.
- **`supportedMimeTypes` GraphQL query is now accessible to anonymous users** (`config/graphql/pipeline_queries.py:205`). Removed the `@login_required` decorator from `resolve_supported_mime_types` so that uploaders, landing pages, and other unauthenticated UI surfaces can advertise the accepted file formats without forcing a login. The data returned is derived purely from the pipeline registry and exposes no user-specific information. Test updated: `opencontractserver/tests/test_pipeline_component_queries.py::test_supported_mime_types_allows_anonymous`.

### Added

- **CAML README references in bulk-import zips** (`opencontractserver/utils/caml_rewrite.py`, `opencontractserver/utils/import_v2.py:285`, `opencontractserver/tasks/import_tasks_v2.py:357`). The corpus README (`md_description`) shipped inside a V2 import zip now supports `oc-import://document/<filename-in-zip>` and `oc-import://annotation/<id-in-data.json>` placeholder URLs that are rewritten to live `/d/<user-slug>/<corpus-slug>/<doc-slug>[?ann=<new-pk>]` URLs after all referenced objects have been created. Lets a zip author hand-write a README that links to bundled documents and annotations without knowing the destination deployment's primary keys or slugs. Document references key off the same filename used as the `annotated_docs` key in `data.json`; annotation references key off the export-time `"id"` already aggregated by the importer as `all_annot_id_maps`. Unresolved references are left intact and warned about (no silent stripping). Revision snapshots are intentionally not rewritten to preserve the existing checksum chain. New tests in `opencontractserver/tests/test_caml_rewrite.py`. Documented in `docs/upload_methods/corpus_export_import.md`.
- **Server-derived `me.canImportCorpus` field** (`config/graphql/user_types.py:43-60`). A new boolean on `UserType` that mirrors the backend permission check enforced by `UploadCorpusImportZip` / `ImportZipToCorpus`: returns `false` for anonymous users and for usage-capped users when `USAGE_CAPPED_USER_CAN_IMPORT_CORPUS` is disabled, otherwise `true`. Exposed through `GET_ME` (`frontend/src/graphql/queries.ts`) so the frontend can gate corpus-import UI without re-implementing the rule. Tests: `opencontractserver/tests/test_user_can_import_corpus.py` covers the three states (capped+disabled, capped+enabled, uncapped).
- **Dedicated `ImportCorpusModal`** (`frontend/src/components/widgets/modals/ImportCorpusModal.tsx`). Replaces the previous hidden `<input type="file">` flow on the Corpuses page with a confirm/upload/progress modal that calls `importOpenContractsZip` to create a brand-new corpus from an OpenContracts export ZIP. The trigger button on `CorpusListView` is now hidden for users whose `me.canImportCorpus` is `false`, eliminating the prior UX where usage-capped users could pick a file and only then receive a backend permission error.

### Security

- **`updatePipelineSettings` mutation rejects plaintext secrets in `parser_kwargs` and `component_settings`** (`config/graphql/pipeline_settings_mutations.py`, related to issue #1515). Schema-declared `SettingType.SECRET` fields (e.g. `api_key`, `secret_token`) can no longer be stored as non-empty values in either of these unencrypted JSON fields; the mutation returns an error directing the operator to `updateComponentSecrets`. Empty-string and `null` placeholders are still permitted as schema markers, paired with the secret-wins merge in `get_parser_kwargs`. Closes the foot-gun where the now-fixed merge bug pressured operators to copy real API keys into `parser_kwargs` (stored unencrypted, returned by GraphQL queries, more likely to leak into logs). The error message echoes the offending field name only — never the supplied value. Tests in `PipelineSettingsGraphQLTestCase` cover both `parser_kwargs` and `component_settings` rejection paths and the empty-placeholder allow-list.
- **WebSocket authentication tokens no longer travel in URL query strings**
  (issue: JWT leakage via nginx logs, browser history, `Referer` headers,
  Sentry breadcrumbs, and CDN/WAF logs). All WS connections now authenticate
  via the standard `Sec-WebSocket-Protocol` handshake header
  (`opencontracts.jwt.v1, <jwt>`). Token rotation is in-band via
  `{"type":"AUTH","token":"..."}` frames — no socket churn on refresh. Hard
  cutover: `?token=` URL parameters are stripped by the new middleware
  (`config/websocket/middleware.py`) and ignored. `Authorization: Bearer`
  headers are also no longer consulted for WS auth (browsers cannot set them
  on the WebSocket constructor anyway). Stale browser tabs from before the
  deploy must reload to recover.

  - **`AuthHandshakeMixin`** (`config/websocket/auth_handshake.py`) — added
    to all three consumers. Refuses user-pk swap mid-connection
    (`USER_MISMATCH` → close 4002), re-runs resource permission checks on
    refresh (`PERMISSION_REVOKED` → close 4003), supports server-nudged
    refresh via `AUTH_REFRESH_REQUIRED` with a grace timer that closes 4001
    on timeout.

  - **Frontend `useWebSocketAuth` hook**
    (`frontend/src/hooks/useWebSocketAuth.ts`) — single shared lifecycle
    owner used by `useAgentChat`, `useNotificationWebSocket`, and
    `CorpusChat`. Token rotation no longer reconnects the socket.

  - **Removed**: deprecated `getDocumentQueryWebSocket` and
    `getCorpusQueryWebSocket` URL helpers (forwarded to the unified
    endpoint and now gone per the no-dead-code rule). Removed the
    `autoReconnect` / `reconnectDelay` options on `useNotificationWebSocket`
    — reconnect is owned by the shared hook.

  - **Hook return-shape change** (`useNotificationWebSocket`): the previous
    `{ connect, disconnect }` actions are replaced by `{ reconnect }`. The
    socket is now opened by the shared `useWebSocketAuth` lifecycle on mount
    and closed on unmount; manual open/close was redundant. External
    consumers that imported the hook directly will fail typecheck.

  - **Tests**: `opencontractserver/tests/test_websocket_auth.py` gains four
    new test classes (`JWTAuthMiddlewareSubprotocolTests`,
    `AuthHandshakeMixinTests`, `UnifiedAgentHandshakeTests`,
    `ThreadUpdatesHandshakeTests`, `NotificationUpdatesHandshakeTests`) —
    ~30 cases covering middleware, mixin, per-consumer integration,
    user-mismatch refusal, and `?token=` regression. Frontend adds
    `useWebSocketAuth.test.ts` and `websocketAuth.test.ts`; the existing
    `useNotificationWebSocket.auth.test.ts` is rewritten as a no-token-in-URL
    regression suite.

  - **Manual test script**:
    `docs/test_scripts/websocket-auth-handshake.md` — verifies subprotocol
    transport, in-band refresh, user-pk-swap refusal, and DevTools sanity
    check that the JWT never appears in the request URL.

  - **Full-stack E2E spec**:
    `frontend/tests/e2e/websocket-auth.spec.ts` drives the real-browser →
    Vite proxy → Daphne ASGI → consumer pipeline for all three websocket
    consumers. Asserts on actual frames captured via Playwright's
    `page.on('websocket')` so the wire protocol (subprotocol echo, AUTH_OK,
    NOTIFICATION_CREATED, in-band refresh) is the source of truth — no
    mocks. Triggers a real BADGE notification via `docker compose exec` to
    prove signal-driven broadcast end-to-end. Companion CI workflow
    `.github/workflows/frontend-e2e-websocket.yml` boots the local stack
    (Daphne, not runserver), runs under VCR replay so the LLM-touching
    test needs no real OpenAI key, and uploads coverage to Codecov under
    flags `frontend-e2e,frontend,websocket` and `backend-e2e,websocket`.
    The agent-chat path of `UnifiedAgentConsumer` (`_stream_agent_response`

    - `_handle_approval_decision`) is now wrapped in `maybe_vcr_cassette()`
      so the cassette covers both branches.

  - **Browser close-code surface is documented**: real browsers see close
    code `1006` (abnormal closure) for any auth rejection that happens
    before the consumer calls `accept()` — Channels translates a pre-accept
    `close()` into an HTTP 403 handshake rejection, which never produces
    a WebSocket close frame with the application code. The
    `WebsocketCommunicator` tests still see `4001`/`4002`/`4003` because
    Channels exposes the intended code to its test runner. The new e2e
    spec asserts the real browser surface (1006) for all pre-accept paths
    so a future change to accept-then-close (which would let the browser
    see the application code) is caught and the test gets updated.

  - **Per-connection AUTH-frame cooldown** in `AuthHandshakeMixin`: a
    1-second floor between accepted AUTH frames per socket prevents a
    malicious client from spamming refresh frames to burn DB queries on
    `_get_user_from_token` + `_validate_resource_permissions`. Auth0 silent
    renewal happens roughly every 50 minutes, so legitimate refreshes are
    unaffected.

  - **Auth-failure close codes are terminal in the frontend**:
    `useWebSocketAuth` treats close codes 4000 (UNAUTHENTICATED), 4001
    (TOKEN_EXPIRED), and 4002 (TOKEN_INVALID) as auth-invalid signals —
    `onAuthInvalid` fires once and the hook stops spawning new sockets.
    Previously 4000/4001 fell through to exponential-backoff reconnect,
    which would just be rejected again in a loop until the user
    re-authenticated. The hook also no longer returns the underlying
    `WebSocket` reference (it was stale on the first render), to remove a
    footgun for downstream consumers.

  - **`CorpusChat` and `ChatTray` migrated to `useWebSocketAuth`**: both
    chat surfaces previously instantiated raw `new WebSocket(url)` without
    subprotocols, so authenticated users would have hit the new middleware
    as anonymous after this deploy. Both now compose the shared hook
    (subprotocol auth on connect, in-band AUTH refresh on token rotation,
    close-code-aware reconnect policy).

  - **Backend `receive()` cleanup**: each consumer now parses the incoming
    text frame once and dispatches AUTH/non-AUTH from the same payload
    (was double-parsing). `thread_updates.py` consolidates redundant
    nested imports of `Conversation` / `Corpus` / `Document` to module
    level.

### Changed

- **Decluttered floating layout around the chat tray input** (`frontend/src/components/knowledge_base/document/FloatingDocumentControls.tsx`, `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`). When the right-hand sliding panel is open in chat mode, the document-tool FABs (extracts, analyses, create-analysis) are now hidden so they no longer crowd the panel boundary next to the chat input and sidebar tabs. The panel-width control remains visible since it directly affects the open panel. Closing the chat tray (or switching the sidebar to a non-chat view) restores the FAB stack. Implemented via a new `hideDocumentTools` prop on `FloatingDocumentControls`, applied to both the desktop FAB column and the mobile speed-dial entries.
- **Tightened chat tray input container so it stops drifting past the panel edge** (`frontend/src/components/knowledge_base/document/ChatContainers.tsx`). The `ChatInputContainer` had `padding: 1.25rem 1.5rem`, a `gap: 0.875rem`, and the `ChatInputWrapper` carried a `2px` border with a `0 0 0 4px` focus halo. On narrower panel widths the halo bled past the wrapper bounds and got clipped by `ChatContainer`'s rounded right edge, while the send button ended up hugging the panel wall. Reduced container padding to `0.875rem 1rem` (with an `env(safe-area-inset-bottom)` aware bottom inset for mobile), shrank the gap to `0.625rem`, dropped the wrapper border to `1px` / `12px` radius, and tightened the focus halo to `0 0 0 2px`. Added `min-width: 0` to both the container and the wrapper plus `flex: 1 1 0` on the wrapper so the textarea actually shrinks within flex instead of forcing horizontal overflow.
- **Corpus-import button gating** (`frontend/src/views/Corpuses.tsx`). The "Import" action (both the dropdown action and the list-view button) is now gated on `REACT_APP_ALLOW_IMPORTS && backendUser.canImportCorpus` instead of `REACT_APP_ALLOW_IMPORTS && Boolean(currentUser)`. Deploy-level kill switch is preserved; user-level permission now matches the backend check. The legacy `corpusUploadRef` / `onImportFileChange` / inline `START_IMPORT_CORPUS` mutation hook in `Corpuses.tsx` are removed in favor of the new modal.
- **`StartImportCorpusExport` TypeScript shape corrected** (`frontend/src/graphql/mutations.ts`). The output type now correctly nests `{ ok, message, corpus }` under `importOpenContractsZip` to match the GraphQL mutation response (the previous flat shape was technically wrong but never accessed because the only consumer relied on `onCompleted` callbacks).
- **Slimmed Django/Celery production image from ~6.3 GB to a target ≤1.5 GB** (Issue #1494). The image previously inherited from `pytorch/pytorch:2.7.1-cuda12.6-cudnn9-runtime`, dragging in CUDA 12.6, cuDNN 9, and a full PyTorch installation that nothing in the Django runtime imports — the only consumer was the optional in-process `CrossEncoderReranker`, which has been removed (see _Removed_ below). Cold pod pulls dropped from ~5+ minutes to seconds, dramatically narrowing the recovery window when nodes reschedule.
  - `compose/production/django/Dockerfile` and `compose/local/django/Dockerfile` now base on `python:3.11-slim-bookworm` for both build and runtime stages. The build toolchain (`build-essential`, `cmake`, `libtesseract-dev`, `libpq-dev`, etc.) is confined to the builder stage; the runtime stage installs only the libraries actually needed at execution time (`libpq5`, `gettext`, `poppler-utils`, `tesseract-ocr` + `tesseract-ocr-eng`, `libgl1`, `libsm6`, `libxext6`).
  - Dead CUDA environment variables (`CUDA_MODULE_LOADING`, `TORCH_CUDA_ARCH_LIST`, `CUDA_VISIBLE_DEVICES`, `PYTORCH_CUDA_ALLOC_CONF`) and the now-unused `ffmpeg` apt package were removed from both Dockerfiles.
  - `.dockerignore` was significantly tightened — `frontend/`, `fixtures/`, `docs/`, `cloudflare-og-worker/`, `tools/`, `model_preloaders/`, `locale/`, `.github/`, `.vscode/`, `.cursor/`, `.claude/`, `.envs/.local/`, `*.md` (except `CHANGELOG.md`), and Python caches are now excluded from the build context, both shrinking image size and reducing accidental secret/source leakage into runtime images.
  - **CI image-size budget** added to `.github/workflows/docker-build-release.yml`: the release build now fails if the Django image exceeds 1.5 GiB (`1610612736` bytes), surfacing regressions immediately rather than at the next cold pod pull.
- **Celery configured for at-least-once delivery so worker death no longer silently loses tasks** (Issue #1493, `config/settings/base.py`). Previously Celery acked messages on receive (at-most-once): if a worker died mid-task — OOM, host failure, deploy, eviction, SIGKILL — the broker had already removed the message and the task was silently lost, leaving long-running ingest/parse/embed work with documents stuck at `backend_lock=True` and no parsed content. With the new settings the broker only acks after the task returns successfully, and hard-killed workers cause redelivery rather than silent disappearance. The redundant per-task `acks_late=True` on `opencontractserver/worker_uploads/tasks.py` is left in place as an explicit declaration of intent (it is now equivalent to the global default). Documented in `docs/architecture/asynchronous-processing.md`, including the contract, known non-idempotent tasks to harden in follow-up work, and the per-task opt-out for tasks that genuinely cannot be made idempotent.

  **Technical details**

  - `CELERY_TASK_ACKS_LATE = True` — broker only removes the message after the task returns successfully.
  - `CELERY_TASK_REJECT_ON_WORKER_LOST = True` — SIGKILL'd workers cause redelivery rather than silent task loss.
  - `CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": CELERY_REDIS_VISIBILITY_TIMEOUT_SECONDS}` (12 hours) — prevents Redis from double-delivering tasks that run longer than the 1-hour Celery default. The constant lives in `opencontractserver/constants/celery.py` (Issue #1504).
  - **Trade-off**: at-least-once delivery means tasks may run twice. All tasks must be idempotent; per-task opt-out is available via `@shared_task(acks_late=False, reject_on_worker_lost=False)`.
  - Regression test in `opencontractserver/tests/test_celery_worker_death_resilience.py` pins all three settings _and_ verifies the namespaced Django→Celery wiring propagates each of them into `celery_app.conf`, so a future settings refactor cannot silently revert the resilience guarantee (Issue #1504).

### Removed

- **In-process `CrossEncoderReranker`** (Issue #1494). The lazily-loaded `sentence_transformers.CrossEncoder` reranker added in PR #845 was the sole reason the Django/Celery image inherited a CUDA-enabled PyTorch base (~4–5 GB of bloat for a feature that few deployments enabled). Reranking is still available via `MicroserviceReranker` (HTTP side-car) and `CohereReranker` (API), both of which are framework-agnostic and require no in-container model weights.
  - Deleted `opencontractserver/pipeline/rerankers/cross_encoder_reranker.py`. The pipeline registry auto-discovers reranker classes from the package, so removal cleanly drops `CrossEncoderReranker` from `get_registry().rerankers` with no further wiring changes.
  - Deleted `CrossEncoderRerankerTest` (`opencontractserver/tests/test_reranker.py`) and `CrossEncoderRerankerTests` (`opencontractserver/tests/test_data_extract_helpers.py`). Both used in-process stubs of `_load_cross_encoder` to avoid downloading weights in CI; with the backend gone, their coverage target is gone too. The orphaned `from unittest.mock import MagicMock` import in `test_data_extract_helpers.py` was removed.
  - Removed orphaned configuration: `SENTENCE_TRANSFORMER_MODELS_PATH` (`config/settings/base.py:986-988`) and the matching `SENTENCE_TRANSFORMER_MODELS_PATH=/models/sentence-transformers` line in `.github/workflows/production-stack.yml:107` were the only consumers of the deleted backend's model-cache directory.
  - Removed the `tokenizers>=0.23.1,<0.24` pin from `requirements/base.txt`. Its inline comment ("Pin to prevent conflicts with transformers") referred to a transitive constraint that was only relevant while `sentence-transformers` was installed; nothing in the codebase imports `tokenizers` directly.

### Fixed

- **`brotli` decoder added to runtime deps** (`requirements/base.txt`). The previous PyTorch base image shipped `brotli` incidentally via conda, so `httpx` registered a `br` decoder and transparently decompressed `Content-Encoding: br` responses. The new `python:3.11-slim-bookworm` base does not, leaving `httpx._decoders.SUPPORTED_DECODERS` at `{identity, gzip, deflate, zstd}` only. Without a brotli decoder, httpx returns the still-compressed bytes to callers and the OpenAI SDK then raises `json.JSONDecodeError: Expecting value: line 1 column 1 (char 0)` when it tries to parse them. This surfaced as `test_structured_response_uses_document_tools` (the only VCR cassette in the repo recorded with `Content-Encoding: br`) failing on the slim image; live OpenAI traffic was equally affected whenever the API negotiated brotli. Adding `brotli>=1.1.0` re-enables `httpx`'s `br` decoder for both VCR replay and production HTTP.
  - Deleted `.github/workflows/docker-build-cuda.yml`. The CUDA-tagged image build had no remaining purpose once the only torch consumer was removed.
  - Deleted `docs/deployment/docker-gpu-setup.md` and its `mkdocs.yml` nav entry. The guide documented the now-removed CUDA/PyTorch base image and was the only page under the _Deployment_ nav section.

### Added

- **Pre-fill defaults in the new-corpus modal**:

  - **Default LabelSet seeded at install** (`opencontractserver/annotations/label_set_seeds.py`, migration `annotations/0069_labelset_is_default_and_seed_default.py`): a public `LabelSet` titled "Default Labels" is now created on install (owned by the first superuser) with a starter palette aimed at corporate operations review — three general research labels (Important, Question, Reference) plus a light contract palette (Definition, Party, Governing Law, Effective Date, Termination Date, Expiration, Termination, Renewal, Limitation of Liability). New `LabelSet.is_default` BooleanField with a partial unique constraint ensures at most one labelset is flagged as the default. Idempotent management command `python manage.py seed_default_labelset` (`opencontractserver/annotations/management/commands/seed_default_labelset.py`) re-runs the seed safely after the first superuser is created and appends any newly-added labels to an already-seeded labelset.
  - **`defaultLabelset` GraphQL query** (`config/graphql/annotation_queries.py`): returns the install-wide default LabelSet visible to the current user (or null). `LabelSetType` exposes the new `is_default` field via `DjangoObjectType`'s default field set.
  - **`CreateCorpusMutation` auto-fills `label_set`** (`config/graphql/corpus_mutations.py`): when the mutation is called without a label_set the install-wide default is substituted, so corpuses created via the API land with a usable starter palette. Defaulting is intentionally at the mutation layer, not in `Corpus.save()`, so direct ORM creates in tests/scripts remain opt-in.
  - **CorpusModal CREATE mode pre-fills** (`frontend/src/components/corpuses/CorpusModal.tsx`, `frontend/src/graphql/queries.ts`): on open, the modal fetches `pipelineSettings.defaultEmbedder` + `defaultLabelset` via a single `GET_CORPUS_CREATE_DEFAULTS` query and seeds the embedder and labelset selectors. License pre-selects `CC-BY-4.0` via the new `DEFAULT_NEW_CORPUS_LICENSE` constant in `frontend/src/assets/configurations/constants.ts`. All three pre-fills are pure UX nudges — users can clear them before submitting.

- **Cross-content Discover search** at `/discover/search`. The Discover hero
  search box now lands on a unified results page that fans out across
  discussions, annotations, collections (corpuses), and notes in parallel,
  rather than redirecting to discussion-only search.

  - Backend: new `searchNotesForMention` resolver in
    `config/graphql/search_queries.py` (visibility via
    `Note.objects.visible_to_user`, optional `corpusId` / `documentId`
    scoping, eager-loads `document`, `corpus`, and creators).
    `NoteType` is exported through the same `DjangoConnectionField` shape
    used by the other mention searches and goes through `NoteType.get_queryset`
    for a second visibility pass.
  - Frontend view: `frontend/src/views/DiscoverSearchResults.tsx` reuses
    `ThreadListItem` for discussions and a single shared `ResultRow` for
    annotations / collections / notes. Tabbed UI: All (preview of 5 each),
    Discussions, Annotations, Collections, Notes.
  - Routing: `?q=` and `?type=` are URL-driven (replace, no history spam).
    Result rows deep-link via `getCorpusUrl` / `getDocumentUrl`; annotation
    rows preserve `?ann=`, note rows pass `?note=` to the document URL.
  - New `?note=<id>` deep-link param wired through `selectedNoteId`
    (`frontend/src/graphql/cache.ts`), `QueryParams.noteId`
    (`frontend/src/utils/navigationUtils.ts`), and the central manager
    Phase 2/4 sync (`frontend/src/routing/CentralRouteManager.tsx`).
    `DocumentKnowledgeBase` now consumes the var: when the loaded
    document contains a note matching the URL id, the existing note
    detail modal opens once and the param is consumed (one-shot
    deep-link, mirrored back to the URL via Phase 4).
  - Hero wiring: `NewHeroSection.handleSearchSubmit` now navigates to
    `/discover/search?q=...`. The legacy `/discussions?search=...` route is
    preserved for callers that want a discussion-only listing.
  - Tests: `MentionSearchTestCase` in
    `opencontractserver/tests/test_mentions.py` covers visibility
    filtering, content-substring matching, corpus / document scoping,
    anonymous-user filtering, wrong-type global-id rejection, the
    `-modified` ordering contract, and both
    `content_preview` paths (DB-annotated `Left('content', 400)` for
    search results, Python fallback for per-id note fetch).
  - Component test: `frontend/tests/DiscoverSearchResults.ct.tsx`
    covers the empty-prompt state, the four-section header render
    after typing, and a populated-results render exercising every
    section's row primitives. Captures both
    `discover--search-results--empty-prompt.png` and
    `discover--search-results--with-results.png` for docs.

- **Loud guardrail against the `system_prompt=` foot-gun in pydantic-ai** (Issue #1451): `pydantic_ai.Agent` accepts both `system_prompt=` and `instructions=`, but the `system_prompt` value is _only_ materialised into the model request when `message_history` is `None`. OpenContracts' `chat()` flow always persists the user's HUMAN message before calling `Agent.run()`, so `message_history` is never empty in practice and any `system_prompt=` argument is silently dropped — the LLM runs without any system instruction. CLAUDE.md pitfall #14 documented the workaround (use `instructions=`), but a future pydantic-ai bump that renames or re-precedences these parameters could re-introduce the regression silently.

  - **Single construction path** (`opencontractserver/llms/agents/pydantic_ai_factory.py`): new `make_pydantic_ai_agent(...)` factory is now the only place in the codebase that instantiates `pydantic_ai.Agent`. The factory uses a sentinel-based check (not `is not None`) to refuse `system_prompt=` outright — even `system_prompt=None` raises `TypeError` so the lesson cannot be re-learned by accident. The error message references issue #1451 and CLAUDE.md pitfall #14.
  - **All call sites refactored** (in `opencontractserver/llms/agents/pydantic_ai_agents.py`: `_run_structured_extraction`, the document-agent factory, and the corpus-agent factory; in `opencontractserver/tasks/memory_tasks.py`: `summarise_agent` and `curation_agent`). Five direct `PydanticAIAgent(...)` constructions in production code now route through the factory.
  - **Self-contained factory** (`opencontractserver/llms/agents/pydantic_ai_factory.py`): the factory imports `from pydantic_ai.agent import Agent as PydanticAIAgent` directly and calls `PydanticAIAgent(...)` with no sideways indirection through `pydantic_ai_agents`. Tests that need to intercept agent construction now patch `opencontractserver.llms.agents.pydantic_ai_factory.PydanticAIAgent` (44 patch sites updated across 7 test files: `test_pydantic_ai_agents.py`, `test_nested_approval_gates.py`, `test_agent_memory.py`, `test_duplicate_tool_registration.py`, `test_tool_approval_gate.py`, `test_long_conversation_api.py`). The chokepoint is now both production-code clean and test-visible at the same path.
  - **Version-pinning regression test** (`opencontractserver/tests/test_pydantic_ai_factory.py`): four tests cover the loud-failure paths (`system_prompt=<str>`, `system_prompt=None`) and the precedence behaviour itself — `test_instructions_survive_non_empty_message_history` constructs an agent with `instructions=<sentinel>` via the factory, runs it against `pydantic_ai.models.test.TestModel` with a non-empty `message_history`, and asserts the sentinel is delivered to the model (either as a `SystemPromptPart` or via `ModelRequest.instructions`). If a future pydantic-ai release changes precedence so `instructions=` is also dropped under non-empty history, this test fails loudly so the regression is caught before silently shipping.
  - **Dependency pin commentary** (`requirements/base.txt:51`): the `pydantic-ai>=1.89.1,<2` line now carries a multi-line comment documenting that this codebase relies on the current `instructions=` precedence rule and pointing future maintainers at the regression test before widening the upper bound.

- **E2E spec for threads/discussions** (`frontend/tests/e2e/threads-discussions.spec.ts`):
  - Anonymous pass renders `/discussions` (filter pills, search box, "Corpus Discussions" section header) and `/threads` (search route) without authentication.
  - Authenticated pass logs in, creates a per-run corpus (suffixed with `Date.now()` to avoid collisions across retries), opens its inline discussions view via `?view=discussions`, fills the `CreateThreadForm` modal (title, optional description, ProseMirror initial message), posts a top-level reply through the `ReplyForm` composer, navigates back to the thread list, and confirms the new thread surfaces in the global `/discussions` "Corpus Discussions" section before clicking through to the detail.
  - Picked up automatically by the existing `frontend-e2e.yml` workflow — no CI changes required.
- **Three reusable E2E helpers** in `frontend/tests/e2e/helpers.ts`:
  - `openCorpusDiscussionsViaUI` — SPA-navigates to a corpus and opens its inline discussions view; waits on the "All" filter pill as the readiness signal.
  - `createThreadViaUI` — clicks the `aria-label="Create new discussion"` CTA, fills the modal (scoped via the `#thread-title` anchor), submits the composer, and waits for the thread-detail header.
  - `postThreadReplyViaUI` — types into the bottom `ReplyForm` ProseMirror editor and clicks Send, returning once the new message text is visible.

### Removed

- **Stray planning artifacts at repo root** (Issue #1453): deleted `plan.md` (CorpusHome query-performance optimization plan) and `plans/routing-audit-report.md` (frontend routing audit). Both described work that has already shipped — `CountableConnection.resolve_total_count()` already calls `.count()` directly (`config/graphql/base.py:88`) and `browseRoutes` already includes `"discussions"` (`frontend/src/utils/navigationUtils.ts:177`). The `plans/` directory is now gone; future planning docs belong under `docs/plans/` or `docs/architecture/` per existing conventions.

### Fixed

- **Discover search section header miscounted threads when soft-deleted ones were present** (`frontend/src/views/DiscoverSearchResults.tsx`): the discussions section displayed `data.conversations.totalCount` as the count, but rendered rows are filtered client-side via `!n?.deletedAt`. With even one tombstoned thread in the results, the header read e.g. "5 threads" while only 4 rows appeared. Switched the count to `threads.length` (post-filter) so the badge always matches what the user sees.

- **`?note=<id>` deep-link could pin in the URL forever for documents with no notes** (`frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`): the auto-open effect early-returned on `notes.length === 0`, which is _also_ true while the document query is loading. The intent comment said "Clear regardless of match" but the early return skipped the clear call entirely. If the underlying document genuinely had no notes (or the loaded notes simply did not contain the deep-linked id) the reactive var stayed set, so `CentralRouteManager` Phase 4 kept writing `?note=<id>` back into the URL on every render — the deep-link became sticky with no user-visible escape hatch. Effect now gates only on `combinedData?.document` being loaded; once the query has resolved, the effect runs `find` (possibly empty) and always clears the var.

- **`NewHeroSection` had a dead `isAuthenticated` prop** (`frontend/src/components/landing/NewHeroSection.tsx`, `frontend/src/views/DiscoveryLanding.tsx`): the prop was declared on the interface and threaded through `DiscoveryLanding` but never read inside the component. Removed from the interface and the call site.

- **Discover search rows silently no-op'd on `"#"` URLs from missing slugs** (`frontend/src/views/DiscoverSearchResults.tsx`): `getDocumentUrl` / `getCorpusUrl` return `"#"` when slugs are missing on the underlying entities. Each section's `onClick` did `if (url !== "#") navigate(url)`, so the row remained visually clickable, hover styles fired, and the click was swallowed with no feedback. `ResultRow` now accepts `disabled` / `disabledReason`; sections compute `unrouteable = url === "#"` once and forward both. Disabled rows render with `opacity 0.55`, `cursor: not-allowed`, native `disabled` (and `aria-disabled`), and a tooltip explaining the missing data — replacing the silent failure with a discoverable one. `disabled:not(:disabled)` scoping on the hover style stops the disabled rows from animating on hover.

- **Annotation deep-links from the corpus-home Table of Contents silently no-op'd** (`frontend/src/components/corpuses/DocumentAnnotationIndex.tsx`, `frontend/src/components/knowledge_base/document/document_kb/RightPanelContent.tsx`): clicking a structural section in the corpus-home document index (e.g. "Subchapter I. Formation, p. 2") was supposed to open the document with the annotation pre-selected and scrolled into view. Instead it appeared to do nothing. Root cause: `DocumentAnnotationIndex` overloaded a single `embedded` prop with two semantics — visual layout ("render without an outer container") _and_ click routing ("we are already on the document page, just rewrite `?ann=`"). The corpus-home call site (`DocumentTableOfContents.tsx:919`) needed the visual flavor but absolutely was _not_ on a document page, so `handleSectionClick` took the wrong branch and wrote `?ann=<id>` onto the corpus URL — no navigation, no scroll. Fix splits the prop: `embedded` is now purely visual, and a new explicit `onDocumentPage` prop controls click routing. The single call site that's actually on a document page (`RightPanelContent.tsx`) opts in. Regression test in `frontend/src/components/corpuses/__tests__/DocumentAnnotationIndex.test.tsx` pins the new contract: a click from a corpus URL must produce a string-form `navigate("/d/.../doc?ann=<id>")` (full deep link), while a click from a document URL with `onDocumentPage` produces `navigate({ search: "...ann=<id>..." }, { replace: true })`.

- **Zip importer reported `success: True` even when sidecars failed (silent annotation loss)** (`opencontractserver/tasks/import_tasks.py:421`, `:1411`): `_read_sidecar` raises `ValueError` when a sidecar exceeds `ZIP_MAX_SIDECAR_SIZE_BYTES`; malformed JSON, schema failures, and missing labels for sidecar-declared annotations all bump `annotation_sidecars_errored` and append to `errors`. The success determinations only checked `files_errored` (`import_zip_with_folder_structure`) or the user-cap message (`process_documents_zip`), so callers observed `success: True, completed: True` while annotations were silently dropped — exactly the silent data-loss path called out in PR #1489 review feedback. `import_zip_with_folder_structure` now requires `annotation_sidecars_errored == 0` in addition to the existing `files_errored == 0` and user-cap check; `process_documents_zip` now requires `error_files == 0` in addition to the user-cap check. `relationship_errors` is intentionally not folded in — those are surfaced separately via `relationships_skipped` + `relationship_errors` and the documents themselves are imported correctly. Two tests in `test_sidecar_import.py` (`test_skip_pipeline_without_labels_json`) and a new regression test (`test_sidecar_error_drops_overall_success_flag`) lock down the new contract.

- **`_create_analysis_notification` was reading a non-existent field on `Analyzer`** (Issue #1471, `opencontractserver/analyzer/views.py:50`). The notification builder for completed/failed analyses was reading `analysis.analyzer.analyzer_id`, but `Analyzer` has no such field — its primary key is `id` (a `CharField(max_length=1024, primary_key=True)`). On a real ORM instance every successful or failed remote-analyzer notification raised `AttributeError` inside the builder before the `Notification` row was written. Production tests in `opencontractserver/tests/test_job_notifications.py` masked the bug by mocking the entire `Analysis` instance with `MagicMock()` and setting `analysis.analyzer.analyzer_id = "test-analyzer"` (MagicMock auto-creates any attribute on access). Fix replaces the bad attribute with `analysis.analyzer.id`. A follow-up integration test using a real `Analyzer` instance is tracked in the issue.
- **Preferred Embedder dropdown showed disabled embedders for superusers** (`frontend/src/components/widgets/CRUD/EmbedderSelector.tsx`, `frontend/src/graphql/queries.ts`): the create/edit corpus form's `EmbedderSelector` listed every embedder returned by `pipelineComponents.embedders`. The backend resolver in `config/graphql/pipeline_queries.py` only filters by configured/preferred class names for non-superusers, so a superuser saw embedders that `PipelineSettings.enabled_components` had explicitly disallowed and could pick one the rest of the system would refuse to use. The `enabled` flag was already computed and exposed on `PipelineComponentType` but the `GET_EMBEDDERS` query did not request it. Added `enabled` to the query and filter `embedder.enabled !== false` on the client (`undefined` from older backends still passes through). Test mocks in `frontend/tests/EmbedderSelector.ct.tsx` and `frontend/tests/corpus-modal.ct.tsx` updated, and a new test case verifies a `enabled: false` embedder is omitted from the dropdown.

### Changed

- **Typing: graduate `opencontractserver.documents.*` from mypy baseline** (refs Issue #1447): removed eight `[mypy-…]` `ignore_errors` blocks (`opencontractserver.documents.{admin,checks,models,query_optimizer,versioning}`, `documents.tests.test_pdf_hash`, `documents.management.commands.{compute_document_hashes,migrate_pipeline_settings}`); added a scoped `disable_error_code = django-manager-missing` block for `documents.models` (DocumentPath self-relation `path_records` / `children` and `IngestionSource.document_paths` are django-tree-queries reverse-relation surfaces django-stubs cannot resolve). Pruned 40 baseline error lines from `docs/typing/mypy_baseline.txt` (7124 → 7084). Per-file fixes:

  - `documents/versioning.py` — replaced the runtime `User = get_user_model()` with a TYPE_CHECKING import of `opencontractserver.users.models.User` and quoted six `user: "User"` annotations; widened `move_document(new_folder=...)` from `Optional[CorpusFolder] = "UNSET"` to `Optional[CorpusFolder] | str = "UNSET"` and added an `assert not isinstance(new_folder, str)` after the sentinel check so `folder_to_use: Optional[CorpusFolder]` narrows correctly.
  - `documents/models.py` — scoped `# type: ignore[misc]` on the `objects = DocumentManager()` manager override; added `author_obj: AbstractBaseUser` annotation in `Document.update_summary` plus `# type: ignore[misc]` on the corresponding `author=author_obj` kwarg passed to `DocumentSummaryRevision.objects.create(...)`; typed two `all_components: list[Any] = []` accumulators in pipeline-component check helpers.
  - `documents/query_optimizer.py` — typed `result: dict[str, list[Any]] = {...}` in the related-object collector; added `Any` to the `typing` import.
  - `documents/checks.py` — typed three `errors: list[Warning] = []` accumulators (one widened to `list[Error | Warning]` because the body appends both severity classes).
  - `documents/admin.py` — scoped `# type: ignore[attr-defined]` on the three Django-admin `method.short_description = "..."` / `admin_order_field = "..."` annotations.
  - `documents/tests/test_pdf_hash.py` — added narrowing `assert computed_hash is not None` before the SHA-256 length assertion; cast `_meta.get_field("pdf_file_hash")` to `Any` in the field-introspection helper since the abstract `Field` return type doesn't expose `db_index` / `max_length`.
  - `documents/management/commands/compute_document_hashes.py` — added `assert document.pdf_file.name is not None` before `storage.exists(...)`; coalesced `old_hash[:8]` to `(old_hash or "")[:8]` for the no-change branch.
  - `documents/management/commands/migrate_pipeline_settings.py` — typed two `all_components: list[Any] = []` accumulators; widened the heterogeneous `stats` dict to `dict[str, Any]`; renamed the inner iteration variable to `group_components` and rebound `components: list[Any] = list(group_components)` so the list-comp re-assignment doesn't conflict with the source tuple type.

- **Annotation sidecar size limit raised and made env-configurable** (`opencontractserver/constants/zip_import.py:49`, `config/settings/base.py:730`): the per-sidecar JSON cap during zip import was 10 MB, which silently dropped annotations for larger corpora — `_read_sidecar` (`opencontractserver/tasks/import_tasks.py:572`) raised `ValueError` and the importer recorded it under `errors` while still returning `success: True, completed: True`, so callers had no signal that annotations were lost. Default raised to 50 MB and the setting is now read from the `ZIP_MAX_SIDECAR_SIZE_BYTES` env var via `django-environ` in `config/settings/base.py`, mirroring the pattern used for `MAX_WORKER_UPLOAD_SIZE_BYTES` and friends. The constants module's `getattr(settings, ...)` fallback was bumped to 50 MB so non-Django importers see the same default. Existing tests in `opencontractserver/tests/test_sidecar_import.py` patch the constant directly and are unaffected.

- **Remaining zip-import limits are now env-configurable** (`config/settings/base.py:730`, `opencontractserver/constants/zip_import.py`): `ZIP_MAX_FILE_COUNT`, `ZIP_MAX_TOTAL_SIZE_BYTES`, `ZIP_MAX_SINGLE_FILE_SIZE_BYTES`, `ZIP_MAX_COMPRESSION_RATIO`, `ZIP_MAX_FOLDER_DEPTH`, `ZIP_MAX_FOLDER_COUNT`, `ZIP_MAX_PATH_COMPONENT_LENGTH`, `ZIP_MAX_PATH_LENGTH`, and `ZIP_DOCUMENT_BATCH_SIZE` previously honored Django settings overrides via `getattr(settings, ...)` in the constants module but were not surfaced in `base.py`, so operators could only override them in code, not via environment variables. Each now has a matching `int(env(...))` entry in `base.py` with the same default the constants module already used, and the constants-module docstring documents the env-var path. No behavior change at the defaults.

- **Typing: graduate `opencontractserver.{agents,analyzer,discovery.views}` from mypy baseline** (refs Issue #1447): removed seven `[mypy-…]` `ignore_errors` blocks (`opencontractserver.agents.{admin,memory,models}`, `opencontractserver.analyzer.{utils,views}`, `opencontractserver.discovery.views`) and pruned 32 baseline error lines from `docs/typing/mypy_baseline.txt`. The `discovery.tests.test_discovery_views` test module stays on the baseline — its 49 errors come from `setUpTestData` class-attribute assignments that mypy can't follow without per-test refactoring. Per-file fixes:

  - `analyzer/views.py` — fixed real bug: replaced `analysis.analyzer.analyzer_id` with `analysis.analyzer.id` (see Fixed above).
  - `analyzer/utils.py` — added `# type: ignore[assignment]` on the `get_doc_analyzer_task_by_name = None` ImportError fallback (mypy sees the original as `Callable`); switched the gating check from `not get_doc_analyzer_task_by_name` (truthy-function warning) to `is None`; cast `AnalyzerManifest` to `dict(...)` before `is_dict_instance_of_typed_dict()` since the helper expects a plain dict at runtime.
  - `agents/models.py` — scoped `# type: ignore[override]` on two `visible_to_user(user)` overrides in `AgentConfigurationManager` / `AgentActionResultManager` (django-stubs flags signature drift from `BaseVisibilityManager`); scoped `# type: ignore[misc]` on the two `objects = ...Manager()` declarations (manager class-vs-instance reassignment); added narrowing assertion on `self.corpus.title` in `AgentConfiguration.__str__` (the model `CheckConstraint` already enforces `corpus__isnull=False` when `scope == "CORPUS"`).
  - `agents/memory.py` — added narrowing `assert corpus.memory_document is not None` in the `IntegrityError` re-read branch (the truthy `corpus.memory_document_id` already proves the FK is set); added `list[str]` annotation on `result_parts` in the no-query memory-injection branch.
  - `agents/admin.py` — scoped `# type: ignore[attr-defined]` on the eight `admin_method.short_description = "..."` and one `admin_order_field = "..."` Django-admin annotation patterns mypy can't model; collapsed `obj.document.title[:30]` truncation into a single `doc_title or ""` binding so the slice no longer hits `str | None`.
  - `discovery/views.py` — scoped `# type: ignore[attr-defined]` on `Document.objects.filter(...).search_by_embedding(...)` (django-stubs returns generic `QuerySet[Document, Document]`, not the project's `DocumentQuerySet` which mixes in `VectorSearchViaEmbeddingMixin`).

- **Typing: graduate `opencontractserver.corpuses.*` from mypy baseline** (refs Issue #1447): removed five `[mypy-…]` `ignore_errors` blocks (`opencontractserver.corpuses.{admin,checks,folder_service,managers,models}`) and pruned 166 baseline error lines from `docs/typing/mypy_baseline.txt`. Per-file fixes:

  - `corpuses/checks.py` — typed `errors: list[Warning] = []` accumulator.
  - `corpuses/folder_service.py` — replaced the `if TYPE_CHECKING: User = get_user_model()` runtime-call-as-type pattern with a direct `from opencontractserver.users.models import User` (cleared 50+ "User is not valid as a type" errors with one edit).
  - `corpuses/managers.py` — added `lightweight: bool = False` parameter to `CorpusActionExecutionManager.visible_to_user` and forwarded it to `super()`, aligning the signature with `BaseVisibilityManager` (proper signature fix instead of suppression).
  - `corpuses/admin.py` — scoped `# type: ignore[attr-defined]` on the eight `admin_method.short_description = "..."` and one `admin_order_field = "..."` Django-admin annotation patterns mypy can't model.
  - `corpuses/models.py` — added `author_obj: AbstractBaseUser` annotation in `update_description` so the `int → User` and `User → User` branches share a unified type; scoped `# type: ignore[misc]` on five `creator=user` kwargs and one `author=author_obj` kwarg passed to `Document` / `DocumentPath` / `CorpusDescriptionRevision.objects.create(...)` (django-stubs widens the FK target type to `User | Combinable` while the call sites pass `AbstractBaseUser`); scoped `# type: ignore[arg-type]` on `set_permissions_for_obj_to_user(user, ...)`; scoped `# type: ignore[misc]` on the lambda in `transaction.on_commit(...)` (mypy cannot infer the captured-default-arg lambda); added narrowing `assert document is not None` in the `remove_document` else-branch (the early `if not document and not path: raise` guard already establishes the invariant); scoped `# type: ignore[assignment]` on the `CorpusActionTemplate.creator = ForeignKey(..., null=True)` field redeclaration (intentional override of `BaseOCModel.creator` to use `SET_NULL`); scoped `# type: ignore[misc]` on the `objects = CorpusActionExecutionManager()` manager override; scoped `# type: ignore[return-value,arg-type]` on the `cls.objects.bulk_create(executions)` return in `bulk_queue_executions` (the `cls(...)` instances are typed as `Self` but the function explicitly returns `list[CorpusActionExecution]`); dropped two now-dead `# type: ignore[arg-type]` comments on `md_description.open(...)` calls and one on `descendants(include_self=True)`.

- **Typing: graduate `opencontractserver.llms.tools.*` from mypy baseline** (refs Issue #1447): removed seven `[mypy-…]` `ignore_errors` blocks (`core_tools` + `core_tools.*` wildcard, `image_tools`, `moderation_tools`, `pydantic_ai_tools`, `tool_factory`, `tool_registry`) and pruned 25 baseline error lines from `docs/typing/mypy_baseline.txt`. Per-file fixes:

  - `core_tools/_helpers.py` — dropped three dead `# type: ignore` comments on `channels.db` / `asgiref.sync` imports (the partial-call concern doesn't surface under current type stubs).
  - `core_tools/text_extracts.py` / `core_tools/descriptions.py` — dropped four dead `# type: ignore[arg-type]` comments on `FieldFile.read()` / `.open()` calls.
  - `core_tools/links.py` — added a trailing `raise ValueError(f"Unhandled entity_type: {entity_type!r}")` to both `create_markdown_link` and `acreate_markdown_link` so mypy sees the function as exhaustive (without converting an unreachable to an opaque `AssertionError`).
  - `core_tools/notes.py` — widened `search_document_notes` / `asearch_document_notes` return type from `list[dict[str, str | int]]` to `list[dict[str, str | int | None]]`. The previous annotation was a real lie: `note.created.isoformat() if note.created else None` could already produce `None` values.
  - `core_tools/document_summaries.py` — replaced `author or author_id` with an explicit `summary_author = author if author is not None else author_id` + narrowing assertion; the early `if author is None and author_id is None: raise ValueError(...)` guard establishes the invariant the assertion locks down.
  - `image_tools.py` — removed redundant `images: list[ImageData] = []` re-annotation that conflicted with an earlier `images = _load_images_from_annotation_file(...)` binding; scoped a `# type: ignore[attr-defined]` on `Document.objects.filter(...).visible_to_user(user)` (django-stubs returns generic `QuerySet[Document, Document]`, not the project's custom `PermissionQuerySet`).
  - `moderation_tools.py` — scoped `# type: ignore[attr-defined]` on `message._skip_signals = True` (runtime convention, not a model field).
  - `pydantic_ai_tools.py` — typed `PydanticAIDependencies.vector_store` as `Optional[CoreAnnotationVectorStore]` (was `CoreAnnotationVectorStore` with `default=None` — a latent Pydantic validation bug); scoped `# type: ignore[attr-defined]` on the four `async_wrapper.<attr> = ...` metadata assignments used for pydantic-ai introspection and approval-gate wiring.
  - `tool_factory.py` — hoisted `param_descriptions = self.metadata.parameter_descriptions or {}` out of the `sig.parameters` loop so `.get()` no longer hits `Optional[dict[str, str]]`.
  - `tool_registry.py` — tightened `tool_class: type | None` to `tool_class: type[BaseTool] | None`; added `TYPE_CHECKING` imports for `BaseTool` / `CoreTool`; dropped the now-unnecessary `# noqa: F821` on `to_core_tool`.

- **Typing: graduate `opencontractserver.shared.{QuerySets,fields,mixins}` from mypy baseline** (refs Issue #1447): removed three `[mypy-…]` `ignore_errors` blocks (`opencontractserver.shared.QuerySets`, `opencontractserver.shared.fields`, `opencontractserver.shared.mixins`) and pruned 10 baseline error lines from `docs/typing/mypy_baseline.txt`. The remaining `Managers` and `decorators` modules still have substantial typing surfaces (django-stubs limitations around dynamic `from_queryset`, `self.model.objects` generic narrowing, etc.) and stay on the baseline for follow-up work. Per-file fixes:

  - `opencontractserver/shared/QuerySets.py` — replaced two `timezone.timedelta(...)` calls with `datetime.timedelta(...)`. `django.utils.timezone` re-exports `timedelta` only as a runtime alias; the django-stubs package does not expose it on the public `timezone` module surface.
  - `opencontractserver/shared/fields.py` — scoped `# type: ignore[override]` on `NullableJSONField.formfield(**kwargs)` since the variadic kwargs signature intentionally differs from `Field.formfield`'s positional defaults.
  - `opencontractserver/shared/mixins.py` — scoped `# type: ignore[attr-defined]` on `self.filter(...)` in `VectorSearchViaEmbeddingMixin.search_by_embedding` (provided by the QuerySet base the mixin is combined with at the concrete subclass level), and on two `self.creator` references in `HasEmbeddingMixin.add_embedding` / `add_embeddings` (provided by the model base the mixin is combined with).

- **Typing: graduate small single-file packages from mypy baseline** (refs Issue #1447): removed eight `[mypy-…]` `ignore_errors` blocks (`config.urls`, `config.jwt_utils`, `config.ratelimit.decorators`, `config.graphql_auth0_auth.utils`, `opencontractserver.conftest`, `opencontractserver.feedback.models`, `opencontractserver.worker_uploads.views`, `opencontractserver.examples.structured_response_example`) and pruned 14 baseline error lines from `docs/typing/mypy_baseline.txt`. Per-file fixes:

  - `config/urls.py` — replaced `settings.USE_SILK` with `getattr(settings, "USE_SILK", False)`. `USE_SILK` is only defined in `config/settings/local.py`, so the bare `settings.USE_SILK` access only worked because `mypy.ini` was muting the file; the `getattr` form is what other optional settings already use.
  - `config/jwt_utils.py` — replaced the runtime `User = get_user_model()` (which mypy treats as a value, not a type) with a `TYPE_CHECKING` import of `opencontractserver.users.models.User` and quoted return-type strings.
  - `config/ratelimit/decorators.py` — added `count: int | str` annotation so the `count, period_name, retry_after = "?", "period", 60` fallback (taken when `parse_rate(rate)` raises `ValueError`/`IndexError`) doesn't conflict with the `int` inferred from the success branch.
  - `config/graphql_auth0_auth/utils.py` — JWKS endpoints publish public keys only, but `cryptography` stubs widen `RSAAlgorithm.from_jwk` to `RSAPrivateKey | RSAPublicKey`. Added a localised `assert isinstance(public_key, RSAPublicKey)` before passing the key to `jwt.decode` so the union narrows correctly.
  - `opencontractserver/conftest.py` — wrapped `UserFactory()` in `cast(User, ...)` since the factory class returns a generic `UserFactory` instance under mypy.
  - `opencontractserver/feedback/models.py` — scoped `# type: ignore[misc]` on the `objects = UserFeedbackManager()` manager override (django-stubs flags the re-declaration as overriding a class variable).
  - `opencontractserver/worker_uploads/views.py` — replaced three `token: CorpusAccessToken = request.auth` assignments with `cast(CorpusAccessToken, request.auth)` so the DRF `Token | Any` return type narrows to the project's `CorpusAccessToken` subclass.
  - `opencontractserver/examples/structured_response_example.py` — added `set[str]` annotation on the `all_parties` aggregator.

- **Typing: graduate `opencontractserver.annotations.*` from mypy baseline** (refs Issue #1447): removed seven `[mypy-…]` `ignore_errors` blocks (`opencontractserver.annotations.{admin,compact_json,models,query_optimizer,utils}` plus the two `management.commands.{migrate_structural_annotations,populate_content_modalities}` blocks); added a scoped `disable_error_code = django-manager-missing` block for `annotations.models` (mirrors the existing `users.models` exception — django-stubs / django-tree-queries cannot resolve the `Corpus.label_set → LabelSet.used_by_corpuses` reverse relation). Pruned 27 baseline error lines from `docs/typing/mypy_baseline.txt` (7124 → 7097). Per-file fixes:

  - `annotations/compact_json.py` — collapsed an `isinstance(bounds, dict)` re-binding into a single `bounds_raw → bounds` assignment so the variable isn't typed `Any | None` first; renamed `token_indices: list[int]` to drop the redundant re-annotation that conflicted with the v2 branch's untyped binding.
  - `annotations/query_optimizer.py` — narrowed `path_query.first()` into a separate `first_path: DocumentPath | None` binding before the `is None` guard so mypy doesn't see the `DocumentPath | None` value flowing into the previously `DocumentPath`-typed `document_path` variable.
  - `annotations/models.py` — added `author_obj: AbstractBaseUser` annotation in `Note.update_content_with_revision`; scoped `# type: ignore[misc]` on three manager-override `objects = ...Manager()` declarations (`Embedding`, `Annotation`, `Note`); scoped `# type: ignore[assignment]` on the `StructuralAnnotationSet.creator` `ForeignKey(..., null=True)` redeclaration (intentional override of `BaseOCModel.creator`); scoped `# type: ignore[misc]` on the lambda inside `transaction.on_commit(...)` and on `author=author_obj` passed to `NoteRevision.objects.create(...)`; dropped the `: str` re-annotation on `embedder_path = CharField(...)` (CharField field declarations don't accept Python type annotations on the LHS).
  - `annotations/utils.py` — typed `all_tokens: list[Any] = []` accumulator.
  - `annotations/management/commands/populate_content_modalities.py` — typed `all_token_refs: list[Any] = []`.
  - `annotations/management/commands/migrate_structural_annotations.py` — widened the heterogeneous `stats` dict to `dict[str, Any]` so the mixed counter / error-list values type-check.

- **Badge tile overflow fix on community leaderboard** (`frontend/src/components/badges/Badge.tsx`, `frontend/src/components/community/Leaderboard.tsx`): long badge names previously broke out of their `BadgeCard` containers in the leaderboard grid because the card was a horizontal flex (`flex-direction: row`) with no flex-shrink hint, and `StyledBadge`/`BadgeName` did not allow word-break. Switched `BadgeCard` to a vertical layout (`flex-direction: column`, `align-items: flex-start`, `gap: 12px`, `min-width: 0`) so the icon stacks above the metadata and the badge can fully consume the card width; added `overflow-wrap: anywhere` plus `max-width: 100%` to `StyledBadge` and `overflow-wrap: anywhere` to `BadgeName` so very long names break at any character. `BadgeMeta` was switched from `flex: 1` to `width: 100%` to size correctly under the new column direction. `Badge.tsx`'s wrapper `<motion.span>` also gets an inline `maxWidth: "100%"` so badges constrained by ancestor flex layouts cannot escape their parent. Affects every screen that renders a `Badge` (the column-direction `BadgeCard` change is leaderboard-scoped).

- **Consolidate inline document READ checks onto `visible_to_user`** (Issue #1450, `config/graphql/document_types.py`): four resolvers (`resolve_page_annotations`, `resolve_page_relationships`, `resolve_relationship_summary`, `resolve_extract_annotation_summary`) each repeated the same hand-rolled `if not self.is_public: ... user != self.creator and not user.is_superuser ... user_has_permission_for_obj(user, self, READ)` block. The fallback to `user_has_permission_for_obj` for a Document READ check is the exact anti-pattern the helper's docstring warns against — it ignores corpus context and other visibility rules. Replaced all four blocks with a single private helper, `DocumentType._assert_user_can_read`, which delegates to `Document.objects.visible_to_user(user).filter(id=self.id).exists()` and raises a `GraphQLError` (`Authentication required` for anonymous, `do not have access` for authenticated). Behavior is preserved for the public/anonymous/owner/superuser/sharee/no-access matrix and is now driven by the canonical visibility manager. New unit suite `opencontractserver/tests/test_document_type_read_permission.py` locks the helper's contract.

- **Leaderboard page redesigned to match the OS Legal design system** (`frontend/src/components/community/Leaderboard.tsx`):

  - Replaced custom gradient/inline-style sections with a hero header, `StatGrid`-based community stats, and named styled components for every compound element (`UsernameCell`, `ScoreCell`, `DetailsCell`, `RisingStarTag`, `BadgeCard`, `BadgeMeta`, `BadgeName`, `BadgeStats`).
  - `RankBadge` palette tokenised into a file-local `RANK_COLORS` map and `RisingStarTag` colours into `RISING_STAR_COLORS`; rank-2 surface/border/text now reuse `OS_LEGAL_COLORS` tokens directly.
  - Bumped `RisingStarTag` font-size from 11px to 12px (WCAG 2.1 small-text floor).
  - Removed unused `GradientSegment` import and `statsLoading` variable.

- **Consistent MCP discoverability across corpus tiles and detail pages** (`frontend/src/components/common/MCPShareButton.tsx`, `frontend/src/components/corpuses/CorpusListView.tsx`, `frontend/src/components/corpuses/CorpusHome/CorpusLandingView.tsx`, `frontend/src/components/corpuses/CorpusHome/CorpusDetailsView.tsx`):

  - The MCP share button now renders for every corpus that has a slug, not just public ones, and the tile overlay is always visible (no longer fades in on hover).
  - `MCPShareButton` accepts a new `isPublic` prop. For private corpora the popover shows a `Lock` icon and explains that the corpus must be made public before an MCP endpoint is exposed (the backend MCP server only serves public corpora — `opencontractserver/mcp/server.py`).
  - The redundant "MCP Endpoint" entry in the corpus tile context menu was removed; the always-visible overlay button is now the single canonical entry point.

- **Extract pipeline cleanup pass** (Issue #1410): hardens the extraction-related plumbing introduced by #1381 / #1380 / #1399 against three medium-impact failure modes and four documentation/observability gaps surfaced during code review.
  - **`opencontractserver/tasks/embeddings_task.py:_batch_embed_text_annotations`** — Transient HTTP errors (`requests.Timeout`, `requests.ConnectionError`, `EmbeddingServerError`) now drop queued sub-batches and shut down the `ThreadPoolExecutor` with `wait=False, cancel_futures=True` before re-raising. Previously the exception propagated out of the `with` block, which calls `shutdown(wait=True, cancel_futures=False)` by default and blocked Celery autoretry by up to ~`max_workers` × the per-sub-batch round-trip latency. In-flight HTTP calls cannot be torn down from Python, but at least we no longer wait on them.
  - **`opencontractserver/benchmarks/loader.py:force_celery_eager`** — Refuses to mutate the global Celery config unless `settings.MODE == "TEST"` _or_ the new `OC_BENCHMARK_CLI` env var is set; outside test mode also refuses when `task_always_eager` is already `True` (concurrent benchmark runs would race on the global save/restore). In test mode an already-`True` flag is the ambient state imposed by `CELERY_TASK_ALWAYS_EAGER=True`, so the helper is a no-op there. The `run_benchmark` management command now sets `OC_BENCHMARK_CLI=1` automatically. Prevents the benchmark helper from silently routing every task in a live web/worker process through the in-process executor.
  - **`opencontractserver/llms/agents/pydantic_ai_agents.py`** — `PydanticAICorpusAgent._build_structured_system_prompt` now says "most legal corpora need multiple targeted queries" instead of "most legal documents …" — corpora-scoped wording for the corpus agent. The document-agent prompt is unchanged.
  - **`config/settings/test.py`** — The `DEFAULT_EMBEDDER` env-var override is now gated behind an explicit `BENCHMARK_MODE=1` env var. Without that opt-in, a stray `DEFAULT_EMBEDDER` value in the CI environment would silently push the regular test suite onto a real embedder and start making live network calls. The default `TestEmbedder` keeps regular `pytest` runs hermetic.
  - **`opencontractserver/tasks/data_extract_tasks.py`** — When `model_override` is accepted in the unrestricted (operator-only) path because `BENCHMARK_ALLOWED_MODEL_OVERRIDES` is unset, a `WARNING`-level log line now records the override + cell ID. WARNING (not INFO) because the open mode is a misconfiguration / abuse signal — anything firing it from a web/GraphQL caller is a real anomaly that should reach log aggregation. Lets operators `grep` production logs to confirm the open mode is fired only by the benchmark tooling and never by an unexpected web/GraphQL caller.
  - **`opencontractserver/pipeline/utils.py:get_default_reranker_instance`** — Docstring now warns that `bulk_update` / `QuerySet.update` / data-migration writes to `PipelineSettings.default_reranker` bypass the `auto_now` `modified` field used as part of the cache key, so callers who mutate the singleton via those paths must also call `invalidate_reranker_cache()` (or touch `modified` explicitly).

### Documentation

- **Auth0 admin-claim namespace mismatch** (`docs/configuration/authentication.md`): documented a silent failure mode where the Post-Login Action's `namespace` constant doesn't match `AUTH0_ADMIN_CLAIM_NAMESPACE` (e.g. `https://opencontracts.opensource.legal/` vs the default `https://contracts.opensource.legal/`). When the namespaces don't match byte-for-byte, the backend's fail-closed sync (`config/graphql_auth0_auth/utils.py:sync_admin_claims_from_payload`) treats the claims as missing and overwrites `is_staff` / `is_superuser` to `False` on every authenticated request, so the frontend admin link in the user dropdown never appears even though `app_metadata.is_superuser = true` is set in Auth0. Added a `!!! danger` callout to the Post-Login Action setup section and expanded the "Admin claim missing, defaulting to False" troubleshooting entry with the namespace-mismatch case and a `!!! info` note explaining why missing claims revoke admin instead of being ignored.

### Security

- **Audit + IDOR-pattern hardening of the four largest GraphQL mutation modules** (Issue #1449): completed the preventative permission audit called for in #1449 across `config/graphql/document_mutations.py` (1,673 lines), `config/graphql/corpus_mutations.py` (1,480 lines), `config/graphql/extract_mutations.py` (1,312 lines), and `config/graphql/annotation_mutations.py` (883 lines). Findings + fixes:
  - **Per-file verdict**: `extract_mutations.py` is fully covered (every user-supplied ID is loaded through `Model.objects.visible_to_user(user)` or guarded by `user_has_permission_for_obj` with creator/public Q-filters, and error messages are uniform). `corpus_mutations.py`, `document_mutations.py`, and `annotation_mutations.py` were mostly safe but had a handful of resolvers that loaded objects via raw `Model.objects.get(pk=...)` and then either skipped the visibility check, deferred it to a service, or returned a _different_ error message for "not found" vs "permission denied" — the exact IDOR-enumeration smell `CLAUDE.md` warns about.
  - **`config/graphql/annotation_mutations.py`** — `RemoveRelationships` (line ~489), `UpdateRelationship` (line ~550), `UpdateRelations` (line ~690), and `UpdateNote` (line ~741) now load the target object through `Relationship.objects.visible_to_user(user)` / `Note.objects.visible_to_user(user)` before any permission gating. All branches ("relationship not found", "permission denied", "note not found", "not creator") collapse to one shared error string per mutation so an attacker cannot distinguish "ID does not exist" from "ID exists but is not yours". `UpdateRelationship` additionally filters add-source / add-target annotation IDs through `Annotation.objects.visible_to_user(user)` instead of `Annotation.objects.filter(id__in=...)`, and short-circuits with the same unified message when any requested ID is invisible. Removal paths now restrict deletion to annotations actually attached to the relationship (via `relationship.source_annotations.filter(id__in=...)`) so removal cannot leak the existence of unrelated annotation IDs even though the relationship-level UPDATE check still gates the operation.
  - **`config/graphql/document_mutations.py`** — `EmptyTrash` (line ~1470), `RestoreDocumentToVersion` (lines ~1529–1530), `UploadDocument` (line ~200, corpus load), `UploadDocumentsZip` (line ~575), and `ImportZipToCorpus` (line ~922) now route corpus/document loads through `Corpus.objects.visible_to_user(user)` / `Document.objects.visible_to_user(user)`. The "corpus not found" and "you don't have permission to add documents to this corpus" branches in the upload mutations were merged into a single unified message, and `RestoreDocumentToVersion`'s separate "document version not found" / "corpus not found" / "you don't have permission to restore this document" / "you don't have permission to modify this corpus" branches were collapsed into one IDOR-safe string.
  - **`config/graphql/corpus_mutations.py`** — `UpdateCorpusDescription` (line ~266) now goes through `Corpus.objects.visible_to_user(user)` and returns the same message whether the corpus does not exist, the caller cannot see it, or the caller can see it but is not the creator. `AddDocumentsToCorpus` (line ~365) and `RemoveDocumentsFromCorpus` (line ~424) likewise route the corpus load through `visible_to_user(user)` instead of relying solely on the `DocumentFolderService` write-permission check that runs immediately afterwards. `RunCorpusAction` (line ~1284) now carries an explicit `assert user.is_superuser` and a comment documenting that its raw `CorpusAction.objects.get(...)` / `Document.objects.get(...)` calls are intentional because the resolver is gated by `@user_passes_test(lambda user: user.is_superuser)`.
  - **Test updates** (`opencontractserver/tests/test_permission_fixes.py`): the existing `RemoveRelationships` and `UpdateRelations` IDOR tests were updated to assert the new unified error string instead of the old per-branch messages. Test _intent_ is unchanged — they still verify that an unauthorized user and a user supplying a non-existent ID receive byte-identical responses — only the expected literal changed because the mutations now use the harmonised wording. No production behaviour outside the audited mutations was modified.
  - **What was not changed**: `extract_mutations.py` was left as-is (already compliant); `RetryDocumentProcessing` in `document_mutations.py` was left as-is because its raw load is followed immediately by a `user_has_permission_for_obj` check that already returns the _same_ "Document not found" string for both branches, so the IDOR pattern is preserved despite the load shape; `AddTemplateToCorpus`'s raw `CorpusActionTemplate.objects.get(pk=...)` is intentional per the in-line docstring (templates are global) and was left in place pending a separate decision on whether template publishing should ever become user-scoped.

### Fixed

- **Extraction grounding follow-up #2** (Issue #1407, follow-up to PR #1397):

  - **DB-level idempotency for grounding `get_or_create`** (`opencontractserver/annotations/migrations/0069_grounding_annotation_unique_constraints.py`, `opencontractserver/annotations/models.py`, `opencontractserver/utils/extraction_grounding.py`): added two partial `UniqueConstraint`s on `Annotation` (`annotation_unique_token_label_grounding_key`, `annotation_unique_span_label_grounding_key`) that exactly match the grounding pipeline's `get_or_create` lookup keys, scoped to a new `Annotation.is_grounding_source` boolean so non-grounding flows producing the same key tuple are unaffected. Promotes the application-level `get_or_create` from best-effort to a correctness invariant against concurrent Celery retries. `creator_id` and `structural=False` were moved into the lookup so the constraints back the application-level call. Migration backfills `is_grounding_source=True` on pre-existing `OC_EXTRACT_SOURCE` rows, dedupes survivors (keeping lowest-pk and re-pointing M2M / FK references on `Datacell.sources`, `ChatMessage.{source,created}_annotations`, `Assignment.resulting_annotations`, `Relationship.{source,target}_annotations`, `UserFeedback.commented_annotation`), then adds the constraints.
  - **Documented PostgreSQL-specific `json` lookup semantics** (`_create_span_annotation` docstring) and the dedup-by-pk loop's rationale (`_create_grounding_annotations`).

- **Blob retention across corpus-isolated Document copies** (Issue #1464): `Corpus.add_document()` and the document-versioning helpers create corpus-isolated _copies_ of a `Document` whose file fields (`pdf_file`, `txt_extract_file`, `pawls_parse_file`, `icon`, `md_summary_file`) intentionally point to the SAME S3 blobs as the source. Until now, exactly one production code path (`update_memory_content` in `opencontractserver/agents/memory.py:237`) called `txt_extract_file.delete(save=False)` unconditionally — meaning any memory document copy-shared between two corpora via `Corpus.add_document()` would see its blob silently destroyed in the sibling corpus on the next memory update. Detection was masked by `FileSystemStorage`'s path-reuse semantics: `delete()` followed by `save(SAME_NAME, …)` writes to the same path, so the sibling's `txt_extract_file.name` still resolves but reads back the _new_ (foreign-corpus) content — a quiet data-corruption failure, not a 404.
- **Defensive coverage spans every Document `FileField`, not just the one current call site.** The fix introduces a generic primitive that protects all five file fields uniformly so that the upcoming orphan-cleanup work (Issue #1492) inherits the sharing-check for free.
- **New manager method** (`opencontractserver/shared/Managers.py`): `Document.objects.unique_blob_paths(doc)` returns the set of file-field blob paths on `doc` that are NOT referenced by any other live `Document` row. The method derives its FileField list from `Document._meta.get_fields()`, so adding a new `FileField` to the model extends coverage automatically.
- **New model method** (`opencontractserver/documents/models.py`): `Document.safe_delete_field_blob(field_name, *, save=False)` is the single, public chokepoint for freeing a blob in storage. Empty field → no-op; unique blob → `FieldFile.delete()`; shared blob → field cleared on this row only, blob retained in storage. Validates that `field_name` resolves to a `FileField` on the model and raises `ValueError` for typos so silent no-ops can't hide. Any code in this codebase that needs to delete a blob from storage MUST go through this primitive instead of calling `FieldFile.delete()` directly.
- **Guard at the one current production call site** (`opencontractserver/agents/memory.py:241`): the inline shared-blob check is replaced with a single call to `locked_doc.safe_delete_field_blob("txt_extract_file")`. The behavioural contract is identical; the logic now lives on the model where every future caller can find it.
- **Regression coverage** (`opencontractserver/tests/test_blob_retention.py`, 4 `TransactionTestCase` classes):
  - `UniqueBlobPathsTestCase` (4 tests) — pins the manager method's contract: solitary docs own all blobs, shared blobs are excluded for both source and copy, per-field uniqueness after partial overwrite, empty fields produce no `""` entries.
  - `SafeDeleteFieldBlobTestCase` (5 tests) — exercises the primitive across **every** `FileField` on Document (parameterized via `DOCUMENT_FILE_FIELDS`): unique blobs are removed from storage, shared blobs are retained for the sibling, empty fields are no-ops, invalid field names raise, non-FileField names raise.
  - `MemoryDocumentBlobRetentionTestCase` (2 tests) — content-equality assertion that catches the actual failure mode: after `update_memory_content` on corpus A, sibling in corpus B still reads its original content. Failed pre-fix, passes post-fix.
  - `DocumentDeleteBlobRetentionTestCase` (2 tests, parameterized across all FileFields) — issue-spec invariant: `Document.delete()` does not destroy a blob still referenced by a sibling. Passes today (default `Model.delete()` doesn't touch FileField blobs) and is the contract test for the row-delete path; it will become live coverage once the orphan-cleanup work in Issue #1492 lands the deletion mechanic.
- **Forward pointer to Issue #1492**: this PR is the _defensive_ half of the blob-retention contract ("don't delete shared blobs"). The complementary half ("reclaim truly orphaned blobs at row-delete time") is tracked separately and will reuse `Document.safe_delete_field_blob` so the sharing-check does not need to be re-implemented.
- **Audit findings** (preserved in PR description): `django-cleanup` is not installed; no `pre_delete`/`post_delete` handlers on `Document` delete blobs (the existing handlers only GC `StructuralAnnotationSet`); `worker_uploads/tasks.py` deletes belong to `WorkerDocumentUpload.file`, a separate model; default `Model.delete()` does not delete FileField blobs.

- **MCP share popover clipped/obscured on corpus list tiles** (`frontend/src/components/common/MCPShareButton.tsx`): the popover was rendered inline inside `Container` with `position: absolute; z-index: 1000`, which left it subject to two ancestor problems on `CorpusListView`. (1) `PageContainer` sets `overflow-x: hidden`, so the popover's right edge was clipped on narrow viewports — particularly in responsive mode where the trigger sits near the right edge of the card. (2) Sibling card stacking contexts (each card's `MCPButtonOverlay` carries its own `z-index: 10` over `position: relative` cards) caused the popover from one card to render _behind_ the next card down. Fix: portal the popover into `document.body`, switch it to `position: fixed`, compute its viewport coordinates from the trigger's `getBoundingClientRect()` (clamped within `POPOVER_VIEWPORT_MARGIN` so it can't overflow the viewport on either edge), and recompute on `scroll` (capture phase, to catch nested scrollers) and `resize`. Click-outside detection now checks both `containerRef` and the new `popoverRef` since the portaled popover is no longer a DOM descendant of the trigger.

- **Slow document loading on corpora with hundreds of documents** — opening the Documents tab on a large corpus blocked behind a 5+ second "Documents Loading…" spinner because the list query did O(N) work per document for relationship metadata:

  - **Frontend** (`frontend/src/graphql/queries.ts`, `frontend/src/components/documents/ModernDocumentItem.tsx`): `GET_DOCUMENTS` no longer eagerly fetches `allDocRelationships` for every document on the page (each list page was pulling source/target/label rows for every linked document — 5,000+ extra fields on a corpus with 100 docs averaging 50 links). Added a new lazy `GET_DOC_RELATIONSHIPS_FOR_DOC` query (resolves through the existing `bulkDocRelationships` field) that fires the first time the user hovers/focuses a relationship badge on a document card. The hover popup's existing "Loading relationships…" placeholder covers the brief network round-trip. The `docRelationshipCount` badge still renders synchronously from the list query.
  - **Backend** (`opencontractserver/documents/query_optimizer.py`, `config/graphql/document_types.py`): replaced the per-document `.count()` call in `resolve_doc_relationship_count` with `DocumentRelationshipQueryOptimizer.get_relationship_counts_by_document` — a single pair of aggregated `GROUP BY` queries (one for source, one for target) that returns `{document_id: count}` for every document the user can see (optionally scoped by corpus). Result is cached on `info.context` keyed by user+corpus so all N badges in a single GraphQL response share the work. Eliminates the `Document.objects.get` + permission re-check + `select_related` query that previously ran once per document. For a 20-item page this collapses ~80 DB round-trips into 2.
  - **Backend** (`config/graphql/filters.py:in_folder`): the folder filter was materialising `DocumentPath` IDs into a Python `set()` (forcing eager evaluation) and running two extra `COUNT` queries purely for `logger.info` debug lines. Switched to a lazy `values()` queryset so Django emits a SQL subquery for the `__in` lookup and dropped the diagnostic counts — keeps the queryset fully lazy for downstream pagination.

- **Backend hardening for GraphQL CSRF / `Authorization` semantics** (Issue #1432, follow-up to #1431): the original fix already normalised empty / whitespace-only `Authorization` headers, but any _non-empty_ value still bypassed CSRF. That left a defense-in-depth gap (a malformed scheme such as `Authorization: Basic …` was treated as evidence of token auth) and the production logs were still drowning in benign `Forbidden (CSRF token missing.)` warnings. Two-part hardening:

  - **Strict scheme validation** (`config/graphql/security.py:45-104`): added `_is_recognised_token_credential` which only accepts the `Authorization` header as a token credential when it splits into exactly `<scheme> <credential>`, the credential is non-empty, and the scheme matches a recognised prefix (currently `Bearer` plus the configured `API_TOKEN_PREFIX` when API-key auth is enabled). Scheme matching is case-insensitive per RFC 7235. Empty, whitespace-only, scheme-only (`Bearer`, `Bearer  `), and unrecognised-scheme (`Basic …`) values now fall through to the cookie-based path so a malformed header cannot smuggle a session-cookie request into the token-auth bypass. `conditional_csrf_exempt` was rewritten on top of this helper; the no-cookie anonymous bypass is preserved unchanged so Bearer-only SPA boot races still get a 200 on cold start.
  - **Log volume control** (`config/graphql/security.py:157-198`, `config/settings/base.py`, `config/settings/production.py`): introduced `CsrfRejectLogFilter`, a `logging.Filter` wired into the `django.security.csrf` logger that demotes the predictable `Forbidden (CSRF token missing. | CSRF cookie not set.)` WARNING to INFO. Genuine anomalies — origin mismatch, bad referer, mismatched token — keep their WARNING level so log shipping continues to surface them. The filter inspects `record.args[0]` against a small allow-list of benign reason strings; anything else is passed through unchanged.
  - **Test matrix expansion** (`opencontractserver/tests/test_security_hardening.py`): added the missing legs called out in #1432. New `TestConditionalCsrfExempt` cases — `test_unrecognized_scheme_with_session_enforces_csrf`, `test_unrecognized_scheme_without_session_bypasses_csrf`, `test_bearer_without_credential_with_session_enforces_csrf` (covers `"Bearer"`, `"Bearer "`, `"Bearer    "`), `test_bearer_with_credential_bypasses_csrf`, `test_bearer_scheme_is_case_insensitive`, `test_api_key_scheme_bypasses_csrf` (with `API_TOKEN_PREFIX` overridden so the test is independent of `ALLOW_API_KEYS`), and `test_session_with_csrf_token_passes` — close the positive session+CSRF path that the suite previously lacked. New `TestCsrfRejectLogVolume` covers the filter contract directly: `test_filter_demotes_csrf_token_missing_warning_to_info`, `test_filter_does_not_demote_other_csrf_reasons` (origin mismatch, bad referer, `CSRF token incorrect.`), and `test_filter_passes_unrelated_records_unchanged`.
  - **Out of scope** (per the issue): no SPA auth-model changes, no CSRF-token plumbing on the frontend.

- **System Settings page failed to load with `Enum 'FileTypeEnum' cannot represent value: 'md'`** (`config/graphql/pipeline_types.py:12`): the GraphQL `FileTypeEnum` was hand-maintained and only declared `PDF`/`TXT`/`DOCX`, but the backend `FileTypeEnum` has `MD = "md"` and `oc_markdown_parser.py` declares `supported_file_types = [FileTypeEnum.MD]`. When the pipeline-components query serialized the markdown parser's supported file types, the GraphQL enum could not represent `"md"` and the entire query failed. Replaced the hand-maintained class with `graphene.Enum.from_enum(BackendFileTypeEnum)` so the GraphQL enum is generated from the backend source of truth and the two can never drift again — adding a member to the backend enum now exposes it through the schema automatically.

- **Document annotator right sidebar: crowded tabs and unresponsive Chat panel**:

  - **Tabs visually merge when panel is open** (`frontend/src/components/knowledge_base/document/styled/SidebarTabs.tsx:78-93`): `SidebarTabsContainer` set `gap: 0` whenever the panel was open, so the four vertical tabs (Index, Chat, Feed, Discussions) butted up against each other along the panel's left edge with no separation. Replaced the conditional gap with a constant `gap: 6px`, restoring per-tab visual identity in both open and closed states.
  - **`NewChatFloatingButton` ignored panel bounds** (`frontend/src/components/knowledge_base/document/ChatContainers.tsx:587-606`): the FAB used `position: fixed; bottom: 2rem; right: 2rem`, anchoring it to the viewport instead of the chat panel — at 50% panel width the "+" button rendered over the document, not the chat tray, and didn't follow panel resizes. Switched to `position: absolute` so it tracks the `SlidingPanel`, with a softer offset (`1.5rem`) and a more visible shadow.
  - **Empty conversation list collapsed to a thin strip** (`frontend/src/components/knowledge_base/document/right_tray/ConversationListView.tsx:87-104`, `frontend/src/components/knowledge_base/document/ChatContainers.tsx:389-399`): the list view's outer wrapper had no `flex: 1`/height, and `ConversationGrid` likewise didn't grow, so when no conversations existed the panel showed only the filter row over a sea of empty white. Made both flex-grow within their parent and added `grid-auto-rows: max-content` plus bottom padding so cards stack from the top and don't collide with the FAB.
  - **No empty-state messaging for Chat** (`frontend/src/components/knowledge_base/document/right_tray/ConversationListView.tsx:185-231`): added a centered empty state with a chat icon, "No conversations yet" heading, and a hint that nudges users toward the new-chat FAB. Gated on `conversations.length === 0` so it disappears as soon as data lands.

- **GraphQL POST 403 storm when Auth0 token is empty** (Issue #1431): production logs were filling with `Forbidden (CSRF token missing.): /graphql/` because the React frontend always sent `Authorization: ""` whenever the Auth0 access token was momentarily missing (startup race, silent-refresh failure, post-expiry re-mount), and `conditional_csrf_exempt` only bypassed CSRF when the header was _truthy_. An empty header fell through to Django's session/CSRF path even though the frontend never carries a CSRF token. Two-sided fix:

  - **Frontend** (`frontend/src/index.tsx:36-50`): conditional-spread `Authorization` so the header is omitted entirely when the token is empty rather than sent as the empty string.
  - **Backend** (`config/graphql/security.py:43-86`): refactored `conditional_csrf_exempt` to skip CSRF when **either** an `Authorization` header is present (Bearer / API-key — browsers don't auto-attach it) **or** no session cookie is present (the request is fully anonymous; without a cookie there is nothing for a cross-origin attacker to ride). Whitespace-only `Authorization` headers are now treated as missing. Session-cookie requests without a Bearer token still go through the existing CSRF check, so authenticated session traffic remains protected.
  - **Backend** (`config/graphql_auth0_auth/settings.py:73-79`): `Auth0JWTSettings.__getattr__` now raises `AttributeError` quietly for any name beginning with `_`, eliminating the `ERROR settings  Auth0JWTSettings.__getattr__() - Invalid setting requested: _user_settings` log noise emitted whenever Python internals (deepcopy / pickle / `hasattr`) probed dunder/private attributes. Unknown public settings still log an error.
  - **Tests** (`opencontractserver/tests/test_security_hardening.py`, `opencontractserver/tests/test_auth0_jwks_cache.py`): updated `test_session_auth_without_csrf_rejected` to inject a session cookie (so the CSRF path is actually exercised); added `test_anonymous_no_session_bypasses_csrf`, `test_empty_authorization_header_treated_as_missing`, and `test_empty_authorization_with_session_still_enforces_csrf` to lock in the new semantics; added `TestAuth0JWTSettingsDunderProbe` to assert that dunder/private probes don't emit error logs while unknown public settings still do.

- **Shared-protocol contract drift surfaced by PR #1400 follow-up review** (Issue #1408):
  - **`PermissionedQueryManagerProtocol.visible_to_user` signature mismatch** (`opencontractserver/types/protocols.py:147`): the protocol declared `visible_to_user(self, user: Any = None)` but `PermissionManager.visible_to_user` and `UserFeedbackManager.visible_to_user` have no default. A caller holding a protocol-typed reference could call `.visible_to_user()` with no args and trigger a runtime `TypeError`, and mypy would also reject the concrete classes as structural matches. Dropped the `= None` default so the protocol pins the strictest contract; the docstring now explains that callers must pass an `AnonymousUser` when no authenticated principal is available. Lenient managers (e.g. `BaseVisibilityManager` with `user=None`) still satisfy the protocol — verified with both `isinstance` and `issubclass` checks.
  - **`ToolProtocol` `@property` descriptors against a `@dataclass`** (`opencontractserver/types/protocols.py:97`): `CoreTool` is a `@dataclass` exposing `name` / `description` / `parameters` / `requires_approval` as plain instance attributes. Mypy accepted dataclass fields against the property-shaped protocol, but the asymmetry was a footgun for future implementors who would reach for `@property` based on the protocol surface. Converted to plain attribute declarations matching the concrete class.
  - **`StreamObserverProtocol` drift hazard** (`opencontractserver/types/protocols.py:152`): the protocol duplicated `opencontractserver.llms.types.StreamObserver` with no automated enforcement. A naive re-export creates a real circular-import risk because `protocols.py` is imported by `opencontractserver.shared.Managers` during Django app loading and `opencontractserver.llms.__init__` eagerly pulls in the heavy LLM stack (`api` → conversation models, agent factories). Kept duplicated definitions but added explicit "Must be kept in sync with…" notices on both sides (`opencontractserver/llms/types.py:18` and `opencontractserver/types/protocols.py:152`) explaining the rationale, and aligned the `__call__` return type on `llms.types.StreamObserver` (`-> None`, dropping the redundant `Awaitable[None]` wrapper that was already implicit in `async def`).
  - **`VectorStoreProtocol` test missed renames** (`opencontractserver/tests/test_protocols.py:24`): the canonical-implementation test used `hasattr` against hard-coded method names instead of the `@runtime_checkable` machinery used by the other three tests in the file. Renaming `search` / `async_search` would have left the test passing as long as the attribute names happened to exist via some other code path. Switched to `issubclass(CoreAnnotationVectorStore, VectorStoreProtocol)`, which executes the protocol's structural check without instantiating the class. The negative test for plain `object` was tightened from a manual `hasattr` to `issubclass(object, VectorStoreProtocol)` for symmetry.

### Added

- **Tests for issue #1410 fixes**:

  - `opencontractserver/tests/test_data_extract_failure_classification.py::PydanticAiSchemaCanaryTests` — Pin the pydantic-ai message-schema discriminators (`ModelResponse.kind == "response"`, `ToolCallPart.part_kind == "tool-call"`, `TextPart.part_kind == "text"`) so a future minor version that renames them surfaces immediately rather than silently flipping every `None` extraction into the `empty_history` mis-classification mode.
  - `opencontractserver/tests/test_batch_embedding.py::TestBatchEmbedTextAnnotations::test_transient_error_does_not_block_on_in_flight_peers` — Asserts the executor fast-fail path completes well inside the 10 s peer-block window. A regression that waits on in-flight peers (the previous default-shutdown behaviour) would take ~10 s and fail this test.
  - `opencontractserver/tests/test_benchmarks.py::ForceCeleryEagerSafetyGuardsTestCase` — Pins the three new `force_celery_eager` safety refusals: non-test mode without the CLI env var, non-test mode _with_ the CLI env var (allowed), and stacked invocations (`task_always_eager` already `True`).

- **Extract iterations & cell-level diff** — surfaces the three eval workflows callers asked for (model drift, document-version drift, fieldset-config sweeps) without forking the data model. Two new fields on `Extract` chain a parent/child relationship and capture run-time model config; everything else reuses the existing extract → fieldset → datacell pipeline so storage, permissions, and the runner are unchanged.

  - **Migration `extracts/0029_extract_iterations.py`** — adds `Extract.parent_extract` (self-FK, indexed) and `Extract.model_config` (`NullableJSONField`). Both nullable so legacy extracts continue to load with no backfill.
  - **`opencontractserver/extracts/diff.py`** — pure helper (no Graphene, no DB writes) that takes two pre-permission-filtered `Datacell` iterables and produces aligned `(row_key, column_key)` rows classified `UNCHANGED | CHANGED | ONLY_IN_A | ONLY_IN_B`. Rows are aligned by `Document.version_tree_id` so a single logical document still appears once even when the two iterations point at different content versions. `corrected_data` wins over `data` when computing equality. `_column_config_signature` gates the `column_config_changed` flag on prompt/instructions/output_type/etc., not name — so a rename alone doesn't trip it.
  - **`config/graphql/extract_mutations.py:CreateExtractIteration`** — single mutation that forks an extract along `MODEL`, `DOCUMENT_VERSIONS`, or `FIELDSET`. MODEL and DOCUMENT_VERSIONS share the parent's fieldset (column definitions stay byte-identical for apples-to-apples comparison); FIELDSET deep-clones `Fieldset` + `Column` rows via `_clone_fieldset_for_iteration` with optional per-column overrides. DOCUMENT_VERSIONS re-resolves each parent doc to the row with `is_current=True` in its `version_tree_id`. `auto_start=True` queues the existing `run_extract` task on commit — no new orchestration code path. `parent_extract` is set on the new extract; default name is `<source> (iteration N)` with N counting existing siblings + 1. Permission gate: caller needs `READ` on the source extract.
  - **`opencontractserver/tasks/extract_orchestrator_tasks.py`** — `run_extract` reads `extract.model_config.get("model")` and forwards it as the existing `model_override` kwarg of `doc_extract_query_task` (originally added for the benchmark runner). New `_task_accepts_kwarg(task_func, name)` (LRU-cached) inspects the column task's signature so the kwarg is only forwarded to tasks that accept it — custom column tasks predating `model_override` keep working unchanged. No model_config → behaviour byte-identical to before.
  - **`config/graphql/extract_types.py`** — exposes `modelConfig`, `parentExtract`, `fullIterationList`, plus a derived `iterationAxis` field (`MODEL` if `model_config` differs from parent, `FIELDSET` if fieldset differs, `DOCUMENT_VERSIONS` if doc set differs, else null) so the frontend can badge iterations without persisting the axis.
  - **`config/graphql/extract_queries.py:compareExtracts(extractAId, extractBId)`** — returns `ExtractDiffType { extractA, extractB, cells, summary }` where `cells` is a list of `ExtractCellDiffType { rowKey, columnKey, document, documentA, documentB, cellA, cellB, status, columnConfigChanged }`. Permission check reuses `ExtractQueryOptimizer.check_extract_permission` so visibility rules stay consistent across the existing `extract` resolver and the diff resolver. Cells pulled via `ExtractQueryOptimizer.get_extract_datacells` — same optimizer (and permission filtering) on both sides of the diff.
  - **Frontend — `frontend/src/components/extracts/iterations/`**:
    - `ExtractIterationsTab.tsx` lives as a 4th tab (`Iterations`) on `ExtractDetailContent`. Lists every extract in the series (parent + current + iterations) with a status bullet, axis chip, and captured-model chip. Clicking two rows toggles a comparison view inline.
    - `NewIterationDialog.tsx` is the focused fork modal — three radio-style cards (model, document versions, schema), an optional name, an optional model identifier, and a "Run immediately" toggle that maps to `autoStart`.
    - `ExtractCompareView.tsx` runs `compareExtracts` and renders the heatmap: rows are documents, columns are the union of column names, each cell is colored by `DiffStatus` and clickable to open a fixed-position side panel showing the A/B JSON. Reuses `formatCellValue` from `frontend/src/utils/formatters.ts` so cell values look identical to the main extract grid. Summary chips (changed / only-in-A / only-in-B / unchanged) above the grid use the same palette as the cell borders so the legend and grid never drift.
  - **Frontend GraphQL** — `REQUEST_GET_EXTRACT` extended with `modelConfig`, `iterationAxis`, `parentExtract`, `fullIterationList`. New `COMPARE_EXTRACTS` query and `REQUEST_CREATE_EXTRACT_ITERATION` mutation live alongside the existing extract operations in `frontend/src/graphql/queries.ts` / `mutations.ts`. `ExtractType` in `frontend/src/types/graphql-api.ts` gained the same four fields.
  - **Tests — `opencontractserver/tests/test_extract_iterations.py`** — `DiffHelperTestCase` exercises the diff algorithm directly (alignment by `version_tree_id`, `corrected_data` precedence, only-in-A/B, summary counts); `CreateExtractIterationMutationTestCase` covers all three axes, the unknown-axis error path, and the auto-incrementing default name; `CompareExtractsResolverTestCase` smoke-tests the GraphQL resolver end-to-end. Respects the `Document.version_tree_id` unique constraint by demoting `is_current` before creating successor versions.
  - **E2E coverage — `frontend/tests/e2e/extract-pdf-workflow.spec.ts`** — extends the existing pipeline spec with `forkExtractIterationViaUI` + `selectIterationsForCompare` helpers (in `frontend/tests/e2e/helpers.ts`) to drive the `Iterations` tab end-to-end after the parent extract finishes: opens `NewIterationDialog`, picks the MODEL axis, names the iteration, submits with `Run immediately` OFF (the cassette only covers parent-extract LLM calls — see helper docstring), and asserts the new row renders with its axis chip. Then selects parent + new iteration to load `ExtractCompareView`, asserting the summary chips show `Only in A: 2 / Changed: 0 / 2 cells compared` and the heatmap header includes the column name. Proves `createExtractIteration → fullIterationList → compareExtracts` end-to-end without burning a second LLM round-trip.

- **VCR.py wrapper for LLM calls in `doc_extract_query_task`** — `opencontractserver/utils/vcr_replay.py` exposes a `maybe_vcr_cassette()` context manager that, when `OC_LLM_VCR_MODE` and `OC_LLM_VCR_CASSETTE` are set on the celery worker, records or replays every HTTP call to LLM provider hosts (currently `api.openai.com` / `api.anthropic.com`). A custom request-body matcher strips volatile values (millisecond timestamps, Django document PKs, OpenAI tool-call IDs, UUIDs) so a cassette recorded against one DB replays cleanly against another. With the env vars unset the wrapper is a no-op — production behavior is unchanged. Pre-recorded cassette for the E2E extract spec lives at `opencontractserver/tests/fixtures/cassettes/e2e_extract_pdf_workflow/extract.yaml`. Replay was verified end-to-end against a deliberately-fake `OPENAI_API_KEY` to confirm no real network call is made. See `docs/development/e2e_vcr.md` for record / replay / debug instructions.
- **`.github/workflows/frontend-e2e-extract.yml`** — CI workflow that runs the E2E extract spec against the full `local.yml` stack with `OC_LLM_VCR_MODE=replay` and a fake `OPENAI_API_KEY`. Triggered on `pull_request` (path-filtered to changes that can affect extract behaviour) in addition to `workflow_dispatch`. PDF parsing routes to the in-stack Docling microservice (`PDF_PARSER=docling`); the workflow forces `PipelineSettings.preferred_parsers["application/pdf"]` to Docling explicitly after `migrate` so a pre-existing DB singleton can't silently route through LlamaParse. No external service credentials required.
- **`frontend/tests/e2e/extract-pdf-workflow.spec.ts`** — full-stack Playwright E2E spec for the extract pipeline: login → create corpus → upload two PDFs (`frontend/tests/fixtures/{usc-title-1,eton-agreement}.pdf`) → wait for parse + embedding → create extract with one column → run with a real OpenAI call → CSV export → assert non-empty cells. Adds new helpers to `frontend/tests/e2e/helpers.ts` (`uploadPdfViaUI`, `waitForDocumentReady`, `createExtractViaUI`, `openExtractByName`, `addColumnViaUI`, `addDocumentsToExtractViaUI`, `runExtractAndWaitForFinish`). Gated on `E2E_RUN_LLM_TESTS=true`; skipped in CI until LLM responses can be mocked over the wire. Runs on the live `local.yml` stack; required tweaks to disable Auth0 (`.envs/.local/.django USE_AUTH0=false`) and to widen the celeryworker `watchfiles --ignore-paths` (in `compose/local/django/celery/worker/start` and the `local.yml` command pointer) so editor / Playwright artifact writes don't hot-reload the worker mid-task. Also adds `data-testid="document-card"` (+ `data-processing` on the `/documents`-view variants) to `frontend/src/views/Documents.tsx` and `data-testid="document-card"` to `frontend/src/components/documents/ModernDocumentItem.tsx`, so tests can poll for the `backendLock` UI signal without depending on hover-only action menus. Cards are matched by `[data-testid="document-card"]` filtered with the visible title text — the standard Playwright pattern.
- **Mypy: type analyzer, shared, agents, badges, worker_uploads; introduce shared protocols** (Issue #1335): Brought the five smaller, interface-rich target packages over the ≥70% return-annotation bar called for by the issue and seeded `opencontractserver/types/protocols.py` with the four protocols requested in the scope:
  - `VectorStoreProtocol` — minimum surface (`search` / `async_search`) implemented by `CoreAnnotationVectorStore` (`opencontractserver/llms/vector_stores/core_vector_stores.py`); imported and re-exported from that module so consumers can annotate against the protocol rather than the concrete dataclass.
  - `PipelineComponentProtocol` — `title` / `description` / `author` / `dependencies` surface that the pipeline registry duck-types against; imported from `opencontractserver/pipeline/base/base_component.py` so any future parser/embedder/thumbnailer registered outside the inheritance hierarchy still type-checks against the same contract.
  - `ToolProtocol` — `name` / `description` / `parameters` / `requires_approval` mirror of `CoreTool` (`opencontractserver/llms/tools/tool_factory.py`); framework adapters can accept any object satisfying it, no inheritance required.
  - `PermissionedQueryManagerProtocol` — `visible_to_user(user) -> QuerySet` contract that `BaseVisibilityManager`, `PermissionManager`, `DocumentManager`, `AnnotationManager`, and `NoteManager` all satisfy (`opencontractserver/shared/Managers.py`); imported there so callers receiving "a permissioned manager" can type against the protocol instead of a concrete class.
  - `StreamObserverProtocol` — duplicated `__call__(event)` shape from `opencontractserver/llms/types.StreamObserver` for callers in non-LLM modules (notifications, websockets) that need the contract without importing `llms.types`.
  - **Coverage delta** (return-annotation coverage measured by AST walk, excluding `__init__.py`):
    - `analyzer/`: 12.5% → **87.5%** (target ≥70%)
    - `shared/`: 38.1% → **96.3%** (target ≥70%)
    - `agents/`: 34.4% → **71.9%** (target ≥70%)
    - `badges/`: 47.1% → **100%** (target ≥70%)
    - `worker_uploads/`: 46.4% → **100%** (target ≥70%)
  - **Files touched** (annotations only, zero behavior changes): `analyzer/{apps,checks,signals,startup,utils,admin,admin_views}.py`, `analyzer/management/commands/sync_doc_analyzers.py`, `agents/{apps,admin,memory}.py`, `badges/{apps,signals,models}.py`, `worker_uploads/{apps,auth,serializers,views,models,tasks}.py`, `shared/{utils,defaults,fields,mixins,Managers,QuerySets,decorators}.py`, plus the four protocol-consumer wirings above.
  - **Bare-generic promotion**: every `dict` / `list` / `set` in newly-touched public signatures was widened to a parametrised form (`dict[str, Any]`, `list[Any]`, `dict[str, int]`, etc.). The `HasEmbeddingMixin` docstring example was tightened to match (`-> dict[str, Any]`).
  - **`# type: ignore` audit**: every bare `# type: ignore` comment in the codebase now carries a specific error code. `opencontractserver/llms/tools/core_tools.py:413,416,418,420` (channels/asgiref import + `partial` kwarg-aware wrapper) tightened to `[import-not-found]` / `[call-arg]` with a one-line comment explaining why; `opencontractserver/utils/embeddings.py:171` to `[attr-defined]`; `opencontractserver/tests/test_pipeline_utils.py:395,423,431,438,446` cleaned up the duplicated `# type: ignore; type: ignore` markers and scoped them to `[import-not-found]`; `opencontractserver/tests/test_core_tool_factory.py:34,35,47` widened from bare to `[attr-defined]`. The total count went from 62 → 61, and the ratio of bare-to-scoped dropped to zero.
- **Return-type annotations across `config/graphql/` resolvers and mutations** (Issue #1332, follow-up to #1331): The largest, least-typed subtree in the backend (459 function definitions, ~4.4% return-annotation coverage at baseline) is now at 100% return-annotation coverage. Touched files include every `*_mutations.py`, every `*_queries.py`, every `*_types.py`, plus `filters.py`, `base.py`, `base_types.py`, `security.py`, `optimized_file_resolvers.py`, `permissioning/permission_annotator/{middleware,mixins,utils}.py`, and the small utility modules. No behavioral changes — annotations only.
  - **`mutate(...)` on `graphene.Mutation` subclasses**: typed as forward references to the enclosing class (`-> "ClassName"`). Discovered and fixed the latent bug in `config/graphql/analysis_mutations.py:179` (`DeleteAnalysisMutation.mutate`) where the success path had no `return` statement; annotation is `-> "DeleteAnalysisMutation | None"` and an explicit `return None` was added to preserve the original implicit-None behavior on success.
  - **`resolve_*` methods**: typed as `-> Any` by default, refined where the GraphQL field type makes the runtime return obvious (e.g. `resolve_in_use -> bool`, `resolve_datacell_count -> int`).
  - **`AnnotatePermissionsForReadMixin`** (`config/graphql/permissioning/permission_annotator/mixins.py`): per the issue's specific guidance, `resolve_my_permissions -> list[str]`, `resolve_is_published -> bool`, `resolve_object_shared_with -> list[dict[str, Any]]`. The pre-existing wrong annotation `list[PermissionTypes]` (an Enum, while the implementation returns plain strings) was corrected to `list[str]`. The now-unused `PermissionTypes` import was removed.
  - **Filter / queryset helpers** (`filter_by_*`, `text_search_method`, `get_node`, `get_queryset`, `_get_*`, etc.) typed as `-> Any` to keep the change conservative; tightening to `QuerySet[Model]` is a follow-up.
  - **`config/graphql/permissioning/permission_annotator/utils.py`** had a broken import (`config.graphql.permission_annotator.middleware` instead of `config.graphql.permissioning.permission_annotator.middleware`) — fixed in passing.
  - **`config/graphql/conversation_types.py`**: replaced `base64.binascii.Error` with a direct `binascii.Error` import (pre-existing — `base64` re-exports `binascii` at runtime but `mypy` doesn't see the re-export).
  - **Var-annotated additions**: `id_to_children: dict[Any, list[Any]]` in `base_types.py`, `read_only_fields: list[str]` in `serializers.py`, `this_model_permission_id_map: dict[int, str]` etc. in middleware.
  - **Five modules graduated from the mypy baseline** (`mypy.ini` → no longer `ignore_errors = True`): `config.graphql.base_types`, `config.graphql.conversation_types`, `config.graphql.permissioning.permission_annotator.middleware`, `config.graphql.permissioning.permission_annotator.utils`, `config.graphql.serializers`. Their entries in `docs/typing/mypy_baseline.txt` (11 lines) were also pruned. Future PRs can graduate the remaining baselined files as the structural issues they expose (custom `visible_to_user` manager method not seen by `django-stubs`, `set_permissions_for_obj_to_user` signature mismatch, mixin `_meta` access) are addressed.
  - **Tooling**: zero new `# type: ignore` markers; black & isort applied; `flake8 config/graphql/` clean. `mypy --config-file mypy.ini opencontractserver config` passes with the updated baseline.
- **Mypy: graduated `opencontractserver/users/tasks.py` out of the baseline** (Issue #1333 follow-up): `tasks.py` was the last `opencontractserver.users` module still suppressed in `mypy.ini`. PR #1370 left it untyped because the file is only loaded when `settings.USE_AUTH0=True`, so it never failed at runtime under the test settings; the typing gap kept the package short of the issue's "all four packages at ≥80% return-annotation coverage" Done-When criterion. Added return + parameter annotations to all five Auth0 sync tasks (`get_new_auth0_token`, `apply_data_to_user`, `sync_remote_user`, `ensure_valid_auth0_token`, `get_user_details_async`), introduced a module-level docstring documenting the `USE_AUTH0` gating, and removed the `[mypy-opencontractserver.users.tasks] ignore_errors = True` section. Local `data` rebound from request body (`dict[str, str]`) to response payload (`dict[str, Any]`) was split into two distinctly-named variables (`request_data` / `payload`) so the types are unambiguous; behavior is unchanged. No callers needed updating — `config/graphql_auth0_auth/utils.py` still consumes `sync_remote_user.delay(...)` exactly as before.

### Fixed

- **Structured extraction agent loops on tool calls instead of committing** (Issue #1414, follow-up to #1381 / #1413): The E2E spec from PR #1413 surfaced a real `failure_mode=no_final_response` (the classifier added by PR #1399). With `gpt-4o-mini`, a strict prompt ("verbatim title from page 1"), and a 9-page PDF, the agent received the answer in its very first `load_document_text(start=0, end=500)` call and then made **97 more sequential 500-byte byte-range reads** through the entire body without ever calling `final_result*`. The structured runner exhausted `output_retries=3` and returned `None`. Two underlying biases were addressed in the structured-extraction system prompts:
  - **Tool-loop without commit** — the agent had the answer but treated reading as the entire task. The `2-3 distinct search queries` rule (added in #1381) was worded as an unconditional precondition to committing, which encouraged exhaustive reading whenever the prompt was constrained. All three `_build_structured_system_prompt` overrides in `opencontractserver/llms/agents/pydantic_ai_agents.py` (`PydanticAICoreAgent`, `PydanticAIDocumentAgent`, `PydanticAICorpusAgent`) now lead with a **COMMIT-EARLY rule**: as soon as a tool result contains a confident answer, the agent MUST stop calling tools and commit by calling the result tool. The 2-3-search rule is now explicitly **negative-case-only** — it bounds giving up, not committing — and rule precedence is stated inline so the model can't conflate the two.
  - **Tool selection bias** — for prompts that map cleanly onto pre-chunked, embedded structural annotations (titles, parties, defined terms, specific clauses), the agent skipped `similarity_search` entirely and went straight to byte-range reads. The same prompt overrides now state that `similarity_search` is the **preferred first step** for fact-finding queries because the search index is already chunked and ranked over structural annotations, and reserve `load_document_text` (and other byte-range readers) for whole-document tasks (summaries, exhaustive review) or as a fallback when search clearly misses. The corpus override carries the matching guidance for document-coordination tools.
  - **Tests** — `opencontractserver/tests/test_pydantic_ai_agents.py` extended: `test_document_agent_structured_prompt_commits_to_result`, `test_corpus_agent_structured_prompt_commits_to_result`, and `test_core_agent_base_structured_prompt_commits_to_result` now assert on the new `COMMIT-EARLY` and `similarity_search` / `load_document_text` guidance, and the base-agent test asserts that the old unconditional "Before concluding the requested information is absent, you MUST issue at least 2-3 distinct search queries…" wording is **gone** — that exact phrasing is what produced the loop in #1414.
  - **Out of scope** — no model-side changes (no temperature override, no `output_retries` change), no rework of the structural-annotation similarity index. The classifier from #1381 / #1399 still surfaces any residual `no_final_response` if the model continues to loop on a different prompt shape.
  - **End-to-end verification (PR follow-up)** — re-recorded `opencontractserver/tests/fixtures/cassettes/e2e_extract_pdf_workflow/extract.yaml` against the new prompt with the in-stack Docling parser, then replayed the spec from PR #1413 cassette-only. The originally-failing USC cell now produces `"TITLE 1-GENERAL PROVISIONS"` (body text from page 1) and the Eton cell commits its description fallback — both non-empty. The cassette has 13 OpenAI interactions and contains zero occurrences of the old `Before concluding the requested information is absent…` wording, confirming the prompt the agent was asked under is the new one.
  - **VCR matcher hardening (`opencontractserver/utils/vcr_replay.py`)** — required for the re-record to replay against fresh DBs. (1) Strip Django auto-increment PKs (`annotation_id` / `corpus_id` / `document_id` / `label_id` / generic `id`) from request bodies; PKs are echoed back into subsequent assistant calls inside JSON-encoded tool messages, so the patterns match both the raw-JSON form and the escaped-string form (`\"annotation_id\\":N`). (2) Strip `similarity_score` / `score` floats — the embedder microservice and pgvector hybrid-search fusion produce slightly different numeric scores even for identical inputs across runs. (3) Replay drops body matching entirely — `record_mode="none"` now matches on `(method, scheme, host, port, path)` only and plays interactions back in cassette order. The vector-embedder microservice is real (not cassette-wrapped because it is not an LLM provider) and returns slightly different similarity orderings against a fresh corpus, which leaks chunk content into subsequent request bodies that no static stripper can normalize. Sequential playback is sufficient because the celery worker processes datacells sequentially, the model is effectively deterministic at the temperatures used, and the cassette captures the full successful conversation per document. Record mode keeps strict body matching so partial re-records still produce a deterministic cassette. New unit tests in `opencontractserver/tests/test_vcr_replay.py` cover the PK stripper (raw + escaped forms) and the similarity-score stripper.
  - **CI activation (`.github/workflows/frontend-e2e-extract.yml`)** — flipped from `workflow_dispatch`-only to `pull_request` (path-filtered to changes that can affect the extract agent or the cassette / matcher / spec). Forces `PDF_PARSER=docling` and explicitly resets `PipelineSettings.preferred_parsers` to Docling after `migrate` so a stale DB singleton can't silently route through LlamaParse. No external service credentials needed.
  - **Test-helper fixes for `frontend/tests/e2e/helpers.ts`** uncovered while replaying the spec end-to-end:
    - `addDocumentsToExtractViaUI` no longer hard-fails when the "Add documents" modal renders zero cards — the corpus auto-populates extract rows on creation, so the second `addDocuments` call legitimately has nothing to add. Probes for cards with a short timeout and bails to "Cancel" on the empty-modal path.
    - `runExtractAndWaitForFinish` now waits for the AG-Grid `Loading…` placeholder to detach after the "Extraction in progress" overlay clears. Previously the helper returned as soon as the backend marked the Extract row complete, but the per-cell `data` payloads were still mid-flight via Apollo, which made cell-text assertions intermittently see whitespace. Source of the "row has no non-empty extracted cell" false-negative when the spec was first re-run.
    - `waitForDocumentReady` no longer gates on the outer "Your documents" page heading. The route can render the document-card grid before its surrounding chrome (the heading lives in an outer layout component that occasionally hydrates last); the helper now waits directly on the document card it cares about with an inner 30s ceiling, since the card is a sufficient signal that the documents view loaded. Source of an intermittent 8-minute hang on `/documents` re-navigation.
- **Anthropic models silently fail in `doc_extract_query_task`** (Issue #1381): When `doc_extract_query_task` was run with an Anthropic / Claude model, ~85% of cells failed with the canonical "extraction returned None — the requested information may not be present" message even though the document contained the answer. Inspecting `Datacell.llm_call_log` for failed cells showed Claude's last assistant message was always `text` + `tool_use` parts, never a final structured response — pydantic-ai's structured-response runner treated this as no result and returned `None`. The error message conflated three distinct outcomes ("agent committed to None", "agent never produced a final structured response", "agent looped on the same tool call") under one ambiguous string. Three coordinated changes:
  - **`opencontractserver/llms/agents/pydantic_ai_agents.py`** — All three `_build_structured_system_prompt` overrides (`PydanticAICoreAgent`, `PydanticAIDocumentAgent`, `PydanticAICorpusAgent`) now explicitly tell the agent that **after gathering enough information from the tools, it MUST commit to the final structured response by calling the result tool**. Wording is universal — harmless for OpenAI and necessary for Claude. `_structured_response_raw` now passes `output_retries=STRUCTURED_OUTPUT_RETRIES` (=3) to `PydanticAIAgent` so pydantic-ai retries the final-result tool call when the model fails to commit on the first pass. When an Anthropic model is used and the caller did not pin a temperature, structured runs force `temperature=0` (Claude is reluctant to commit; non-zero temperature pushes it toward more exploratory text). **Cost note**: bumping `output_retries` from pydantic-ai's default of 1 to 3 means a cell that fails the first commit attempt can incur up to 3 result-tool round trips. In the previous behaviour those cells silently returned `None` instead of retrying, so the worst-case per-cell LLM cost on Anthropic models can roughly triple compared to the broken baseline. This is the correct tradeoff — the previous baseline produced no answer — but operators should anticipate the cost shift in billing for Anthropic-driven extractions.
  - **`opencontractserver/constants/llm.py`** — Hosts `STRUCTURED_OUTPUT_RETRIES`, `TOOL_LOOP_THRESHOLD`, `EXTRACT_DEFAULT_TEMPERATURE`, and the `NONE_RESULT_*` vocabulary used by the classifier (see below).
  - **`opencontractserver/utils/llm.py`** — New `is_anthropic_model()` helper so call sites outside the agents layer (notably `data_extract_tasks.doc_extract_query_task`) can decide whether to pass `temperature=None` and let the Anthropic guard activate.
  - **`opencontractserver/tasks/data_extract_tasks.py`** — New `_classify_none_result(messages)` helper inspects the captured pydantic-ai message log and returns one of `agent_committed_none` (a `final_result*` tool call appears — legitimate "data not present"), `no_final_response` (no `final_result*` anywhere — pipeline integration failure), `tool_loop_no_output` (same tool call repeated ≥ `TOOL_LOOP_THRESHOLD`× without final — pipeline bug), or `unknown`. The `Datacell.stacktrace` now records `failure_mode=<classification>` plus a human-readable message (the integration-failure variants reference issue #1381) so operators can `grep failure_mode=` to separate legitimate "not present" outcomes from pipeline bugs. New `_resolve_extract_temperature(model_name)` helper picks the temperature passed to the structured runner: returns `None` for Anthropic models so `_structured_response_raw`'s `temperature=0` override fires automatically, and `EXTRACT_DEFAULT_TEMPERATURE` (0.3) otherwise. This closes the latent footgun where flipping `DEFAULT_EXTRACT_MODEL` to a Claude model would have silently bypassed the reliability fix because `temperature=EXTRACT_DEFAULT_TEMPERATURE` was passed unconditionally.
  - **`opencontractserver/tests/test_data_extract_failure_classification.py`** — `SimpleTestCase` suite covering the classifier (empty input, `final_result` detection, `final_result_<TypeName>` suffix variants, no-tool-calls path, tool-call-without-final path, repeated-call loop detection, threshold-minus-one boundary, loop-then-commit precedence, mixed text + tool path, non-`ModelResponse` skip, JSON-string `args` normalisation, malformed JSON `args` defensive path, unhashable `args` `repr` fallback), `is_anthropic_model` (prefix, bare-name, OpenAI / `gpt-4` / `o1` rejection, empty/None), and `_resolve_extract_temperature` (Anthropic→None, OpenAI→default, unknown→default, current-default sanity check).
  - **`opencontractserver/tests/test_pydantic_ai_agents.py`** — New `_structured_response_raw` tests pinning the Anthropic temperature override (forces 0 when caller passes `temperature=None`, respects function-level pin, respects `config.temperature` pin, leaves OpenAI runs untouched), and three smoke tests for the strengthened `_build_structured_system_prompt` overrides covering the document, corpus, and core base agents.
- **Extraction grounding follow-up** (Issue #1246, follow-up to original #1245 grounding pipeline):
  - **Bug — silent `page=1` fallback corrupted multi-page PDF grounding** (`opencontractserver/utils/extraction_grounding.py`, `_create_pdf_annotation`): when PlasmaPDF could not determine a page for a span, the previous code logged a warning and saved the annotation on page 1 anyway. For multi-page PDFs this produced a structurally incorrect annotation pinned to the wrong page (and therefore the wrong bounding box context), so users clicking through to the source landed on a different page than the one containing the extracted text. Fixed: `_create_pdf_annotation` now raises `ValueError` inside its `transaction.atomic()` savepoint, the savepoint rolls back, and the outer per-result `try/except` in `_create_grounding_annotations` logs it as a failed grounding attempt. Best-effort grounding is preserved (other annotations in the batch are unaffected) but no annotation is ever saved with a wrong page.
  - **Bug — label-set lookup outside the per-annotation guard caused all-or-nothing failure** (`opencontractserver/utils/extraction_grounding.py`, `_create_grounding_annotations`): `corpus.ensure_label_and_labelset(...)` was invoked once before the per-annotation `try/with transaction.atomic()` loop. A failure to materialise the label-set (e.g. a transient DB error or a pre-existing constraint conflict) propagated out, was caught by the outer `try/except` in `data_extract_tasks.py`, and silently dropped _all_ groundings for the datacell. Moved the call inside the savepoint so a label-lookup failure only skips the affected annotation.
  - **Bug — duplicate `OC_EXTRACT_SOURCE` annotations on Celery retry** (`opencontractserver/utils/extraction_grounding.py`, `_create_pdf_annotation` & `_create_span_annotation`): nothing prevented the grounding pipeline from creating fresh annotations and re-linking them via `datacell.sources.add(*annotations)` if `ground_extraction_to_annotations` ran twice on the same datacell (Celery retry after partial failure was the realistic trigger). Replaced the construct-then-`save()` flow with `Annotation.objects.get_or_create()` keyed on `(document, annotation_label, annotation_type, raw_text, …)` so retries reuse existing rows. `datacell.sources` is a `ManyToManyField`, so re-linking the same row is already a no-op once the row is shared.
  - **Constant — extracted `DOCX_MIME_TYPE`** (`opencontractserver/constants/document_processing.py`): the long `application/vnd.openxmlformats-officedocument.wordprocessingml.document` literal previously lived inline in `_load_document_text_and_layer`. Per the project's no-magic-strings rule it now sits next to `MARKDOWN_MIME_TYPE` and is imported from one place.
  - **Type annotations** (`opencontractserver/utils/extraction_grounding.py`): `Document`, `Corpus`, `Datacell`, `Annotation`, and `AnnotationLabel` parameters and return types added via a `TYPE_CHECKING` block on every public and helper function. No runtime change.
  - **Documentation** — the `page=1` placeholder for SPAN_LABEL annotations (text/DOCX) is now documented in the function docstring, explaining that the `txt_extract_file` pipeline does not preserve a page-break map and the actual location lives in the character offsets in `json`.
  - **Tests** — `opencontractserver/tests/test_extraction_grounding.py`:
    - `TestGroundingPipelinePDFIntegration` (new class): builds a synthetic two-page PAWLS payload (no real PDF binary needed), runs grounding through `build_translation_layer`, and verifies (a) annotations land on the correct page, (b) re-running grounding is idempotent, and (c) when PlasmaPDF returns `page=None` the annotation is **skipped** instead of being saved on page 1.
    - `test_ground_text_document_is_idempotent`: regression for the duplicate-annotation bug on the SPAN_LABEL path.
- **`CreateCorpusActionModal` opened with the wrong default agent instructions for document triggers** (Issue #1385, `frontend/src/components/corpuses/CreateCorpusActionModal.tsx:136-144,168-171`): the `inlineAgentInstructions` state was initialised with `DEFAULT_MODERATOR_INSTRUCTIONS` even though the default trigger is `add_document` (a document trigger). The trigger-change handler at line 611 swaps to `DEFAULT_DOCUMENT_AGENT_INSTRUCTIONS`, but a user who created an inline agent on the default-selected trigger without first re-selecting the trigger would submit the moderator copy as the new agent's system instructions. Initialised both the `useState` default and `resetForm()` to `DEFAULT_DOCUMENT_AGENT_INSTRUCTIONS` so the pre-interaction value matches the default trigger. Updated `frontend/tests/CreateCorpusActionModal.ct.tsx` "inline-agent create: full happy path" mutation mock to expect `DEFAULT_DOCUMENT_AGENT_INSTRUCTIONS` — the previous mock variable masked this bug because `MockedProvider` was matching the stale moderator default rather than the trigger-appropriate one.

### Changed

- **Test/type cleanup follow-ups from the PR #1383 review** (Issue #1385):

  - Pinned the `isProcessing` contract for SYNC_CONTENT in `frontend/tests/CorpusChat.ct.tsx` "SYNC_CONTENT renders a complete message immediately": added an `expect(input).toBeEnabled()` assertion after the reply renders, locking the documented invariant that `setIsProcessing(true)` is owned solely by `ASYNC_START` and that a SYNC_CONTENT-only reply must never disable the input.
  - Consolidated the duplicated `::: oc-component` fence dispatcher: extracted `OcComponentBlock` interface and a new `buildOcComponentCustomBlocks(renderMarkdown)` helper into `frontend/src/utils/camlComponents.ts`. Both `frontend/src/hooks/useCamlComponentRenderer.tsx` and `frontend/src/components/corpuses/caml/CamlDirectiveRenderer.tsx` now share the same helper instead of each casting `block` independently.
  - Replaced `route: any` and `page: any` escape hatches with the proper `Route` and `Page` types from `@playwright/test` in `frontend/tests/CorpusDescriptionEditor.ct.tsx` (`setupMdRoute` and the abort-route test).
  - Migrated `.version-number` CSS-class locators in `frontend/tests/CorpusDescriptionEditor.ct.tsx` to a semantic `data-testid="version-number"` matcher (`page.getByTestId("version-number")`); added the test id to the rendered version-row in `frontend/src/components/corpuses/CorpusDescriptionEditor.tsx`.

- **`test_superuser_sees_all_queryset` miscounts personal corpuses by 1** (Issue #1394, `opencontractserver/tests/test_visibility_managers.py`, `opencontractserver/tests/test_resolvers.py`): Two `VisibleToUserTests.test_superuser_sees_all_queryset` cases asserted that `Corpus.objects.visible_to_user(superuser).count() == 4` (public + private + 2 personal), but the actual count is 5 because the test DB starts with a pre-existing personal corpus owned by django-guardian's `AnonymousUser` (created during fixture setup before/around the username-based skip in `opencontractserver/users/signals.py::user_created_signal`). The assertion is now scoped to corpuses created by the test's two users (`creator__in=[self.user, self.superuser]`), making it resilient to any fixture-level corpuses that exist at test DB init time. Production code is unchanged.
- **Merged `frontend` Codecov flag drops to ~33% on every commit where Frontend CI's CT job fails** (`frontend/package.json` `test:coverage:ct`): the script chained `playwright test ... && mkdir -p ... && nyc report ...`, so a failing CT run short-circuited before `nyc report` could turn the per-test JSON files in `.nyc_output` into an `lcov.info`. The downstream `Upload CT Coverage to Codecov` step (`if: success() || failure()`) then errored with "No coverage reports found" and `frontend-component` did not upload for that SHA. Codecov's server-side aggregation of the `frontend` flag was left with only `frontend-unit` (~23%) and `frontend-e2e` (~24%), pulling the merged number down to ~33% even though the previous commit was at ~67% — observed on six consecutive main commits 2026-04-26T01:02..02:58Z (`2d7033f8`..`be5bcfc8`) before recovering on `30298391`. Mirrored the existing `test:e2e:coverage` pattern (`; CT_EXIT=$?; nyc report ... || echo "No coverage data to report"; exit $CT_EXIT`) so `nyc report` runs regardless of test outcome and the lcov ships even on red CT runs. `frontend-component` will still report a slightly lower number when tests fail (failed tests register fewer hits), but it will report — keeping the merged `frontend` flag's denominator stable.
- **`User.__init__` shared-state mutation re-introduced by branch merge** (`opencontractserver/users/models.py:172-180` removed): PR #1374 (commit `50ed6740`) deleted the `User.__init__` override that mutated `Field.validators[0]` on every instantiation, but a subsequent merge (`b68c1cb4 → 6d2cddbf`) resurrected the override along with its mypy-narrowing changes. The current main on commit `6d2cddbf` therefore reproduced the original `#1358` bug: `User(...)` rebound `username_field.validators[0]` and clobbered any third-party validator prepended to the list. Removed the `__init__` override entirely; the class-body declaration `validators=[UserUnicodeUsernameValidator()]` on the `username` field (still present from PR #1374) is the canonical and only declaration. Also dropped the now-unused `Field` import. Regression coverage from PR #1374 (`opencontractserver/tests/test_user_username_validator.py`) was already on main and is what surfaced the regression in CI.
- **Decomposed `opencontractserver/llms/tools/core_tools.py` into a package** (Issue #1445): The 2,981-line monolith has been split into the `opencontractserver/llms/tools/core_tools/` package with one submodule per tool category — `md_summaries`, `notes`, `text_extracts`, `descriptions`, `document_summaries`, `annotations`, `document_indexing`, `search`, `page_images`, `links`, `documents`, `memory`, plus a private `_helpers` for shared utilities (`_token_count`, `_db_sync_to_async`, `_apply_ndiff_patch`). The package's `__init__.py` re-exports every previously top-level name (functions, classes, `_DOC_TXT_CACHE`, and the `Document`/`Note`/`Corpus`/`CorpusDescriptionRevision`/`NoteRevision` model references that tests patch directly), so no consumer import paths change. Behavior-preserving — each function body is identical to the original; this is purely a structural refactor to make individual tool families reviewable in isolation. Documentation references in `docs/architecture/pawls-format.md`, `docs/architecture/document_annotation_index.md`, `docs/architecture/llms/README.md`, `docs/walkthrough/step-7-query-corpus.md`, and the `opencontractserver/corpuses/template_seeds.py` comment were updated to point at the new package paths.

### Security

- **`RemoveAnnotationLabelsFromLabelsetMutation` allowed any user to strip labels from public labelsets** (`config/graphql/label_mutations.py`): The resolver used `Q(creator=user) | Q(is_public=True)` with no further check, so `is_public` (a read flag) effectively granted write access to non-owners. Replaced with `user_has_permission_for_obj(user, labelset, PermissionTypes.UPDATE, include_group_permissions=True)`. Denial raises `LabelSet.DoesNotExist` so the response is byte-identical to the not-found case (no IDOR information leak). Coverage: `test_remove_labels_rejects_non_owner_even_when_labelset_is_public`.
- **`CreateLabelForLabelsetMutation` ignored guardian UPDATE grants** (`config/graphql/label_mutations.py`): The resolver was creator-only, so collaborators with explicit `update_labelset` guardian permission (or a group grant) could not add labels even though the consolidated permissioning guide says edit rights on a `LabelSet` should permit add/remove/edit of its labels. Now uses the same `PermissionTypes.UPDATE` gate as `Remove`, with the matching IDOR-safe deny path. Coverage: `test_non_owner_with_explicit_update_permission_can_create`. Both mutations now have a dedicated `LabelSet.DoesNotExist` handler that logs at WARNING (not ERROR) so auth probes don't pollute logs with stack traces.
- **Cross-corpus structural-annotation leak in `CoreAnnotationVectorStore`** (`opencontractserver/llms/vector_stores/core_vector_stores.py:296-326,371-413`): The corpus-wide retrieval path (`corpus_id` set, `document_id=None`) returned every structural annotation in the database regardless of corpus. Two collaborating defects caused the leak:
  1. `Q(structural=True)` in the corpus-only branch had **no corpus constraint** — parser-produced structural annotations have `Annotation.document_id = corpus_id = NULL`, so corpus membership is only knowable through `structural_set → Document.structural_annotation_set (reverse FK) → DocumentPath.corpus_id`, a join the previous code did not perform.
  2. The `check_corpus_deletion` block (default `True`) added `Q(document_id__in=active_doc_ids)`, and `__in` lookups never match `NULL`, so structural annotations were silently dropped on the production-default path. Bypassing this filter with `check_corpus_deletion=False` exposed defect #1 directly.
  - **Impact**: Multi-tenant deployments leaked structural annotations across tenant corpora — a real security boundary violation since the upfront IDOR check only validated the _requested_ `corpus_id`, not the _returned_ rows. Single-tenant deployments saw it as a corpus-scoping / search-quality bug (e.g. corpus-wide benchmark runs returned chunks from abandoned corpora). The standard `Annotation.objects.visible_to_user()` permission filter was bypassed entirely because the vector store builds its own filter chain rather than going through that manager method.
  - **Fix** (corpus boundary + per-document visibility for the structural class): the corpus-only branch now requires `structural_set_id__in=<sets reachable from a document in this corpus that is visible to the user>`, joining through `Document.objects.visible_to_user(user).filter(path_records__corpus_id=...)`. The deletion-aware filter accepts both `document_id__in=active` AND structural rows whose set links to one of those active documents, so parser-produced structural annotations remain reachable on the default path. `CoreAnnotationVectorStore.global_search()` was already correct (it explicitly joins via `structural_set__documents__in=accessible_doc_ids`) and is unchanged.
  - Regression coverage: `opencontractserver/tests/test_corpus_isolation_vector_store.py` — six tests covering cross-corpus leak, deletion-aware drop, orphan-set leak, document-scoped retrieval still returns structural rows, viewer-without-doc-permission excluded, creator still sees own row.
- **Test-only**: `opencontractserver/tests/test_pydantic_ai_agents.py`, `opencontractserver/tests/test_structural_annotation_portability.py` — `Document.objects.create(...)` calls in `TransactionTestCase` setUp now pass `processing_started=timezone.now()` to short-circuit `process_doc_on_create_atomic`, which would otherwise eagerly chain a Celery PDF-ingest task that fails on the (file-less) test document and aborts the whole test class. Pre-existing failure, exposed cleanly when the regression suite was added.

### Fixed

- **`Embedding.embedder_path` could be NULL but was typed `str`** (Issue #1357, `opencontractserver/annotations/models.py:461-465`, `opencontractserver/annotations/models.py:584-585`, `opencontractserver/annotations/migrations/0068_enforce_embedder_path_not_null.py`): The Django field was declared `null=True, blank=True` while the Python annotation claimed `str`, causing a long-standing mypy `assignment` error and — more importantly — silently gutting the partial unique constraints added in migration 0059. Each `unique_embedding_per_{document,annotation,note,conversation,message}_embedder` constraint is conditioned on `<parent>__isnull=False` and keys on `(embedder_path, <parent>)`, so any row with `embedder_path IS NULL` bypassed duplicate prevention for its parent. Every production code path that creates an `Embedding` (`Embedding.objects.store_embedding()`, `HasEmbeddingMixin.add_embedding()`, `worker_uploads._store_embeddings()`) already supplies a concrete `embedder_path` or skips creation when empty, so enforcing non-null at the DB level matches actual behaviour rather than constraining it. New migration 0068 backfills any legacy NULL rows with `settings.DEFAULT_EMBEDDER` (deleting rows that would collide with an existing `(default_embedder_path, parent)` row under the partial unique constraint — they were previously unreachable via any query path since all call sites filter on a concrete embedder path), then `AlterField`s the column to `NOT NULL`. Removed the now-unreachable `or 'Unknown Model'` fallback in `Embedding.__str__`. Migration runs with `atomic = False` so the RunPython backfill commits before `AlterField` takes the `ACCESS EXCLUSIVE` lock to set `NOT NULL`, matching the pattern established by migration 0059.

### Added

- **Frontend `any`-baseline gate wired into CI** (Issue #1448): The frontend has no ESLint pipeline, so explicit `any` accretion was silent — ~449 type-position uses across 123 files at the time of this change. New script `frontend/scripts/check-any-baseline.js` walks `frontend/src/**/*.{ts,tsx}`, counts `: any | as any | <any> | any[] | Array<any> | Promise<any> | ReadonlyArray<any>` (skipping comment-only lines), and compares against the committed `frontend/.any-baseline.json` snapshot (total + per-area breakdown across `knowledge_base`, `annotator`, `widgets_chat`, `widgets_other`, `components_other`, `graphql`, `hooks`, `atoms`, `routing`, `utils`, `types`, `other`). New scripts: `yarn any:check` (regression-only), `yarn any:check:strict` (also fails when the count drops without the baseline being lowered, keeping the snapshot honest), `yarn any:write` (regenerate after a reduction). The Lint job in `.github/workflows/frontend.yml` runs `any:check:strict` on every PR. Workflow + rationale documented in `docs/frontend/any-baseline.md`. Per the issue, follow-up PRs are expected to drain the prioritised areas (`knowledge_base` 33 → … → 0, `annotator` 90 → …, `widgets_chat` 2 → 0) by replacing `any` with types from `frontend/src/types/graphql-api.ts` / GraphQL codegen output, calling out the per-area delta in each PR.
- **Pluggable text chunking strategies for `TxtParser`** (Issue #1348, alongside PR #1239): Introduced `opencontractserver/pipeline/parsers/text_chunkers.py` — a small registry-backed abstraction (`BaseTextChunker` + `TextChunk` + `get_chunker`) with three built-in strategies: `SentenceChunker` (spaCy `doc.sents`, preserves pre-#1348 behaviour and emits the existing `SENTENCE` label), `ParagraphChunker` (blank-line split with optional `min_chars` filter and `max_chars` oversize-paragraph fallback, emits `PARAGRAPH`), and `SlidingWindowChunker` (fixed-character window with configurable `overlap` and optional `respect_word_boundaries` snap, emits `WINDOW`). `TxtParser` now declares a `Settings` dataclass with a `chunkers: list[ChunkerSpec]` field (default `[{"name": "sentence"}]`) that can be overridden via `PipelineSettings` _or_ per-call via a `chunkers=[...]` kwarg on `parse_document`; the parser iterates the configured strategies and emits one structural SPAN_LABEL annotation per chunk under each strategy's label, so stacked configurations (e.g. sentence + paragraph) index multiple retrieval granularities simultaneously. Motivates the benchmark work in #1239: the LegalBench-RAG `probe_recall_at_10` gap on `privacy_qa` (0.22 observed vs 0.5–0.8 paper floor) is the thesis for needing paragraph-granularity retrieval units, but this PR is strategy-neutral — which chunker wins for which subset is a follow-up optimisation to be driven by the benchmark harness itself. Regression coverage in `opencontractserver/tests/test_text_chunkers.py` (pure-Python, no Django DB) exercises offset/whitespace invariants, overlap arithmetic, word-boundary snapping, argument validation and registry lookup; `test_txt_ingestor_pipeline.py` gains two integration tests that parse the live fixture with a paragraph-only and a stacked paragraph+sliding_window recipe. Existing sentence-only ingestion path is unchanged.
- **Global post-retrieval reranker for vector search** (Issue #1349): Adds an optional cross-encoder reranking stage that runs after first-stage vector / hybrid retrieval, so OpenContracts can close the gap between vanilla HNSW recall and the accuracy achievable with a cross-encoder scoring pass.
  - New abstract base class `opencontractserver.pipeline.base.reranker.BaseReranker` wired into the existing `PipelineComponentBase` settings machinery: concrete subclasses declare a `Settings` dataclass (loaded from `PipelineSettings` at runtime) and implement `_rerank_impl(query, passages, **kwargs)`. A default `_arerank_impl` wraps the sync implementation via `sync_to_async` so every backend has a working async path without duplicating logic.
  - Fault-tolerant helpers `safe_rerank` / `safe_arerank` swallow reranker failures and return `None` so retrieval degrades gracefully to the first-stage ordering — critical because a misconfigured reranker must never take down semantic search.
  - Four shipped backends in `opencontractserver/pipeline/rerankers/`:
    - `NoopReranker` — identity pass-through for tests and benchmark control conditions.
    - `CrossEncoderReranker` — in-process `sentence_transformers.CrossEncoder` (default `BAAI/bge-reranker-v2-m3` per the issue); lazy model load cached by `(model_name, device)` so workers pay the ~300 MB cost once and reuse it on every query. `sentence-transformers` / `torch` are treated as optional dependencies; a missing install surfaces a clear `ImportError` only when this backend is actually selected.
    - `MicroserviceReranker` — HTTP client that mirrors the shape of `MicroserviceEmbedder` (URL, optional API key, Cloud-Run IAM auth, retry-friendly timeouts). Operators can run any reranker model behind a `/rerank` endpoint and point OpenContracts at it via `RERANKER_MICROSERVICE_URL` (+ secret `RERANKER_MICROSERVICE_API_KEY`).
    - `CohereReranker` — hosted Rerank API (`rerank-v3.5` by default) via the REST endpoint directly (no hard dep on the `cohere` SDK). API key stored in the encrypted `PipelineSettings.encrypted_secrets` bag under `cohere_api_key` (env var `COHERE_API_KEY` at migration time).
  - New `ComponentType.RERANKER` enum value and `rerankers/` auto-discovery in `opencontractserver.pipeline.registry`; `PipelineComponentRegistry` now exposes `.rerankers` / `get_all_rerankers_cached()` alongside parsers, embedders, thumbnailers, and post-processors.
  - `PipelineSettings.default_reranker` (CharField, max_length=512, `documents/models.py:852-980`) — empty string disables reranking; any value is a full class path resolved at runtime. Seeded from `DEFAULT_RERANKER` Django setting at migration time (`documents/migrations/0037_add_default_reranker_to_pipeline_settings.py`). Helpers `get_default_reranker_path()` / `get_default_reranker_class()` / `get_default_reranker_instance()` in `opencontractserver.pipeline.utils`, with a process-local instance cache (cross-encoder model weights are expensive) invalidated via `invalidate_reranker_cache()` on every settings update.
  - `CoreAnnotationVectorStore` (`opencontractserver/llms/vector_stores/core_vector_stores.py:120-1041`) now accepts an optional `reranker` override + `rerank_oversample_factor` kwarg. Every search path — `search`, `async_search`, `hybrid_search`, `async_hybrid_search`, `global_search`, `async_global_search` — oversamples candidates by `RERANK_OVERSAMPLE_FACTOR` (default 3× the requested `top_k`, hard-capped by `RERANK_MAX_CANDIDATES = 128`) when a reranker is active and re-orders results through `_apply_rerank` / `_aapply_rerank` before returning the final `top_k`. All new plumbing is a no-op when `default_reranker` is empty, so zero behavior change for existing deployments.
  - GraphQL surface: `PipelineComponentsType.rerankers`, `PipelineSettingsType.default_reranker`, and `UpdatePipelineSettingsMutation.default_reranker` (validated against the registry, invalidates the reranker instance cache on change).
  - New constants in `opencontractserver.constants.search`: `RERANK_OVERSAMPLE_FACTOR`, `RERANK_MAX_CANDIDATES`, `RERANK_DEFAULT_TOP_K`. New `RERANKER_REQUEST_TIMEOUT_SECONDS` in `opencontractserver.constants.document_processing`.
  - Tests in `opencontractserver/tests/test_reranker.py` cover the base-class contract (sorting, top_k trim, out-of-range indices, max-candidates, async fallback), `safe_rerank` / `safe_arerank` fault-tolerance, all three HTTP backends with mocked `requests.post`, pipeline utility resolution + instance caching, registry auto-discovery, and vector-store integration (oversample factor, reranker failure fallback, re-ordering effects).
- **Mypy graduation: typed GraphQL resolvers, mutations, and filters** (Issue #1332): Raised return-annotation coverage in `config/graphql/` from ~4.8% at the start of #1331 to **91.5%** (421/460 function defs) and removed 22 modules from the `mypy.ini` baseline allow-list.
  - **Root-cause annotation fixes in `opencontractserver/utils/permissioning.py`**: `set_permissions_for_obj_to_user`, `user_has_permission_for_obj`, `get_users_permissions_for_obj`, and `get_permission_id_to_name_map_for_model` were previously annotated with `instance: type[django.db.models.Model]` (a class) despite every call site passing an instance — and with `user: type[User]` instead of the `User` runtime instance. These were annotation bugs (the code was correct, the annotations were inverted), which compounded: every mutation calling `set_permissions_for_obj_to_user(user, obj, ...)` was a single `[arg-type]` error each. Corrected to `instance: django.db.models.Model` / `user: UserModel` (forward-referenced via `TYPE_CHECKING` import of `opencontractserver.users.models.User`). Also added the missing `dict[int, str]` annotation on `this_model_permission_id_map` and removed the `user_instance=User` (class) default on `get_users_group_ids`, which would have exploded at runtime if any caller ever omitted the argument. Module graduated out of the baseline.
  - **Graduated from `mypy.ini` baseline** (22 modules): `config.graphql.{action_queries, agent_mutations, badge_mutations, base_types, conversation_mutations, conversation_types, corpus_types, document_queries, filters, ingestion_source_mutations, moderation_mutations, og_metadata_queries, pipeline_queries, security, serializers, slug_queries, smart_label_mutations, social_types, user_queries, user_types, voting_mutations}` and `opencontractserver.utils.permissioning`. Each had the underlying mypy errors fixed first (root-cause in `permissioning.py` cleared the `set_permissions_for_obj_to_user` cluster across every mutation file above).
  - **Per-file type fixes**:
    - `config/graphql/slug_queries.py` & `config/graphql/user_types.py` & `config/graphql/social_types.py` & `config/graphql/corpus_types.py`: Reversed `.filter(...).visible_to_user(user)` → `.visible_to_user(user).filter(...)` so the custom manager method (typed on the manager) resolves before `.filter()` flattens to the base `QuerySet[Model]` that django-stubs doesn't know carries `visible_to_user`. Semantics are preserved — both orderings AND the conditions. The CLAUDE.md permissioning docs already recommend the manager-first pattern.
    - `config/graphql/og_metadata_queries.py`: Guarded against `Extract.corpus` being `None` (the FK uses `on_delete=SET_NULL`, so `corpus` is nullable in the DB but the OG metadata resolver was treating it as non-null).
    - `config/graphql/pipeline_queries.py`: Narrowed the `mimetype` optional before passing to `get_components_by_mimetype_cached` (which required `str`), and typed `components_data` as `dict[str, Sequence[PipelineComponentDefinition]]` so both branches (which return `list[...]` vs `tuple[...]`) type-check against the unified annotation.
    - `config/graphql/ingestion_source_mutations.py`: Replaced `if error:` with `if pk is None:` in two call sites — the error-then-continue pattern left mypy unable to narrow `pk: str | None` through the conditional. Functionally equivalent (`error is None ⟺ pk is not None` by construction of `_parse_ingestion_source_global_id`).
    - `config/graphql/conversation_types.py`: Fixed `base64.binascii.Error` → `binascii.Error` with an explicit `import binascii` — `base64` doesn't re-export `binascii` as an attribute, so the reference was broken under `warn_unused_ignores`.
    - `config/graphql/filters.py`: Coerced `from_global_id(value)[1]` (returns `str`) to `int` before passing to `folder_id` lookup.
    - `config/graphql/security.py`: Replaced `CsrfViewMiddleware(lambda req: None)` with a typed `_csrf_noop_get_response(request) -> HttpResponse` so the middleware's `get_response` contract is satisfied in-types; switched `wrapped_view.csrf_exempt = True` and `request._dont_enforce_csrf_checks = True` to `setattr(...)` to avoid typing-only attribute errors against Django stubs that don't carry these internal flags. Behaviour identical.
    - `config/graphql/moderation_mutations.py`: Added an explicit `Union[ChatMessage, Conversation, None]` annotation on `target` where the surrounding `if/else` mixes the two types (mypy can't unify across branches without the hint).
    - `config/graphql/action_queries.py`: Coerced `from_global_id(...)[1]` to `int` at the three call sites where it feeds `for_corpus` / `for_document` custom queryset methods (which expect `int` PKs).
  - **Docs & baseline maintenance**: `mypy.ini` baseline section for `config.graphql` trimmed from 35 modules to 14; 63 matching error lines pruned from `docs/typing/mypy_baseline.txt` so the reference file matches the live baseline.
  - **Known remaining bugs surfaced by mypy** (filed as separate issues per the scope rules of #1332):
    - #1359 — `RemoveLabelsFromLabelsetMutation` calls non-existent `labelset.documents`. Silent runtime failure (swallowed by a broad `except Exception`). Blocks `config.graphql.label_mutations` graduation; one-line fix + test needed.
    - #1360 — `DRFMutation.IOSettings` declares `model: django.db.models.Model = None` and `serializer = None`. Non-trivial refactor of the base mutation class; blocks `config.graphql.base` graduation.
- **Coverage: raise Corpus Chat & Agent Management component tests** (Issue #1276): added 36 new Playwright CT tests across the four lowest-ROI corpus components to drive coverage toward the ≥60% target. Breakdown:
  - `frontend/tests/CorpusChat.ct.tsx` (+13 tests): `initialQuery` auto-send, tool-call timeline entries (ASYNC_THOUGHT), ASYNC_SOURCES merge, SYNC_CONTENT rendering, ASYNC_RESUME, ask_document sub-tool approval remapping, unknown-type default branch, back-to-list navigation, server-message-with-sources rendering, title-filter debounce, and additional navigation-header coverage. Extended the shared `StubSocket` in `beforeEach` with new query-triggered frame sequences.
  - `frontend/tests/CreateCorpusActionModal.ct.tsx` (+8 tests): analyzer-path validation, inline-agent validation (empty name / empty instructions), existing-agent-selection validation, successful inline-agent mutation, backend error toast, analyzer edit-mode pre-population, and legacy trigger-casing normalization fallback.
  - `frontend/tests/CorpusAgentManagement.ct.tsx` (+8 tests): query loading state, query error state, multi-tool badge overflow, inactive-status badge, update-mutation happy path, create-mutation backend-error toast, tool deselection, and edit-modal cancel.
  - `frontend/tests/CorpusDescriptionEditor.ct.tsx` (+7 tests): save failure (`ok: false`), save network-error path, reapply of snapshot-less version, twice-click collapse, Cancel Version Edit reset, fetch-md URL failure, and version-count pluralization.
  - **Follow-up review polish**: moved `DEFAULT_DOCUMENT_AGENT_INSTRUCTIONS` from `CreateCorpusActionModal.tsx` into `frontend/src/assets/configurations/constants.ts` so both default-instruction strings (moderator + document agent) live in the single constants module per the project's no-magic-strings rule.
- **Return-type annotations across core models and import/export pipeline** (Issue #1334, follow-up to #1331): The mypy gate wired in by #1331 recorded a 7208-error baseline frozen across 357 files. This PR pays down the annotation deficit on the core domain models and the bulk import/export tasks without touching runtime behavior or adding validators. Coverage jumped from the pre-issue numbers to:
  - `opencontractserver/corpuses/` 61.5% → 88.4% (target ≥80%)
  - `opencontractserver/annotations/` 48.1% → 93.8% (target ≥80%)
  - `opencontractserver/documents/` 51.9% → 84.9% (target ≥80%)
  - `opencontractserver/extracts/` 47.1% → 94.1% (target ≥75%)
  - `tasks/import_tasks.py`, `tasks/import_tasks_v2.py`, `tasks/export_tasks.py`, `tasks/export_tasks_v2.py` all at 100% function-signature coverage.
  - **Files touched** (annotations only — zero behavior changes, zero new comments, zero renames): `annotations/{models,signals,admin}.py`, `corpuses/{models,signals,managers}.py`, `documents/{models,signals}.py`, `extracts/{models,signals}.py`, `tasks/{import_tasks,import_tasks_v2,export_tasks,export_tasks_v2}.py`.
  - Each file adopts `from __future__ import annotations` so forward references work without string quoting, and uses a `TYPE_CHECKING` block for the handful of cross-app imports (`AbstractBaseUser`, `Corpus`, `Document`, `Annotation`, etc.) that would otherwise be circular at runtime.
  - **TypedDict adoption** — the import/export task signatures now consume existing TypedDicts from `opencontractserver/types/dicts.py` directly instead of bare `dict`: `OpenContractsExportDataJsonPythonType` / `OpenContractsExportDataJsonV2Type` for the data.json payloads, `OpenContractDocExport` for per-document payloads, `OpenContractsAnnotationPythonType` / `OpenContractsRelationshipPythonType` for annotation/relationship lists, `IngestionSourceExport` / `DocumentPathExport` / `StructuralAnnotationSetExport` / `CorpusFolderExport` / `AgentConfigExport` / `DescriptionRevisionExport` / `ConversationExport` / `ChatMessageExport` / `MessageVoteExport` for V2-specific shapes, and `OpenContractsAnnotatedDocumentImportType` for single-document imports. No new TypedDicts were introduced — all were already defined and previously unused in callers.
  - Signal handlers (`annotations/signals.py`, `corpuses/signals.py`, `documents/signals.py`, `extracts/signals.py`) now have typed `sender` / `instance` / `created` / `**kwargs` parameters with `TYPE_CHECKING` imports of the sender model classes.
  - **Manager methods**: `CorpusActionExecutionManager.visible_to_user` now types the optional `user` param as `Optional[AbstractBaseUser]` (previously bare default), and `summary_by_status` / `summary_by_action` tightened from bare `dict` / `QuerySet` to `dict[str, int]` / `QuerySet[Any]`.
  - **Model method signatures**: `save()`/`clean()`/`delete()` overrides are now `-> None` (Django's `Model.save` returns None) or `-> tuple[int, dict[str, int]]` (Django's `Model.delete` signature); `__str__` returns `str`; `@property` accessors have explicit scalar returns; classmethod factories return `"ClassName"`; tree/versioning helpers on `CorpusFolder` type their descendant queries as `QuerySet[CorpusFolder]` / `list[CorpusFolder]` where applicable (falling back to `Any` where the CTE queryset class isn't easily importable).
  - **Graduation from the mypy baseline is deferred.** The `mypy.ini` `[mypy-…] ignore_errors = True` sections for these modules still contain pre-existing Django-plugin-specific errors (field descriptor `assignment` mismatches, `misc` lambdas in model field declarations, base-class variable override warnings on `embedder_path`/`creator`) that are not caused by missing annotations and therefore cannot be fixed under this issue's "annotations only, no bugfixes" constraint. The annotations landed here make those modules ready for a follow-up PR that either `# type: ignore`s or refactors the remaining Django-model type issues and then removes the baseline entries.
- **Mypy type-checking wired into pre-commit and CI** (Issue #1331): The existing `[mypy]` block in `setup.cfg` and the `mypy==1.20.1` / `django-stubs==6.0.2` / `djangorestframework-stubs==3.16.9` pins in `requirements/local.txt` were never actually enforced, so the investment was drifting (48 pre-existing `# type: ignore` markers, many modules at 0% annotation coverage). This PR turns on the gate without requiring the 7208 existing errors across 357 files to be fixed first — `mypy.ini` lists each of those files under its own `[mypy-<module>] ignore_errors = True` section, so new modules added outside the baseline **are** type-checked and CI / the hook fails on their errors.
  - `mypy.ini` (split out of `setup.cfg` because the per-module baseline is ~1000 lines): `python_version` bumped `3.9` → `3.11` to match the runtime, plugins kept, `django_settings_module` pointed at the new `config/settings/mypy.py` (dummy `DATABASE_URL` default so contributors don't need env vars).
  - `.pre-commit-config.yaml`: new `mirrors-mypy@v1.20.1` hook with `pass_filenames: false` and fully pinned stubs + Django runtime in `additional_dependencies` (pre-commit autoupdate only bumps `rev`, so unpinned stubs would drift).
  - `.github/workflows/backend.yml`: `Run mypy` step added to the `linter` job; the preceding `pip install -r requirements/local.txt` step satisfies the django-stubs plugin.
  - `docs/typing/README.md`: how to run locally / via pre-commit / in the test container, plus the per-file graduation workflow.
  - `docs/typing/mypy_baseline.txt`: frozen error list (sorted for stable diffs) so follow-up issues can measure progress.
- **Mypy: typed the auth, users, and notifications packages** (Issue #1333, follow-up to #1331): Brought `config/admin_auth`, `config/graphql_api_token_auth`, `opencontractserver/users`, and `opencontractserver/notifications` to full return-annotation coverage and graduated them out of the mypy baseline. These packages sit on hot paths for authentication, token verification, session handling, and notification dispatch; typing them documents invariants that used to live only in `CLAUDE.md`.
  - Signal handlers (`opencontractserver/users/signals.py`, `opencontractserver/notifications/signals.py`) now declare `sender: type[Model]`, `instance: Model`, `created: bool`, `**kwargs: Any`, and `-> None` so drive-by edits can't silently change the signal contract.
  - `opencontractserver/notifications/__init__.py` gained a module-level docstring calling out that the app deliberately diverges from `AnnotatePermissionsForReadMixin` — a single `recipient` FK is the authoritative visibility gate. The design note in `CLAUDE.md` is referenced so changes to one file flag the other.
  - `mypy.ini`: dropped `[mypy-config.admin_auth.*]`, `[mypy-config.graphql_api_token_auth.backends]`, and `[mypy-opencontractserver.users.models]` `ignore_errors` sections. Added a narrow `disable_error_code = django-manager-missing` on `opencontractserver.users.models` only, documented inline: `django-tree-queries`' `as_manager(with_tree_fields=True)` pattern used by `Corpus`, `CorpusFolder`, and `DocumentPath` creates manager classes at runtime that the `mypy_django_plugin` can't introspect, so reverse `_set` accessors on `User` can't resolve — a known plugin limitation, not a code error. Graduating those three models with explicit manager typing will let us delete the disable.
  - `docs/typing/mypy_baseline.txt`: pruned all 20 entries for the graduated modules.
  - Bug fixes uncovered by typing:
    - `config/graphql_api_token_auth/backends.py`: `ApiKeyBackend.authenticate_header` referenced `self.keyword` but the attribute was never declared. DRF calls this on 401 responses to build the `WWW-Authenticate` header, so an unauthenticated request would have hit `AttributeError` at response time. Added `keyword: str = "Token"` as a class attribute.
    - `opencontractserver/notifications/signals.py`: `create_moderation_notification` dereferenced `action.moderator.username` unconditionally, but the `ModerationAction.moderator` FK permits `NULL`. A moderation action created without a moderator would crash the signal handler (and therefore the originating save). Added an explicit early return with a debug log when `action.moderator is None`.

### Changed

- **Decomposed `ModernDocumentItem.tsx`** (Issue #1446, `frontend/src/components/documents/ModernDocumentItem.tsx`, `frontend/src/components/documents/ModernDocumentItem.styles.ts` (new), `frontend/src/components/documents/DocumentRelationshipList.tsx` (new)): The component had grown to 1,699 lines with ~755 lines of `styled-components` definitions and a relationship-popup body duplicated verbatim across the card and list view branches. Extracted all styled-components into a sibling `ModernDocumentItem.styles.ts` (re-exported by name) and lifted the duplicated relationship popup body into a shared `DocumentRelationshipList` component used by both views. Net: main component file dropped from 1,699 → 870 lines (49% reduction) while remaining behaviorally identical — the public `ModernDocumentItem` export, prop signature, and DOM structure are unchanged. All 42 component tests in `frontend/tests/ModernDocumentItem.ct.tsx` and `frontend/tests/document-failure-overlay.ct.tsx` pass without modification, confirming the refactor is semantics-preserving. First slice of the multi-PR decomposition tracked by issue #1446.
- **Centralized extract and user-profile route resolution in `CentralRouteManager`** (`frontend/src/routing/CentralRouteManager.tsx`, `frontend/src/components/routes/ExtractDetailRoute.tsx`, `frontend/src/components/routes/UserProfileRoute.tsx`, new `frontend/src/components/routes/ProfileRedirect.tsx`, `frontend/src/utils/navigationUtils.ts`, `frontend/src/graphql/cache.ts`): `ExtractDetailRoute` and `UserProfileRoute` were each calling `useParams()`, running their own GraphQL resolution queries, and writing entity reactive vars — duplicating the four-phase flow documented in `docs/frontend/routing_system.md` and racing the manager during back-navigation. Both are now thin consumers that read `openedExtract` / `openedUser` / `routeLoading` / `routeError`. `parseRoute` learned `/users/:slug` and `/extracts/:extractId`; the manager added a `GET_USER` lazy query and a Phase 1 user branch alongside the existing extract handling. New `openedUser` reactive var (typed `OpenedUserProfile`) joins the routing-owned set. `/profile` is now served by a small `ProfileRedirect` that uses `backendUserObj` to redirect to `/users/<slug>` — auth-driven, not URL-driven, so legitimately outside the manager. Smaller fixes that ride along: `views/Corpuses.tsx:1807` swaps `window.history.replaceState` for `navigate({ pathname, search }, { replace: true })` so query-param mutations stay inside React Router; redundant `openedLabelset(null)` calls in `LabelSetDetailPage` and `LabelSetLandingRoute` handlers were removed (Phase 1 already clears the var on browse navigation).
  - Test coverage: 8 new `parseRoute` tests, 3 new manager tests (user resolve, user not-found, extract by id), updated `beforeEach` to reset `openedExtract` / `openedUser`. New `frontend/src/routing/__tests__/centralRouteDiscipline.test.ts` is a static regression test that grep-walks `frontend/src/` and fails if any production file outside the manager and `cache.ts` SETs one of the 15 routing-owned reactive vars — caught three pre-existing `openedLabelset` writes during development that were also fixed in this PR. New `frontend/tests/e2e/user-and-extract-routes.spec.ts` deep-links `/users/<slug>`, verifies the `/profile` redirect, and exercises the dumb-consumer error path on `/extracts/<unknown-id>`. `tests/e2e/helpers.ts` `VIEWS` catalog now includes `/users/admin` so the existing login-and-navigation walk also covers the user route.
  - Doc updates: `docs/frontend/routing_system.md` route-pattern table now lists `/users/:slug`, `/extracts/:extractId`, and `/label_sets/:labelsetId`; the "ONLY CentralRouteManager may SET" list and the four critical RULE blocks were extended to include `openedThread`, `openedLabelset`, and `openedUser`; the new discipline test is referenced as the CI enforcement mechanism.
- **Pre-existing `tsc` failure on `frontend/src/components/corpuses/caml/CamlDirectiveRenderer.tsx`** resolved by refreshing `@os-legal/caml-react` to `0.1.0` (latest on the registry). The lockfile had been pinned to `0.0.1`, whose published `dist/index.d.ts` lacks the `resolveImageSrc` prop that the consumer passes through; `0.1.0` ships the prop on `CamlArticleProps`, `CamlChapterRendererProps`, and the block-renderer prop interfaces. `yarn upgrade @os-legal/caml-react@^0.1.0 @os-legal/caml@^0.1.0` updates the lockfile; no code changes needed.
- **Simplified `RelationGroup.updateForAnnotationDeletion` pruning logic** (Issue #1317, follow-up to #1314, `frontend/src/components/annotator/types/annotations.ts:40-60`): The method previously branched on four near-duplicate conditions (`sourceEmpty && nowTargetEmpty`, `targetEmpty && nowSourceEmpty`, `!sourceEmpty && nowSourceEmpty`, `!targetEmpty && nowTargetEmpty`) each returning `undefined`. All four are equivalent to a single `nowSourceEmpty || nowTargetEmpty` check (`filter` is monotonic, so an originally-empty side stays empty after filtering). Collapsed the branches and removed the now-unused `sourceEmpty` / `targetEmpty` locals per the project's DRY guideline. Behavior is unchanged; existing regression tests in `frontend/src/components/annotator/types/__tests__/annotations.test.ts` still pass unmodified, confirming the simplification is semantics-preserving.
- **Frontend Vitest unit coverage provider switched from V8 to Istanbul** (`frontend/vite.config.ts:210-230`, `frontend/package.json:154`, `frontend/yarn.lock`): The merged `frontend` Codecov flag was landing at ~44% even though `frontend-component` alone was at ~61% on the same code — impossible for a union-aggregated metric unless the three per-suite uploads were measuring on different yardsticks. Root cause: Vitest's `@vitest/coverage-v8` provider emits ~183 `DA:` records per source file (~86k across ~480 files) because V8's native coverage API reports hits for nearly every executable line — imports, declarations, block-closing `}`, etc. — as needed for engine profiling and DevTools. `vite-plugin-istanbul` (used by Playwright CT and E2E suites) emits ~61 `DA:` per file (~27k total) because it only instruments statements. Same code, ~3× denominator mismatch. When Codecov unions multiple uploads under one flag it keeps V8's larger line-number set; Istanbul hits from CT/E2E land on a subset of those line numbers and can never close the gap. Swapped the unit-coverage provider to Istanbul so all three suites report on the same universe. Swapped the `@vitest/coverage-v8` devDep for `@vitest/coverage-istanbul` at the same `^3.1.2` version to match the `vitest` major.minor, changed `coverage.provider` to `"istanbul"` in the Vitest config, regenerated `yarn.lock`, and updated the `all: true` rationale comment to stop singling out V8. The per-suite `flags:` tagging in `frontend.yml` / `frontend-e2e.yml` is unchanged — server-side aggregation still handles the merge. Verified locally: the new unit lcov reports ~451 `SF:` and ~24k `DA:` (previously ~86k), matching the CT/E2E scale. Trade-off: unit-test runtime under coverage grows somewhat (Istanbul transforms source pre-execution), likely 10–30s on this suite. `tsc --noEmit` clean.

### Removed

- **Dead GraphQL operations and result types removed** (Issue #1244, `frontend/src/graphql/landing-queries.ts`, `frontend/src/graphql/queries/folders.ts`, `frontend/src/graphql/metadataOperations.ts`, `frontend/src/graphql/queries.ts`, `frontend/src/graphql/mutations.ts`): Removed GraphQL operations and associated TypeScript Input/Output types that were not referenced anywhere in `frontend/src/` or `frontend/tests/`.
  - `landing-queries.ts`: `GET_TRENDING_CORPUSES`, `GET_RECENT_DOCUMENTS` (+ `GetRecentDocumentsOutput`), `GET_COMMUNITY_STATS` (query only — `GetCommunityStatsOutput` interface is still referenced internally by `GetDiscoveryDataOutput`), `GET_GLOBAL_LEADERBOARD` (+ `GetGlobalLeaderboardOutput`).
  - `queries/folders.ts`: `GET_CORPUS_FOLDER` (singular — kept the live `GET_CORPUS_FOLDERS` plural query), `MOVE_DOCUMENTS_TO_FOLDER` (plural — kept the live `MOVE_DOCUMENT_TO_FOLDER` singular), plus the `GetCorpusFolderInputs/Outputs`, `MoveDocumentsToFolderInputs/Outputs`, and `GetDeletedDocumentsInputs/Outputs` type pairs. Kept `GET_DELETED_DOCUMENTS_IN_CORPUS` and `DeletedDocumentPathType` (used by `TrashFolderView.tsx`).
  - `metadataOperations.ts`: `GET_METADATA_COMPLETION_STATUS`, `GetDocumentMetadataDatacellsInput/Output`, `GetMetadataCompletionStatusInput/Output`, `DocumentMetadataResult` (inlined into `GetDocumentsMetadataBatchOutput`).
  - `queries.ts`: `USER_BY_SLUG`, `CORPUS_BY_SLUGS`, `DOCUMENT_BY_SLUGS` (kept live `DOCUMENT_IN_CORPUS_BY_SLUGS` and `RESOLVE_*_FULL` variants), `GET_LABELSET_BY_ID_FOR_REDIRECT`, `REQUEST_PAGE_ANNOTATION_DATA`, `GET_EXPORT` (kept live `GET_EXPORTS`), `GET_FIELDSET` (kept live `GET_FIELDSETS`), `GET_DOCUMENT_ANNOTATIONS_AND_RELATIONSHIPS`, `getAnnotationsByDocumentId`, `listAnnotations`, `GET_DOCUMENT_DETAILS` (kept live `GET_DOCUMENT_KNOWLEDGE_AND_ANNOTATIONS`), `GET_UNREAD_NOTIFICATION_COUNT`, `GET_DOCUMENT_RELATIONSHIP_COUNT`, plus all associated Input/Output interfaces.
  - `mutations.ts`: `UPDATE_LABELSET`, `CREATE_ANNOTATION_LABEL` (kept live `CREATE_ANNOTATION_LABEL_FOR_LABELSET`), `SMART_LABEL_LIST` (kept live `SMART_LABEL_SEARCH_OR_CREATE`), `REMOVE_ANNOTATION_LABELS_FROM_LABELSET`, `DELETE_ANNOTATION_LABEL` (kept live `DELETE_MULTIPLE_ANNOTATION_LABELS`), `DELETE_DOCUMENT` (kept live `DELETE_MULTIPLE_DOCUMENTS`), `REQUEST_DELETE_DOC_TYPE_ANNOTATION`, `UPDATE_CORPUS_SETTINGS`, `UPDATE_BADGE`, `AWARD_BADGE`, `REVOKE_BADGE` (kept live `DELETE_BADGE`), `DELETE_CONVERSATION`, `MARK_NOTIFICATION_READ`, `MARK_NOTIFICATION_UNREAD`, `MARK_ALL_NOTIFICATIONS_READ`, `DELETE_NOTIFICATION` (entire notification-mutations block had no call sites), `PERMANENTLY_DELETE_DOCUMENT`, `UPDATE_DOCUMENT_RELATIONSHIP`, `DELETE_DOCUMENT_RELATIONSHIPS` (kept live `DELETE_DOCUMENT_RELATIONSHIP` singular), plus all associated Input/Output interfaces.
  - Verification: every deletion was preceded by a word-boundary grep (`\bSYMBOL\b`) across both `frontend/src/` and `frontend/tests/`; interfaces that were only referenced by the deleted operations were removed, while interfaces still referenced internally (e.g. `GetCommunityStatsOutput` used by `GetDiscoveryDataOutput`) were preserved. `tsc --noEmit` and `yarn build` pass cleanly.
  - `types/graphql-api.ts` and `types/graphql-slug-queries.ts` were intentionally left untouched per the issue scope — test wrappers and mock fixtures import many type names from these files, and auto-generated structural types are too risky to prune by hand.

### Fixed

- **DRFMutation/DRFDeletion `IOSettings` misconfiguration surfaces as a generic internal error** (Issue #1360, `config/graphql/base.py`): `DRFMutation.IOSettings` declared `model: django.db.models.Model = None`, `graphene_model: DjangoObjectType = None` and `serializer = None` — the type annotations were wrong (instance types, not class types, holding `None`), and there was no runtime guard. A subclass that forgot to override one of those would fail deep inside `mutate()` with an `AttributeError` (`'NoneType' object has no attribute 'objects'`) or `TypeError` (`'NoneType' object is not callable`), which the broad `except Exception` then masks as a generic "internal error" message back to the API caller. `DRFDeletion.IOSettings` had the inverse problem — it didn't declare `model` at all, even though `DRFDeletion.mutate` dereferences `cls.IOSettings.model.objects`. Discovered while working on #1332 (typing): mypy flagged four errors on `base.py:128/204/238/252` plus `base.py:93 "type[IOSettings]" has no attribute "model"`.
  - Retyped both `IOSettings` declarations with `ClassVar[Optional[type[...]]] = None` so the annotation matches reality (holding a class, not an instance, and nullable by default). `pk_fields` narrowed from `list[str | int]` to `list[str]` — the only call sites use it as a dict key.
  - Added `_require_io_setting(mutation_cls, name)` helper that fails fast with `NotImplementedError("<Subclass>.IOSettings.<name> must be set by the subclass.")` when a required attribute is `None` or missing. Called for `model`/`serializer`/`graphene_model` in `DRFMutation.mutate` and for `model` in `DRFDeletion.mutate`. In `DRFMutation` the error is caught by the broad `except Exception` and surfaced as "internal error" to the API, with the full descriptive message in the server log via `logger.error(traceback.format_exc())`. `DRFDeletion.mutate` has no equivalent `try/except` (matching its pre-PR behavior), so the `NotImplementedError` propagates to graphene as a GraphQL execution error — also a fail-fast outcome, just on a different surface.
  - Tightened `DRFDeletion.mutate` to raise `ValueError` when `kwargs[lookup_field]` is missing (the `id` GraphQL argument is `required=False`), instead of passing `None` into `from_global_id` and getting an opaque `AttributeError`. Same propagation surface as the `_require_io_setting` guard above.
  - Fixed pre-existing `to_global_id(graphene_model.__class__.__name__, obj.id)` in both `DRFMutation` create/update return paths: `graphene_model` is the `DjangoObjectType` class, so `.__class__` referenced graphene's metaclass (`SubclassWithMeta_Meta`) and the resulting global id used the wrong type name. Switched to `graphene_model.__name__` to match the GraphQL type name (e.g. `"CorpusType"`) used everywhere else in `config/graphql/`. Surfaced by the new explicit `ClassVar[Optional[type[...]]]` annotations — both call sites stayed dormant because the returned `obj_id` is rarely consumed by the frontend (which re-queries by global id from its own cache).
  - Graduated `config.graphql.base` out of the mypy baseline: removed the `[mypy-config.graphql.base]` section from `mypy.ini` and pruned the ten corresponding lines from `docs/typing/mypy_baseline.txt`. `mypy --config-file mypy.ini` is now clean on this module.
  - Regression tests in `opencontractserver/tests/test_security_hardening.py::TestIOSettingsRequiredFieldsGuard`: the helper raises `NotImplementedError` for each of `model`/`serializer`/`graphene_model` when missing or `None`, returns the configured value when present, and the base classes expose the `None` defaults the guard relies on.
- **`CorpusChat` dropped `SYNC_CONTENT` messages from the visible chat** (Issue #1276, `frontend/src/components/corpuses/CorpusChat.tsx:468-505`): The `SYNC_CONTENT` WebSocket frame is a standalone, non-streaming assistant reply used for synchronous server responses. `ChatTray` (document chat) appends these directly to its `chat` state; the corpus-level chat only forwarded the content to `handleCompleteMessage`, which stores sources in `ChatSourceAtom` but never pushes a message to the visible list. As a result, any `SYNC_CONTENT` the backend sent over the corpus socket rendered nothing. Fixed by mirroring the `ChatTray` pattern — push a new complete assistant message into `chat` before persisting sources/timeline. The fallback `crypto.randomUUID()` is also now captured in a single local variable so the visible chat entry and the `ChatSourceAtom` record share the same id when the server omits `message_id`. New regression test in `frontend/tests/CorpusChat.ct.tsx` ("SYNC_CONTENT renders a complete message immediately") pins the behavior.
- **CAML article preview crashed when inserting an extract grid embed** (`frontend/src/utils/camlComponents.ts`, `frontend/src/hooks/useCamlComponentRenderer.tsx`, `frontend/src/components/corpuses/CamlArticleEditor.tsx`, `frontend/src/components/corpuses/caml/CamlDirectiveRenderer.tsx`): the editor wrapped each newly-inserted `[component:TYPE ...]` marker in a `::: prose` fence, but `@os-legal/caml`'s parser has no `case "prose"` in `parseBlock`, so the resulting block carried `body` instead of `content`. `ProseBlock` then crashed inside `splitPullquotes(undefined)`, which unmounted the entire editor modal and made the "ArrowDown then Enter inserts the extract-grid component marker" CT test fail. Switched the fence to a project-specific `::: oc-component` block and routed it through `CamlArticle`'s `customBlocks` slot, where the marker text is handed back to the existing `[component:...]` resolver. The keyboard handler in `CamlArticleEditor` was also tightened to read the active picker index from a `useRef` mirror so back-to-back ArrowDown/Enter keystrokes don't observe a stale closure value of `-1` and bail out before insertion.
- **PR #1177 follow-up: CAML extract embed polish** (Issue #1227):
  - **`fullDatacellList` payload now bounded server-side**: `ExtractType.full_datacell_list` accepts optional `limit` / `offset` arguments and the resolver clamps `limit` to `MAX_FULL_DATACELL_LIST_LIMIT` (`opencontractserver/constants/extracts.py`, currently `500`) after permission filtering (`config/graphql/extract_types.py`). `GET_EXTRACT_GRID_EMBED` passes `limit: EXTRACT_GRID_EMBED_CELL_LIMIT` (mirrored at `500` in `frontend/src/assets/configurations/constants.ts`) so pathological extracts no longer transmit thousands of cells just to trigger the too-many-rows guard (`frontend/src/graphql/queries.ts`, `frontend/src/components/extracts/ExtractGridEmbed.tsx`). Full server-side pagination is still tracked in #1204.
  - **`resolveComponentMarker` now receives a stable React key from both call sites**: `useCamlComponentRenderer` and `CamlDirectiveRenderer` pass the marker string as the `key` argument so multiple `[component:...]` blocks in a single article reconcile correctly without React's "missing key prop" warnings (`frontend/src/hooks/useCamlComponentRenderer.tsx`, `frontend/src/components/corpuses/caml/CamlDirectiveRenderer.tsx`). Added regression tests in `frontend/src/utils/__tests__/camlComponents.test.ts`.
  - **Code-point-safe cell value truncation**: `formatCellValue` in `ExtractGridEmbed` now slices on `Array.from(json)` instead of `String.substring`, so cell values containing emoji or other non-BMP characters are no longer truncated mid-surrogate-pair (which previously emitted `U+FFFD` replacement glyphs).
  - **Keyboard navigation in CAML extract picker**: Added a listbox keyboard handler to the "Insert Extract Grid" dropdown in `CamlArticleEditor` — Arrow keys / Home / End move the focused option, Enter selects, Escape closes and returns focus to the trigger button. Active option is reflected via `aria-activedescendant` and an `$active` highlight on `ExtractPickerItem`.
  - **`buildSourceLink` page-indexing convention documented**: Added an explanatory comment clarifying that `Annotation.page` is 1-based (model default=1) and is only used for the chip label (`p.{page}`) — the document viewer navigates by `annotationId` alone, so there is no URL-layer indexing convention to worry about (`frontend/src/components/extracts/ExtractGridEmbed.tsx`).
  - **Verified `GET_EXTRACTS` already selects `fullDocumentList`** (`frontend/src/graphql/queries.ts:1901`) and that `CamlDirectiveRenderer` still wires `resolveImageSrc` through to `MarkdownMessageRenderer` — both no-ops requested by the issue's verification items.
- **`User.__init__` mutated the shared `Field.validators` list on every instantiation** (Issue #1358, `opencontractserver/users/models.py:150-153`): `User.__init__` overwrote `self._meta.get_field("username").validators[0]` with a fresh `UserUnicodeUsernameValidator()` on every `User(...)` construction — including every ORM hydration, form save, and QuerySet materialisation. Two problems: (1) per-instance allocation + assignment on a hot path, and (2) a hard-coded `validators[0]` index that would silently flip the slot back to Django's default `UnicodeUsernameValidator` if any third-party code or future migration prepended a validator, breaking usernames that rely on `\` or `|`. Fix: declare the validator on the `username` field at class-body time (`opencontractserver/users/models.py:78-90`) and drop the `__init__` override. Added migration `0026_alter_user_username_validator.py` to update the stored field definition. Regression tests in `opencontractserver/tests/test_user_username_validator.py` verify the validator survives repeated instantiation and continues to accept `\`, `|`, `*`.
- **`RemoveLabelsFromLabelsetMutation` silently did nothing** (Issue #1359, `config/graphql/label_mutations.py:296`): The resolver referenced `labelset.documents.filter(pk__in=label_pks)`, but `LabelSet` has no `documents` relation — the M2M to labels is `annotation_labels` (see `opencontractserver/annotations/models.py:1284`). Every invocation therefore raised `AttributeError`, which was swallowed by the surrounding `except Exception as e:` block and returned as a generic `"Error removing label(s) from labelset: ..."` with `ok=False`. Because the frontend `REMOVE_ANNOTATION_LABELS_FROM_LABELSET` mutation was itself unused (#1244 swept it out), this bug went unnoticed in production for an unknown length of time; discovered while grading mypy errors for #1332 (`config/graphql/label_mutations.py:296: error: "LabelSet" has no attribute "documents"  [attr-defined]`). Swapped `documents` → `annotation_labels`, removed the now-resolved error from `docs/typing/mypy_baseline.txt`, and added `opencontractserver/tests/test_label_mutations.py` with four regression cases covering: labels are actually removed from the M2M, IDs not in the labelset are silently ignored, a non-owner / non-public caller cannot mutate the labelset, and a public labelset remains editable (pinning the current `Q(creator=user) | Q(is_public=True)` resolver behaviour so any future permission hardening is explicit).
- **`package_annotated_docs` silently corrupted exports when a document failed to burn** (Issue #1356, `opencontractserver/tasks/export_tasks.py:150-212`, `opencontractserver/utils/etl.py:198-463`, `opencontractserver/tasks/doc_tasks.py:463-504`): `build_document_export()` returns `("", "", None, {}, {})` when a per-document export fails (e.g. the underlying file cannot be loaded). The V1 consumer `package_annotated_docs` had no guard for that placeholder — it ran `doc[1].encode("utf-8")` on the empty string (harmless, no crash), wrote an empty-named entry into the zip, and inserted `annotated_docs[""] = None` into the final `data.json`, so a single failed document silently poisoned the export. The V2 pipeline in `export_tasks_v2.py:126-128` already has this guard; V1 did not. Added `if not doc_name or doc_export is None: continue` (mirroring V2's check) and logged a warning identifying the skipped doc. Also tightened the return-type annotations on `build_document_export` and `burn_doc_annotations`: slots 0 and 1 are always `str` (never `None`; they are empty strings on the failure path), so the signature is now `tuple[str, str, OpenContractDocExport | None, dict[...], dict[...]]`. Corrected the `burned_docs` parameter annotation on `package_annotated_docs` from a single-element tuple-of-tuples to a variadic `tuple[tuple[...], ...]` — the runtime iteration is variadic, and the previous annotation was a red herring uncovered while typing for #1334. Regression test in `opencontractserver/tests/test_package_annotated_docs.py` covers both the mixed-success and all-failed scenarios, asserting the zip contains no empty-named entries and `annotated_docs` holds no `None` values.
- **Frontend coverage badge stuck on "unknown"** (`README.md:12`, `.github/workflows/codecov-notify.yml`, `.github/workflows/frontend.yml`, `.github/workflows/frontend-e2e.yml`, `frontend/package.json`, `frontend/yarn.lock`): PR #1322 pointed the README badge at a new merged `frontend` flag fed by a cross-workflow `lcov-result-merger@5` step inside `codecov-notify.yml`. Every `frontend-merged-coverage` upload since has landed at Codecov with `state: error` / `totals: null`, so the badge rendered "unknown" even though `frontend-unit` (31%), `frontend-component` (61%), and `frontend-e2e` (24%) were all processing correctly. Two defects in the merged lcov confirmed by local repro of the CI merge step: (1) `lcov-result-merger@5` emits a stripped lcov containing only `SF:`, `DA:`, `BRDA:`, `end_of_record` — it drops `TN:`, `FN`, `FNDA`, `FNF`, `FNH`, `LF`, `LH`, `BRF`, `BRH`, so line-summary fields required by Codecov's parser are absent; (2) Vitest v8 emits `src/...` (relative to `frontend/`) while `vite-plugin-istanbul` + `nyc report` emit `/home/runner/work/OpenContracts/OpenContracts/frontend/src/...` (absolute), and the merger keys on the literal path string so the same file appears as two records with conflicting hit counts. `codecov-notify.yml` also ran without `actions/checkout`, which Codecov's action docs explicitly recommend against. Fix: stop merging client-side and let Codecov aggregate server-side, since that is what flags are for. Each per-suite upload now declares two flags — `frontend-unit,frontend`, `frontend-component,frontend`, `frontend-e2e,frontend` — so the `frontend` flag total is the union of the three uploads computed by Codecov. `codecov-notify.yml` is reduced to its original gate-and-notify role (no artifact downloads, no `lcov-result-merger`, no merged upload). Deleted the `frontend-{unit,ct,e2e}-lcov` artifact publishes in `frontend.yml` / `frontend-e2e.yml`, removed the `lcov-result-merger@^5.0.1` devDep and the `coverage:merge` script from `frontend/package.json`, and pruned the orphaned entries from `frontend/yarn.lock`. README badge URL unchanged (`flag=frontend`).

### Changed

- **PR #1297 follow-up — tighten component tests and unit-test coverage** (Issue #1321):
  - `frontend/src/components/documents/__tests__/DocumentRelationshipModal.test.ts`: added a test that pins the `labelType: undefined` edge case for `filterRelationshipLabels` (previously only `null` non-relationship labels were covered), so the strict-equality guard against non-`RelationshipLabel` types is explicit.
  - `frontend/tests/ModernDocumentItem.ct.tsx`:
    - Extended the `__reactProps$` upgrade-risk comments on `clickViaReact` / `openContextMenu` to call out the build-hash suffix explicitly — the suffix rotates on every React build, so `Object.keys(el).find(k => k.startsWith("__reactProps$"))` is the only correct lookup pattern.
    - `openContextMenu` now falls back to a full-document scan for an element with a React `onContextMenu` prop when the `.checkbox` anchor is absent (e.g. future conditional rendering of read-only items). The primary checkbox path is unchanged.
    - Added `NOTE` comments on the relationship-popup `toBeAttached()` assertions explaining that they only verify DOM presence (the popup uses `visibility: hidden` + hover reveal) and when to switch to `toBeVisible()` instead.
  - `frontend/tests/DocumentRelationshipModal.ct.tsx`:
    - Replaced the DOM-order `removeButtons.nth(1)` assertion in the "removes document from target column" test with an XPath-based walk from the pill's visible title, so re-ordering the source/target layout can no longer silently test the wrong button.
    - Strengthened the mutation-failure negative assertion: after submitting a failing mutation, wait for the submit button to re-enable (a positive signal that the `isSubmitting` finally-block has executed) before polling `onSuccessCalled === false`. Prevents the poll from passing immediately on the initial `false` value.
  - `frontend/tests/utils/ReactiveVarObserver.tsx`: expanded the doc comment with a three-step recipe for extending the observer to additional reactive vars, matching the existing `viewingDocument` / `editingDocument` convention.

### Fixed

- **Backend CI build aborted on transient SSL failure mid-wheel-download** (`compose/local/django/Dockerfile`, `compose/production/django/Dockerfile`): Backend CI run `24646911374` on commit `233a9b67` (push to `main`) failed in the `Build the Stack` step with `ssl.SSLError: [SSL] record layer failure (_ssl.c:2590)` at 22 MB of 60.4 MB while `pip wheel` was downloading `opencv-python-headless`, aborting the entire build and cascading skips through `Run DB Migrations`, `Verify Docker Containers`, and `pytest`. Pip's default `--retries 5` only covers connection setup and does not resume broken mid-stream downloads — that behaviour is gated on `--resume-retries`, added in pip 24.1 (env var `PIP_RESUME_RETRIES`). Added `ENV PIP_RETRIES=10 PIP_TIMEOUT=60 PIP_RESUME_RETRIES=5` to both the `python-build-stage` and `python-run-stage` of both Dockerfiles so every `pip install` / `pip wheel` invocation (wheel build, `--upgrade pip`, spacy model downloads) picks up the hardened settings. Verified the base image (`pytorch/pytorch:2.7.1-cuda12.6-cudnn9-runtime`) ships pip 25.1.1 and recognises all three env vars (`pip config list` reports `:env:.resume-retries='5'`, `pip install --help` shows `--timeout (default 60.0 seconds)`).
- **`fullDatacellList` no-args path was unbounded** (Issue #1256, follow-up to PR #1235, `config/graphql/extract_types.py:131-148`): `ExtractType.resolve_full_datacell_list` capped the `limit` and `offset-only` branches at `MAX_FULL_DATACELL_LIST_LIMIT` but returned the entire queryset when called with no arguments. Authenticated callers hitting `fullDatacellList` directly (no `limit`, no `offset`) could bypass the payload bound. Collapsed all three code paths so every call — no-args, offset-only, or `limit`+`offset` — returns `qs[start : start + min(limit_or_max, MAX_FULL_DATACELL_LIST_LIMIT)]`. The embed UI is unaffected (it already passes `limit: 500`); direct API callers now receive at most 500 cells and must paginate via `offset` to walk the rest. Added regression test `test_full_datacell_list_no_args_capped_at_server_max` in `opencontractserver/tests/test_extract_queries.py` which creates 501 cells and asserts the no-args response returns exactly `MAX_FULL_DATACELL_LIST_LIMIT` while `datacellCount` reports the true total.

### Changed

- **Dual-constant sync warnings for `MAX_FULL_DATACELL_LIST_LIMIT` / `EXTRACT_GRID_EMBED_CELL_LIMIT`** (Issue #1256, `opencontractserver/constants/extracts.py`, `frontend/src/assets/configurations/constants.ts:351-364`): Both constants now carry explicit `IMPORTANT: ... CI sync-check tracked in issue #1256` warnings so future edits are obvious. Updated the GraphQL schema description on `fullDatacellList(limit:)` to document the server cap rather than claiming the omitted-limit case returns everything. Corrected the stale "N+1: issues a COUNT(\*) per ExtractType instance" comment on `resolve_datacell_count` to reflect that the embed path issues a `COUNT(*)` in addition to the main list query. Added a `TODO` reference to #1256 next to the `ExtractQueryOptimizer` circular-import inline import in `_get_datacell_qs`.
- **Backend CI `changes` job failed on every push to main** (`.github/workflows/backend.yml`): `dorny/paths-filter@v3` was declared without a prior `actions/checkout`, on the stated premise that the action "fetches diffs via the GitHub API". That is only true for `pull_request` events; on `push` events the action shells out to `git branch --show-current`, which fails with `fatal: not a git repository` when the workspace is empty. Every push to a protected branch therefore produced a red X on the `changes` job, silently masked by `continue-on-error: true` plus a `|| github.event_name == 'push'` fail-open gate on downstream jobs. Scoped the `changes` job to `pull_request` events (`if: github.event_name == 'pull_request'`) — where paths-filter actually works — and rewrote the downstream gates on `linter` / `pytest` as `if: always() && (github.event_name == 'push' || needs.changes.outputs.backend != 'false')` so the skip cascade from a `needs`-target that is now intentionally skipped on push does not block the real test jobs. Behaviour preserved: push always runs `linter` + `pytest`; PRs with no backend changes still skip both; PRs where the filter errors transiently (outputs.backend == '') still fail open.

- **Frontend CI tippy-debug step removed** (`.github/workflows/frontend.yml`): Dropped the `Debug - Check for tippy references` step in the `lint` job — a leftover investigation artifact that ran on every PR/push and produced noise with no diagnostic value today. Flagged in Issue #1319.

- **Codecov-notify `matching[0]` relied on undocumented API ordering** (`.github/workflows/codecov-notify.yml`): The cross-workflow coordinator picked the "most recent" workflow run per expected name by indexing `matching[0]` on the filter result, leaning on the GitHub Actions REST API returning results newest-first. That ordering is not guaranteed. Added an explicit `created_at` descending sort before taking `matching[0]`, so re-run detection is correct regardless of API-side quirks. Flagged in Issue #1319.

- **Frontend coverage badge reported ~31% despite months of added tests** (`README.md:12`, `.github/workflows/codecov-notify.yml`, `.codecov.yml`, `frontend/vite.config.ts:210-223`): The README's "Frontend coverage" badge was pointing at `flag=frontend-unit` — the Vitest slice only. The three frontend suites (Vitest unit, Playwright component via `vite-plugin-istanbul`, Playwright E2E via `vite-plugin-istanbul`) upload to separate Codecov flags (`frontend-unit`, `frontend-component`, `frontend-e2e`) and were never merged into a single lcov. Recent PRs almost exclusively added Playwright component and E2E tests, so their coverage landed in `frontend-component`/`frontend-e2e` while the badge stayed stuck reading the Vitest slice.
  - Added an `Upload {unit,CT,E2E} lcov artifact` step (using `actions/upload-artifact@v7`) to each producing job in `.github/workflows/frontend.yml` (`component-test` and `unit-test`) and `.github/workflows/frontend-e2e.yml` (`e2e`). The existing per-flag Codecov uploads are untouched, so per-suite drill-in still works.
  - Extended the existing cross-workflow coordinator at `.github/workflows/codecov-notify.yml` to download the three artifacts by run id (via `actions/download-artifact@v7` with `continue-on-error: true` to tolerate path-filtered skips and upload failures), merge them with `npx lcov-result-merger@5`, and upload the combined lcov to Codecov under a new `frontend` flag before calling `send-notifications`. The existing `listWorkflowRunsForRepo` check already discovers the producing runs by SHA — it now also emits `frontend_ci_run_id` and `frontend_e2e_run_id` outputs for the downloads to consume.
  - Added the `frontend` flag to `.codecov.yml` (paths: `frontend/src/`, `carryforward: true`). Intentionally does not match the `frontend-.*` regex used by the `frontend` component's `flag_regexes`, so the component keeps aggregating only the three per-suite flags and does not double-count the merged upload.
  - Pointed the README badge at `flag=frontend` so the displayed number reflects the union of all three suites.
  - Fixed a secondary v8 issue at `frontend/vite.config.ts:210-223`: added `all: true` to the Vitest coverage block. Without it, v8 silently drops files not imported by any unit test, inflating the `frontend-unit` ratio and misaligning the v8 lcov's file universe with the Istanbul-based component/E2E lcovs (which do enumerate all source files). Aligning the two is required for the merged `frontend` lcov to be meaningful.
  - Added `lcov-result-merger@^5.0.1` as a `frontend/package.json` devDependency with a local `coverage:merge` script so the merge can be reproduced locally (`yarn test:coverage:unit && yarn test:coverage:ct && yarn test:e2e:coverage && yarn coverage:merge`).

### Changed

- **`fullDatacellList` caps caller-supplied `limit` at `MAX_FULL_DATACELL_LIST_LIMIT` (500)** (Issue #1227): The resolver for `ExtractType.fullDatacellList` now clamps the `limit` argument to `MAX_FULL_DATACELL_LIST_LIMIT = 500` (`opencontractserver/constants/extracts.py`) regardless of what the caller requests. Callers that pass no `limit` still get every datacell (full pagination is tracked in #1204); the cap only applies once `limit` is supplied, so a caller asking for `limit: 10_000` is transparently answered with at most 500 rows per request.

- **Simplify `RelationGroup.updateForAnnotationDeletion` pruning branches** (Issue #1316, `frontend/src/components/annotator/types/annotations.ts:40-59`): The four conditional branches that each returned `undefined` collectively covered exactly `newSourceIds.length === 0 || newTargetIds.length === 0`, and the `sourceEmpty` / `targetEmpty` pre-filter variables were only referenced inside those branches. Collapsed the four conditions into a single `if` and removed the now-unused pre-filter variables. No behavior change — existing `RelationGroup > .updateForAnnotationDeletion()` regression tests still pass unchanged (all 28 tests in `annotations.test.ts` pass; `tsc --noEmit` clean).
- **Harden backend CI path-filter job** (Issue #1290, `.github/workflows/backend.yml`): Tightened the `changes` path-filter job so transient `dorny/paths-filter` failures no longer silently skip `linter` / `pytest`.
  - Dropped the redundant `actions/checkout` step in the `changes` job — `dorny/paths-filter@v3` fetches diffs via the GitHub API and does not need a local clone.
  - Added `Dockerfile*` to the backend path filter so root-level Dockerfile changes (which directly affect the backend build environment) correctly trigger backend CI.
  - Marked the `changes` job `continue-on-error: true` so a transient GitHub API failure does not fail the workflow.
  - Changed the downstream gate from `needs.changes.outputs.backend == 'true'` to `needs.changes.outputs.backend != 'false'` on both `linter` and `pytest`. With `continue-on-error: true`, a failed `changes` job produces an empty-string output; the old `== 'true'` gate evaluated to `false` for PR events and silently skipped downstream jobs — the exact failure mode the "fail open" comment claimed to prevent. The new `!= 'false'` gate runs downstream jobs for both the happy path (`'true'`) and the transient-failure path (`''`), only skipping when the filter explicitly reports no backend changes (`'false'`).
  - Reworded the linter path-filter comment to clarify that gating only affects PRs with no backend changes; push events to protected branches still run unconditionally.

### Fixed

- **`ChatMessage` source indicator no longer renders "1 sources"** (`frontend/src/components/widgets/chat/ChatMessage.tsx:1993`): the source indicator always used the plural noun regardless of count, while every other pluralized label in the component ("1 Source", "1 step", "1 tool") switched correctly. Updated to `"1 source" / "N sources"` to match.
- **`DocumentRelationshipModal` "Create label" button never appeared** (Issue #1280, `frontend/src/components/documents/DocumentRelationshipModal.tsx`): `@os-legal/ui`'s Dropdown only fires `onSearchChange` in `searchable="async"` mode — with `searchable="local"` the parent's `labelSearchTerm` state never updated, so the `labelSearchTerm`-gated "Create label: ..." empty-state button was permanently hidden. Switched the Dropdown to `async`, moved option filtering into a `useMemo(filteredRelationshipLabels, [relationLabels, hasCorpus, labelSearchTerm])`, and added the missing dep. Now typing a novel label name surfaces the Create button as designed.
- **`RelationGroup.updateForAnnotationDeletion` pre-filter length check** (Issue #1288): `frontend/src/components/annotator/types/annotations.ts:49-50` computed `nowSourceEmpty` / `nowTargetEmpty` from the pre-filter `this.sourceIds` / `this.targetIds`, so the "now empty" conditions were identical to the "before" conditions and the pruning branches that return `undefined` were dead code. Deleting the sole source or sole target of a relation left the relation orphaned, pointing at a deleted annotation id. Fixed by reading from the post-filter `newSourceIds` / `newTargetIds`. Called from `PdfAnnotations.undoAnnotation()`, so undo now properly drops any relation whose last source or target was the popped annotation. New regression tests under `RelationGroup > .updateForAnnotationDeletion()` cover all four pruning branches plus the survive-with-updated-ids and unchanged cases, and the existing `undoAnnotation` test that pinned the wrong behaviour was corrected.

  - Additionally fixed a surviving-relation identity drop in the same method: the post-prune return previously constructed `new RelationGroup(newSourceIds, newTargetIds, this.label)` without forwarding `this.id` / `this.structural`, so relations that merely lost a member were silently reassigned a fresh uuid and had their `structural` flag cleared. Both fields are now preserved, and the survival tests assert `updated!.id === rel.id` (plus `structural === true` where applicable) to pin the behaviour.

- **`useAgentChat` WebSocket reconnects on every approval-gate transition** (Issue #1296): `frontend/src/hooks/useAgentChat.ts` had `pendingApproval` in the main WebSocket `useEffect` dependency array, so every `ASYNC_APPROVAL_NEEDED` / `ASYNC_APPROVAL_RESULT` / `ASYNC_CONTENT` / `ASYNC_FINISH` message that set or cleared approval state tore the socket down and recreated it mid-conversation. Impacts: approval decisions could race the reconnect and silently fail (`isConnected` flipped to `false` before the fresh socket opened), in-flight streaming tokens were dropped whenever an approval gate opened/closed, and the server saw a brand-new consumer attaching mid-run. Fix: mirror `pendingApproval` in a `pendingApprovalRef` updated by a separate effect (`useAgentChat.ts:291-299`), read `pendingApprovalRef.current` from the `ASYNC_CONTENT` / `ASYNC_APPROVAL_RESULT` / `ASYNC_FINISH` branches of `onmessage` (`useAgentChat.ts:651-752`), and drop `pendingApproval` from the socket effect's dependency array (`useAgentChat.ts:802-818`). Tests in `frontend/src/hooks/__tests__/useAgentChat.test.tsx`: removed the previous `_open()` workarounds around approval events, replaced the old "reconnects on approval state change" regression test with a new `does not reconnect the socket when approval state changes (issue #1296)` guard that asserts `wsInstances.length` is unchanged, `originalSocket.close` was not called, and the approval decision is dispatched through the original socket. All 23 tests still pass.
- **Forced-by-selected-relation visibility now respects explicit user selection regardless of `showStructural`** (Issues #1289, #1294): In `frontend/src/components/annotator/hooks/useVisibleAnnotations.ts:54-62`, the IDs forced visible by a selected relation were previously only merged into `forcedIds` inside the `if (showStructural)` branch. With structural annotations toggled off, clicking a relation in the sidebar failed to highlight its member annotations — making the relation sidebar effectively unusable in that mode. Moved the `forcedBySelectedRelationIds` merge outside the structural branch so explicit user selection always wins (consistent with the existing `forcedBySelection` treatment). Structural-relationships auto-forcing (`showStructuralRelationships`) remains gated on `showStructural`, since that path is implicit rather than user-initiated.

  - Updated `frontend/src/components/annotator/hooks/__tests__/useVisibleAnnotations.test.tsx`: replaced the pinning test that documented the bug ("does NOT apply forced-by-selected-relation when showStructural is false") with two assertions verifying the corrected behavior — forced-by-selected-relation now applies to both non-structural annotations (overriding a label filter) and structural annotations when `showStructural` is false. All 16 tests in the file pass; `tsc --noEmit` clean.

- **`useUpdateAnnotation` lost sibling annotations on update** (`frontend/src/components/annotator/hooks/AnnotationHooks.tsx:386`): The hook called `replaceAnnotations([updatedAnnotation])`, which collapses the whole `annotations` array down to the single passed-in element. On any document with more than one annotation, updating one annotation silently dropped the others. Fixed by using `setPdfAnnotations` with a `.map` that swaps only the matching id; regression test added in `AnnotationHooks.test.tsx` (see `useUpdateAnnotation > updates one annotation in place without dropping siblings`).

### Added

- **Frontend coverage for widgets, modals, and icon-picker** (Issue #1279): Added one vitest suite and three Playwright CT suites targeting ~2,150 uncovered lines in `frontend/src/components/widgets/`:
  - `frontend/src/components/widgets/icon-picker/__tests__/icons.test.ts` — 450 tests (≥90% target). Validates every entry in the Lucide catalog has a valid kebab-case `name`, non-empty trimmed `label`, and a `category` that belongs to the declared `IconCategory` union; asserts name uniqueness, that every declared category contains at least one icon, that the derived `LUCIDE_ICON_NAMES` set and `findIconEntry` map stay in sync with `LUCIDE_ICONS`, and a floor of 300 curated icons as a snapshot guard.
  - `frontend/tests/ChatMessageCoverage.ct.tsx` — 16 tests pushing `ChatMessage.tsx` past the 60% floor (from 50%). Covers user vs assistant header rendering, markdown (bold/code/lists/GFM tables), `onSelect` wiring, source-indicator visibility, all three approval-pill states, sources-preview expand/collapse + per-chip expand + chip `onClick`, timeline collapsed-by-default for long completed messages + header expansion + short-timeline default-expanded path + streaming `showTimelineOnly` branch.
  - `frontend/tests/FieldsetModal.ct.tsx` + `frontend/tests/FieldsetModalTestWrapper.tsx` + `frontend/tests/FieldsetModalMocks.ts` — 11 tests. Covers create-mode header/empty state/disabled-save, footer validation transitions (name-required → column-required), column add via the embedded `CreateColumnModal`, column delete with counter update, edit-mode prefill from `REQUEST_GET_FIELDSET`, and all three close paths (Cancel / X / overlay).
  - `frontend/tests/SelectAnalyzerOrFieldsetModal.ct.tsx` + `frontend/tests/SelectAnalyzerOrFieldsetModalTestWrapper.tsx` + `frontend/tests/SelectAnalyzerOrFieldsetModalMocks.ts` — 16 tests. Covers analyzer-grid rendering, document vs corpus subtitle, result-count pill, debounced search filter, "no match" and "none available" empty states, selection flipping Run↔Configure, schema preview toggle, tab switching (analyzer ↔ fieldset), pagination across 12 analyzers, and all three close paths.
  - Verification: 450/450 vitest + 43/43 Playwright CT (stable across 3 consecutive runs); `tsc --noEmit` clean; prettier clean.
- **Knowledge-base document viewer CT coverage remediation** (Issue #1277): Added ~1,400 lines of new Playwright component tests across the four highest-uncovered files in `frontend/src/components/knowledge_base/document/` — `DocumentKnowledgeBase.tsx` (1,961 LOC / 25.5%), `right_tray/ChatTray.tsx` (1,417 / 25.3%), `unified_feed/RelationshipActionModal.tsx` (549 / 16.2%), and `unified_feed/UnifiedContentFeed.tsx` (577 / 29.8%). All new tests use the existing `*TestWrapper` pattern and drive the UI through `--reporter=list` as required by CLAUDE.md.

  - `frontend/tests/RelationshipActionModal.ct.tsx` — Expanded from 3 to 16 specs. New coverage: corpus-loaded rendering, role-picker + add-to-existing flow with callback assertion, structural-relationship filtering, create-mode search + label-list filtering, "no labelset" warning, create-label form round-trip (open, cancel, submit via `SMART_LABEL_SEARCH_OR_CREATE` mock, change selected label), create-mode submit enablement, full submission (source/target pill assignment → `onCreate(labelId, sourceIds, targetIds)` verification), cancel → `onClose`, `getAnnotationPreview` ellipsis truncation at 30 chars, and singular/plural `Selected: N annotation(s)` rendering.
  - `frontend/tests/RelationshipActionModalTestWrapper.tsx` — Extended to accept `withCorpus` / `hasLabelset` / `relationLabels` / `onAddToExisting` / `onCreate` / `onClose` / `corpusId` props. Seeds `corpusStateAtom` via an internal `CorpusSetupInner` children-wrapping effect component (children-wrapping rather than null-returning to satisfy Playwright CT's babel transform, which silently fails to mount otherwise).
  - `frontend/tests/RelationshipActionModalFixtures.ts` — New plain `.ts` fixtures file hosting `buildRelationLabel`. Kept separate from the `.tsx` wrapper to avoid the Playwright CT split-import rule (pitfall #16 in `CLAUDE.md`).
  - `frontend/tests/UnifiedContentFeed.ct.tsx` — Expanded with 8 new specs: multi-select selection-toolbar appearance & count, Select All, Clear, "Add to Relationship" opens `RelationshipActionModal`, read-only hides the checkbox, `noCorpus` hides the toolbar entirely, sort-by-type, note search-query filter, annotation rawText search-query filter, structural filter, `STRUCTURAL_LABEL_PREFIX` always-hidden invariant, and content-type routing (relationship-only).
  - `frontend/tests/UnifiedContentFeedTestWrapper.tsx` — Added `noCorpus`, `showStructural`, `searchText`, and `textSearchMatches` props so the new tests can exercise the no-corpus and structural branches without mounting the component directly.
  - `frontend/tests/ChatTray.ct.tsx` — Added 10 new specs covering empty conversation list, Back-to-Conversations (exitConversation + refetch), ASYNC_ERROR → Reconnect banner, new-chat FAB path, no-op Enter on empty input, sub-90% no character counter, Shift+Enter newline, context-meter rendering via `ASYNC_FINISH` `context_status`, compaction banner via `ASYNC_THOUGHT` with `compaction` metadata, SYNC_CONTENT standalone assistant message, and readOnly-style start path.
  - `frontend/tests/DocumentKnowledgeBase.ct.tsx` — Added 2 new specs covering the `!documentId` invalid-document error modal and its Close-button `handleClose` callback.
  - Verification: full CT suite (`yarn test:ct --reporter=list`) — **1,523 passed, 3 skipped, 0 failed**.

- **Component test coverage for `ModernDocumentItem` and `DocumentRelationshipModal`** (Issue #1280): Added ~1,600 lines of Playwright CT coverage for two high-traffic document components that were lacking happy-path tests.
  - `frontend/tests/ModernDocumentItem.ct.tsx` — New 750-line suite covering card/list view rendering (thumbnails, badges, metadata), selection state, permission-gated action buttons (CAN_UPDATE / read-only), action behaviors (view, edit, remove, open, download), backend-locked processing state, context menu actions, and version-history / relationship badges.
  - `frontend/tests/DocumentRelationshipModal.ct.tsx` — Extended suite now covers state transitions (moves / removes across columns), Add Source/Target search flow, label picker with pre-populated labels, label change, RELATIONSHIP and NOTES submit mutations, mutation failure handling, inline create-label flow, `SMART_LABEL_SEARCH_OR_CREATE`, the missing-corpus error state, and the useMemo filter bodies (`availableDocuments`, `filteredRelationshipLabels`).
  - `frontend/tests/DocumentRelationshipModalTestWrapper.tsx` — Added `makeMockRelationLabel` helper, `extraMocks` prop for per-test Apollo mocks, `relationLabels` / `withoutLabelset` / `corpusIdOverride` props, and a DRY `buildDocumentsMock()` factory (replaces duplicated mock blocks).
  - `frontend/tests/utils/ReactiveVarObserver.tsx` — New helper that exposes Apollo reactive-var state via DOM data attributes so Playwright assertions can observe cross-process (browser-side) state without reaching into the Node fixture.
  - Added workarounds for DndContext pointer-event interception (`clickViaReact` / `openContextMenu` using React 18 `__reactProps$` fiber access) with explicit upgrade-risk comments.
- **Route component and ExtractDetail view test coverage** (Issue #1285): Closed coverage gaps for zero-coverage route wrappers and the 7%-covered `ExtractDetail.tsx` orchestrator.

  - `frontend/src/components/routes/__tests__/ExtractDetailRoute.test.tsx` — 6 vitest specs covering missing-ID, reactive-var reuse, loading, error, not-found, and success paths for the slug-resolving extract route.
  - `frontend/src/components/routes/__tests__/ExtractLandingRoute.test.tsx` — 4 specs for the legacy `/e/:user/:extract` route, including the redirect to `/extracts/:id` when the reactive var is populated.
  - `frontend/src/components/routes/__tests__/LabelSetLandingRoute.test.tsx` — 5 specs for loading/error/success state and `onClose` cleanup.
  - `frontend/src/components/routes/__tests__/GlobalDiscussionsRoute.test.tsx` — smoke test for the thin route wrapper.
  - `frontend/src/components/routes/__tests__/NotFound.test.tsx` — renders 404 UI and verifies the "Go to Corpuses" button navigates back.
  - `frontend/src/components/routes/__tests__/UserProfileRoute.test.tsx` — 7 specs covering `/profile` redirects (anonymous → `/login`, logged-in → `/users/<slug>`), loading, error, not-found, and own-vs-other profile flags.
  - `frontend/tests/ExtractDetail.ct.tsx` + `ExtractDetailTestWrapper.tsx` + `ExtractDetailFixtures.ts` — 17 Playwright CT specs exercising the extract-detail view's header, status chips, StatBlocks, running/failed/completed state swaps, Data/Documents/Schema tabs, CreateColumnModal/ConfirmModal integration, and the Export-CSV enable/disable gating.
  - `frontend/tests/e2e/routing-round-trip.spec.ts` — Playwright E2E spec covering the URL → entity → back round-trip for corpuses, documents, deep-link replay, and the unknown-route `NotFound` fallback. Runs after the corpus-workflow spec so the necessary seed data exists.
  - Verification: 36 route vitest specs pass (`yarn test:unit --run src/components/routes/__tests__/`), 17 ExtractDetail CT specs pass (`yarn test:ct --reporter=list ExtractDetail.ct.tsx`), `tsc --noEmit` clean.
  - Surfaced Issue #1295 (UserProfileRoute Rules-of-Hooks violation) as a follow-up.

- **Frontend coverage: admin panel CT tests** (Issue #1281): Added 32 new Playwright component tests to close the coverage gap on the three admin-panel files called out in `docs/coverage/frontend-roi-ranking.md` (ranks 10, 24, 27).
  - `frontend/tests/pipeline-icons.ct.tsx` (+ `PipelineIconsTestWrapper.tsx`) — 5 tests covering `components/admin/PipelineIcons.tsx` (5.1% → target ≥60%). Mounts the full icon catalog (Docling, LlamaParse, TextParser, PdfThumbnail, TextThumbnail, ModernBert, SentenceTransformer, Multimodal, Generic), asserts brand icons render as `<img>` with alt text while inline SVG icons render as `<svg>`, verifies `size`/`className` prop forwarding, exhaustively exercises every branch of the `getComponentIcon` className dispatcher (including the compound `pdf+thumb`, `text+thumb`, and `modern_bert` patterns that must precede the single-token fallbacks), and covers `getComponentDisplayName` including the `title` override and `KNOWN_ACRONYMS` replacement path.
  - `frontend/tests/global-agent-management.ct.tsx` — 15 tests covering `components/admin/GlobalAgentManagement.tsx` (8.8% → target ≥60%). Adds loading/error states, list-rendering branches (missing slug, >3 tools with `+N` overflow badge, long-description truncation, inactive status), and full create/update/delete flows: submit-button disabling until required fields are filled, invalid-JSON `badgeConfig` validation toasts, successful create/update mutations with refetch, server-side `ok=false` error toasts, the `ConfirmModal` cancel path, and confirmed-delete mutation + refetch. Introduces a new `GlobalAgentManagementWithToastsWrapper` in `tests/AdminComponentsTestWrapper.tsx` that provides `ToastContainer` + `MemoryRouter` so tests can assert on react-toastify output without affecting pre-existing wrapper consumers.
  - `frontend/tests/system-settings-flows.ct.tsx` — 12 tests covering additional `components/admin/SystemSettings.tsx` flows not exercised by `admin-components.ct.tsx` (17.9% → target ≥60%): assigning/clearing parsers via the filetype-default dropdown, opening the Default Embedder modal and saving both list-selected and hand-typed class paths, non-secret config save (`num_workers` int + `verbose` bool select) through `AdvancedSettingsPanel`, mobile-tab keyboard navigation (`ArrowLeft`/`ArrowRight`/`Home`/`End` including wrap-around), mutation error branches (network error toast + server-side `ok=false` message), and the `handleToggleEnabled` transition from empty `enabledComponents` ("all enabled") to an explicit list.
  - Verification: 77/77 tests pass (`yarn test:ct --reporter=list admin-components.ct.tsx pipeline-icons.ct.tsx global-agent-management.ct.tsx system-settings-flows.ct.tsx`) with no changes to pre-existing admin-components tests. `tsc --noEmit` and `prettier --check` clean.
- **Annotator hook / renderer / label-selector coverage** (Issue #1284): Added vitest coverage for the annotator hook layer plus Playwright CT coverage for the text and docx renderers and the enhanced label selector.

  - `frontend/src/components/annotator/hooks/__tests__/AnnotationHooks.test.tsx` — 25 tests covering state-wrapper helpers, guard clauses, success paths, and relation side effects. Includes a regression that keeps a sibling annotation in state across a `useUpdateAnnotation` call.
  - `frontend/tests/TxtAnnotator.ct.tsx`, `frontend/tests/DocxAnnotator.ct.tsx`, and `frontend/tests/EnhancedLabelSelector.ct.tsx` — CT suites driving prop-driven scenarios (visibility filtering, search highlights, chat sources, structural filtering, read-only mode).

- **Frontend component-test coverage for Corpus Chat & Agent Management** (Issue #1276): Added Playwright CT coverage for the four corpus components called out as the highest-uncovered surface area in `docs/coverage/frontend-roi-ranking.md` (CorpusChat, CreateCorpusActionModal, CorpusAgentManagement, CorpusDescriptionEditor — combined ~3,300 lines, ~2,800 previously uncovered).
  - `frontend/tests/CorpusChat.ct.tsx`: extended from 3 to 13 tests. Adds a Playwright `WebSocket` stub (mirroring the `ChatTray.ct.tsx` pattern) so we can drive ASYNC_START/CONTENT/FINISH, ASYNC_THOUGHT (with compaction notice), ASYNC_APPROVAL_NEEDED, ASYNC_APPROVAL_RESULT, ASYNC_ERROR (deferred via `setTimeout` to avoid being clobbered by the synchronous `setWsError(null)` after `socket.send`), and CONTEXT_EXHAUSTED. Covers approve/reject flows, context-meter rendering, send-button gating, conversation loading, navigation callbacks, and GraphQL error surface.
  - `frontend/tests/CorpusChatTestWrapper.tsx`: now sets `authToken`/`userObj` synchronously (matches `ChatTrayTestWrapper`) to avoid a re-mount race that closes the stub socket between renders. Forwards `initialQuery`, `onNavigateHome`, and `onMessageSelect` to the component.
  - `frontend/tests/CreateCorpusActionModal.ct.tsx` + `CreateCorpusActionModalTestWrapper.tsx`: 13 tests covering create/edit modes, modal open/close lifecycle, all three action types (fieldset/analyzer/agent), trigger-driven query switching (document tools vs. moderation tools), inline-vs-existing-agent toggle, validation toasts, Clear-All / Select-All tool toggles, and the create + update mutation happy paths.
  - `frontend/tests/CorpusAgentManagement.ct.tsx` + `CorpusAgentManagementTestWrapper.tsx`: 9 tests covering the `canUpdate=false` permission notice, empty state, agent row rendering with status badge and slug, create modal (with disabled-submit gating), edit modal pre-population, delete confirmation flow (yes / no), and the Permission Required Tools section unlocking when an available tool is selected.
  - `frontend/tests/CorpusDescriptionEditor.ct.tsx` + `CorpusDescriptionEditorTestWrapper.tsx`: 10 tests covering closed-modal behavior, fetching markdown via `page.route` interception, empty-corpus state, "Unsaved changes" indicator, Show/Hide history panel, version-row expansion + reapply mutation, edit-from-version state with badge, and unsaved-aware close.
  - All test wrappers mount a `<ToastContainer />` so `react-toastify` calls produce visible DOM in CT (Playwright `index.tsx` does not mount one globally).
  - Verification: 45/45 new + updated tests pass via `yarn test:ct CorpusChat.ct.tsx CreateCorpusActionModal.ct.tsx CorpusAgentManagement.ct.tsx CorpusDescriptionEditor.ct.tsx --reporter=list`; `tsc --noEmit` clean on the new files.

### Added

- **Extracts DataGrid & Detail component test coverage** (Issue #1282): Expanded Playwright component tests for the two biggest uncovered files in `frontend/src/components/extracts/`. Previously at 32.5% and 23.0% line coverage respectively (~990 uncovered lines combined); the new tests exercise the high-signal branches listed in the issue.

  - `frontend/tests/DataGrid.ct.tsx`: grew from 4 to 19 tests. New coverage: loading overlay (idle vs running copy), Document/column sort toggles, row-selection bulk-delete bar + callback, add-column modal, per-column edit modal, per-column delete confirmation (including the `fieldset.inUse` warning copy), add-documents FAB, a 4-type cell-rendering matrix (text/number/boolean/JSON object), corrected-data precedence, and `exportToCsv` via the imperative handle.
  - `frontend/tests/ExtractDetailContent.ct.tsx` + `frontend/tests/ExtractDetailContentTestWrapper.tsx` (both new): 16 tests covering the loading overlay, not-found state, stats panel, Data/Documents/Schema tabs, running-state spinner, failed-state Retry button + `startExtract` mutation, schema tab empty-vs-populated + Add Column + Delete Column confirmation, Documents tab empty state, and both imperative-handle methods (`exportToCsv`, `startExtract`) plus the `onExtractLoaded` callback.
  - `frontend/tests/DataGridTestWrapper.tsx`: added optional callback-spy props (`onAddDocIds`, `onRemoveDocIds`, `onRemoveColumnId`, `onAddColumn`) and a `withExportButton` flag that renders a test-only button bound to the grid's imperative `exportToCsv` handle, so tests can verify the handle without reaching into internal refs.

- **Frontend coverage tests for useAgentChat / LabelSetDetailPage / ModerationDashboard** (Issue #1286): Added 47 new tests lifting the three high-ROI files from ~11–28% line coverage toward the ≥60% target.

  - `frontend/src/hooks/__tests__/useAgentChat.test.tsx` — 23 new Vitest unit tests wiring a mock `WebSocket` through `renderHook`. Cover connection lifecycle (open, error, close), every streaming message type (`ASYNC_START`, `ASYNC_CONTENT`, `ASYNC_FINISH`, `ASYNC_THOUGHT`, `ASYNC_SOURCES`, `ASYNC_APPROVAL_NEEDED`, `ASYNC_APPROVAL_RESULT`, `ASYNC_RESUME`, `ASYNC_ERROR`, `SYNC_CONTENT`), malformed JSON handling, approval flow including `sendApprovalDecision`, `sendMessage` guards (empty, disconnected, while-processing, send-throws), `clearError`, and `setSelectedMessageId`. Local coverage now 82.9% lines for `useAgentChat.ts`.
  - `frontend/tests/LabelSetDetailPage.coverage.ct.tsx` — 15 new Playwright CT tests covering previously-uncovered mutation success paths: `UPDATE_ANNOTATION_LABEL` inline edit save, `CREATE_ANNOTATION_LABEL_FOR_LABELSET` submission, `DELETE_MULTIPLE_ANNOTATION_LABELS` delete flow, `DELETE_LABELSET` confirm modal → Yes → mutation, `handleExportJSON` blob download handler, Overview tab Delete button (visible/hidden per permission), Edit Details footer button visibility, Relationships / Doc Labels / Sharing tabs, error state when the query fails, and the empty-state "Add First Label" branch.
  - `frontend/tests/ModerationDashboard.coverage.ct.tsx` — 9 new Playwright CT tests: `ROLLBACK_MODERATION_ACTION` mutation with reason, action-type filter refetch, automated-only toggle refetch, time-range dropdown change refetch, `Load More` cursor pagination via `fetchMore`, actions-query error state, metrics-query error state, rollback-modal cancel, and the System / "No reason provided" rendering branches.
  - **Issue #1296 discovered and filed**: `useAgentChat.ts:810` includes `pendingApproval` in the main WebSocket `useEffect` dependency array, so the socket tears down + reconnects every time approval state changes. This drops in-flight tokens and can race `sendApprovalDecision` against `isConnected=false` during reconnect. A regression test in `useAgentChat.test.tsx` documents the current behaviour; the workaround comments reference #1296 so they can be removed after the fix lands.

- **Codecov components for backend and frontend** (`.codecov.yml`): Added `component_management` with two components — Backend (`opencontractserver/**`, `config/**`) and Frontend (`frontend/src/**`) — grouping all five flags so PR comments and the Codecov UI show a single frontend number (union of unit + component + e2e) instead of five overlapping flag numbers. Component-level patch/project statuses now gate PRs per-area.
- **Frontend coverage ROI audit** (`docs/coverage/frontend-roi-ranking.md`): Ranked every frontend source file by uncovered line count against the Codecov project-level (flag-union) report. Top 50 files, files at 0% coverage with >=100 lines, and per-area totals. Source for prioritizing targeted coverage work.
- **Frontend error-path test coverage** (Issue #1270): Added targeted unit tests covering error boundaries, the Apollo error link, and `catch` blocks in utility modules. Previously these branches were effectively untested, leaving regressions in auth error handling, file download failures, and error boundary fallback UI undetectable in CI.
  - `frontend/src/components/widgets/__tests__/ErrorBoundary.test.tsx` — 7 new tests for `ErrorBoundary.tsx`: default fallback UI on child-thrown errors, `onError` callback invocation, console logging in `componentDidCatch`, custom `fallback` render prop, recovery via the reset button, and persistent error state when the child keeps throwing.
  - `frontend/src/graphql/errorLink.test.ts` — Rewrote the existing smoke test (which only toggled reactive vars) into a full suite that executes the real `errorLink` through an `ApolloLink.from([errorLink, terminating])` chain. Now asserts all auth-error branches (401/403/`UNAUTHENTICATED`, message-based detection, JWT-expired reload via `setTimeout` + `window.location.reload`), the non-auth logging fall-through, and both network-error branches (401/403 vs generic).
  - `frontend/src/utils/__tests__/files.test.ts` — Added tests for `downloadFile` success path (axios → Blob → anchor click) and its `catch (e)` branch (rejects with the original error and logs before re-throwing), plus `toBase64` success and `FileReader.onerror` rejection path.
  - `frontend/src/utils/graphqlGuards.test.ts` — Added explicit coverage for `createSafeQueryExecutor`'s `catch (error)` branch (line 129, logs via `console.error` and surfaces the error) and the validation-blocked `console.warn` branch.
  - Verification: 49/49 new + updated tests pass via `yarn vitest run` on all four files; `tsc --noEmit` and `prettier --check` both clean.

### Fixed

- **`PdfAnnotations.undoAnnotation()` mutated the caller's annotations array** (Issue #1291): `frontend/src/components/annotator/types/annotations.ts:311` called `this.annotations.pop()` before constructing the new `PdfAnnotations`, silently shortening the original instance's `annotations` array even though the field is declared `readonly`. This violated the immutability contract the other methods (`saved()`, `update()`, `fromObject()`) uphold and is a footgun for the upcoming PdfAnnotator package extraction (#1283) — any consumer holding a reference to the pre-undo instance would see a stale length without a state change. The method now derives `popped` via index access and `remaining` via `slice(0, -1)`, so the caller's array is untouched. The pinning test in `frontend/src/components/annotator/types/__tests__/annotations.test.ts` was updated to assert non-mutation (`pdf.annotations` stays `[a, b]` after `pdf.undoAnnotation()`), and the "known wrinkle" comment was removed.

### Removed

- **Frontend dead Jotai atoms and Apollo reactive vars** (Issue #1243): Removed unused state management exports after triple-verifying each against both `frontend/src/` and `frontend/tests/` — atoms consumed only by test wrappers (e.g. `hideLabelsAtom`, `rawPermissionsAtom`, the deprecated `showAnnotation*Atom` set) were left in place per the issue's scope-correction note.

  - **`frontend/src/atoms/folderAtoms.ts`**: Removed 8 unused exports — `currentFolderAtom` (only read by the two dead permission atoms below), `canUpdateCurrentFolderAtom`, `canDeleteCurrentFolderAtom`, `draggingDocumentIdAtom`, `enableDragDropAtom`, `collapseAllFoldersAtom`, `expandAllFoldersAtom`, `openMoveFolderModalAtom`.
  - **`frontend/src/atoms/threadAtoms.ts`**: Removed 5 unused atoms — `selectedCorpusIdAtom`, `currentThreadIdAtom`, `expandedMessageIdsAtom`, `showCreateThreadModalAtom`, `editingMessageIdAtom` — plus the unused `ConversationType` / `ChatMessageType` imports. Unexported the `ThreadFilterOptions` type since it is only used internally by `threadFiltersAtom`.
  - **`frontend/src/components/annotator/context/DocumentAtom.tsx`**: Removed 7 unused atoms and 7 unused hooks: `fileTypeAtom` + `useFileType`, `isLoadingAtom` + `useIsLoading`, `canUpdateDocumentAtom` + `useCanUpdateDocument`, `canDeleteDocumentAtom` + `useCanDeleteDocument`, `hasDocumentPermissionAtom` + `useHasDocumentPermission`, `pageSelectionAtom`, `pageSelectionQueueAtom` + `usePageSelectionQueue`, plus the dead `useViewState` and `usePermissions` hooks (the underlying `viewStateAtom` and `permissionsAtom` are kept because they remain live via `useSetViewStateError` / `useDocumentPermissions` respectively). Dropped the now-unused `BoundingBox` import.
  - **`frontend/src/components/annotator/context/UISettingsAtom.tsx`**: Removed `hasScrolledToAnnotationAtom` (zero references anywhere in the tree).
  - **`frontend/src/graphql/cache.ts`**: Removed 7 unused reactive vars — `addingColumnToExtract`, `editingColumnForExtract`, `editingExtract`, `analyzerSearchTerm`, `editMode`, `allowUserInput`, `filterToAnnotationLabelId` — plus the now-unused `ColumnType` import. Unexported the `LinkDocumentsModalState` interface since it is only used internally as the type argument for `linkDocumentsModalState`.
  - Verification: `tsc --noEmit` passes, `vitest run` passes (935/935 unit tests).

- **Frontend dead styled components in `knowledge_base/document/styled/`** (Issue #1241): Removed unused styled-components from the `frontend/src/components/knowledge_base/document/styled/` folder. None of the deleted exports are referenced by any production code, test wrapper, or `.ct.tsx` test:

  - **`Relationships.tsx` deleted entirely** — all 3 exports (`RelationshipPanel`, `RelationshipCard`, `RelationshipType`) were unused. Note: a GraphQL type with the same name `RelationshipType` lives in `types/graphql-api.ts` and is unaffected.
  - **`LoadingStates.tsx`**: removed `DocumentLoadingContainer` (truly dead — no internal or external use). The other exports (`PlaceholderBase`, `PlaceholderItem`, `SummaryPlaceholder`, `NotePlaceholder`, `RelationshipPlaceholder`) are kept because they are used internally by `LoadingPlaceholders` in the same file.
  - **`RightPanel.tsx`**: removed 5 unused exports (`ControlButtonGroupLeft`, `ControlButtonWrapper`, `ControlButton`, `ChatIndicator`, `ControlButtonGroup`). Kept the only consumed exports: `ConnectionStatus` and `SlidingPanel`.
  - **`styled/index.ts`** barrel pruned to drop the dead re-exports listed above.
  - Verification: `yarn build` succeeds, `yarn test:unit` passes (927/927), `yarn test:ct -g "DocumentKnowledgeBase" --reporter=list` passes (62/62).
  - **Deferred to a follow-up**: dead exports in `ResizeControls.tsx` (`ResizeHandleControl`, `ResizeHandleButton`, `WidthControlBar`, `WidthControlMenu`, `WidthControlToggle`, `WidthButton`, `AutoMinimizeToggle`) — although these are also unreferenced anywhere in the tree, deleting them deterministically exposes a pre-existing race in `tests/DocumentKnowledgeBase.ct.tsx:348 "fullscreen modal covers the entire viewport"` when the suite runs in parallel. The test passes in isolation but fails in the full suite, suggesting the deletion shifts bundle ordering enough to alter timing for the modal mount/measure window. Will be addressed in a separate PR that either patches the test to wait for the modal animation to settle, or finds the underlying race.

- **Frontend dead code cleanup, pass 1** (Issue #1240): Deleted 13 unreachable source files identified by static reachability analysis from `App.tsx`/`index.tsx` and the test entry points. No production importers and no test importers exist for any of these files:
  - Hooks orphaned by refactors: `frontend/src/hooks/useThreadWebSocket.ts` (thread real-time streaming via `ws/thread-updates/` was never wired into any thread component; the backend consumer exists but the frontend hook had no importers — see Issues #623, #697 for future work), `frontend/src/hooks/useUrlAnnotationSync.ts` (the latter was already documented as removed in a comment at `DocumentKnowledgeBase.tsx:1858` but the file itself was never deleted; the stale comment is also removed)
  - Duplicate styled helpers: `frontend/src/components/common.tsx` (duplicated `VerticallyCenteredDiv`/`SidebarContainer` from `components/layout/Wrappers.tsx`) and `frontend/src/components/annotator/common.tsx` (only consumed by the dead file above; also contained a `console.log` that would have appeared in production builds)
  - Dead types: `frontend/src/components/annotator/types/ui.ts` (unused `TabVisibility`) and `frontend/src/components/annotator/renderers/txt/types.ts` (duplicate stub of `MultipageAnnotationJson`; the real one lives in `components/annotator/types/annotations.ts`)
  - Dead barrels (every consumer imports inner files directly, never through these): `frontend/src/components/widgets/icon-picker/index.ts`, `frontend/src/components/corpuses/CorpusAbout/index.ts`, `frontend/src/components/corpuses/CorpusHome/index.ts` (file `CorpusHome.tsx` shadows the directory in Node module resolution), `frontend/src/components/corpuses/caml/index.ts`, `frontend/src/components/corpuses/folders/index.ts`
  - Dead utility: `frontend/src/utils/history.ts` (`createBrowserHistory()` singleton, never imported)
  - Dead mock factory file: `frontend/src/tests/utils/factories.ts` (mock GraphQL response factories with no consumers; its parent directory `src/tests/` is also removed since it's now empty)
  - Updated documentation referencing deleted `useThreadWebSocket`: removed hook section and references from `docs/architecture/websocket/frontend.md`, annotated implementation plan entries in `docs/features/agent_mentions_implementation_plan.md`, removed row from `docs/commenting_system/IMPLEMENTATION_GUIDE.md`
  - Verification: `yarn build` succeeds, all 927 unit tests pass (`yarn test:unit`), and the 16 icon picker component tests still pass (`IconDropdown`, `IconPickerModal`, and `icons.ts` are kept because they have full Playwright `.ct.tsx` test suites in `frontend/tests/`)

### Added

- **Benchmark harness for external RAG datasets** (new app `opencontractserver/benchmarks/`): Generate an OpenContracts corpus from a third-party benchmark (LegalBench-RAG today, pluggable for CUAD/MAUD/etc. via a small adapter interface), run the production extract-grid pipeline against the benchmark's queries with a configurable LLM, probe retrieval independently via `CoreAnnotationVectorStore`, and compute standard metrics (SQuAD-style exact match / token F1 for answers; character-span recall@k / precision@k / IoU for retrieval). Results are written as `report.json` / `report.csv` / `config.json` / `gold.json` under a run directory.
  - Adapter interface and `LegalBenchRAGAdapter` at `opencontractserver/benchmarks/adapters/` (reads the authoritative ZeroEntropy schema — `{"tests": [{"query", "snippets": [{"file_path", "span": [start, end]}], "tags"}]}`)
  - Loader, runner, evaluator, and report modules under `opencontractserver/benchmarks/`
  - Django management command: `python manage.py run_benchmark --benchmark legalbench-rag --path /data/legalbench-rag --user admin --model openai:gpt-4o-mini --top-k 10`
  - Micro fixture under `fixtures/benchmarks/legalbench_rag_micro/` for end-to-end tests without downloading the full dataset
  - Test coverage: `opencontractserver/tests/test_benchmarks.py` (metric unit tests, adapter unit tests, loader materialization test, runner end-to-end test with mocked structured-response agent)
- **`model_override` kwarg on `doc_extract_query_task`** (`opencontractserver/tasks/data_extract_tasks.py`): Optional, backward-compatible kwarg that lets callers override the hardcoded `openai:gpt-4o-mini` default for a single invocation. Consumed by the benchmark runner to sweep models without affecting production defaults; still defaults to `openai:gpt-4o-mini` when not supplied.
- **Frontend unit tests for utils and hooks** (Issue #1267): Added 14 new `*.test.ts(x)` files covering previously-untested utilities and hooks to raise `frontend-unit` coverage on high-ROI pure functions:

  - **Utils**: `formatters.test.ts`, `arrayUtils.test.ts`, `colorUtils.test.ts`, `parseOutputType.test.ts`, `annotationGuards.test.ts`, `env.test.ts`, `extractUtils.test.ts`, `layout.test.ts`, `persistentVar.test.ts`, `routingLogger.test.ts`, `navigationCircuitBreaker.test.ts`, `performance.test.ts`, `jobNotificationCacheUpdates.test.ts`, `compactAnnotationJson.test.ts`.
  - **Hooks**: `useAuthReady.test.tsx`, `useFeatureAvailability.test.ts`, `useMessageBadges.test.tsx`, `useBadgeCelebration.test.tsx` (render-hook based with vi.useFakeTimers/MockedProvider).
  - Adds ~210 new assertions across file-size/date/initial formatting, hex→RGB(A) conversions, Pydantic output-type parsing, per-annotation runtime guards, runtime env coercion, extract status, viewport clamping, session-storage-backed reactive vars, debug logger toggling, navigation circuit breaker tripping/reset/window pruning, performance monitor metric lifecycle, Apollo cache field-level mutation dispatch for job notifications, and v1↔v2 compact annotation JSON round-tripping.
  - Verification: full unit suite passes (1118/1118) and `tsc --noEmit` is clean.

- **Unit tests for Jotai atoms and Apollo reactive vars** (Issue #1268): Added vitest coverage for the global state layer per the ROI audit in PR #1266.

  - **`frontend/src/atoms/__tests__/folderAtoms.test.ts`**: 46 tests covering every primitive atom (initial value + write), every derived atom (`folderTreeAtom`, `folderBreadcrumbAtom`, `folderMapAtom`, `canCreateFoldersAtom`) across multiple dependency states, every write-only action atom (toggle/expand/open/close helpers), and `atomWithStorage` persistence paths (Set ↔ JSON round-trip, malformed-JSON fallback, SSR-safe `sidebarCollapsedAtom` default for mobile vs desktop viewports).
  - **`frontend/src/atoms/__tests__/threadAtoms.test.ts`**: 9 tests covering all five atoms plus a localStorage round-trip and re-hydration path for `threadContextSidebarExpandedAtom`.
  - **`frontend/src/graphql/__tests__/cache.test.ts`**: 24 tests covering initial values of every reactive var exported from `cache.ts` (routing, modals, entity refs, search terms, collections), round-trip writes for representative vars, `mergeArrayByIdFieldPolicy` id-based merge + default-empty branches, `InMemoryCache` presence, and `showKnowledgeBaseModal` (`persistentVar`) first-write persistence, re-hydration from sessionStorage, and malformed-JSON fallback.
  - Coverage (v8 provider): `src/atoms` → 99.23% statements / 93.54% branches; `frontend/src/graphql/cache.ts` → 95.6% statements / 100% branches. Well above the 80% target in the issue.
  - Verification: all 79 new tests green; full unit suite (1014/1014) still passes; `tsc --noEmit` clean.

- **Frontend permission-gating tests** (Issue #1269): Added branch-exhaustive coverage for the permission predicates that gate annotation write UI.

  - New utility `frontend/src/utils/annotationPermissions.ts` centralizes three pure predicates (`canEditAnnotationsInCorpus`, `canDeleteAnnotation`, `canUpdateAnnotation`) that were previously inlined in `DocumentKnowledgeBase.tsx:471` and `HighlightItem.tsx:242`. Both consumers now delegate to the helper, eliminating duplicated logic.
  - New unit tests `frontend/src/utils/__tests__/annotationPermissions.test.ts` cover every branch of the predicates, including the full 2×2 truth table for the corpus/document effective-edit check and the full 2³ truth table for the structural × read-only × `CAN_REMOVE` delete gate (30 assertions).
  - New Playwright CT tests `frontend/tests/HighlightItemPermissions.ct.tsx` (+ harness `HighlightItemPermissionsTestWrapper.tsx`) verify that the sidebar delete affordance renders only on the intersection of the required conditions, and that structural annotations are delete-locked even with `CAN_REMOVE` (6 scenarios).
  - Production behavior is unchanged — the helper is a faithful extraction of the existing inline logic.

- **Frontend E2E integration tests with coverage**: Added a Playwright integration spec (`frontend/tests/e2e/login-and-navigation.spec.ts`) that exercises the full Vite + Django + Postgres stack. The spec logs in via the password form against a real backend and walks every routed view in `src/views/` (Discovery, Corpuses, Documents, LabelSets, Annotations, Extracts, GlobalDiscussions, ThreadSearchRoute, PrivacyPolicy, TermsOfService, UserProfile, Login). New supporting files:
  - `frontend/tests/e2e/fixtures.ts` — Playwright fixture that dumps `window.__coverage__` to `frontend/coverage/e2e/.nyc_output/` after every test (mirrors the CT pattern in `tests/utils/coverage.ts`).
  - `frontend/tests/e2e/helpers.ts` — `VIEWS` catalog, `loginViaUI`, `spaNavigate`, and `expectViewVisible` helpers. Uses `history.pushState` + `popstate` dispatch for SPA navigation so the in-memory `authToken` reactive var survives between routes.
  - `frontend/playwright.config.ts` — Rewritten to manage a `webServer` block that boots vite with `COVERAGE=true` and `REACT_APP_USE_AUTH0=false`, runs the chromium project against `http://127.0.0.1:5173`, and matches `tests/e2e/**/*.spec.ts` (existing `*.spec.tsx` files under `tests/` keep running inside the CT runner).
  - `frontend/package.json` — New `test:e2e:coverage` script that runs the spec with Istanbul instrumentation and merges the per-test JSON dumps into `coverage/e2e/lcov.info` via `nyc report`.
  - `.github/workflows/frontend-e2e.yml` — New CI job that builds the django image from `test.yml`, brings up postgres + redis + django with `--no-deps` (skipping the multi-GB parser/embedder images), waits on `/api/health/`, verifies migration `0003_create_initial_superuser` produced the `admin` user, then runs `yarn test:e2e:coverage` and uploads the lcov to Codecov under the `frontend-e2e` flag.
  - `.codecov.yml` — Declared the `frontend-e2e` flag with `paths: [frontend/src/]` and `carryforward: true` so the E2E upload participates in project coverage aggregation alongside `frontend-unit` and `frontend-component`. Without this declaration the flag was accepted by Codecov but not aggregated into the main-branch project total, which is why coverage did not visibly rise after PR #1251 merged.
- **Bounded `fullDatacellList` payload for extract grid embeds** (Issue #1204): `ExtractType.fullDatacellList` now accepts optional `limit` and `offset` arguments, and a new `ExtractType.datacellCount` field returns the total visible datacell count ignoring pagination. Resolver `resolve_full_datacell_list` applies a deterministic `order_by("document_id", "column_id", "id")` before slicing so limits/offsets produce stable windows across requests (`config/graphql/extract_types.py`, `schema.graphql`). Frontend `GET_EXTRACT_GRID_EMBED` (`frontend/src/graphql/queries.ts`) now passes `limit: EXTRACT_GRID_EMBED_CELL_LIMIT` (500) and fetches `datacellCount`; `ExtractGridEmbed` (`frontend/src/components/extracts/ExtractGridEmbed.tsx`) renders a "Showing N of M cells" / "Showing N of M documents" partial-data footer instead of the previous hard "exceeds limit" error screen. Adds the `EXTRACT_GRID_EMBED_CELL_LIMIT` constant (`frontend/src/assets/configurations/constants.ts`), a new `partial` state to `ExtractGridEmbedTestWrapper`, and backend coverage in `test_extract_queries.py::test_full_datacell_list_limit_offset_and_count`.

- **Agent memory system**: Per-corpus memory that lets agents accumulate reusable insights from conversations. Memory is stored as a first-class markdown Document in the corpus (visible and editable by users). Features include:
  - Corpus model fields: `memory_enabled` toggle and `memory_document` FK (`opencontractserver/corpuses/models.py`)
  - Memory CRUD utilities with hybrid retrieval (full injection for small memory, keyword-scored section filtering for large) (`opencontractserver/agents/memory.py`)
  - End-of-conversation curation via two-stage Celery task: privacy-preserving summarisation followed by LLM-based pattern extraction (`opencontractserver/tasks/memory_tasks.py`)
  - Automatic memory injection into agent system prompts via `UnifiedAgentFactory` (`opencontractserver/llms/agents/agent_factory.py`)
  - New agent tools: `get_corpus_memory` (read) and `suggest_memory_update` (write) (`opencontractserver/llms/tools/core_tools.py`)
  - GraphQL `ToggleCorpusMemory` mutation and `memoryActiveWarning` field with privacy notice (`config/graphql/corpus_mutations.py`, `config/graphql/corpus_types.py`)
  - Periodic Celery beat task to detect idle conversations eligible for curation (`config/settings/base.py`)
  - Constants and configuration in `opencontractserver/constants/agent_memory.py`
- **Frontend code coverage reporting** (PR #1205): Added coverage collection and Codecov integration for both unit tests (Vitest + v8) and component tests (Playwright CT + Istanbul). Coverage is opt-in via `COVERAGE=true` environment variable to avoid performance overhead during normal test runs. Custom Playwright fixture (`frontend/tests/utils/coverage.ts`) extracts Istanbul `__coverage__` data from the browser after each test. CI uploads separate coverage flags (`frontend-unit`, `frontend-component`) with `carryforward: true` for granular tracking. Added `.codecov.yml` configuration, `vite-plugin-istanbul` for build-time instrumentation, and `nyc` for LCOV report generation from collected coverage data.

### Fixed

- **Rules-of-Hooks violation in `UserProfileRoute`** (Issue #1295): `frontend/src/components/routes/UserProfileRoute.tsx` called `useQuery` _after_ a conditional early return for the `!slug` redirect case, so when the same component fiber was reused across the `/profile` → `/users/:slug` redirect (e.g. both routes rendering `UserProfileRoute` in a single `Routes` tree) React threw `Rendered more hooks than during the previous render`. Production flows that unmount/remount across the redirect masked the bug, but any future refactor that kept the fiber alive would start crashing, and test suites that mounted both routes together could not exercise the full redirect → render path. Moved `useQuery` above the early-return block and pass `{ slug: slug ?? "" }` as variables while keeping the existing `skip: !slug` gate on the network call.
- **`PdfAnnotations.undoAnnotation()` mutated the source array** (Issue #1291): `frontend/src/components/annotator/types/annotations.ts` called `this.annotations.pop()` before constructing the returned instance, silently mutating the caller's `annotations` array even though the field is declared `readonly`. Every other method on this reducer-style class returns without touching the input, so the leak was a footgun for downstream consumers of the upcoming PdfAnnotator package extraction (Issue #1283) — any reference to the pre-undo instance would observe a stale shorter array without re-rendering. Replaced `.pop()` with a non-mutating `slice(0, -1)` + index read. Updated `frontend/src/components/annotator/types/__tests__/annotations.test.ts` to remove the pinning comment and add an explicit assertion that the original instance's `annotations` array is unchanged after `undoAnnotation()`.
- **`RelationGroup.updateForAnnotationDeletion` never pruned orphaned relations** (Issue #1292): `frontend/src/components/annotator/types/annotations.ts:49-50` computed `nowSourceEmpty` / `nowTargetEmpty` against `this.sourceIds` / `this.targetIds` (the **pre-filter** arrays) instead of the `newSourceIds` / `newTargetIds` arrays produced a few lines above. Because the "now empty" flags were identical to the "was empty" flags, all four `return undefined` branches intended to prune orphaned relations were unreachable, and the method always returned a fresh `RelationGroup` — even when `PdfAnnotations.undoAnnotation()` had just removed the relation's only source or target annotation. That left zombie relations in `PdfAnnotations.relations` with empty `sourceIds` or `targetIds` and required defensive handling downstream. Fix swaps the two comparisons to the post-filter arrays so the documented deletion conditions fire correctly. `frontend/src/components/annotator/types/__tests__/annotations.test.ts` drops the bug-pinning header note and replaces the ambiguous "prunes any relations" assertion with two explicit cases: the survivor is kept when it still has a source and a target, and a relation whose only target references the popped annotation is now removed (21/21 unit tests pass).
- **O(N) query regression in bulk folder operations** (Issue #1199): `DocumentFolderService.move_documents_to_folder()` and `DocumentFolderService.delete_folder()` in `opencontractserver/corpuses/folder_service.py` previously issued ~3 DB round-trips per document (an EXISTS-style disambiguation fetch, an `UPDATE` via `save(update_fields=…)`, and an `INSERT` via `DocumentPath.objects.create()`). For a 100-document batch that was ~300 queries instead of the single-query `.update()` the old non-lineage code used. Both methods now:
  1. Pre-fetch all occupied paths in the target directory in a **single** query via the new `_fetch_occupied_paths_in_directory` helper and pass the shared mutable set to `_disambiguate_path` via a new `occupied_override` parameter (replaces the single-purpose `extra_occupied` kwarg).
  2. Batch-deactivate every superseded `DocumentPath` row with one `.filter(pk__in=…).update(is_current=False)` call.
  3. Batch-insert every successor row with one `.bulk_create()` call, then manually dispatch `post_save` (`created=True`) via the new `_dispatch_document_path_created_signals` helper so the document-text embedding side effect wired up in `documents/signals.py::process_doc_on_document_path_create` still fires (bulk_create normally bypasses per-row signals).
  4. Use `select_related("document")` + `select_for_update(of=("self",))` on the affected-path query so `current.document` accesses inside the build loop no longer N+1, and the row lock stays scoped to the `DocumentPath` table. Net result: a 100-document bulk move now executes roughly 4 DB round-trips instead of ~300, and the old batched `.update()` performance is restored without sacrificing the Path Tree history nodes introduced in PR #1195.
- **TOCTOU race on `DocumentPath` uniqueness** (Issue #1200): `DocumentFolderService.move_document_to_folder()`, `move_documents_to_folder()`, and `delete_folder()` previously caught `IntegrityError` from the `unique_active_path_per_corpus` partial unique constraint and either bubbled it up to the caller as a "Path conflict, please retry" error (single move) or rolled back the entire batch (bulk move / folder delete). Under concurrent moves of different documents to the same target folder, two transactions could both observe a candidate path as free in `_disambiguate_path()` and race to insert it; the loser hit the partial unique index and the operation failed. New helper `DocumentFolderService._create_successor_path_with_retry()` (`opencontractserver/corpuses/folder_service.py`) wraps the deactivate-then-create pair in a savepoint and retries with a freshly disambiguated path on `IntegrityError`, treating each losing path as occupied. Up to `MAX_PATH_CREATE_RETRIES + 1` attempts (`opencontractserver/constants/document_processing.py`) run before propagating the conflict. The partial unique index added in migration `0023_documentpath_documentpathgroupobjectpermission_and_more` remains the authoritative correctness guarantee. Test class `TestMoveDocumentIntegrityRecovery` covers single-document transient-failure recovery, disambiguated retry path selection, and exhausted-retry rollback. Bulk operations (`move_documents_to_folder`, `delete_folder`) now use the batch approach from Issue #1199 instead of per-row retry, so their former test classes (`TestBulkMoveIntegrityRecovery`, `TestDeleteFolderIntegrityRecovery`) have been replaced by `TestCoverageGapBulkMoveIntegrityErrorRollback` and `TestCoverageGapDeleteFolderIntegrityErrorRollback` which verify full-batch rollback on `IntegrityError` (`opencontractserver/tests/test_document_folder_service.py`).
- **Bulk move loop recomputed target folder path per document** (Issue #1202): `DocumentFolderService.move_documents_to_folder()` (`opencontractserver/corpuses/folder_service.py`) called `_compute_moved_path()` once per document, and each call invoked `target_folder.get_path()` — which walks ancestors via a recursive CTE query. For an N-document bulk move to the same folder, this issued N redundant CTE queries for an O(1) value. The target folder path is now resolved once before the loop and threaded through `_compute_moved_path()` via a new optional `target_folder_path` parameter.
- **Backend coverage disappearing from Codecov dashboard** (`.codecov.yml`): Added a `backend` flag with `carryforward: true` (paths: `opencontractserver/`, `config/`), mirroring the pattern already in place for `frontend-unit` / `frontend-component`. Without carryforward, any commit whose backend `pytest` job fails or times out (e.g. the merge commit `f166a59`, where the push-to-main `pytest` check failed after PR #1213 merged) causes Codecov to record backend files at 0% for that commit, which then replaces the healthy ~90% backend coverage on the dashboard — making it look as if only frontend coverage (~39%) is being collected. With carryforward, a flaky/failing backend run inherits the parent commit's backend coverage so the dashboard continues to reflect both suites.
- **IngestionSource follow-up hardening** (Issue #1228):

  - Added `@graphql_ratelimit` to `CreateIngestionSourceMutation`, `UpdateIngestionSourceMutation`, and `DeleteIngestionSourceMutation` (`config/graphql/ingestion_source_mutations.py`). Previously these mutations only carried `@login_required`, allowing an authenticated user to create thousands of rows (plus Guardian permission entries) in a tight loop. Creates/updates use `WRITE_MEDIUM`, deletes use `WRITE_LIGHT`, matching other CRUD mutations in the codebase.
  - Guarded the fallback `.get()` inside the `except IntegrityError` handler in `_import_ingestion_sources` (`opencontractserver/tasks/import_tasks_v2.py` ~line 463). In the rare scenario where a concurrent request created-then-deleted the row between the `IntegrityError` and the fallback, the unguarded `.get()` would raise `IngestionSource.DoesNotExist` and abort the entire corpus import. Now logs a warning and continues.
  - Added an explicit `fields` allowlist to `IngestionSourceType.Meta` (`config/graphql/document_types.py`) to prevent `user_lock`, `backend_lock`, and `is_public` from leaking through the GraphQL API. `user_lock` in particular would leak the username of whoever currently holds the lock — an information-disclosure issue.
  - Added a clarifying comment documenting the intentional export/import asymmetry for `ingestion_metadata` in `_reconstruct_document_paths` (`opencontractserver/tasks/import_tasks_v2.py`).

- **Document lineage fields dropped on move/delete/restore** (PR #1197): `move_document`, `delete_document`, and `restore_document` in `opencontractserver/documents/versioning.py` now forward `ingestion_source`, `external_id`, and `ingestion_metadata` from the parent path record to newly-created path nodes. Previously these fields were silently dropped, losing lineage provenance on any path operation.

- **`ingestion_metadata` truthiness check** (PR #1197): Changed `if ingestion_metadata:` to `if ingestion_metadata is not None:` in `config/graphql/document_mutations.py` so that an empty dict `{}` is correctly stored rather than silently discarded.

- **Lazy logging in export_v2.py** (PR #1197): Replaced remaining f-string logger calls with `%s`-style lazy formatting in `opencontractserver/utils/export_v2.py` for corpus folders, relationships, conversations, structural annotations, and markdown description error handlers.

- **TOCTOU race in `_import_ingestion_sources`** (PR #1197): Wrapped `get_or_create` in `transaction.atomic()` savepoint so that an `IntegrityError` from a concurrent insert doesn't abort the outer PostgreSQL transaction, which would cause the fallback `.get()` to raise `TransactionManagementError`.

- **`DRFMutation` validation error formatting extracted** (PR #1197): Extracted inline validation-error formatting logic from `DRFMutation.mutate()` into a reusable `format_validation_error()` static method (`config/graphql/base.py`). Updated `TestDRFMutationValidationError` tests to call the real method instead of duplicating the logic inline.

- **Renamed `EXPECTED_GLOBAL_ID_TYPE` to `INGESTION_SOURCE_GLOBAL_ID_TYPE`** (PR #1197): Renamed the generic constant to be self-documenting across `config/graphql/document_types.py`, `document_queries.py`, `document_mutations.py`, and `ingestion_source_mutations.py`.

- **Conversation pagination cache collisions** (PR #1206): Added `keyArgs` configuration for the `conversations` relay pagination cache entry in `frontend/src/graphql/cache.ts`. Previously `conversations` used `relayStylePagination()` with no key arguments, causing all conversation queries (across different corpora, documents, and conversation types) to share a single cache entry. This led to paginated results from one context bleeding into another. Now uses `["documentId", "corpusId", "conversationType", "hasCorpus", "hasDocument"]` to isolate cache entries by filter dimensions.

- **GraphQL security hardening cleanup** (Issue #1198):

  - Corrected misleading `DisableIntrospection` docstring that claimed the class checks DEBUG, when it unconditionally blocks introspection (`config/graphql/security.py`)
  - Rewrote `test_introspection_allowed_in_debug` to use graphql-core's `validate()` directly, as graphene's test Client does not apply validation rules (`opencontractserver/tests/test_security_hardening.py`)
  - Added backslash-prefix check to `_get_safe_redirect_url()` to prevent open-redirect bypass via browser backslash-to-slash normalization (`config/admin_auth/views.py`)

- **Batch embedding cleanup** (Issue #1226): Follow-up on PR #1148:
  - Added backward-compatible re-exports of `EmbeddingClientError` / `EmbeddingServerError` from `opencontractserver/pipeline/embedders/multimodal_microservice.py` so external embedders that imported the symbols from their original module keep working. Canonical location remains `opencontractserver.pipeline.base.exceptions`.
  - `EmbeddingClientError` is now actually raised: batch embedding methods (`MicroserviceEmbedder.embed_texts_batch` and `BaseMultimodalMicroserviceEmbedder.embed_texts_batch` / `embed_images_batch`) now raise `EmbeddingClientError` on 4xx responses instead of silently returning `None`. Callers can now distinguish a client-side failure ("we sent bad data") from a parsing error that still returns `None`. `_batch_embed_text_annotations` in `opencontractserver/tasks/embeddings_task.py` catches `EmbeddingClientError` explicitly and records a permanent per-annotation failure without triggering Celery retries — preventing retries from burning on invalid input that will never succeed. The single-text paths (`_embed_text_impl`, `_embed_image_impl`) are intentionally unchanged because their error semantics must not abort remaining annotations in a task. Tests updated in `test_batch_embedding.py` and `test_multimodal_embedder_unit.py`.
  - Moved `import requests` from inside `BaseEmbedder.embed_texts_batch()` to the top of `opencontractserver/pipeline/base/embedder.py`. Local imports are reserved for breaking circular-import cycles; `requests` is already a listed dependency so module-level import is cheaper on repeated calls.
  - Refactored `_create_embedding_for_annotation` in `opencontractserver/tasks/embeddings_task.py` to remove a misleadingly-indented `else` block. The text-only fallthrough now clearly separates the image-drop debug log from the `return _create_text_embedding(...)` statement.
  - Documented in the `calculate_embeddings_for_annotation_batch` docstring that the returned `result` dict counts reflect only the last attempt on Celery retry (not cumulative work across retries). `add_embedding()` is idempotent so there is no correctness risk, but any external monitoring consuming `result` should read Celery task state for cumulative counts.

### Added

- **Document lineage tracking** (IngestionSource + DocumentPath enrichment): New `IngestionSource` model (`opencontractserver/documents/models.py`) registers named integrations/crawlers/pipelines that produce documents. Three new fields on `DocumentPath` — `ingestion_source` (FK), `external_id` (indexed CharField), and `ingestion_metadata` (JSON) — record which source produced each version of a document and with what context. Lineage kwargs flow through `import_document()` (`opencontractserver/documents/versioning.py`), `Corpus.add_document()` (`opencontractserver/corpuses/models.py`), and the `UploadDocument` GraphQL mutation. New GraphQL types (`IngestionSourceType`, `IngestionSourceTypeEnum`), queries (`ingestionSources`, `ingestionSource`), and CRUD mutations (`createIngestionSource`, `updateIngestionSource`, `deleteIngestionSource`) in `config/graphql/`. Migration: `0036_add_ingestion_source_and_lineage_fields`.

- **Document path history tracking** (PR #1195): Document moves via `DocumentFolderService.move_document_to_folder()`, `move_documents_to_folder()`, and `delete_folder()` now create immutable `DocumentPath` history nodes instead of silent in-place updates. Each move produces a new `DocumentPath` record linked to the previous one via `parent`, enabling full audit trail traversal. Path conflicts are resolved by appending numeric suffixes (e.g. `report_1.pdf`). `versioning.py` `determine_action()` now detects folder-only moves (same path string, different `folder_id`). **Atomic guarantees**: Both `delete_folder()` and `move_documents_to_folder()` use `transaction.atomic()` so that if any document fails to relocate, ALL changes are rolled back (no partial-success state). Within-batch path conflict detection prevents two documents with the same filename from colliding during bulk moves. Failed operations are safe to retry. Comprehensive test coverage in `test_document_folder_service.py` for move tracking, path conflicts, bulk moves, delete-folder tracking, atomic rollback, retry-after-failure, and full lifecycle integration.
- **Move-document agent tool** (PR #1196): New `move_document` / `amove_document` tool that moves a document between folders within the same corpus. Delegates to `DocumentFolderService.move_document_to_folder` for permission checks and path updates. Categorized as `ToolCategory.CORPUS` (corpus-level agents only), requires write permission and approval. Pass `target_folder_id` for a specific folder or omit for corpus root. Uses `visible_to_user()` for document and corpus lookups to prevent IDOR enumeration (`opencontractserver/llms/tools/core_tools.py`, `opencontractserver/llms/tools/tool_registry.py`).

### Changed

- **`move_documents_to_folder()` return value semantics** (PR #1195): The returned `moved_count` now reflects only documents that were actually relocated; documents already in the target folder are skipped and not counted. Previously returned `len(document_ids)` unconditionally.
- **Web search agent tool** (PR #1174): New `aweb_search` tool for agents with pluggable provider backends (Brave Search and Tavily). Introduces `BaseTool` base class (`opencontractserver/llms/tools/base_tool.py`) for tools with database-backed settings and encrypted secrets via `PipelineSettings`. Tools declare a nested `Settings` dataclass with `PipelineSetting` metadata; `is_configured()` gates tool availability at resolution time. `ToolFunctionRegistry` supports `tool_class` entries for database-level enablement gating. New GraphQL mutations `UpdateToolSecretsMutation` and `DeleteToolSecretsMutation` (`config/graphql/pipeline_settings_mutations.py`) allow superusers to manage tool API keys. Includes per-process rate limiting, provider-specific response parsing, and comprehensive test coverage.

### Fixed

- **Quickstart blockers** (PR #1179): Fixed several issues preventing first-time setup from working:
  - Missing `EMBEDDINGS_MICROSERVICE_URL` and `DOCLING_PARSER_SERVICE_URL` env var defaults crash Django on startup (`config/settings/base.py`)
  - Celerybeat race condition on first boot — added Django healthcheck and `service_healthy` dependency for celery services (`local.yml`)
  - Password login broken — JWT middleware was gated behind `USE_AUTH0=True` so `tokenAuth` tokens were never validated (`config/settings/base.py`)
  - Admin user login 500 — `User.save()` re-validated slug on `last_login` updates, and `"admin"` is a reserved slug (`opencontractserver/users/models.py`)
  - Django admin panel 500 on static files — overrode `StaticFilesStorage` in local settings (`config/settings/local.py`)
  - Nginx duplicate `.js` MIME type warning (`frontend/conf/conf.d/default.conf`)
  - Updated README quickstart with missing steps and corrected `docker compose` v2 syntax

### Added

- **CAML Interactive Article System** (PR #1156): Frontend support for corpus articles using CAML (Corpus Article Markup Language). Includes a two-pass parser (tokenizer + block parsers), typed intermediate representation, and composable renderer supporting hero sections, cards, pills, tabs, timelines, CTAs, signup blocks, corpus stats, and pullquotes. Frontend adds `CamlArticleEditor` modal with live preview, `CorpusArticleView` for rendered article display, and `CorpusLandingView` integration for article discovery. Playwright component tests with `docScreenshot` captures for all block types.
- **Extract grid embed in CAML articles** (PR #1177): New `ExtractGridEmbed` component renders extract data tables inline in CAML articles via `[component:extract-grid extractId=...]` marker syntax. Includes `useCamlComponentRenderer` hook and `camlComponents` utility for a generic component marker system, `CAML_COMPONENTS` shared registry (`frontend/src/utils/camlComponentRegistry.ts`), and `ExtractPickerDropdown` toolbar in the editor for inserting extract grids. Query (`GET_EXTRACT_GRID_EMBED`) fetches extract fieldset columns, datacells, and source annotations with deep links to the document viewer.

### Fixed

- **CAML YAML parser nested key bug**: Fixed `parseYamlFrontmatter` in `frontend/src/caml/parser/tokenizer.ts` where `content` used `line.trimEnd()` instead of `line.trimEnd().trimStart()`, causing the key-value regex (`^[a-zA-Z_]...`) to fail for indented nested keys (e.g., `hero.kicker`). Nested frontmatter properties were silently dropped, producing empty objects.

### Changed

- **Replaced hardcoded hex colors with OS_LEGAL_COLORS tokens** in `CamlArticleEditor.tsx` and `CorpusArticleView.tsx`: All hardcoded hex values (`#e2e8f0`, `#fafbfc`, `#f8fafc`, `#64748b`, `#94a3b8`, `#ffffff`, `#fef3c7`, `#92400e`, `#475569`, `#f1f5f9`, `#cbd5e1`) replaced with semantic design tokens from `osLegalStyles.ts`.
- **Extracted `isExternalHref` helper** in `frontend/src/caml/renderer/safeHref.ts`: Deduplicated `href.startsWith("http")` checks across `CamlBlocks.tsx` and `CamlFooter.tsx` into a shared utility function.
- **Removed redundant `articleStats` useMemo** in `CorpusArticleView.tsx`: The memo mirrored its input without transformation; `stats` is now passed directly to `CamlArticle`.

### Fixed

- **Semantic UI removal cleanup** (Closes #1123): Fixed `folderIconAlpha` color mismatch in `ExtractCellFormatter.tsx` StatusDot box-shadows — replaced hardcoded `rgba(245, 158, 11, ...)` with `folderIconAlpha()` to match `getCellBackground()`. Changed Revoke button in `WorkerTokensSection.tsx` from `variant="secondary"` with inline danger color to `variant="danger"` for proper destructive-action styling. Removed conflicting `box-shadow` and `border-radius` from `StyledModalInner` in `EditMessageModal.tsx` to prevent doubled styling with `OsModal` wrapper. Added `role="menu"`, `aria-label`, `aria-haspopup`, keyboard support, and arrow key navigation (roving tabindex) to custom popup in `ExtractCellFormatter.tsx`. Removed redundant Escape key handler from `StatusDot` `onKeyDown` (already handled by global listener). Extracted inline `maxHeight: "70vh"` in `SelectDocumentsModal.tsx` to `MODAL_BODY_MAX_HEIGHT` constant in `constants.ts`.
- **Annotation rendering cleanup** (Closes #1144): Replaced hardcoded `"4px 4px 0 0"` border-radius with `ANNOTATION_BOUNDARY_RADIUS` constant in `SearchResult.tsx` and `ChatSourceResult.tsx`. Removed dead `border` prop from `SelectionInfo` interface and call sites. Used `APPROVED_RGB` constant for approved-state box-shadow in `SelectionBoundary.tsx` (matching `REJECTED_RGB` pattern). Added `TOKEN_EXPANSION_PX` constant and replaced magic `-1`/`+2` token expansion values across `SelectionTokenGroup.tsx`, `Tokens.tsx`, and `ChatSourceTokens.tsx`. Consolidated `[Previous Unreleased]` CHANGELOG section into single `[Unreleased]` block. Removed debug `console.log` statements from `DocumentKnowledgeBase.ct.tsx`, `SearchResult.tsx`, and `SelectionTokenGroup.tsx`. Moved annotation display reactive var initialization into `useEffect` in `DocumentKnowledgeBaseTestWrapper.tsx`.

### Added

- **Index tab in document sidebar**: Added a new "Index" tab (BookOpen icon) to the document viewer sidebar, positioned first before Chat, Feed, and Discussions. The Index tab reuses the `DocumentAnnotationIndex` component to display the document's OC_SECTION-based table of contents directly in the sidebar (`frontend/src/components/knowledge_base/document/document_kb/RightPanelContent.tsx`, `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`).
- **OpenAI Embedder pipeline component** (`opencontractserver/pipeline/embedders/openai_embedder.py`): New `OpenAIEmbedder` class supporting `text-embedding-3-small` (default), `text-embedding-3-large`, and `text-embedding-ada-002` models. Features configurable output dimensions, custom API base URL for Azure/proxy support, and graceful error handling. Dynamic `vector_size` property reflects runtime model/dimension configuration to prevent pgvector column size mismatches.
- **OpenAI embedding constants** (`opencontractserver/constants/embeddings.py`): Centralized model dimension mappings and defaults for the OpenAI embedder.

### Fixed

- **BulkImportModal AlertBox hardcoded colors** (Closes #1145): Replaced `OS_LEGAL_COLORS.warning*` / `OS_LEGAL_COLORS.info*` constants with CSS custom properties (`var(--oc-warning-*)`, `var(--oc-info-*)`). Added the six new tokens to `index.css :root` so they participate in theme switching (`frontend/src/index.css`, `frontend/src/components/widgets/modals/UploadModalStyles.ts`).
- **CloudUpload icon inline magic numbers** (Closes #1145): Replaced `style={{ width: 16, height: 16, marginRight: 8 }}` on the Start Import button icon with a new `ButtonIcon` styled component that uses `var(--oc-font-size-md)` and `var(--oc-spacing-xs)` tokens (`frontend/src/components/widgets/modals/BulkImportModal.tsx`, `UploadModalStyles.ts`).
- **Remaining pixel values in styled components** (Closes #1145): Replaced `HeaderIcon` 28px/15px with `calc()` expressions using spacing/font tokens, `DropZone` 200px/160px min-heights with `calc(var(--oc-spacing-xl) * N)`, and `AlertBox` SVG 2px margin-top with `calc(var(--oc-spacing-xs) / 4)` with a documenting comment (`UploadModalStyles.ts`).
- **Missing InMemoryCache in BulkImportTestWrapper** (Closes #1145): Added `InMemoryCache` with relevant type policies and a `mocks` prop to match the `DocumentKnowledgeBaseTestWrapper` pattern (`frontend/tests/wrappers/BulkImportTestWrapper.tsx`).

### Added

- **Sidecar JSON schema validation before annotation import** (Closes #1127): Added `_validate_sidecar_schema()` in `opencontractserver/tasks/import_tasks.py` that validates container types (`labelled_text`, `doc_labels`, `relationships` must be lists) and required keys per entry (`annotationLabel`/`rawText`/`annotation_json` for annotations; `relationshipLabel`/`source_annotation_ids`/`target_annotation_ids` for relationships) before any database work. Invalid sidecars now increment `annotation_sidecars_errored` and log clear error messages instead of crashing or silently skipping data. Comprehensive unit and integration tests added in `opencontractserver/tests/test_sidecar_import.py`.
- **Progress step test coverage for BulkImportModal** (Closes #1145): New test exercises the progress UI (spinner, progress bar, "Importing Documents..." heading) by mocking `IMPORT_ZIP_TO_CORPUS` with a long delay, verifying the loading state is rendered and footer buttons are hidden during import (`frontend/tests/bulk-import-modal.ct.tsx`).

### Added

- **Schema validation for `labels.json` before processing annotation labels** (Closes #1128): Added `validate_labels_data()` in `opencontractserver/utils/importing.py` that validates the structure of `labels.json` before `prepare_import_labels()` is called. Checks that `text_labels`/`doc_labels` are dicts (not lists), each label entry is a dict with a required non-empty `text` field, and optional fields (`label_type`, `color`, `icon`, `description`) have correct types. Validation errors are logged and appended to `results["errors"]` with `labels_loaded` set to `False`. Added unit tests (`TestValidateLabelsData`) and integration tests (`TestMalformedLabelsImport`) in `opencontractserver/tests/test_sidecar_import.py`.

### Changed

- **Batch embedding API calls for annotation embeddings** (PR #1148): Refactored `calculate_embeddings_for_annotation_batch` Celery task to use `embed_texts_batch()` for text-only annotations instead of embedding one-at-a-time, reducing HTTP requests from one-per-annotation to one-per-sub-batch. Key changes:
  - New `embed_texts_batch()` extension point on `BaseEmbedder` (`opencontractserver/pipeline/base/embedder.py`): default sequential fallback that re-raises transient HTTP exceptions; subclasses override for true batch API endpoints.
  - `MicroserviceEmbedder.embed_texts_batch()` calls `/embeddings/batch` with per-item NaN detection, vector-count mismatch validation, and 5xx → `EmbeddingServerError` for Celery retry (`opencontractserver/pipeline/embedders/sent_transformer_microservice.py`).
  - Moved `EmbeddingClientError` / `EmbeddingServerError` to `opencontractserver/pipeline/base/exceptions.py` (breaking import path change for external embedders that imported from `multimodal_microservice`).
  - Django system check `documents.E001` validates `EMBEDDING_API_BATCH_SIZE <= MICROSERVICE_EMBEDDER_MAX_BATCH_SIZE` at startup (`opencontractserver/documents/checks.py`).
  - New constants in `opencontractserver/constants/document_processing.py`: `EMBEDDING_API_BATCH_SIZE` (50), `MICROSERVICE_EMBEDDER_MAX_BATCH_SIZE` (100), `EMBEDDER_SINGLE_REQUEST_TIMEOUT_SECONDS` (30), `EMBEDDER_BATCH_REQUEST_TIMEOUT_SECONDS` (60).
  - Extracted `_get_service_config()` helper in `MicroserviceEmbedder` to eliminate config duplication.
  - Comprehensive test suite (938 lines) covering sequential fallback, partial failures, vector count mismatch, per-item NaN handling, sub-batching, transient error propagation through the full Celery task, and dual-strategy fallback (`opencontractserver/tests/test_batch_embedding.py`).
- **Deferred structural annotation loading for PDF performance**: Structural annotations (headers, sections, paragraphs — often 4,000-6,000 for large documents) are no longer fetched in the initial GraphQL queries. They are loaded lazily via `GET_DOCUMENT_STRUCTURAL_ANNOTATIONS` only when the user toggles structural visibility on. Also removed redundant structural annotation re-fetching during analysis/extract switching (structural annotations are analysis-independent). For a 200-page document, this eliminates ~5-20MB of JSON from the initial payload (`frontend/src/graphql/queries.ts`, `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`, `frontend/src/components/annotator/context/AnnotationAtoms.tsx`).
- **OC\_\* annotations filtered from Feed**: Platform-generated annotations with labels prefixed `OC_` (e.g., `OC_SECTION`) are now always hidden from the UnifiedContentFeed, preventing structural index entries from cluttering the annotation feed (`frontend/src/components/knowledge_base/document/unified_feed/UnifiedContentFeed.tsx`).
- **Improved sidebar tab spacing**: Reduced tab dimensions (76px/88px height, 36px/44px width), gap (4px), padding, icon size, and font size to prevent overlap when additional tabs are present (`frontend/src/components/knowledge_base/document/styled/SidebarTabs.tsx`).
- **Softened PDF annotation bounding boxes to diffuse highlighter-pen aesthetic**: Replaced hard-edged borders on annotation boundaries, tokens, and label pills with multi-layer box-shadow glows. Boundaries use a three-layer shadow (outer, mid, inset) that feathers into the page. Tokens use a single-layer soft blur. Approved/rejected states pulse with matching diffuse glows instead of solid borders. Extracted shared `computeAnnotationBoxShadow` utility (`frontend/src/utils/colorUtils.ts`) to eliminate duplicated shadow logic between `SelectionBoundary` and `ResultBoundary`. Added named constants for all shadow radii, opacity levels, border-radius tiers, and status colors (`frontend/src/assets/configurations/constants.ts`). Removed dead code: `getBorderWidthFromBounds`, unused `$border` prop, unused `$isSelected` prop. Fixed `pulseMaroon` animation color mismatch (was `rgba(180, 40, 40)`, now matches static rejected state).
- **Reorganized upload documentation into dedicated `docs/upload_methods/` section**: Consolidated scattered upload-related docs into 8 user-facing reference pages covering single upload, bulk ZIP import, corpus export/import, annotated document import, worker uploads, supported formats, and annotation side effects. Simplified `docs/walkthrough/step-1-add-documents.md` and `docs/walkthrough/advanced/export-import-corpuses.md` to reference the new guides. Trimmed `docs/architecture/bulk-import.md` to focus on internal implementation. Removed obsolete `docs/features/zip_import_with_folders_design.md` design doc (feature is fully implemented).

### Fixed

- **`skip_pipeline=True` silently ignored `meta.csv` metadata** (Closes #1131): Documents created via the `skip_pipeline` path in `import_zip_with_folder_structure` now have their title, description, custom_meta, and is_public fields applied from `meta.csv` and task-level parameters, matching the behavior of the normal pipeline path (`opencontractserver/tasks/import_tasks.py`).
- **Incomplete assertion in `test_malformed_labels_json_records_error`** (Closes #1131): The test now also asserts that `annotation_sidecars_errored` is incremented when labels.json is malformed and a sidecar contains annotations (`opencontractserver/tests/test_sidecar_import.py`).

### Added

- **Test for `skip_pipeline=True` with no `labels.json`** (Closes #1131): New test `test_skip_pipeline_without_labels_json` verifies that pipeline-skipped documents are created successfully even when no labels file is present, and that annotation import errors are recorded properly (`opencontractserver/tests/test_sidecar_import.py`).
- **Test for `skip_pipeline=True` with `meta.csv` metadata** (Closes #1131): New test `test_skip_pipeline_applies_metadata_from_csv` verifies that pipeline-skipped documents correctly receive title and description overrides from `meta.csv`.
- **Per-sidecar JSON size limit** (Closes #1131): Added `ZIP_MAX_SIDECAR_SIZE_BYTES` constant (10 MB default) in `opencontractserver/constants/zip_import.py`. Sidecars exceeding this limit are skipped with an error, preventing oversized JSON from consuming excessive memory during import (`opencontractserver/tasks/import_tasks.py`).

## [Unreleased] - 2026-03-18

### Added

- **`COMMUNITY_STATS_CACHE_TTL` env var for community stats caching** (`config/graphql/social_queries.py`, `opencontractserver/constants/community_stats.py`): The `communityStats` GraphQL resolver now caches results using Django's cache framework with a configurable TTL (default 1 hour). This avoids re-running 7+ COUNT queries on every landing page load. Cache is keyed by user type (anonymous vs authenticated) and optional corpus scope. Also includes an N+1 fix using `in_bulk()` for badge distribution and a subquery optimization in `resolve_corpus_categories`.

### Changed

- **Refactored FullScreenModal to use native `size="fullscreen"` variant** (Closes #1073): Replaced the `createGlobalStyle` workaround in `frontend/src/components/knowledge_base/document/LayoutComponents.tsx` with the native `size="fullscreen"` prop from `@os-legal/ui`, which handles viewport positioning, sizing, border-radius removal, and overlay padding natively. Only minimal body styling overrides (background color, padding, and overflow) remain.
- **Added `overlayClassName` to CorpusModal** (`frontend/src/components/corpuses/CorpusModal.tsx`): Uses the new `overlayClassName` prop to properly apply the `.corpus-modal-overlay` styles to the portal-rendered overlay element, fixing mobile overlay styles that were previously unreachable.
- **Complete removal of Semantic UI React dependency**: Migrated all 19 remaining files that imported from `semantic-ui-react` to use `@os-legal/ui` equivalents and native HTML elements. Component mappings: `Label` → `Chip`, `Modal/ModalHeader/ModalContent/ModalActions` → `Modal/ModalHeader/ModalBody/ModalFooter`, `Button` → `Button`, `Confirm` → custom confirm dialog via `Modal`, `Loader/Dimmer` → `Spinner`, `Dropdown` → `Dropdown`, `Icon` → lucide-react icons, `Segment/Card/Form` → styled HTML elements. Removed `semantic-ui-css` and `semantic-ui-react` from `package.json`, deleted the semantic-ui CSS import from `App.tsx` and `playwright/index.tsx`, and removed the orphaned `semantic.css` asset file. Also replaced the `SemanticICONS` type with `string` in `graphql/mutations.ts`.

### Fixed

- **Undefined ordering in async note retrieval** (Closes #1107): Added missing `.order_by("created")` to `aget_notes_for_document_corpus` in `opencontractserver/llms/tools/core_tools.py`, matching the sync counterpart `get_notes_for_document_corpus` which already had deterministic ordering.
- **Tool name breaking change from async migration** (Closes #1107): `create_document_tools()` in `opencontractserver/llms/tools/tool_factory.py` derived tool names from `func.__name__`, causing names to silently change from e.g. `load_document_md_summary` to `aload_document_md_summary` when the registry switched to async functions. Added explicit `name=` parameters to all `CoreTool.from_function()` calls to preserve the original tool names.
- **Test username typo**: Fixed "toklenuser" → "tokenuser" in `opencontractserver/tests/test_agent_search_tools.py`.
- **Misleading CleanViewContainer comment** (`frontend/src/views/Corpuses.tsx`): Corrected the styled-component comment that claimed height constraints were "intentionally removed" when `height: 100%`, `min-height: 0`, and `overflow: hidden` were still present. The comment now accurately documents the height model and explains that only `max-height: 100dvh` was removed. Closes #1044 (items 1 & 2).

### Added

- **Annotation sidecar import for bulk zip upload** (`opencontractserver/tasks/import_tasks.py`, `opencontractserver/utils/zip_security.py`): The `ImportZipToCorpus` mutation now supports importing pre-calculated annotations alongside source documents. When a document file (e.g. `contracts/master.pdf`) has a co-located `.json` sidecar (e.g. `contracts/master.json`) conforming to `OpenContractDocExport`, annotations and intra-document relationships are imported from the sidecar instead of running the parser pipeline. A root-level `labels.json` file provides label definitions. Documents without sidecars continue through the normal pipeline. This enables mixed-mode imports where some documents have pre-computed annotations and others are parsed fresh.
- **Test coverage for Focus/Power mode toggle** (`frontend/tests/CorpusHome.ct.tsx`, `frontend/tests/CorpusHomeTestWrapper.tsx`): Added four Playwright component tests exercising the `onModeToggle` and `isPowerUserMode` props: toggle hidden when callback absent, toggle visible in focus mode, toggle reflects power-user state, and click fires callback. Updated `CorpusHomeTestWrapper` to forward mode-toggle props. Closes #1044 (item 5).
- **Tighten MCP telemetry input validation and test coverage** (Closes #1106):
  - Bound User-Agent storage to 512 characters (`MAX_USER_AGENT_LENGTH`) in both `set_request_context()` and `get_user_agent_from_scope()` to prevent attacker-controlled multi-megabyte values from reaching telemetry backends (`opencontractserver/mcp/telemetry.py`)
  - Empty-string slugs no longer reach telemetry — changed `if slug is not None` to falsy checks (`if slug`) consistently across all sync and async recording functions
  - Documented the duplicate-header-dropping assumption when converting ASGI header lists to dicts in `get_user_agent_from_scope()` and `get_claimed_client_ip_from_scope()`
  - Added async telemetry tests for `arecord_mcp_tool_call`, `arecord_mcp_resource_read`, and `arecord_mcp_request` (`opencontractserver/tests/test_mcp_extended.py`)
  - Added tests for User-Agent truncation and empty-string slug filtering

## [Unreleased] - 2026-03-15

### Breaking Changes

- **Removed `tokensJsons` and `boundingBox` from GraphQL API**: The `tokensJsons` and `boundingBox` fields have been removed from GraphQL annotation queries and mutations (PR #1100). External API consumers and integrations must update to use the `json` field instead. The `json` field contains either v1 (legacy page-keyed format) or v2 (compact format) annotation data. Use `iter_page_annotations()` (Python) or `iterPageAnnotations()` (TypeScript) for format-agnostic access.

### Added

- **Dynamic MIME type support from pipeline components** (Closes #1059): Supported file types are now derived dynamically from registered pipeline components instead of being hardcoded. Changes include:
  - New `get_supported_mime_types()` and `get_allowed_mime_types()` functions in `opencontractserver/pipeline/registry.py` that compute supported file types by intersecting component coverage across parser, embedder, and thumbnailer stages
  - New `supportedMimeTypes` GraphQL query (`config/graphql/pipeline_queries.py`) returning per-file-type support level with stage coverage details
  - New `SupportedMimeTypeType` and `StageCoverageType` GraphQL types (`config/graphql/pipeline_types.py`)
  - Centralized MIME ↔ file type mappings (`MIME_TO_FILE_TYPE`, `FILE_TYPE_TO_MIME`, `FILE_TYPE_LABELS`, `LEGACY_MIME_ALIASES`) in `opencontractserver/pipeline/base/file_types.py`, replacing scattered inline dicts
  - Upload validation in `document_mutations.py`, `folder_service.py`, and `import_tasks.py` now uses the dynamic registry instead of `settings.ALLOWED_DOCUMENT_MIMETYPES`
  - Frontend `FiletypeDefaults` component fetches supported MIME types via GraphQL query instead of using hardcoded `SUPPORTED_MIME_TYPES` constant
  - Warning icon displayed for partially-supported file types (e.g., DOCX which lacks a thumbnailer)
  - `FileTypeEnum` gains `.mimetype` and `.label` properties and supports legacy MIME aliases in `from_mimetype()`
- **Compact PAWLs v2 format for ~67% storage reduction** (PR #1112): New v2 compact format for PAWLs files (per-page token bounding boxes) that reduces storage from ~500+ KB to ~180 KB for a typical 9-page PDF. Changes include:

  - Core encode/decode in `opencontractserver/utils/compact_pawls.py` (Python) and `frontend/src/utils/compactPawls.ts` (TypeScript)
  - Array-based tokens `[x, y, w, h, "text"]` instead of verbose dicts, shortened page keys, implicit page index, coordinate precision normalization
  - Write paths (parser, import, worker upload) auto-compact on save; read paths transparently expand v2 → v1
  - Safety limit: `COMPACT_PAWLS_MAX_TOKENS_PER_PAGE` (100,000) in `opencontractserver/constants/pawls.py` — graceful fallback to v1 when exceeded
  - Comprehensive Python (5 test classes) and TypeScript (8 tests) unit tests including roundtrip and image token coverage

- **Compact annotation JSON v2 format for ~75% storage reduction** (PR #1100): New compact v2 format for annotation JSON payloads that range-encodes consecutive token indices, compacts bounds to arrays, and drops redundant `pageIndex` and `rawText` from per-page data. Changes include:
  - Core encode/decode in `opencontractserver/annotations/compact_json.py` (Python) and `frontend/src/utils/compactAnnotationJson.ts` (TypeScript)
  - Safety limits: `COMPACT_JSON_MAX_RANGE_SPAN` (10,000) and `COMPACT_JSON_MAX_TOTAL_TOKENS` (50,000) in `opencontractserver/constants/annotations.py`
  - `Annotation.save()` auto-compacts v1 → v2 on write (lazy migration) with exception guard for malformed legacy data
  - Migration `0066` removes redundant `tokens_jsons` and `bounding_box` columns (includes `RunPython` backfill step to preserve any legacy data before column removal — **irreversible migration**)
  - 60+ Python and 48+ TypeScript unit tests for the codec
  - All GraphQL queries/mutations updated to use `json` field instead of `tokensJsons` and `boundingBox`
  - Format-agnostic accessor layer: `iter_page_annotations()`, `offset_annotation_json()`, `has_any_tokens()` (Python) and `iterPageAnnotations()`, `hasAnyTokens()` (TypeScript) — all production code migrated off `expand_annotation_json()`
  - `ServerTokenAnnotation` constructor now accepts both v1 and v2 formats, normalizing internally

### Added

- **Document index feature (within-document TOC)**: Hierarchical, navigable section index for long documents with markdown descriptions. Changes include:
  - New `long_description` (TextField, nullable) on the Annotation model for markdown section summaries (`opencontractserver/annotations/models.py`)
  - Database migration `0066_annotation_long_description`
  - `OC_SECTION` label constant (`opencontractserver/constants/annotations.py`) — `OC_` namespace for platform-generated labels
  - New `create_document_index` agent tool for building hierarchical indexes from exact string matches (`opencontractserver/llms/tools/core_tools.py`)
  - Tool registered in tool registry with approval gate (`opencontractserver/llms/tools/tool_registry.py`)
  - `long_description` exposed in GraphQL AnnotationType (auto-resolved), UpdateAnnotation mutation, and AnnotationSerializer
  - Export/import extended: `long_description` in `OpenContractsAnnotationPythonType`, import reads it, export includes it
  - New `DocumentAnnotationIndex` frontend component with tree rendering, expandable markdown descriptions, page badges, and filter support (`frontend/src/components/corpuses/DocumentAnnotationIndex.tsx`)
  - `DocumentTableOfContents` now renders annotation indexes under each document node; single-doc corpora skip the document header and show sections directly
  - New `GET_DOCUMENT_ANNOTATION_INDEX` GraphQL query and TypeScript types (`frontend/src/graphql/queries.ts`)
  - Frontend constants: `DOCUMENT_ANNOTATION_INDEX_LIMIT`, `OC_SECTION_LABEL` (`frontend/src/assets/configurations/constants.ts`)
- **First-class DOCX document support via Docxodus pipeline**: Added a complete parallel ingestion pipeline and rendering tree for DOCX files, bringing Word document support alongside existing PDF and TXT pipelines. Changes include:
  - **Backend: Docxodus microservice** (`docxodus-service/`): .NET 8 minimal API wrapping `OpenContractExporter.Export()` to produce OpenContractDocExport-compatible JSON with structural annotations and character offsets from DOCX files. Multi-stage Docker build exposed on port 8080.
  - **Backend: DocxodusServiceParser** (`opencontractserver/pipeline/parsers/docxodus_parser.py`): REST parser that sends base64-encoded DOCX to the microservice, normalizes camelCase→snake_case response fields, and handles transient/permanent error classification.
  - **Backend: DocxThumbnailGenerator** (`opencontractserver/pipeline/thumbnailers/docx_thumbnailer.py`): Two-tier thumbnail approach — extracts embedded thumbnails from DOCX ZIP archives (`docProps/thumbnail.jpeg`), falling back to text-based thumbnails via XML parsing of `word/document.xml`.
  - **Frontend: DocxAnnotator** (`frontend/src/components/annotator/renderers/docx/DocxAnnotator.tsx`): WASM-powered DOCX renderer using `docxodus` npm package's `convertDocxToHtmlWithExternalAnnotations()` for annotation projection onto native DOCX HTML output. Supports text selection for new annotation creation via `findTextOccurrences()`.
  - **Frontend: DocxAnnotatorWrapper** (`frontend/src/components/annotator/components/wrappers/DocxAnnotatorWrapper.tsx`): State management wrapper mirroring TxtAnnotatorWrapper pattern — manages annotation CRUD, chat sources, text search, and ref registration.
  - **Frontend: DocumentKnowledgeBase integration**: DOCX loading flow (fetches raw bytes + extracted text) and renderer dispatch added to both query handlers.
  - **Frontend utilities**: `isDocxFileType()` in `frontend/src/utils/files.ts`, `DOCX_MIME_TYPE` constant, `docxBytesAtom` / `useDocxBytes()` hook in DocumentAtom, `getDocxBytes()` in cachedRest.
  - **Docker Compose**: `docxodus-parser` service added to `local.yml`, `production.yml`, and `test.yml` with dependency wiring.
  - **Dependencies**: `docxodus@5.5.0` and `dompurify@3.3.3` added to frontend.
  - **Backend tests**: `test_doc_parser_docxodus.py` with parser unit tests (success, timeout, connection error, normalization) and thumbnailer tests (text preview, embedded thumbnail, invalid DOCX handling).
  - **Frontend tests**: `DocxAnnotator.ct.tsx` component test with `docScreenshot` captures.
- **Richer social media link previews for corpus and document links**: Improved the Cloudflare OG worker to generate better social tags. Changes include:
  - Corpus descriptions are now included in OG/Twitter description tags, combined with document count (e.g. "Corpus description — 15 documents")
  - Document-in-corpus links now surface the parent corpus description when the document lacks its own description
  - Added Twitter structured data tags (`twitter:label1`/`twitter:data1`, `twitter:label2`/`twitter:data2`) showing author, document count, corpus name, etc.
  - New `corpus_description` field on `OGDocumentMetadataType` GraphQL type (`config/graphql/og_metadata_types.py`)
  - Backend resolver returns corpus description for document-in-corpus queries (`config/graphql/og_metadata_queries.py`)
  - Fixed pre-existing TypeScript type errors in worker test suite and metadata extraction
- **Creative Commons license support for corpuses**: Corpuses can now have a license applied, choosing from standard Creative Commons licenses (CC BY, CC BY-SA, CC BY-NC, CC BY-NC-SA, CC BY-ND, CC BY-NC-ND, CC0) or a custom license with a URL. Changes include:
  - New `license` (CharField with SPDX identifiers) and `license_link` (URLField) fields on the Corpus model (`opencontractserver/corpuses/models.py`)
  - License constants in `opencontractserver/constants/licenses.py` and `frontend/src/assets/configurations/constants.ts`
  - Database migration `0047_corpus_license_fields` (single migration with URL validation)
  - GraphQL create/update mutations accept `license` and `licenseLink` arguments
  - New `LicenseSelector` frontend component (`frontend/src/components/widgets/CRUD/LicenseSelector.tsx`)
  - CorpusModal updated with License section for create/edit flows
  - CorpusInfoSection displays the selected license in corpus settings

### Fixed

- **Token bounding box drift in PDF viewer**: Fixed progressive misalignment between token bounding boxes (blue outlines) and rendered PDF text. PAWLs token coordinates are extracted by the parser in its own coordinate system, which can differ from PDF.js's viewport coordinate system. The frontend now normalizes token coordinates by comparing PAWLs page dimensions (`page.width`/`page.height`) against the PDF.js viewport dimensions at scale 1, rescaling tokens when they differ. Added `normalizeTokensToPdfViewport()` utility in `frontend/src/utils/transform.tsx` and applied it in both PDF loading paths in `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`.
- **Auth0 SDK blocked by CSP on admin login page** (Closes #1077): The admin login template loads the Auth0 SPA SDK from `cdn.jsdelivr.net`, but the Content-Security-Policy `script-src` directive did not include this origin, causing browsers to block the script. Added `https://cdn.jsdelivr.net` to `script-src` when Auth0 is enabled. (`config/settings/base.py`)
- **PostgreSQL HNSW config warning on startup** (Closes #1074): Fixed `invalid configuration parameter name 'hnsw.iterative_scan'` warning caused by database-level `ALTER DATABASE SET` GUC defaults persisting in `pg_db_role_setting`. The docker-entrypoint-initdb.d phase runs in a temporary postgres without the user's command-line args, so `hnsw.*` GUCs aren't registered when the database-level defaults are later applied on connections. Consolidated shared postgres settings (`shared_preload_libraries`, HNSW, I/O tuning) into `compose/postgres/shared.conf` with a custom entrypoint wrapper that injects them as `-c` flags. Per-environment memory settings remain in compose files. Removed `ALTER DATABASE SET` from `init.sql`. Added migration 0066 to RESET stale database-level defaults. **Upgrade note:** Existing users must rebuild the postgres image (`docker compose build postgres`) to pick up the new entrypoint wrapper and shared config file. (`compose/postgres/shared.conf`, `compose/postgres/docker-entrypoint-wrapper.sh`, `compose/production/postgres/Dockerfile`, `compose/production/postgres/init.sql`, `opencontractserver/annotations/migrations/0066_reset_database_level_hnsw_settings.py`)
- **Single-select Dropdown missing onBlur in MetadataCellEditor** (PR #1076): Two single-select `Dropdown` instances (STRING with choices, CHOICE type) silently dropped the `onBlur` callback during the SUI→@os-legal/ui migration, causing metadata grid cells to never exit edit mode after selection. Initially worked around with `onClose={onBlur}` in v0.1.13; upgraded to proper `onBlur={onBlur}` prop in v0.1.14. (`frontend/src/components/metadata/editors/MetadataCellEditor.tsx`)
- **Dropdown test selectors outdated after @os-legal/ui upgrade**: Updated CSS selectors in `DocumentMetadataGrid.ct.tsx` from `.oc-select-trigger`/`.oc-select-search` to `.oc-dropdown__trigger`/`.oc-dropdown__search-input` to match the v0.1.13 Dropdown component's class names. Changed Enter-key selection to direct `.oc-dropdown__option` click. (`frontend/tests/DocumentMetadataGrid.ct.tsx`)

### Changed

- **Enforced async-only tool registry for LLM agent tools**: Audited and converted the entire tool registry to reject sync functions. Previously `ToolRegistryEntry` carried both `sync_func` and `async_func` fields, `FUNCTION_MAP` registered both versions, and `PydanticAIToolWrapper` had a sync wrapper path that called sync functions without a thread pool (risking `SynchronousOnlyOperation`). Changes include:

  - Removed `sync_func` field from `ToolRegistryEntry` — only `async_func` remains (`opencontractserver/llms/tools/tool_registry.py`)
  - Simplified `FUNCTION_MAP` from 3-tuples `(sync, async, aliases)` to 2-tuples `(async, aliases)`, removing all sync imports from `_populate()`
  - Replaced sync wrapper path in `PydanticAIToolWrapper` with a `TypeError` guard that rejects sync functions at construction time (`opencontractserver/llms/tools/pydantic_ai_tools.py`)
  - Created async versions of `get_note_content_token_length` and `get_partial_note_content` (`aget_note_content_token_length`, `aget_partial_note_content`) which previously only had sync versions that used `Note.objects.get()` (`opencontractserver/llms/tools/core_tools.py`)
  - Updated `create_document_tools()` to use only async functions (`opencontractserver/llms/tools/tool_factory.py`)
  - Updated `__init__.py` exports to async-only (`opencontractserver/llms/tools/__init__.py`)
  - Updated affected tests to use async functions and added new tests for sync rejection

- **SUI Tab migration to @os-legal/ui FilterTabs** (Closes #1022 — tab portion): Removed unused `Tab` and `Menu` imports from `semantic-ui-react` in `frontend/src/views/Corpuses.tsx`. Replaced custom styled `Tab`/`TabContainer` components and custom `SearchInput` in `frontend/src/views/GlobalDiscussions.tsx` with `FilterTabs` and `SearchBox` from `@os-legal/ui`, consistent with the rest of the codebase.

- **Migrated frontend from Semantic UI React to @os-legal/ui design system**: Replaced Modal, Button, Input, and other UI components across the entire frontend with `@os-legal/ui` equivalents. Updated icon system from Semantic UI icons to lucide-react. Introduced `OS_LEGAL_COLORS`, `OS_LEGAL_TYPOGRAPHY`, and `OS_LEGAL_SPACING` design tokens for consistent styling. Upgraded `@os-legal/ui` from 0.1.8 to 0.1.14.
- **Replaced hardcoded hex colors with design system tokens**: Consolidated ~50 color tokens and replaced hardcoded hex color literals across 150+ files with `OS_LEGAL_COLORS` constants from `osLegalStyles.ts`.
- **Extracted shared corpus object collection logic** into `opencontractserver/utils/corpus_collector.py`: New `collect_corpus_objects()` utility and `CorpusObjectCollection` dataclass consolidate duplicated corpus forking/export collection logic (Issue #816)

### Added

- **Dynamic discovery endpoints for crawlers and AI agents**: Replaced static `robots.txt`, `llms.txt`, and `llms-full.txt` files with Django views that dynamically generate content with live data from the database. New endpoints:
  - `robots.txt`: Includes explicit `Allow` directives for AI crawler user-agents (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, Bytespider, cohere-ai) and a proper `Sitemap:` reference to `sitemap.xml`
  - `llms.txt` / `llms-full.txt`: Now auto-populate an "Available Collections" section listing all public corpuses with titles, slugs, document counts, and descriptions. Hostnames are resolved from the request instead of using placeholder text. Links use proper inline Markdown format per the llmstxt.org spec
  - `sitemap.xml`: New XML sitemap listing homepage, public corpuses, their documents (via DocumentPath), and discovery endpoints
  - `.well-known/mcp.json`: New MCP server discovery endpoint listing the global MCP server and per-corpus scoped servers
    (`opencontractserver/discovery/views.py`, `opencontractserver/discovery/urls.py`, `config/urls.py`)
- **Traefik routing for discovery endpoints**: Updated production and CI Traefik configs to route `/robots.txt`, `/llms.txt`, `/llms-full.txt`, `/sitemap.xml`, and `/.well-known/*` to Django instead of the frontend nginx container (`compose/production/traefik/traefik.yml`, `compose/production/traefik/traefik-ci.yml`)
- **MCP discovery link in HTML head**: Added `<link rel="alternate" type="application/json" href="/.well-known/mcp.json">` to `frontend/index.html` for agent discovery
- **Comprehensive test suite** for all five discovery endpoints covering content types, spec conformance, public/private corpus filtering, hostname resolution, and edge cases (`opencontractserver/discovery/tests/test_discovery_views.py`)

### Removed

- **Unused `django-crispy-forms` and `crispy-bootstrap5` dependencies**: These were cookiecutter-django boilerplate never used by the project (React frontend uses its own form components). Removed packages from `requirements/base.txt`, `INSTALLED_APPS`, and `CRISPY_*` settings from `config/settings/base.py`.

### Fixed

- **Duplicate enabled-field logic**: `ComponentLibrary` now reads the backend-computed `component.enabled` field instead of recalculating enablement from the `enabledComponents` list, eliminating divergent sources of truth (issue #1036 item 1). (`frontend/src/components/admin/system_settings/ComponentLibrary.tsx`)
- **Silent toggle failure**: Toggle clicks while components are loading now show a toast warning instead of silently failing (issue #1036 item 2). (`frontend/src/components/admin/SystemSettings.tsx`)
- **MIME-type fallback flaw**: `FiletypeDefaults` now falls back to the full MIME string when the short-label lookup misses, preventing all availability checks from failing for unmapped types (issue #1036 item 3). (`frontend/src/components/admin/system_settings/FiletypeDefaults.tsx`)
- **Unused postProcessors fetch**: Removed `postProcessors` from the `GET_PIPELINE_COMPONENTS` GraphQL query since no frontend component consumes it (issue #1036 item 4). (`frontend/src/components/admin/system_settings/graphql.ts`)
- **Potential duplicate class names**: Toggle handler now deduplicates component paths when transitioning from all-enabled to an explicit list (issue #1036 item 5). (`frontend/src/components/admin/SystemSettings.tsx`)
- **Replicated empty-list-as-all-enabled semantics**: Extracted `isComponentEnabled` and `isComponentAvailable` into a shared `utils.ts` module, consolidating the repeated logic into one place (issue #1036 item 6). (`frontend/src/components/admin/system_settings/utils.ts`)
- **Implicit test dependency**: Added explicit class-level constants for test component paths and expanded `test_pipeline_components_query_non_superuser_filters_configured` to verify all component stages, not just parsers (issue #1036 item 8). (`opencontractserver/tests/test_pipeline_component_queries.py`)
- **Corpus preferred_embedder not set when default is empty** (pre-existing): `Corpus.save()` used `if not self.preferred_embedder and default_embedder` which skipped assignment when `get_default_embedder_path()` returned `""`. Changed to `if self.preferred_embedder is None` so the field is always populated consistently. (`opencontractserver/corpuses/models.py:426`)
- **Stale postProcessors in PipelineComponentsType**: Removed unused `postProcessors` field from the `PipelineComponentsType` TypeScript type to match the system settings GQL query. (`frontend/src/types/graphql-api.ts`)

### Added

#### Unified Rate Limiting for WebSocket and MCP (Closes #730, #745)

- Replaced `django-ratelimit` with a custom protocol-agnostic rate limiting engine (`config/ratelimit/`) supporting GraphQL, WebSocket, MCP, and Django views through a single shared infrastructure
- **Engine** (`config/ratelimit/engine.py`): Fixed-window counter algorithm using Django cache (Redis in production), with sync `is_rate_limited()` and async `ais_rate_limited()` APIs
- **Identity resolution** (`config/ratelimit/keys.py`): Unified IP extraction from both `HttpRequest` (GraphQL/views) and ASGI scope (WebSocket/MCP), plus key building with `user_or_ip`, `ip`, and `user` strategies
- **Rate categories** (`config/ratelimit/rates.py`): `RateLimits` singleton with 17 categories including new `WS_CONNECT` (10/m), `WS_HEARTBEAT` (120/m), and `MCP_GLOBAL` (100/m)
- **WebSocket rate limiting**: All 3 consumers (`UnifiedAgentConversationConsumer`, `NotificationConsumer`, `ThreadUpdatesConsumer`) now enforce connection-rate and per-message limits. Rate-limited messages receive a JSON `RATE_LIMITED` error; the connection stays open
- **MCP rate limiting**: Two-layer check — global cap (`MCP_GLOBAL`) plus per-tool limits mapped to existing categories (e.g., `search_corpus` → `READ_HEAVY`). ASGI scope threaded into tool handlers via `ContextVar`
- **Django view adapter** (`view_ratelimit`): Drop-in replacement for `django_ratelimit.decorators.ratelimit`, used in admin login views
- **Backward-compatible re-exports**: `config/graphql/ratelimits.py` remains a valid import path for all existing GraphQL files
- Comprehensive test suite (`opencontractserver/tests/test_unified_rate_limiting.py`) covering engine, keys, rates, and all protocol adapters

- Backend mutation test for the all-enabled-to-explicit toggle transition, verifying the mutation succeeds and the query reflects the change (issue #1036 item 7). (`opencontractserver/tests/test_pipeline_settings.py`)

### Changed

- **`add_annotations_from_exact_strings` API simplified to single-document** (breaking): Replaced per-item `(label, text, doc_id, corpus_id)` tuples with a flat `document_id`/`corpus_id` pair plus `AnnotationItem` TypedDict items containing only `label_text` and `exact_string`. Callers that previously annotated multiple documents in one call must now make separate calls per document. (`opencontractserver/llms/tools/core_tools.py`)

### Added

#### Action Library (Corpus Action Templates)

- **CorpusActionTemplate model** for reusable, agent-based action definitions that users can browse and add to individual corpuses (`opencontractserver/corpuses/models.py`)
- **5 default action templates** seeded via data migration: Document Description Updater, Corpus Description Updater, Document Summary Generator, Key Terms Annotator, Document Notes Generator — each with a dedicated `AgentConfiguration` and curated tool set (`opencontractserver/agents/migrations/0010_create_default_action_templates.py`)
- **Action Library UI**: "Add from Library" picker in Corpus Settings lets users browse available templates and add them to a corpus on demand. New corpora start empty — no auto-cloning (`frontend/src/components/corpuses/settings/CorpusActionsSection.tsx`)
- **`addTemplateToCorpus` mutation**: Clones a template into a `CorpusAction` for a given corpus with duplicate prevention (`config/graphql/corpus_mutations.py`)
- **`source_template` FK on CorpusAction**: Links cloned actions back to their source template for provenance tracking (`opencontractserver/corpuses/models.py`)
- **GraphQL query `corpusActionTemplates`**: Read-only query exposing available templates with optional `isActive` filter (`config/graphql/action_queries.py`)
- **`sourceTemplate` field on CorpusActionType**: Exposes template provenance in the existing corpus actions GraphQL type (`config/graphql/agent_types.py`)
- **`seed_action_templates` management command**: Idempotent command for seeding default templates on fresh databases (`opencontractserver/corpuses/management/commands/seed_action_templates.py`)

#### Optimize Vector Search and Index Scalability for Million-Scale Corpora

- **HNSW indexes on all Embedding vector columns** (384–4096 dimensions): Approximate nearest neighbor search reduces vector queries from O(n) sequential scan to O(log n). Created via `AddIndexConcurrently` to avoid table locks during index creation (`opencontractserver/annotations/models.py`, `opencontractserver/annotations/migrations/0063_add_hnsw_indexes_and_search_vector.py`)
- **Eliminated Python-side materialization** in `VectorSearchViaEmbeddingMixin.search_by_embedding()`: Previously materialized ALL matching rows into Python, sorted, and sliced. Now uses PostgreSQL `ORDER BY + LIMIT` so only top-k rows cross the wire. The unique constraint from migration 0059 guarantees no JOIN duplicates, removing the need for `DISTINCT ON` (`opencontractserver/shared/mixins.py`)
- **PostgreSQL full-text search** on Annotation: Added `search_vector` (`SearchVectorField`) with GIN index and a database trigger that auto-populates tsvector from `raw_text` on INSERT/UPDATE. Replaces `LIKE '%term%'` (`icontains`) with indexed tsvector matching for 100x+ faster text search at scale (`opencontractserver/annotations/models.py`)
- **Hybrid search with Reciprocal Rank Fusion (RRF)**: New `CoreAnnotationVectorStore.hybrid_search()` method runs vector similarity and full-text search in parallel, then fuses results using RRF (k=60). Semantic search GraphQL resolver now uses hybrid search for improved result quality (`opencontractserver/llms/vector_stores/core_vector_stores.py`, `config/graphql/search_queries.py`)
- **RRF utility function** (`opencontractserver/utils/search.py`): Generic `reciprocal_rank_fusion()` that merges any number of ranked lists
- **Search constants** (`opencontractserver/constants/search.py`): `HNSW_M`, `HNSW_EF_CONSTRUCTION`, `RRF_K`, `HYBRID_SEARCH_OVERSAMPLE_FACTOR`, `FTS_CONFIG`
- **pgvector PostgreSQL extension upgraded to 0.8.0** (installed via Docker, not the Python package): Enables iterative index scans for filtered ANN queries (prevents result loss when combining vector search with `WHERE` clauses like `embedder_path` filtering). Database default set via `ALTER DATABASE ... SET hnsw.iterative_scan = 'relaxed_order'` in `init.sql` (`compose/production/postgres/Dockerfile`, `compose/production/postgres/init.sql`)
- **PostgreSQL tuning for vector workloads**: Added `shared_buffers`, `maintenance_work_mem`, `effective_cache_size`, `work_mem`, `max_parallel_maintenance_workers`, `random_page_cost`, and `effective_io_concurrency` to all Docker Compose files. Added `shm_size` for parallel HNSW index builds (`local.yml`, `production.yml`, `test.yml`)
- **Pinned pgvector Python package** to `>=0.4.0` for `HnswIndex` support (`requirements/base.txt`)
- **Expanded valid embedding dimensions** in `CoreAnnotationVectorStore`: Search methods now accept 1024 and 2048 dimensions in addition to existing 384, 768, 1536, 3072, 4096
- **Added 2048-dimension support** to conversation and message vector search methods (`opencontractserver/conversations/models.py`)
- **GraphQL annotation mention search** now uses full-text search via `SearchQuery` on GIN-indexed `search_vector` instead of `raw_text__icontains` (`config/graphql/search_queries.py`)

- **IconPicker Lucide Rebuild (SUI Migration Step 3)**: Replaced the Semantic UI icon catalog with a curated Lucide icon catalog (~440 icons across 16 categories). New `IconPickerModal` component displays Lucide icons in a searchable, category-filtered grid with live preview. New `IconDropdown` component provides a compact trigger that opens the modal. Updated `iconCompat.ts` with dynamic fallback resolution so `resolveIcon()` supports all Lucide icons via `kebabToPascal` conversion against the full `lucide-react` export. Legacy SUI icon names remain backward-compatible through the existing `SEMANTIC_TO_LUCIDE` mapping. Includes 16 Playwright component tests and automated documentation screenshots. (`frontend/src/components/widgets/icon-picker/icons.ts`, `frontend/src/components/widgets/icon-picker/IconPickerModal.tsx`, `frontend/src/components/widgets/icon-picker/IconDropdown.tsx`, `frontend/src/utils/iconCompat.ts`)

### Changed

- Removed `django-ratelimit` dependency from `requirements/base.txt` — all rate limiting now handled by `config.ratelimit`
- `config/admin_auth/views.py`: Replaced `django_ratelimit.decorators.ratelimit` with `config.ratelimit.decorators.view_ratelimit`
- `opencontractserver/mcp/server.py`: Replaced custom `RateLimiter` class with unified `check_mcp_rate_limit`; added `ContextVar` for ASGI scope propagation
- `opencontractserver/mcp/permissions.py`: Removed `RateLimiter` class (moved to shared package)
- `opencontractserver/mcp/telemetry.py`: `get_client_ip_from_scope` now delegates to shared `config.ratelimit.keys`
- Removed `X-RateLimit-*` response headers that `django-ratelimit` previously set on GraphQL responses — the new engine communicates rate limit state through `RateLimitExceeded` errors (GraphQL), `RATE_LIMITED` WebSocket frames, or HTTP 429 status (views)
- Removed `RATELIMIT_ENABLE` dead-code setting from `config/settings/ratelimit.py` — the engine only checks `RATELIMIT_DISABLE`

- **Pipeline Component Management Redesign**: Separated component management from filetype default assignment in the Pipeline Configuration UI. New `enabled_components` JSON field on `PipelineSettings` tracks which components are available. Frontend splits into two sections: ComponentLibrary (flat filterable list with enable/disable toggles, search, stage filter chips) and FiletypeDefaults (MIME type rows with parser/embedder/thumbnailer dropdowns). Removed old stage-centric layout with MIME type filter buttons. (`opencontractserver/documents/models.py`, `config/graphql/pipeline_types.py`, `config/graphql/pipeline_queries.py`, `config/graphql/pipeline_settings_mutations.py`, `frontend/src/components/admin/SystemSettings.tsx`, `frontend/src/components/admin/system_settings/ComponentLibrary.tsx`, `frontend/src/components/admin/system_settings/FiletypeDefaults.tsx`)
- **Clean corpus landing page with Power User mode toggle**: Default corpus view is now a full-page landing without sidebar navigation, providing a cleaner experience for anonymous browsers and casual users. Users with edit permissions see a "Power User" toggle (`?mode=power` URL param) to access the full sidebar+tabs layout (`frontend/src/views/Corpuses.tsx`)
- **Recent discussions feed on corpus landing page**: New `RecentDiscussions` component shows 2-3 latest discussion threads below the "View Details" button, making community activity visible even to anonymous users (`frontend/src/components/corpuses/CorpusHome/RecentDiscussions.tsx`)
- **Inline discussions view**: New `?view=discussions` URL state enables viewing the full discussion thread list and thread detail directly from the corpus home, without switching to the Discussions tab (`frontend/src/components/corpuses/CorpusHome/CorpusDiscussionsInlineView.tsx`)
- `updateModeParam` navigation utility for managing the `?mode=` URL parameter (`frontend/src/utils/navigationUtils.ts`)
- Extended `CorpusDetailViewType` to include `"discussions"` alongside `"landing"` and `"details"` (`frontend/src/graphql/cache.ts`)
- Screenshot tests for new landing view states: clean view, discussion feed, empty discussions, and power user mode (`frontend/tests/CorpusHome.ct.tsx`, `frontend/tests/CorpusTabs.ct.tsx`)

#### Security Headers Middleware (CSP, Referrer-Policy, Permissions-Policy)

- Added `SecurityHeadersMiddleware` in `config/middleware.py` that attaches `Content-Security-Policy` and `Permissions-Policy` headers to every HTTP response; Referrer-Policy is handled by Django's built-in `SecurityMiddleware` via `SECURE_REFERRER_POLICY`
- Middleware positioned after `SecurityMiddleware` so it is the final authority on security headers in the response phase
- CSP directives configured via `SECURE_CSP_DIRECTIVES` dict in `config/settings/base.py` — covers `default-src`, `script-src` (with `blob:` for PDF.js worker fallback), `style-src`, `img-src`, `font-src`, `connect-src`, `worker-src`, `object-src`, `frame-ancestors`, `base-uri`, and `form-action`
- When `USE_AUTH0=True`, the Auth0 tenant domain is automatically added to `connect-src` and `script-src` with input sanitization via `validate_csp_domain()` (rejects domains containing spaces or semicolons to prevent CSP injection)
- Auth0 domain validation extracted into `config.middleware.validate_csp_domain()` for independent testability
- Permissions-Policy disables `camera`, `microphone`, `geolocation`, `payment`, `usb`, `magnetometer`, `gyroscope`, and `accelerometer` via `SECURE_PERMISSIONS_POLICY`
- Header values are pre-built once at middleware init for zero per-request overhead
- Local dev CSP (`config/settings/local.py`) scopes WebSocket `connect-src` to `localhost` instead of bare `wss:`/`ws:` scheme sources, with documented Auth0 domain interaction
- Tests in `opencontractserver/tests/test_security_headers_middleware.py` — unit tests, integration test against `/api/health/`, and `validate_csp_domain()` tests

#### Expand Corpus Import Test Coverage (Closes #999)

- Rewrote `test_corpus_import.py` with proper `TransactionTestCase` base class (previously `ImportCorpusTestCase` with no parent, never discovered by test runners)
- Fixed `FieldFile.save()` call signature in test setup helper (was passing `ContentFile` as `name` instead of `(name, content)`)
- Grouped read-only assertions into 2 test methods using `subTest` to reduce import pipeline executions from 15 to 5
- **Label integrity**: Validates all 107 labels (79 text + 28 doc) with correct color, icon, description, type, and labelset membership
- **Annotation validation**: Verifies raw text, page numbers, bounding box coordinates, token references, and label associations for all 6 annotations
- **Relationship verification**: Tests `import_relationships()` with single and multiple source/target annotations, plus structural flag preservation

- Expanded corpus forking test suite with field-level data integrity checks (Closes #998)

### Fixed

#### Missing embedding dimensions in VALID_EMBEDDING_DIMS

- `VALID_EMBEDDING_DIMS` was missing dimensions 1024, 2048, and 4096, causing validation failures for embedders that produce these common dimensions (e.g., some OpenAI and Cohere models). Added missing entries to `VALID_EMBEDDING_DIMS` and `DIM_TO_FIELD_MAP` in `opencontractserver/constants/search.py`.

#### Fix My Documents Corpus Not Navigable Due to Missing Slugs

- **Root cause**: Migration 0038 created personal corpuses using historical models which bypass `Corpus.save()` slug auto-generation, leaving `slug=NULL`. The frontend requires both `corpus.slug` and `creator.slug` to build navigation URLs (`/c/<user>/<corpus>`), so clicking "My Documents" logged "Cannot navigate to corpus without slugs" and did nothing.
- **Fix (model)**: `Corpus.get_or_create_personal_corpus()` now detects when a returned corpus lacks a slug and triggers `save()` to backfill it on access (`opencontractserver/corpuses/models.py:518-521`).
- **Fix (migration)**: Added data migration `0043_backfill_corpus_slugs` that backfills slugs for all existing corpuses and users missing them (`opencontractserver/corpuses/migrations/0043_backfill_corpus_slugs.py`).

#### Skip redundant document re-parsing during corpus import

- Set `processing_started` on standalone documents created via `create_document_from_export_data()` to prevent the post_save signal from triggering `ingest_doc` (`opencontractserver/utils/importing.py:323`)
- Imported documents already have PAWLS data from the export; re-parsing wasted resources and failed in environments without a parser service

#### Tighten JSON Field Validation for Malformed Input (Closes #1001)

- **Root cause**: `CustomJSONFieldFormTests.TestForm` used `NullableJSONField()` (a model field) instead of `UTF8JSONFormField` (a form field). Django's `Form` metaclass silently ignores model fields, so the form had zero fields and `is_valid()` always returned `True` — masking the fact that malformed JSON was never validated.
- **Fix**: Changed `TestForm.json_field` to `UTF8JSONFormField(required=False)` so form validation actually runs through Django's `forms.JSONField.to_python()`, which raises `ValidationError` on `json.JSONDecodeError` (`opencontractserver/tests/test_custom_fields.py:76`).
- **Re-enabled**: `test_form_with_invalid_json` now asserts that `'not json'` is correctly rejected (`opencontractserver/tests/test_custom_fields.py:90-92`).
- **Added**: `test_formfield_rejects_invalid_json` and `test_formfield_accepts_valid_json` integration tests on `NullableJSONFieldTests` to verify the model field's `formfield()` method produces a form field that properly validates JSON (`opencontractserver/tests/test_custom_fields.py:63-71`).

### Changed

#### Deployment: migration 0063 backfill may need a maintenance window for large deployments

- Operators with >1M annotations should consider running the search_vector backfill (Phase 4 of migration 0063) during a maintenance window. The migration now emits `RAISE NOTICE` progress messages so operators can monitor backfill progress in PostgreSQL logs.
- Production `maintenance_work_mem` reduced from `2GB` to `512MB`. The higher value is only needed during the initial HNSW index build; operators should temporarily increase it for that migration, then revert (`production.yml`).
- `hnsw.ef_search` increased from `40` to `64` to match `ef_construction`, improving recall for legal document search (`compose/production/postgres/init.sql`, migration 0063).

#### Annotation text search now uses PostgreSQL full-text search with English stemming

- The annotation mention search (`resolve_search_annotations_for_mention`) and the semantic search resolver now use `SearchQuery` on a GIN-indexed `search_vector` column instead of `raw_text__icontains`. This means text queries now match English-stemmed forms (e.g., searching "contract" also matches "contracting" and "contracted") rather than requiring exact substring matches. This is a semantic behavior change for users accustomed to exact substring matching on `raw_text`. See `config/graphql/search_queries.py`.

#### Triage and Clean Up TODO/FIXME Comments (Closes #971)

- Removed 62 TODO/FIXME/HACK annotations across 43 backend and frontend files
- Replaced vague TODOs with `NOTE(deferred):` comments explaining deferral reasoning
- Deleted stale comments referencing non-existent files, already-implemented features, and empty test stubs
- Fixed typo: "whould" → "should" in `test_permissioning.py`
- Removed `console.log` debug statement in `ModernDocumentItem.tsx`
- Deleted empty test stub file `test_doc_analysis_tasks.py`
- Consolidated redundant `logger.debug()` calls in `utils/files.py`

#### Extract Magic Numbers to Constants Files (Closes #970)

- Replaced hardcoded upload limit, truncation lengths, DPI, and title limits with named constants in `constants/document_processing.py` and `constants/llm_tools.py`
- Reused existing `MAX_PROCESSING_ERROR_LENGTH`/`MAX_PROCESSING_TRACEBACK_LENGTH` in `corpuses/models.py`

#### GraphQL Module Modularization (Closes #972)

- Split `graphene_types.py` (3,717→107 lines), `mutations.py` (6,229→405 lines), `queries.py` (4,408→54 lines) into domain-specific files
- Full backward compatibility via re-exports; no logic changes

#### Consolidate Duplicate String Truncation Utilities (Closes #976)

- Added `truncate()` helper in `opencontractserver/utils/text.py` and named constants in `constants/truncation.py`
- Replaced inline truncation across `core_tools.py`, `doc_tasks.py`, and `corpuses/models.py`

#### Break Up Large Frontend Components (Closes #977)

- Split 5 large components: StyledContainers (2,115→12), SystemSettings (2,616→1,108), CorpusChat (2,347→1,346), DocumentKnowledgeBase (3,363→2,322), ChatTray (2,215→1,772)
- Extracted shared chat WebSocket types into canonical `chat/types.ts`; renamed duplicate ConversationListView components; replaced `any` types with explicit typed properties

### Removed

#### Deprecated Semantic UI React Components and Icon Picker (PR #1009)

- Deleted 103 files (~20,900 lines) of deprecated frontend components, hooks, and tests that had been fully replaced by OS Legal styled equivalents
- Removed icon picker widget (`IconSelector.tsx`, `IconDropdown.tsx`, `IconPickerModal.tsx`, `icons.ts`, `styles.module.css`)
- Removed unused Semantic UI wrapper components: `DocTypeLabelDisplay`, `DocTypeLabels`, `LabelSelector`, `SemanticSidebar`, and related CSS
- Removed deprecated layout components: `AnnotatorSidebar.tsx`, `DropdownActionButton.tsx`, `CorpusCards.tsx`, `TreeItemDisplay.tsx`
- Removed dead modals: `SelectCorpusAnalyzerModal.tsx`, `SelectExtractFieldsModal.tsx`, `EditExtractModal.tsx`, `NewEditAnalysisModal.tsx`, `SelectDocumentFieldsetModal.tsx`
- Removed deprecated annotator hooks and display components: `useVisibleRelationships`, `useAnnotationDisplay`, `useAnnotationSelection`, `usePageAnnotations`, `RelationshipList`, `ActionBar`, `AnnotationSummary`, and others
- Removed legacy notification components: `NotificationBell`, `NotificationCenter`, `NotificationDropdown`, `NotificationItem`
- Removed deprecated thread components: `MentionPicker`, `ResourceMentionPicker`, `ModerationControls`, `ModeratorBadge`, `ReputationDisplay`, and associated hooks
- Removed orphaned CSS files: `DocTypeLabelDisplayStyles.css`, `DocTypeLabels.css`, `LabelSelector.css`
- Removed associated test files for deleted components

### Added

#### Replace Mock Data with Real User Query in @mention Dropdown (Closes #1002)

- **useMentionUsers hook** (`frontend/src/components/threads/hooks/useMentionUsers.ts`): Replaced hardcoded mock users with real `SEARCH_USERS_FOR_MENTION` GraphQL query. Added 300ms debounced input to reduce excessive API calls and minimum character threshold (2 chars). Hook now returns `{ users, loading, error }` instead of just `MentionUser[]`.
- **MentionPicker component** (`frontend/src/components/threads/MentionPicker.tsx`): Added loading and error state rendering. Shows "Searching users..." during query execution and "Failed to load users" on errors. Added `loading` and `error` optional props to `MentionPickerProps`.

#### Deep Linking and Context Menu for Text/PDF Annotators (Closes #958)

- Copy Link actions in PDF (`SelectionLayer.tsx`) and TXT (`TxtAnnotator.tsx`) context menus encode selections as `?tb=` deep link URLs
- URL-driven annotation selection from chat sources (`ChatTray.tsx`); delete button for processing documents (`ModernDocumentItem.tsx`)

#### Corpus Export Test Coverage (Closes #997)

- Added `test_exported_document_structure` to validate exported document data structure: top-level keys, PAWLS page schema, annotation structure with bounding boxes and token references, and PDF burn-in validity (`opencontractserver/tests/test_corpus_export.py`)
- Added `test_round_trip_consistency` to compare exported data against original import fixture: document title, content, PAWLS page dimensions and token counts, annotation count, raw text, label names (mapped through label lookups), and bounding box coordinates (`opencontractserver/tests/test_corpus_export.py`)
- Added `test_exported_label_names_match_fixture` to verify exported label name sets match the labels actually used in the import fixture (`opencontractserver/tests/test_corpus_export.py`)
- Loaded import fixture data in setUp for round-trip comparison, replacing the previous TODO placeholder (`opencontractserver/tests/test_corpus_export.py:62`)
- Cleaned up existing tests by removing verbose print statements and TODO comments

#### Expand burn_doc_annotations Test (Closes #1000)

- Added `test_burn_doc_annotations_with_text_labels` to exercise the text-label PDF burning code path with TOKEN_LABEL fixtures and bounding-box annotation data (`opencontractserver/tests/test_doc_tasks.py`)
- Validates output PDF contains highlight annotations with correct subtype, label text, and non-empty base64-encoded content
- Validates `doc_export` JSON contains expected `doc_labels` and `labelled_text` entries
- Renamed existing test to `test_burn_doc_annotations_doc_labels_only` for clarity

#### Test Coverage for Untested Backend Modules (Closes #975)

- Unit tests for feedback, shared utils, constants, types, and MCP extended modules (`opencontractserver/tests/`)

### Fixed

#### Code Review Fixes for Text Block Deep Linking (#958)

- Document resolution via corpus membership (`DocumentPath`) instead of `creator=owner`; simplified default path to return already-resolved doc
- Cross-document source click flash fix; `useClearTextBlockOnInteraction` hook consolidation; clipboard `.catch()` for non-HTTPS; dead code removal

#### Document Version Selector UI Cleanup (Closes #964)

- Removed unused query fields (`versionCount`, `hasVersionHistory`, etc.); added WAI-ARIA keyboard navigation; safe `v?` fallback during load
- Backend validation for invalid version numbers (≤ 0); isCurrent JSDoc; updated test mocks and new keyboard nav tests

#### Rollup Vulnerability (Closes #973)

- Pinned `rollup: "^4.59.0"` via yarn resolutions to fix 3 high-severity path traversal advisories
- **Result**: rollup updated from 4.53.1 to 4.59.0, eliminating all 3 rollup-related audit advisories

### Added

#### Worker Upload Management UI and Documentation (#955)

- **GraphQL queries**: `workerAccounts`, `corpusAccessTokens`, `workerDocumentUploads` resolvers with proper permission checks (superuser-only for accounts, superuser/corpus-creator for tokens and uploads) (`config/graphql/queries.py`)
- **ReactivateWorkerAccount mutation**: Allows superusers to re-enable previously deactivated worker accounts (`config/graphql/worker_mutations.py`)
- **Worker Account management page**: New admin page at `/admin/worker-accounts` for creating, listing, and activating/deactivating worker service accounts (`frontend/src/components/admin/WorkerAccountManagement.tsx`)
- **Worker Access Tokens section in Corpus Settings**: Corpus creators and superusers can view, create, and revoke access tokens scoped to their corpus. Includes one-time key display with copy-to-clipboard (`frontend/src/components/corpuses/settings/WorkerTokensSection.tsx`)
- **Documentation walkthrough**: End-to-end guide covering account creation, token management, document upload (with curl/Python examples), metadata format reference, rate limiting, error handling, and security model (`docs/worker_uploads/walkthrough.md`)
- **Component tests**: Playwright component tests for WorkerAccountManagement with automated documentation screenshots

#### Document Version Selector End-to-End Documentation (Closes #954)

- **User-facing guide**: `docs/features/document_versioning.md` — covers version creation workflow, visual status indicators (gray/blue/orange badges), Version History Panel usage, and Trash folder recovery
- **Documentation screenshots**: Added `docScreenshot` calls to capture five key UI states:
  - `versioning--badge--single-version` — gray badge for documents without history (`frontend/tests/VersionBadge.ct.tsx`)
  - `versioning--badge--latest-version` — blue badge showing version count (`frontend/tests/VersionBadge.ct.tsx`)
  - `versioning--badge--older-version` — orange badge for outdated versions (`frontend/tests/VersionBadge.ct.tsx`)
  - `versioning--history-panel--with-versions` — already captured in `frontend/tests/VersionHistoryPanel.ct.tsx`
  - `versioning--trash-folder--restore-ui` — deleted document recovery interface (`frontend/tests/TrashFolderView.ct.tsx`)

### Changed

#### Worker Upload Permission Expansion (#955)

- `CreateCorpusAccessTokenMutation` and `RevokeCorpusAccessTokenMutation` now allow corpus creators (not just superusers) to manage tokens scoped to their own corpora
- GlobalSettingsPanel refreshed with OS Legal design tokens and lucide-react icons, replacing Semantic UI dependencies

#### Auth0 Refresh Token Migration (#955)

- **DEPLOYMENT NOTE**: `useRefreshTokens: true` is now enabled in the Auth0 SDK configuration (`frontend/src/index.tsx`). Deployments using Auth0 **must** enable "Refresh Token Rotation" in the Auth0 dashboard before deploying this change, or silent authentication will fail for all users.

### Fixed

#### Document Version Structural Annotation Set Inheritance

- **Bug**: When a document was updated with new content (different hash), `import_document()` unconditionally inherited the old version's `structural_annotation_set`. This caused the parser's `_create_structural_annotation_set()` to short-circuit (early return at `pipeline/base/parser.py:299`), leaving freshly-parsed structural annotations orphaned — never migrated into a set.
- **Fix**: `opencontractserver/documents/versioning.py:224-231` — `structural_annotation_set` is now only inherited when the content hash is unchanged. When content changes, the field is set to `None` so the parser creates a fresh `StructuralAnnotationSet` during ingestion.
- **Tests**: `opencontractserver/tests/test_structural_annotation_portability.py` — replaced single test with two: one verifying `None` on changed content, one verifying inheritance on identical content.

### Added

#### Annotation Versioning and Document Version-Aware Deep Linking

- **Version-aware document resolution**: `documentInCorpusBySlugs` GraphQL query now accepts optional `versionNumber` parameter to resolve a specific historical version of a document (`config/graphql/queries.py`)
- **Corpus versions field**: New `corpusVersions(corpusId)` field on `DocumentType` returns all versions of a document in a corpus with version number, document ID, slug, creation date, and current status (`config/graphql/graphene_types.py`)
- **`CorpusVersionInfoType` GraphQL type**: New type for version selector data returned by `corpusVersions` field
- **`?v=N` URL parameter**: Deep links to documents now support a `?v=N` query parameter to view a specific version (e.g., `/d/user/corpus/doc?v=1&ann=123`)
- **`selectedDocVersion` reactive var**: New URL-driven state variable in `frontend/src/graphql/cache.ts` synced bidirectionally via CentralRouteManager Phases 2 and 4
- **CentralRouteManager version support**: Phase 1 passes version to GraphQL resolution, Phase 2 parses `?v=` into reactive var, Phase 4 syncs back to URL (`frontend/src/routing/CentralRouteManager.tsx`)
- **`DocumentVersionSelector` component**: Inline version badge and dropdown in document header that shows available versions and allows switching between them (`frontend/src/components/documents/DocumentVersionSelector.tsx`)
- **Navigation utilities**: `QueryParams` interface and `buildQueryParams` now support `version` field for URL construction (`frontend/src/utils/navigationUtils.ts`)
- **Routing documentation**: Updated `docs/frontend/routing_system.md` with version parameter documentation, examples, and reactive var listing

#### Worker Document Upload System

- **New Django app** `opencontractserver.worker_uploads` — enables external document-processing workers to upload fully ingested, annotated, and embedded documents to a target corpus via REST API
- **Service account model** (`WorkerAccount`): dedicated machine identity with auto-created Django User for permission compatibility. Created via `createWorkerAccount` GraphQL mutation (superuser only)
- **Corpus-scoped access tokens** (`CorpusAccessToken`): cryptographically random 256-bit tokens scoped to a single corpus, with configurable expiry and per-token rate limiting. Created via `createCorpusAccessToken` GraphQL mutation
- **Hashed token storage**: tokens are stored as SHA-256 hashes — plaintext shown only once at creation via `create_token()`. Auth backend hashes incoming keys before DB lookup (`opencontractserver/worker_uploads/models.py`, `auth.py`)
- **DRF authentication backend** (`WorkerTokenAuthentication`): validates `Authorization: WorkerKey <token>` headers, hashes token and checks validity, expiry, and account status (`opencontractserver/worker_uploads/auth.py`)
- **REST upload endpoint** (`POST /api/worker-uploads/documents/`): accepts multipart form data (file + JSON metadata), stages uploads in database, returns 202 Accepted immediately. Status polling via `GET /api/worker-uploads/documents/<id>/` and listing via `GET /api/worker-uploads/documents/list/`
- **Upload format** (`WorkerDocumentUploadMetadataType`): extends V2 export format with pre-computed embeddings (`embedder_path` + document/annotation vectors), target path/folder placement, and inline label definitions for auto-creation (`opencontractserver/types/dicts.py`)
- **Database-backed queue** (`WorkerDocumentUpload`): staging table with PENDING/PROCESSING/COMPLETED/FAILED status tracking, avoids Redis saturation for high-volume uploads (millions of documents)
- **Batch processor task** (`process_pending_uploads`): Celery task on dedicated `worker_uploads` queue using `SELECT ... FOR UPDATE SKIP LOCKED` for concurrent processing without conflicts. Configurable batch size via `WORKER_UPLOAD_BATCH_SIZE` setting. Self-reschedules when more work exists
- **Multi-queue architecture**: worker upload processing runs on dedicated `worker_uploads` Celery queue, preserving capacity on the default queue for regular user operations
- **Pre-computed embedding storage**: workers can include embeddings in upload metadata; stored directly via bulk_create without re-running embedder models. Supports all vector dimensions (384–4096)
- **Corpus creator ownership**: all documents, annotations, and labels created via worker uploads are owned by the corpus creator (not the service account), ensuring correct permission inheritance
- **GraphQL management mutations**: `createWorkerAccount`, `deactivateWorkerAccount`, `createCorpusAccessToken`, `revokeCorpusAccessToken` (all superuser-only) in `config/graphql/worker_mutations.py`
- **Celery task routing**: `CELERY_TASK_ROUTES` canonicalized in one place with guard comment (`config/settings/base.py`)
- **Settings-based Beat schedule**: `CELERY_BEAT_SCHEDULE` for worker upload drain (60s interval), replacing fragile data migration approach
- **File size limit**: `MAX_WORKER_UPLOAD_SIZE_BYTES` setting (default 256 MB) enforced at upload endpoint
- **Filename sanitization**: worker-supplied document titles are sanitized before use as filenames, stripping path traversal characters and null bytes

### Technical Details

- New files: `opencontractserver/worker_uploads/{models,views,auth,serializers,tasks,urls,apps}.py`, `config/graphql/worker_mutations.py`
- Migrations: `0001_initial.py` (models), `0002_setup_beat_schedule.py` (cleanup old DB schedule), `0003_hash_token_keys.py` (SHA-256 token hashing)
- Settings: `WORKER_UPLOAD_BATCH_SIZE` (default 50), `MAX_WORKER_UPLOAD_SIZE_BYTES` (default 256 MB), `CELERY_TASK_ROUTES` for queue isolation, `CELERY_BEAT_SCHEDULE` for periodic drain
- Tests: `opencontractserver/tests/test_worker_uploads.py` covering models, hashed token auth, REST endpoints, file size limits, batch processor, filename sanitization, null corpus creator guard, and GraphQL mutations

#### Corpus Export Format Specification and Validation Utility

- **Format specification**: `docs/architecture/corpus-export-format-spec.md` — complete reference for V1 and V2 corpus export ZIP format covering all data.json fields, PAWLs structure, referential integrity rules, security limits, and import behavior
- **Standalone validator**: `opencontractserver/utils/validate_export.py` — checks structural and referential integrity of export ZIPs without requiring Django or a database. Usable as CLI (`python -m opencontractserver.utils.validate_export corpus.zip`) or library (`validate_export()` / `validate_data_json()`)
- **Validation checks**: ZIP↔data.json file consistency, label definitions and type constraints, annotation token/page index bounds, annotation bounds non-negativity, structural set hash consistency, folder hierarchy (circular reference detection, path consistency), document path references, relationship label type enforcement (including structural relationships), V2 required top-level fields, conversation/message/vote cross-references, unknown version warnings
- **Test suite**: `opencontractserver/tests/test_validate_export.py` — 48 pure-Python tests covering all validation paths including CLI entry point

### Changed

#### Migrate from deprecated PyPDF2 to pypdf (Closes #938)

- Replaced `PyPDF2==3.0.1` with `pypdf` in `requirements/base.txt`
- Removed redundant `pypdf` entry from `requirements/local.txt` (now provided by base)
- Updated imports in `opencontractserver/utils/files.py`, `opencontractserver/utils/etl.py`, and `opencontractserver/tests/test_pdf_redaction.py`
- Removed unused `add_highlight_to_page` function from `opencontractserver/utils/files.py` (used deprecated `_addObject` API, never called)

#### Django 4.2 → 5.2 LTS Upgrade

- **Django version**: Upgraded from Django 4.2.24 to 5.2.11 (LTS)
  - `requirements/base.txt`, `requirements/local.txt`, `requirements/production.txt`
- **STORAGES migration**: Replaced deprecated `STATICFILES_STORAGE` and `DEFAULT_FILE_STORAGE` settings with the unified `STORAGES` dict (required since Django 5.1)
  - `config/settings/base.py` — LOCAL, AWS, and GCP storage backends all migrated
  - `config/settings/test.py` — test storage configuration migrated
  - `opencontractserver/tests/base.py` — test `@override_settings` migrated
  - `opencontractserver/tests/test_agent_search_tools.py` — all `@override_settings` decorators migrated
  - `opencontractserver/tests/test_storage_backends.py` — assertions updated to check `STORAGES` dict
- **Removed `USE_L10N` setting**: This setting was removed in Django 5.0 (localization is always enabled)
  - `config/settings/base.py:78`
- **Removed `SECURE_BROWSER_XSS_FILTER` setting**: This setting was removed in Django 5.0 (modern browsers handle XSS filtering natively)
  - `config/settings/base.py:503-504`
- **Replaced `pytz` with `datetime.timezone`**: Django 5.0+ uses `zoneinfo` instead of `pytz`
  - `opencontractserver/users/tasks.py` — replaced `pytz.utc.localize()` with `datetime.datetime.now(datetime.timezone.utc)`
  - Removed `pytz` from direct requirements in `requirements/base.txt`
- **Updated third-party packages for Django 5.2 compatibility**:
  - `graphene-django`: 3.2.2 → 3.2.3 (adds Django 5.1+ support; Django 5.2 not officially supported — tracked via TODO)
  - `django-stubs`: 4.2.7 → 5.2.0
  - `djangorestframework-stubs`: 1.8.0 → 3.15.4
  - `django-celery-beat`: 2.6.0 → 2.8.1 (adds Django 5.2 support)
  - `django-filter`: 24.3 → 25.1 (adds Django 5.2 support)
  - `django-model-utils`: 4.3.1 → 5.0.0 (adds Django 5.x support; no direct imports in codebase — transitive dependency)
  - `django-crispy-forms`: 2.4 → 2.5 (adds Django 5.2 support)
  - `django-cte`: 2.0.0 → 3.0.0 (adds Django 5.2 support, fixes ambiguous column names; LOUTER breaking change does not affect this project — no `_join_type` usage found)
  - `django-environ`: 0.12.0 → 0.13.0 (adds Django 5.2 support)
- **Removed `django-debug-toolbar`**: Was never wired into INSTALLED_APPS or MIDDLEWARE; removed unused dependency and associated INTERNAL_IPS config from `config/settings/local.py`
- **Replaced Collectfast with Collectfasta** (production static file collection):
  - `Collectfast==2.2.0` was archived/unmaintained (last release 2020), incompatible with Django 5.x `STORAGES` setting
  - Switched to `collectfasta>=3.2.0`, an actively maintained fork tested with Django 5.2.3
  - `requirements/production.txt` — package swap
  - `config/settings/production.py` — updated INSTALLED_APPS reference
  - `config/settings/base.py` — renamed `COLLECTFAST_STRATEGY` to `COLLECTFASTA_STRATEGY` and updated paths from `collectfast.strategies.*` to `collectfasta.strategies.*`

### Security

#### Dependency Security Updates

- **Django 4.2.24 → 4.2.28 (now 5.2.11)**: CVEs fixed by the 5.2.11 LTS release include multiple SQL injection vectors (CVE-2025-59681, CVE-2025-64459, CVE-2025-13372, CVE-2026-1312, CVE-2026-1287, CVE-2026-1207), directory traversal (CVE-2025-59682), DoS attacks (CVE-2025-64458, CVE-2025-64460), and user enumeration timing attack (CVE-2025-13473)
  - Updated in `requirements/base.txt`, `requirements/local.txt`, `requirements/production.txt`
- **cryptography 46.0.3 → 46.0.5**: Fixes CVE-2026-26007 — missing subgroup validation in ECDSA/ECDH public key loading for SECT curves, enabling signature forgery and private key leakage
  - Updated in `requirements/base.txt`
- **axios ^1.12.0 → ^1.13.5**: Fixes DoS vulnerability via `__proto__` key in `mergeConfig`
  - Updated in `frontend/package.json`
- **Removed unused `worker-loader`**: Webpack-specific package unused in Vite project; removal eliminates transitive `ajv@6.12.6` ReDoS vulnerability (via `worker-loader > schema-utils > ajv`)
  - Removed from `frontend/package.json`

### Fixed

#### TxtAnnotator Infinite Re-render Loop (Closes #933)

- **Unstable default parameter**: `chatSources = []` in `TxtAnnotator` component props created a new array reference on every render, triggering infinite re-renders via `useEffect` dependency arrays when the prop was not explicitly passed (`frontend/src/components/annotator/renderers/txt/TxtAnnotator.tsx:335`)
- Extracted `ChatSourceHighlight` interface and defined module-level `EMPTY_CHAT_SOURCES` constant as the default value, ensuring referential stability across renders

#### Follow-up Text Annotation Fixes (Closes #911)

- **Double-scroll bug**: `toggleSelectedAnnotation` in `AnnotatorSidebar.tsx:758` and `RelationshipList.tsx:106` called `scrollIntoView` for all annotation types, including text span annotations which already scroll via `TxtAnnotator`'s own `selectedAnnotations` useEffect. This caused two competing scroll animations. Fixed by guarding with `instanceof ServerTokenAnnotation` check.
- **Phantom ID tracking**: `TxtAnnotator.tsx:366` built `currentIds` from all visible annotations before verifying DOM elements existed. Annotations without rendered spans became "ghost" IDs tracked in `registeredAnnotationIdsRef` but never actually registered. Fixed by only adding IDs to the tracking set after confirming a DOM element was found and registered.
- **Page number display regression**: `HighlightItem.tsx` and `RelationHighlightItem.tsx` now use `(annotation instanceof ServerTokenAnnotation || annotation.page > 0)` to show page labels. PDF token annotations always display page labels (page is always meaningful), while span annotations only display them when `page > 0` (since `page=0` is a sentinel for "no page concept applies").
- **TypeScript type narrowing**: `HighlightItem.tsx:176` stored `instanceof` check in an intermediate boolean variable, preventing TypeScript's control-flow narrowing. Inlined the `instanceof` check directly in the conditional.

#### BaseChunkedParser Robustness and Consistency (Closes #926)

- **Config ValueError not wrapped**: `calculate_page_chunks` raises `ValueError` for invalid `max_pages_per_chunk`/`min_pages_for_chunking`, but the call in `_parse_document_impl` was unwrapped. Now caught and re-raised as `DocumentParsingError(is_transient=False)` (`opencontractserver/pipeline/base/chunked_parser.py`)
- **Small-document annotations unprefixed**: Single-chunk documents returned directly from `_parse_chunk_with_retry` without passing through `_reassemble_chunk_results`, resulting in unprefixed annotation/relationship IDs. Now all results consistently receive `c0_` prefixed IDs (`opencontractserver/pipeline/base/chunked_parser.py`)
- **Uncovered backoff cap branch**: `MAX_CHUNK_RETRY_BACKOFF_SECONDS` cap was never exercised by tests. Added test with `chunk_retry_limit=4` that verifies backoff values `[5, 10, 20, 30]` where the 4th retry hits the 30s cap (`opencontractserver/tests/test_chunked_parser.py`)
- **Theoretical race in concurrent test**: `slow_chunks_started.is_set()` assertion could fail on heavily loaded CI. Added `slow_chunks_started.wait(timeout=2)` before the assertion (`opencontractserver/tests/test_chunked_parser.py`)

#### Context Guardrails for LLM Conversation Management (Closes #907)

- **`truncate_tool_output` negative slice defense** (`opencontractserver/llms/context_guardrails.py`): Replaced fragile guard clause with explicit `char_budget = max(0, max_chars - len(notice))` to prevent negative slice indices when `max_chars` is smaller than the truncation notice length
- **Token double-counting across compaction cycles** (`opencontractserver/llms/context_guardrails.py`, `opencontractserver/llms/agents/pydantic_ai_agents.py`): Added `stored_summary_tokens` parameter to `compact_message_history()` so the stored summary is counted in `total_before` (threshold check) but not double-counted in `total_after` (the new summary replaces the old one)
- **Repeating prefix in compaction summaries** (`opencontractserver/llms/agents/pydantic_ai_agents.py`): Successive compaction cycles no longer accumulate duplicate `COMPACTION_SUMMARY_PREFIX` headers — the merge logic now strips the prefix from both old and new summaries before re-adding it once
- **Fragile sentence extraction in deterministic summary** (`opencontractserver/llms/context_guardrails.py`): Extended the first-sentence regex to split on double-newlines (paragraph boundaries) and newlines before markdown list markers (`-`, `*`, `•`, numbered lists), preventing entire bullet-list responses from being treated as a single sentence
- **Deprecated asyncio pattern in tests** (`opencontractserver/tests/test_context_guardrails.py`): Converted `TestPersistCompactionOptimisticLock` from `asyncio.run()` wrapper calls to native `async def` test methods, removing the unused `asyncio` import
- **Weak truncation test assertions** (`opencontractserver/tests/test_context_guardrails.py`): Strengthened `test_custom_max_chars` and `test_truncation_notice_contains_limit` to assert exact upper-bound length (`<= max_chars`) and verify content starts from the beginning of the input string; added `test_truncated_content_from_beginning_not_end` test
- **CHARS_PER_TOKEN_ESTIMATE docstring inconsistency** (`opencontractserver/constants/context_guardrails.py`): Clarified that the constant is intentionally 3.5 (not 4) to over-count tokens slightly for conservative compaction triggering
- **Missing integrity constraint documentation** (`opencontractserver/conversations/models.py`): Added comment and expanded `help_text` on `compacted_before_message_id` explaining why `BigIntegerField` (not `ForeignKey`) is safe — the `id__gt` filter remains correct even if the cutoff message is deleted
- **Unreachable defensive code** (`opencontractserver/llms/context_guardrails.py`): Added clarifying comment on the `recent_count < 1` guard explaining it is unreachable with default `MIN_RECENT_MESSAGES` but protects against callers passing `min_recent=0`
- **Missing compaction bookmark filter tests** (`opencontractserver/tests/test_context_guardrails.py`): Added `TestCompactionBookmarkDatabaseFilter` with two async tests verifying `get_conversation_messages()` applies `id__gt` filtering when a bookmark is set and skips it when `None`
- **New sentence extraction tests** (`opencontractserver/tests/test_context_guardrails.py`): Added `test_markdown_bullet_list_split` and `test_double_newline_paragraph_split` covering the improved regex

#### MCP Documentation Accuracy (Closes #924)

- **Missing `created` field in tool return docs**: `list_public_corpuses`, `list_documents`, and `list_annotations` all return a `created` ISO 8601 timestamp, but `llms-full.txt` omitted it from the documented return shapes
- **Incorrect annotation label shape**: `list_annotations` return docs showed `label` (string) but the actual response uses `annotation_label: { text, color, label_type }` (object) — updated to match `format_annotation()` in `opencontractserver/mcp/formatters.py`
- **Underdocumented `document://` resource**: The resource description only said "Document metadata and full extracted text" without listing the actual fields. Added field inventory including `text_preview` (first 500 chars), `full_text`, `corpus`, and `created` — critical for agents choosing between preview and full text under context window constraints
- **File**: `frontend/public/llms-full.txt`

#### BaseChunkedParser Cleanup (Closes #914)

- **Duplicate test line**: Removed redundant `PdfReader` assignment in `test_pdf_splitting.py:95`
- **Infinite loop guard**: Added input validation for `max_pages_per_chunk` and `min_pages_for_chunking` in `calculate_page_chunks()` (`opencontractserver/utils/pdf_splitting.py`); added `max_concurrent_chunks` validation in `_parse_document_impl` (`opencontractserver/pipeline/base/chunked_parser.py`)
- **Dead code / ID inconsistency**: Removed single-chunk fast-path short-circuit in `_reassemble_chunk_results()` that returned unprefixed IDs, creating inconsistency with multi-chunk results
- **Flaky test**: Replaced wall-clock timing assertion in `test_concurrent_failure_cancels_remaining` with a shorter sleep to reduce CI flakiness
- **Type safety**: Replaced `type: ignore[return-value]` in `_dispatch_concurrent` with explicit `cast()` call
- **Noisy logging**: Downgraded orphaned parent-child reference log from `warning` to `debug` level — these are expected on virtually every large hierarchical document
- **Backoff cap**: Added `MAX_CHUNK_RETRY_BACKOFF_SECONDS` constant (30s) to cap exponential backoff in per-chunk retries (`opencontractserver/constants/document_processing.py`)
- **Missing boundary test**: Added test for exact `min_pages_for_chunking` threshold (75 pages) and clarified docstring semantics
- **Memory trade-off documented**: Added comment explaining concurrent dispatch memory implications
- **Cross-chunk limitation documented**: Enhanced class docstring with follow-up improvement suggestion for section-aware chunk boundaries

### Added

- Unit tests for `HighlightItem` scroll behavior and page label display (`frontend/src/components/annotator/sidebar/__tests__/HighlightItem.scroll.test.tsx`)

### Security

#### Resolve Dependabot Security Advisories (pydantic-ai + ajv)

- **pydantic-ai 0.2.x → 1.x migration**: Upgraded from pydantic-ai 0.2.20 to >=1.56.0,<2 to resolve CVE in older version. Migration includes:
  - `End` import moved from `pydantic_ai.agent` to `pydantic_graph` (`opencontractserver/llms/agents/pydantic_ai_agents.py`)
  - All 3 `PydanticAIAgent` creation sites migrated from `system_prompt=` to `instructions=` to use the 1.x-recommended parameter that is always included in model requests regardless of message history
  - `griffe>=1.3.2,<2` pin removed (was a transitive workaround only needed for pydantic-ai 0.2.x)
  - Test file updated: `result.data` → `result.output`, `result_type` → `output_type`, `system_prompt` → `instructions` (`opencontractserver/tests/test_pydantic_ai_agents.py`)
  - `openai` bumped from ==1.102.0 to >=2.11.0,<3 (pydantic-ai 1.x requires openai >=2.11.0)
  - `pdf2image` pinned to >=1.16.0 (ancient 0.1.x versions have broken setup.py, caused cascading build failures in CI)
- **ajv ReDoS fix (CVE in ajv <8.17.1)**: Added scoped Yarn resolutions for `@rjsf/validator-ajv8/ajv` and `ajv-formats/ajv` to pin ajv 8.18.0, avoiding conflict with schema-utils which requires ajv 6.x (`frontend/package.json`)

#### IDOR Vulnerabilities Fixed in 4 GraphQL Mutations

- **HIGH**: Fixed information leakage allowing object ID enumeration via different error messages
  - `RemoveAnnotation` (`config/graphql/mutations.py`)
  - `RejectAnnotation` (`config/graphql/mutations.py`)
  - `ApproveAnnotation` (`config/graphql/mutations.py`)
  - `RemoveRelationship` (`config/graphql/mutations.py`)
- **Attack Vector**: Unauthorized users could distinguish between "object doesn't exist" and "object exists but you can't access it" by observing different error responses
- **Impact**: Allowed enumeration of valid annotation/relationship IDs
- **Solution**: All mutations now use `visible_to_user()` pattern with unified error messages; secondary permission checks also return the same unified message
- **Information leakage fix**: Outer exception handlers no longer return `str(e)` to clients; errors are logged server-side only
- **Test Coverage**: Added IDOR protection tests in `test_permission_fixes.py` and `test_voting_mutations_graphql.py`

#### QuerySet Permission Filtering Gaps Fixed

- `DocumentQuerySet.visible_to_user()` and `NoteQuerySet.visible_to_user()` inherited from `PermissionQuerySet` which had guardian permission checks commented out — only checking `is_public` and `creator`
  - `opencontractserver/shared/QuerySets.py` (classes `DocumentQuerySet`, `NoteQuerySet`)
- `AnnotationQuerySet.visible_to_user()` checked document/corpus visibility via `is_public` and `creator` only, missing guardian permission lookups for documents and corpuses
  - `opencontractserver/shared/QuerySets.py` (class `AnnotationQuerySet`)
- **Bug**: Code calling `Model.objects.filter(...).visible_to_user(user)` or `Model.objects.visible_to_user(user)` skipped guardian permission checks, making objects invisible to users with explicit share permissions
- **Impact**: Documents shared via `set_permissions_for_obj_to_user()` were invisible through the QuerySet chain code path; annotations on shared documents/corpuses were invisible; Notes on accessible documents were not visible
- **Fix**: All three QuerySets now override `visible_to_user()` with proper guardian permission table lookups. Documents and Annotations check guardian tables directly; Notes inherit from document + corpus permissions

### Fixed

#### Corpus Export/Import V2: Audit and Roundtrip Fixes

- **SPAN_LABEL and RELATIONSHIP_LABEL missing from label export**: `build_label_lookups()` in `opencontractserver/utils/etl.py` only exported TOKEN_LABEL and DOC_TYPE_LABEL labels. SPAN_LABEL and RELATIONSHIP_LABEL labels were silently dropped, causing annotation and relationship import to fail. Now all four label types are exported.
- **Relationship labels not gathered from Relationship model**: `build_label_lookups()` only queried labels from `Annotation` objects. Labels used exclusively on `Relationship` objects (RELATIONSHIP_LABEL type) were never collected. Added Relationship model queries to capture these labels.
- **Label lookup key mismatch for structural annotations and relationships**: Structural annotations and relationships reference labels by TEXT in exports, but the import label_lookup was keyed by PK strings. Created a text-keyed label lookup (`label_lookup_by_text`) in `_import_corpus()` for use by `import_structural_annotation_set()` and `_import_v2_relationships()`.
  - File: `opencontractserver/tasks/import_tasks_v2.py`
- **Document file_type not preserved**: Non-PDF documents (text/plain, etc.) lost their MIME type during export/import since `file_type` was not included in `OpenContractDocExport`. Added `file_type` to export data and import logic.
  - Files: `opencontractserver/types/dicts.py`, `opencontractserver/utils/etl.py`, `opencontractserver/utils/importing.py`
- **Document-level conversations not exported**: `package_conversations()` only exported corpus-level conversations (`chat_with_corpus=corpus`). Document-level conversations (`chat_with_document`) were completely missed. Now both types are exported.
  - File: `opencontractserver/utils/export_v2.py`
- **Conversation export missing permission filtering**: `package_conversations()` exported ALL conversations regardless of the exporting user's permissions. Added `visible_to_user()` filtering for both conversations and messages.
  - File: `opencontractserver/utils/export_v2.py`
- **Conversation fields missing from export/import**: `description`, `is_locked`, `is_pinned` were not exported or imported. Added to both export and import.
- **Message fields missing from export/import**: `parent_message` (threaded replies), `data` (JSON metadata) were not exported or imported. Added to both export and import with two-pass parent re-linking.
- **Timestamps silently discarded on conversation/message import**: Django's `auto_now_add=True` on `created_at` and `auto_now=True` on `updated_at` fields ignored values passed to `create()`. Fixed by using `QuerySet.update()` after creation to patch timestamps.
  - File: `opencontractserver/utils/import_v2.py`
- **include_conversations/include_action_trail not exposed in export mutation**: The V2 export task accepted these parameters but the GraphQL mutation never passed them, hardcoding `False`. Added both as optional mutation arguments.
  - File: `config/graphql/mutations.py`
- **DocumentPath version trees not reconstructed on import**: Exported DocumentPath data (paths, version numbers, folder assignments) was never used during import. Added `_reconstruct_document_paths()` to update auto-created DocumentPaths to match exported path structure.
  - File: `opencontractserver/tasks/import_tasks_v2.py`

### Technical Details

- All label types (TOKEN_LABEL, SPAN_LABEL, RELATIONSHIP_LABEL) are now exported in the `text_labels` dict with their actual `label_type` preserved for correct deserialization
- Conversation document hash (`chat_with_document_hash`) is exported alongside the document ID for cross-system re-linking
- Timestamp patching uses `Model.all_objects.filter(pk=obj.pk).update()` to bypass `auto_now`/`auto_now_add` behavior
- Comprehensive test coverage added: `TestLabelTypeExportCompleteness`, `TestDocumentFileTypeRoundTrip`, `TestConversationExportEnhancements`

### Added

#### Chunked Document Processing for Large PDFs

- **New `BaseChunkedParser` abstract class** (`opencontractserver/pipeline/base/chunked_parser.py`): Extends `BaseParser` to transparently split large PDF documents into page-range chunks for independent parsing and reassembly. Documents below a configurable page threshold are processed as a single request (zero overhead). Features:
  - Automatic PDF splitting via pypdf with configurable `max_pages_per_chunk` (default: 50) and `min_pages_for_chunking` (default: 75)
  - Optional concurrent chunk dispatch via `ThreadPoolExecutor` (`max_concurrent_chunks`, default: 3)
  - Per-chunk retry with exponential back-off before escalating to Celery-level retry
  - Correct reassembly of PAWLs page indices, annotation page references, `tokensJsons.pageIndex`, annotation/relationship IDs, and parent-child relationships across chunk boundaries
  - `_post_reassemble_hook()` for document-wide post-processing (e.g., image extraction on the full PDF)
- **New PDF splitting utility** (`opencontractserver/utils/pdf_splitting.py`): `get_pdf_page_count()`, `split_pdf_by_page_range()`, `calculate_page_chunks()` — pure functions for PDF page manipulation
- **New chunking constants** (`opencontractserver/constants/document_processing.py`): `DEFAULT_MAX_PAGES_PER_CHUNK`, `DEFAULT_MIN_PAGES_FOR_CHUNKING`, `DEFAULT_MAX_CONCURRENT_CHUNKS`, `DEFAULT_CHUNK_RETRY_LIMIT`
- **DoclingParser now extends `BaseChunkedParser`** (`opencontractserver/pipeline/parsers/docling_parser_rest.py`): Large documents are automatically split and parsed in chunks. Configurable via `PipelineSettings` (`DOCLING_MAX_PAGES_PER_CHUNK`, `DOCLING_MIN_PAGES_FOR_CHUNKING`, `DOCLING_MAX_CONCURRENT_CHUNKS`). Image extraction runs once on the full PDF after reassembly via `_post_reassemble_hook`.

#### Context Guardrails & Conversation Compaction (Closes #898)

- **Context guardrails constants** (`opencontractserver/constants/context_guardrails.py`): Centralized configuration for model context windows (OpenAI, Anthropic, Google), compaction thresholds, tool output limits, and token estimation parameters. Covers 20+ model variants with sensible defaults.
- **Token estimation** (`opencontractserver/llms/context_guardrails.py`): Fast heuristic token counter (~3.5 chars/token) for estimating conversation size without importing heavyweight tokeniser libraries. Intentionally over-estimates to trigger compaction conservatively.
- **Model context window lookup** (`context_guardrails.py`): Resolves model names to context window sizes via exact match then longest-prefix matching, with a 128K default fallback for unknown models.
- **Conversation compaction** (`context_guardrails.py`): When conversation history approaches the context window limit (default 75%), older messages are replaced by a concise summary while preserving recent turns (4–20 messages) verbatim. Supports pluggable summary functions for LLM-based summarization.
- **Tool output truncation** (`context_guardrails.py`, `opencontractserver/llms/tools/pydantic_ai_tools.py`): String outputs from tools are automatically truncated to 50K characters with a notice directing the LLM to use range parameters. Applied at the PydanticAI tool wrapper level so all tools benefit transparently.
- **Per-agent compaction configuration** (`CompactionConfig` dataclass): Added `compaction` field to `AgentConfig` allowing per-conversation overrides of threshold ratio, recent message counts, and tool output limits. Enabled by default.
- **Automatic compaction in message history retrieval** (`opencontractserver/llms/agents/pydantic_ai_agents.py`): `_get_message_history()` now checks conversation size against model limits and injects a compaction summary as a system message when needed, transparent to the agent framework.
- **Persisted compaction bookmarks** (`opencontractserver/conversations/models.py`): Added `compaction_summary` and `compacted_before_message_id` fields to the `Conversation` model. Compaction is computed once and persisted — subsequent reads skip old messages at the DB level via `id__gt` filter, making long conversations cheap to load.
  - Migration: `opencontractserver/conversations/migrations/0015_add_compaction_fields.py`
  - `CoreConversationManager.persist_compaction()` writes the bookmark with optimistic locking (concurrent requests are safely resolved)
  - `CoreConversationManager.get_conversation_messages()` honours the cutoff automatically
- **Comprehensive test suite** (`opencontractserver/tests/test_context_guardrails.py`): 30+ unit tests covering token estimation, model lookup, truncation, compaction triggers, summary generation, message proxy conversion, configuration defaults, and Conversation model field definitions. Uses `SimpleTestCase` for fast parallel execution.

### Changed

#### Pipeline Registry: Deduplicate and Filter Abstract Components

- **Removed `MultimodalMicroserviceEmbedder` backwards-compatibility alias**: The module-level alias `MultimodalMicroserviceEmbedder = CLIPMicroserviceEmbedder` in `opencontractserver/pipeline/embedders/multimodal_microservice.py` has been removed. Use `CLIPMicroserviceEmbedder` directly.
- **Fixed duplicate embedder entries in pipeline registry**: `_discover_subclasses()` in `opencontractserver/pipeline/registry.py` now deduplicates discovered classes by identity and skips abstract intermediate base classes via `inspect.isabstract()`, preventing aliases and abstract bases from appearing in the get-embedders query endpoint.

### Fixed

#### Prompt Injection via User-Controlled Content in Agent Prompts

- **Root cause**: Thread and document action prompt builders injected user-controlled content (message bodies, thread titles, document titles) directly into Markdown-structured LLM prompts without any sanitisation boundary. A user who can post a message to a moderated thread could craft content that overrides agent instructions.
  - File: `opencontractserver/tasks/agent_tasks.py` (lines 808-848)
  - File: `opencontractserver/llms/agents/core_agents.py` (lines 963-983, 1036-1054)
  - File: `opencontractserver/llms/agents/pydantic_ai_agents.py` (lines 1655-1669, 2301-2315)
- **Fix**: All user-generated content injected into agent prompts is now wrapped in `<user_content>` / `</user_content>` XML fence tags. An explicit `UNTRUSTED_CONTENT_NOTICE` instruction block is added to thread moderation prompts telling the LLM to treat fenced content as raw data and ignore any embedded directives. A size-threshold warning (`UNTRUSTED_CONTENT_SIZE_WARNING_THRESHOLD = 1000 chars`) logs a `[PromptInjection]` warning for abnormally large user content.
  - New file: `opencontractserver/utils/prompt_sanitization.py` — `fence_user_content()`, `warn_if_content_large()`, `UNTRUSTED_CONTENT_NOTICE`
  - New constant: `opencontractserver/constants/moderation.py` — `UNTRUSTED_CONTENT_SIZE_WARNING_THRESHOLD`
  - New tests: `opencontractserver/tests/test_prompt_sanitization.py`
  - Updated tests: `opencontractserver/tests/test_thread_corpus_actions.py` — added `test_async_thread_action_prompt_fences_user_content`

#### Prompt Injection Mitigation Follow-up (Closes #913)

- **Dead code fix**: `warn_if_content_large()` was called on truncated message previews (max 203 chars) but checks against a 1000-char threshold, making the warning ineffective. Moved the call to run on full content before truncation.
  - File: `opencontractserver/tasks/agent_tasks.py` (`_build_thread_action_system_prompt`)
- **Inconsistent monitoring**: Document and corpus titles embedded in system prompts in `core_agents.py` and `pydantic_ai_agents.py` now have `warn_if_content_large()` calls for consistent size monitoring.
  - File: `opencontractserver/llms/agents/core_agents.py` (`CoreDocumentAgentFactory`, `CoreCorpusAgentFactory`)
  - File: `opencontractserver/llms/agents/pydantic_ai_agents.py` (`PydanticAIDocumentAgent`, `PydanticAICorpusAgent`)
- **Null safety**: All title values passed to `fence_user_content()` and `warn_if_content_large()` now use `or "untitled"` fallback to prevent `TypeError` when `document.title` or `corpus.title` is `None`.
  - Affected files: `agent_tasks.py`, `core_agents.py`, `pydantic_ai_agents.py`
- **Documentation mismatch**: `UNTRUSTED_CONTENT_NOTICE` now describes the labeled tag variant (`<user_content label="...">`) matching the actual implementation, and explains that the label attribute does not change handling.
  - File: `opencontractserver/utils/prompt_sanitization.py`

#### Frontend: Most views show legacy corpus.description instead of versioned mdDescription (Closes #892)

- **Backend description sync**: `Corpus.update_description()` now keeps the plain-text `description` field in sync when `md_description` is updated via the versioned markdown system. A new `_markdown_to_plain_text()` static method strips markdown formatting for the plain-text field.
  - File: `opencontractserver/corpuses/models.py` (lines 249-272, `update_description` method)
- **New `useCorpusMdDescription` hook**: Reusable React hook that fetches markdown content from a corpus's `mdDescription` URL and returns the raw text for rendering with `SafeMarkdown`.
  - File: `frontend/src/hooks/useCorpusMdDescription.ts`
- **CorpusContextSidebar**: Now fetches and renders the versioned markdown description instead of the stale plain-text `description` field.
  - File: `frontend/src/components/threads/CorpusContextSidebar.tsx`
- **DocumentKnowledgeBase**: Corpus info display now fetches `mdDescription` content and renders it as markdown. Added `title`, `description`, and `mdDescription` fields to the `GET_DOCUMENT_KNOWLEDGE_AND_ANNOTATIONS` query's corpus selection.
  - File: `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`
  - File: `frontend/src/graphql/queries.ts` (line 3028)
- **CorpusHeader (settings)**: Now fetches and renders the versioned markdown description via `useCorpusMdDescription` hook with `SafeMarkdown`. Added `mdDescription` to prop chain through `CorpusSettings` and `Corpuses.tsx`.
  - File: `frontend/src/components/corpuses/settings/CorpusHeader.tsx`
  - File: `frontend/src/components/corpuses/CorpusSettings.tsx`
  - File: `frontend/src/views/Corpuses.tsx`
- **TypeScript type update**: Added `mdDescription` optional field to `RawCorpusType`.
  - File: `frontend/src/types/graphql-api.ts`

#### Edit Description Modal Does Not Save on Update (Issue #899)

- **Root cause**: The edit document CRUDModal in `App.tsx` had a no-op `onSubmit` handler that only closed the modal without calling the `UPDATE_DOCUMENT` mutation, so changes were silently discarded
  - File: `frontend/src/App.tsx` (lines 128-149, 398)
- **Fix**: Added `useMutation` hook for `UPDATE_DOCUMENT` in `App.tsx` with proper `onCompleted`/`onError` handlers and `refetchQueries: "active"` to refresh displayed data
- **Removed duplicate modals**: `Documents.tsx` rendered its own edit/view CRUDModals controlled by the same `editingDocument` reactive var as `App.tsx`, causing potential double-modal rendering. Removed the duplicates from `Documents.tsx` and consolidated into the global `App.tsx` handler
  - File: `frontend/src/views/Documents.tsx` (removed ~45 lines of duplicate modal + mutation code)

### Changed

#### Import/Export Pipeline Consolidation

- **DRY refactor of import/export code**: Extracted shared helpers into `opencontractserver/utils/importing.py`:
  - `prepare_import_labels()` - eliminates 4x duplicated label loading boilerplate
  - `create_document_from_export_data()` - eliminates 3x duplicated document creation
  - `import_doc_annotations()` - eliminates 3x duplicated doc+text annotation import loops
- **V1 import now delegates to V2 machinery**: `import_corpus()` in `import_tasks.py` delegates to `import_corpus_v2()` which handles both V1 and V2 formats through a unified `_import_corpus()` handler
- **V2 import fixed to use `corpus.add_document()`**: Previously created documents directly without corpus isolation; now properly uses the versioning API for correct DocumentPath records and corpus isolation
- **V2 import now sets permissions on annotations**: Previously skipped `set_permissions_for_obj_to_user` on annotations
- **V2 import now handles `content_modalities`**: Via the shared `import_annotations()` helper
- **Export finalization DRYed up**: New `finalize_export()` in `export_tasks.py` replaces 4x repeated save/timestamp/notification pattern in `package_annotated_docs`, `package_funsd_exports`, `on_demand_post_processors`, and `package_corpus_export_v2`
- **Removed duplicate `import_relationships` and `import_document_paths`** from `utils/import_v2.py` - relationship import handled inline in `_import_v2_relationships`, DocumentPaths created by `corpus.add_document()`
- **Deleted empty `opencontractserver/utils/export.py`**

### Added

#### Store Model Name in ChatMessage Metadata (#897)

- **Automatic model name persistence**: The LLM model name from `AgentConfig` is now stored in the `data` JSON field of every `ChatMessage` produced by an agent, enabling debugging, auditing, and reproducibility
  - `opencontractserver/llms/agents/core_agents.py` — all five `CoreConversationManager` message-writing methods now persist `data["model_name"]`:
    - `create_placeholder_message()` and `store_llm_message()` — unconditional write at creation time
    - `complete_message()`, `update_message()`, `mark_message_error()` — use `setdefault` to backfill without overwriting placeholder values
- **Tests**: Seven new async tests verifying model name storage across all message lifecycle paths
  - `opencontractserver/tests/test_core_agents.py` — covers explicit model name, default model name, all five methods, and `setdefault` preservation semantics

#### Nested Approval Gates for Corpus Agent Sub-Agents

- **Sub-agent approval propagation**: When a corpus agent delegates a question to a document sub-agent via `ask_document`, and the sub-agent encounters a tool requiring approval, the approval request now propagates up to the corpus agent level and is surfaced to the user via WebSocket (`ASYNC_APPROVAL_NEEDED`)
  - File: `opencontractserver/llms/agents/pydantic_ai_agents.py` (ask_document_tool closure)
- **Frontend sub-tool unwrapping**: CorpusChat's approval modal now displays the actual sub-tool name/arguments instead of the generic `ask_document` wrapper, with validation for malformed metadata
  - File: `frontend/src/components/corpuses/CorpusChat.tsx` (ASYNC_APPROVAL_NEEDED handler)
- **Comprehensive nested approval test suite**: 10 async tests covering approval propagation, metadata stripping, bypass flag lifecycle, malformed event handling, and schema safety
  - File: `opencontractserver/tests/test_nested_approval_gates.py`
- **Architecture documentation**: Added "Nested Approval Gates" section to LLM framework docs with flow diagrams and security notes
  - File: `docs/architecture/llms/README.md`

#### Expose Tool Usage in Chat UI

- **Tool Usage Badge** (`frontend/src/components/widgets/chat/ChatMessage.tsx:1180-1288`): Assistant messages that use tools now display a wrench icon badge ("X tools used") in the message header, visible in both document and corpus chat views. Users can quickly see AI tool usage without expanding the full timeline, improving agent transparency.
- **Tool Call Popover** (`ChatMessage.tsx:1222-1286`): Hovering over the badge opens a popover listing each tool call's formatted name, JSON input arguments, and output result. Keyboard accessible (Enter/Space to toggle, Escape to close) with full ARIA attributes.
- **Tool result content in timeline**: Backend now captures tool result/output content in timeline `tool_result` entries (previously only stored tool name)
  - `opencontractserver/llms/agents/timeline_schema.py:52` — added `result` field
  - `opencontractserver/llms/agents/timeline_utils.py:77-92` — captures result from metadata, truncated to 500 chars
  - `opencontractserver/llms/agents/pydantic_ai_agents.py:123-155` — `_extract_tool_result_summary()` extracts and truncates at source
- **Tool result entries for search tools** (`pydantic_ai_agents.py:642-657, 686-702, 807-813`): `similarity_search`, `search_exact_text`, and `ask_document` now emit `tool_result` timeline entries with result summaries (e.g., "Found 3 matching annotations"). Other tools use a generic extractor with "Completed" fallback.

#### Automated Documentation Screenshots

- **Screenshot capture utility** (`frontend/tests/utils/docScreenshot.ts`): Captures screenshots during Playwright component tests using an enforced `{area}--{component}--{state}` naming convention
- **CI workflow** (`.github/workflows/screenshots.yml`): Automatically runs component tests on PRs touching `frontend/` or `docs/`, then commits updated screenshots back to the PR branch
- **Initial screenshot coverage**: Landing page components (hero, stats bar, trending corpuses, call-to-action) and badge components (celebration modal, toast)

#### V2 Export Format

- **`OPEN_CONTRACTS_V2` export format**: New export type available in `StartCorpusExport` mutation that includes structural annotation sets, folder hierarchy, relationships, agent config, markdown descriptions, and conversations
- **`content_modalities` now exported**: Annotations with IMAGE or other modalities now survive export/import round-trips (`opencontractserver/utils/etl.py:build_document_export`)
- **Migration `0025_alter_userexport_format_add_v2`**: Adds `OPEN_CONTRACTS_V2` to UserExport format choices

#### Edge Case Tests for Personal Corpus (Issue #839)

- **Concurrent creation race condition test**: Verifies that 5 concurrent threads calling `get_or_create_personal_corpus()` all return the same corpus with no duplicates or errors
  - File: `opencontractserver/tests/test_personal_corpus.py` (`TestConcurrentPersonalCorpusCreation`)
- **Delete and recreate flow tests**: Verifies that after deleting a personal corpus, recreation produces a new corpus with correct attributes and permissions
  - File: `opencontractserver/tests/test_personal_corpus.py` (`TestDeleteAndRecreatePersonalCorpus`)
- **Embedding task queue failure tests**: Verifies graceful degradation when Redis/Celery is unavailable during embedding task queuing, including partial batch failure scenarios
  - File: `opencontractserver/tests/test_personal_corpus.py` (`TestEmbeddingTaskQueueFailure`)

### Fixed

#### MCP Telemetry in Async Context

- **`SynchronousOnlyOperation` in MCP server** (`config/telemetry.py`, `opencontractserver/mcp/telemetry.py`, `opencontractserver/mcp/server.py`): Added async telemetry functions (`arecord_event`, `arecord_mcp_tool_call`, `arecord_mcp_resource_read`, `arecord_mcp_request`) that use `sync_to_async` to safely run Django ORM lookups in a thread pool. Prevents "You cannot call this from an async context" errors on every MCP request.
- **Installation ID caching** (`config/telemetry.py:91-113`): Added process-lifetime cache for installation UUID to eliminate redundant database queries on every telemetry call, particularly beneficial for high-frequency MCP requests.

#### Security: LLM Prompt Injection Protection for Approval Bypass

- **Replaced `skip_approval` function parameter with `config._approval_bypass_allowed` flag**: The previous design exposed a `skip_approval` parameter in `ask_document_tool`'s function signature that a malicious LLM could set to `True` to bypass approval gates. Now uses a runtime flag on `AgentConfig` that only `resume_with_approval()` can set, wrapped in a `try/finally` block to guarantee reset
  - File: `opencontractserver/llms/agents/pydantic_ai_agents.py`

#### Inconsistent Approval Status Handling in CorpusChat

- **Added `updateMessageApprovalStatus` to CorpusChat**: Previously, `ASYNC_APPROVAL_RESULT` handler in CorpusChat only cleared pending state without updating message `approvalStatus`, unlike ChatTray and useAgentChat which both call `updateMessageApprovalStatus`. Now consistent across all components
  - File: `frontend/src/components/corpuses/CorpusChat.tsx`
- **Added message `approvalStatus: "awaiting"` on ASYNC_APPROVAL_NEEDED**: CorpusChat now marks messages as awaiting approval in both `chat` and `serverMessages` state arrays, matching ChatTray/useAgentChat behavior
  - File: `frontend/src/components/corpuses/CorpusChat.tsx`

#### Defensive Handling of Malformed Approval Events

- **Backend**: `ask_document_tool` now validates `pending_tool_call` is a dict with a non-empty `name` key before raising `ToolConfirmationRequired`; malformed events are logged and skipped
  - File: `opencontractserver/llms/agents/pydantic_ai_agents.py`
- **Frontend**: `_sub_tool_name` validation checks type is string and non-empty; `_sub_tool_arguments` validates type is object before use
  - File: `frontend/src/components/corpuses/CorpusChat.tsx`

#### Corpus Agent Action Failure: griffe/pydantic-ai Incompatibility

- **Pin `griffe>=1.3.2,<2`** (`requirements/base.txt`): griffe 2.0.0 (released 2026-02-09) removed the `**options` catch-all from all docstring parsers. pydantic-ai 0.2.x unconditionally passes `returns_named_value` and `returns_multiple_items` as parser options to all parsers (including numpy), causing `TypeError: parse_numpy() got an unexpected keyword argument 'returns_named_value'`. This broke all `run_agent_corpus_action` tasks during agent creation. Pinning griffe below 2.0 restores the `**options` parameter that absorbs these Google-specific options harmlessly.

#### SynchronousOnlyOperation in Vector Store Construction from Async Context

- **Wrap vector store construction in `sync_to_async`** (`opencontractserver/llms/vector_stores/pydantic_ai_vector_stores.py:392`): `create_vector_search_tool()` now wraps `PydanticAIAnnotationVectorStore(...)` in `sync_to_async` so the sync ORM calls inside `CoreAnnotationVectorStore.__init__` (embedder resolution via `get_embedder()`) run in a thread pool instead of triggering Django's `SynchronousOnlyOperation`.
- **Pre-resolve embedder_path in `PydanticAIDocumentAgent.create()`** (`opencontractserver/llms/agents/pydantic_ai_agents.py:1529`): Added async embedder pre-resolution using `aget_embedder()` before constructing the vector store, matching the existing pattern in `PydanticAICorpusAgent.create()`. This prevents the sync `get_embedder()` fallback from hitting the ORM in an async context.
- **Defensive `sync_to_async` fallback in both agent `create()` methods** (`pydantic_ai_agents.py`): If `aget_embedder()` fails and `embedder_path` remains `None`, the `PydanticAIAnnotationVectorStore(...)` constructor is wrapped in `sync_to_async` so the ORM calls inside `get_embedder()` run in a thread pool. Applied to both `PydanticAIDocumentAgent.create()` and `PydanticAICorpusAgent.create()`.
- **Fix async test** (`opencontractserver/tests/test_pydantic_ai_agents.py:412`): Wrapped `PydanticAIAnnotationVectorStore(...)` construction in `sync_to_async` in `test_pydantic_ai_vector_store_search`.

### Changed

#### Streamlined Agentic Corpus Action Configuration

- **Renamed `agent_prompt` to `task_instructions`** on `CorpusAction` model (`opencontractserver/corpuses/models.py`): Single, clearly-named field for describing what the agent should do. Migration `0041` handles the rename.
- **Goal-oriented system prompt assembly** (`opencontractserver/tasks/agent_tasks.py`): Agent corpus actions now auto-generate a structured system prompt with automation guardrails ("you MUST use tools"), execution context (trigger type, document metadata, corpus info), and the user's task instructions. Agents no longer receive raw `system_instructions` as the system prompt — the system wraps everything in a goal-oriented format that prevents conversational responses.
- **Document context injection**: Document-based agent actions now inject document title, ID, corpus title, and current description into the system prompt so agents don't waste tool calls loading basic metadata.
- **Thread context injection refactored**: Thread-based agent actions now use the same structured prompt pattern as document actions, with thread context, recent messages, and triggering message content all included in the system prompt rather than the user message.
- **`AgentConfiguration` is now optional for agent actions**: `CorpusAction` can be created with just `task_instructions` (no `agent_config` required). The DB constraint (`valid_action_type_configuration`) now allows lightweight agent actions. `AgentConfiguration` is still supported for custom persona/tool defaults.
- **Default tool selection by trigger type** (`opencontractserver/constants/corpus_actions.py`): When no tools are specified on `agent_config.available_tools`, the system auto-selects trigger-appropriate defaults (document tools for add/edit triggers, moderation tools for thread/message triggers).
- **`pre_authorized_tools` semantics changed** (`opencontractserver/tasks/agent_tasks.py`): `pre_authorized_tools` now only controls which tools skip approval gates. Tool availability is determined by `agent_config.available_tools` (if set) or trigger-appropriate defaults. See **Breaking Changes** below for migration guidance.
- **GraphQL API updated**: `CreateCorpusAction` and `UpdateCorpusAction` mutations use `taskInstructions` instead of `agentPrompt`. The `taskInstructions` field alone (without `agentConfigId`) is now sufficient to create an agent action.
- **Frontend updated**: `CreateCorpusActionModal` and `CorpusActionsSection` use "Task Instructions" labeling instead of "Agent Prompt".
- **Unified Quick Create flow**: `CreateCorpusActionModal` now supports inline agent creation for document triggers (add_document, edit_document) in addition to thread triggers. The modal auto-selects trigger-appropriate tools and default instructions.
- **Manual action trigger** (`config/graphql/mutations.py`): New `RunCorpusAction` mutation allows superusers to manually trigger agent actions on specific documents. Uses `transaction.on_commit()` to dispatch Celery tasks after the DB transaction commits, and `force=True` to bypass dedup checks for manual triggers.
- **ToolFunctionRegistry** (`opencontractserver/llms/tools/tool_registry.py`): Centralized singleton registry mapping tool names to sync/async function implementations. Replaces 3 manually-curated dicts in `_resolve_tools()`. Adding a new tool now requires edits in 2 files instead of 4+.

#### Mobile Navigation & URL-Driven Corpus Navigation

- **Detail view switching now pushes browser history**: `updateDetailViewParam()` in `frontend/src/utils/navigationUtils.ts` now pushes new history entries instead of replacing, so browser back/forward navigates between landing and details views. `updateTabParam()` retains replace semantics so tab switches don't accumulate history entries
- **Thread selection now URL-driven**: Discussions tab thread selection uses the existing `?thread=` query parameter (synced by CentralRouteManager) instead of the local `inlineSelectedThreadIdAtom` Jotai atom. Clicking a thread pushes `?thread=<id>` to the URL; browser back returns to the list
  - Files: `frontend/src/components/discussions/CorpusDiscussionsView.tsx`, `frontend/src/utils/navigationUtils.ts`, `frontend/src/views/Corpuses.tsx`
- **Tab-specific params cleared on tab switch**: `updateTabParam()` now removes `thread` and `message` params when switching tabs to prevent stale state persisting across tab changes
- **Removed `inlineSelectedThreadIdAtom`**: Replaced by URL-driven `selectedThreadId` reactive var from `frontend/src/graphql/cache.ts`; dead `onViewModeChange` callback removed from `CorpusDiscussionsView`

### Added

#### Mobile Menu Access in Corpus Home Views

- **Mobile navigation menu buttons**: Added `MobileMenuButton` (kebab icon, visible ≤600px) to `CorpusLandingView` (breadcrumb row) and `CorpusDetailsView` (header row), allowing mobile users to open the sidebar bottom sheet from the home tab — previously only accessible from non-home tabs
  - Files: `frontend/src/components/corpuses/CorpusHome/CorpusLandingView.tsx`, `frontend/src/components/corpuses/CorpusHome/CorpusDetailsView.tsx`, `frontend/src/components/corpuses/CorpusHome/styles.ts`
- **`updateThreadParam()` utility**: New navigation utility for setting/clearing the `?thread=` URL param with push semantics, following the same pattern as `updateTabParam()` and `updateMessageParam()`
  - File: `frontend/src/utils/navigationUtils.ts`

### Breaking Changes

- **`pre_authorized_tools` no longer controls tool availability**: Previously, `pre_authorized_tools` was used as both the tool set AND the approval gate — if set, it replaced `agent_config.available_tools` entirely. Now it only controls which tools skip the approval gate. **Migration**: If you relied on `pre_authorized_tools` to restrict which tools an agent can access, move those tool names to `agent_config.available_tools` instead. `pre_authorized_tools` should only list tools that are safe to run without human approval.

### Known Limitations

- **Orphaned QUEUED executions if Celery broker is unavailable**: The `RunCorpusAction` mutation creates a `CorpusActionExecution` with `QUEUED` status and dispatches the Celery task via `transaction.on_commit()`. If the Celery broker is down at commit time, the task dispatch silently fails and the execution record stays `QUEUED` indefinitely. This is a general characteristic of the `on_commit` + Celery pattern used throughout the codebase. Monitor for stale `QUEUED` records if broker reliability is a concern.

#### Edge Case Tests for Personal Corpus (Issue #839)

- **Concurrent creation race condition test**: Verifies that 5 concurrent threads calling `get_or_create_personal_corpus()` all return the same corpus with no duplicates or errors
  - File: `opencontractserver/tests/test_personal_corpus.py` (`TestConcurrentPersonalCorpusCreation`)
- **Delete and recreate flow tests**: Verifies that after deleting a personal corpus, recreation produces a new corpus with correct attributes and permissions
  - File: `opencontractserver/tests/test_personal_corpus.py` (`TestDeleteAndRecreatePersonalCorpus`)
- **Embedding task queue failure tests**: Verifies graceful degradation when Redis/Celery is unavailable during embedding task queuing, including partial batch failure scenarios
  - File: `opencontractserver/tests/test_personal_corpus.py` (`TestEmbeddingTaskQueueFailure`)

### Fixed

#### Security Hardening: Authentication & Permissioning Audit Remediation

- **Analysis callback DoS prevention** (`opencontractserver/analyzer/views.py`): Invalid callback tokens no longer mark analyses as FAILED; uses `hmac.compare_digest()` for timing-safe token comparison; unified error messages prevent analysis ID enumeration
- **User lock logic inversion** (`config/graphql/base.py`): Fixed `==` to `!=` in DRFDeletion and DRFMutation user lock checks — previously blocked the lock holder and allowed everyone else
- **IDOR prevention in base mutations** (`config/graphql/base.py`): DRFDeletion and DRFMutation now use `visible_to_user()` filtering before `.get()` to prevent object existence leakage
- **Open redirect prevention** (`config/urls.py`): `home_redirect` now validates the Host header against `ALLOWED_HOSTS` before constructing the redirect URL
- **Cross-corpus data leakage** (`config/graphql/graphene_types.py`): Document summary resolvers (`resolve_summary_revisions`, `resolve_current_summary_version`, `resolve_summary_content`) now verify corpus visibility before returning data
- **CSRF trusted origins** (`config/settings/production.py`): Fixed missing comma causing implicit string concatenation in `CSRF_TRUSTED_ORIGINS`
- **HSTS enforcement** (`config/settings/production.py`): Increased `SECURE_HSTS_SECONDS` from 60 to 518400 (6 days)
- **Analyzer visibility default** (`opencontractserver/analyzer/models.py`): Changed `Analyzer` and `GremlinEngine` `is_public` default from `True` to `False` to prevent accidental data exposure
- **IDOR prevention in GraphQL mutations** (`config/graphql/mutations.py`): Added `visible_to_user()` filtering to 11 previously unprotected `.objects.get()` calls in `StartDocumentExtract`, `DeleteAnalysisMutation`, `CreateColumn`, `CreateExtract`, `CreateCorpusAction`, `UpdateCorpusAction`, and `CreateNote`
- **IDOR prevention in conversation mutations** (`config/graphql/conversation_mutations.py`): `CreateThreadMutation` and `CreateThreadMessageMutation` now use `visible_to_user()` instead of fetch-then-check pattern
- **IDOR prevention in folder mutations** (`config/graphql/corpus_folder_mutations.py`): All folder mutations now verify corpus visibility before operating on folders; folder lookups scoped to validated corpus
- **IDOR prevention in voting mutations** (`config/graphql/voting_mutations.py`): `VoteMessageMutation` and `RemoveVoteMutation` now use `ChatMessage.objects.visible_to_user()` instead of fetch-then-check
- **IDOR prevention in badge mutations** (`config/graphql/badge_mutations.py`): `AwardBadgeMutation` now uses `Badge.objects.visible_to_user()` for badge lookup

### Fixed

#### Enable Relationships for Span-Based (Text) Annotations (Closes #281)

- **File type detection inconsistency**: Multiple frontend components checked for text file types using only `startsWith("text/")`, missing documents with `application/txt` MIME type. Created centralized `isTextFileType()` and `isPdfFileType()` utilities in `frontend/src/utils/files.ts` and updated all callers.
- **Label initialization race condition**: The `initialized.current` ref in `UISettingsAtom.tsx` (line 288) could be set to `true` after span label initialization, preventing relationship labels from auto-initializing on subsequent effect runs. Replaced with separate `spanLabelInitialized` and `relationLabelInitialized` refs.
- **Type restrictions blocking span annotations in relationship UI**: `RelationItem`, `RelationHighlightItem`, `HighlightItem`, and `annotationSelectedViaRelationship()` only accepted `ServerTokenAnnotation` (PDF annotations). Updated all to accept `ServerTokenAnnotation | ServerSpanAnnotation` union type, enabling the sidebar relationship display and creation flow for text documents.
- **Files changed**: `frontend/src/utils/files.ts`, `frontend/src/components/annotator/context/UISettingsAtom.tsx`, `frontend/src/components/annotator/sidebar/RelationItem.tsx`, `frontend/src/components/annotator/sidebar/RelationHighlightItem.tsx`, `frontend/src/components/annotator/sidebar/HighlightItem.tsx`, `frontend/src/components/annotator/utils.ts`, `frontend/src/components/annotator/hooks/AnnotationHooks.tsx`, `frontend/src/components/annotator/labels/EnhancedLabelSelector.tsx`, `frontend/src/components/annotator/labels/UnifiedLabelSelector.tsx`, `frontend/src/components/annotator/labels/label_selector/LabelSelector.tsx`, `frontend/src/components/knowledge_base/document/DocumentKnowledgeBase.tsx`, `frontend/src/components/widgets/chat/ChatMessage.tsx`

## [3.0.0.b4] - 2026-02-08

### ⚠️ Important Migration Notes

**Migration 0040 (`corpus_created_with_embedder`) backfills existing corpuses**

This release includes a data migration that:

- Adds a `created_with_embedder` audit field to all corpuses
- Backfills `preferred_embedder` on existing corpuses that don't have one set (uses current `DEFAULT_EMBEDDER`)
- Backfills `created_with_embedder` from `preferred_embedder`

This migration is safe and non-destructive. Existing corpuses with explicit `preferred_embedder` values are unchanged.

**Migration 0038 (`create_personal_corpuses`) is IRREVERSIBLE**

This release includes a data migration that creates personal "My Documents" corpuses for all users and moves standalone documents into them. This migration **cannot be rolled back** via `python manage.py migrate`. Attempting to reverse will raise `NotImplementedError`.

**Before deploying to production:**

- Ensure you have a database backup
- Test the migration in a staging environment first
- Plan for this being a one-way migration

If rollback is required after deployment, you must write a custom migration to handle your specific data preservation needs.

### Added

#### Document Processing Failure Indicators and Retry Controls (Issue #825)

- **Processing status display**: Document cards and list items now show distinct states for processing (spinner) vs. failed (error overlay with message) instead of a generic "Processing..." overlay for all locked documents
- **Retry button**: Failed documents display a retry button that triggers the `RetryDocumentProcessing` GraphQL mutation, allowing users to re-process documents without backend access
- **Context menu retry**: "Retry Processing" option added to the document context menu for failed documents
- **Permission-aware**: Retry controls only appear when the user has permission to retry (`canRetry` field from backend)
- **Error messages**: Processing error messages from the backend are displayed on the failure overlay (truncated for readability)
- Files: `frontend/src/components/documents/ModernDocumentItem.tsx`, `frontend/src/components/documents/DocumentItem.tsx`, `frontend/src/graphql/queries.ts`, `frontend/src/graphql/mutations.ts`, `frontend/src/types/graphql-api.ts`

#### Embedder Consistency Management (Issue #437)

- **Frozen embedder binding at corpus creation**: `preferred_embedder` is now auto-populated from `DEFAULT_EMBEDDER` when a corpus is created without an explicit embedder. This decouples existing corpuses from future changes to the global setting.
  - Files: `opencontractserver/corpuses/models.py` (save method)
- **Audit trail field `created_with_embedder`**: Records which embedder was active at corpus creation. Never changes, even after re-embedding.
  - Files: `opencontractserver/corpuses/models.py`, migration `0040_corpus_created_with_embedder.py`
- **Immutability guard on `preferred_embedder`**: `UpdateCorpusMutation` rejects changes to `preferred_embedder` after documents have been added to a corpus, preventing inconsistent embeddings.
  - Files: `config/graphql/mutations.py` (UpdateCorpusMutation.mutate)
- **`reEmbedCorpus` mutation**: Controlled migration path for changing a corpus's embedder. Locks the corpus, queues background re-embedding for all annotations, and unlocks when complete.
  - Files: `config/graphql/mutations.py` (ReEmbedCorpus), `opencontractserver/tasks/corpus_tasks.py` (reembed_corpus)
- **Fork with embedder override**: `forkCorpus` mutation now accepts optional `preferredEmbedder` argument to create the fork with a different embedder.
  - Files: `config/graphql/mutations.py` (StartCorpusFork)
- **Corpus-scoped search uses corpus embedder**: `resolve_semantic_search` now uses `corpus.preferred_embedder` for corpus-scoped queries instead of the global `DEFAULT_EMBEDDER`, ensuring consistent results.
  - Files: `config/graphql/queries.py` (resolve_semantic_search)
- **Startup system check**: Django system check warns at startup if `DEFAULT_EMBEDDER` has changed since existing corpuses were created, preventing silent search inconsistencies.
  - Files: `opencontractserver/corpuses/checks.py`, `opencontractserver/corpuses/apps.py`

#### Auth0 Authentication for Django Admin

- **Auth0 admin login support**: Django admin now supports Auth0 authentication when `USE_AUTH0=True`
  - Custom login view displays Auth0 "Sign in" button with password fallback
  - Custom logout view properly clears both Django session and Auth0 session
  - Backward compatible: password authentication always available
  - Files: `config/admin_auth/views.py`, `config/admin_auth/backends.py`
- **Admin claims synchronization**: Admin privileges can be set via Auth0 token claims

  - Supports `{namespace}is_staff` and `{namespace}is_superuser` claims
  - Claims synced on API requests with 5-minute cache TTL (configurable via `ADMIN_CLAIMS_CACHE_TTL` constant)
  - Immediate sync during admin login ensures fresh permissions for admin access
  - Handles boolean, string ("true"/"false"), and numeric (0/1) claim values
  - Configurable namespace via `AUTH0_ADMIN_CLAIM_NAMESPACE` env var
  - Files: `config/graphql_auth0_auth/utils.py:269-360`
  - **Required Auth0 Action** (Post-Login): Set up the following Auth0 Action to include admin claims in tokens:

    ```javascript
    exports.onExecutePostLogin = async (event, api) => {
      const namespace = "https://opencontracts.opensource.legal/";
      const appMetadata = event.user.app_metadata || {};

      // Add admin claims to access token
      if (appMetadata.is_staff !== undefined) {
        api.accessToken.setCustomClaim(
          `${namespace}is_staff`,
          appMetadata.is_staff
        );
      }
      if (appMetadata.is_superuser !== undefined) {
        api.accessToken.setCustomClaim(
          `${namespace}is_superuser`,
          appMetadata.is_superuser
        );
      }
    };
    ```

    Then set `app_metadata.is_staff` and `app_metadata.is_superuser` on users via Auth0 Management API or Dashboard.

- **Auth0AdminBackend**: Dedicated authentication backend for admin login via Auth0
  - Validates user exists, is active, and has `is_staff=True`
  - Files: `config/admin_auth/backends.py:18-88`
- **Security hardening**:
  - Open redirect prevention using `url_has_allowed_host_and_scheme()`
  - Host header injection prevention for Auth0 logout `returnTo` URL
  - CSRF protection on all login/logout endpoints
  - Files: `config/admin_auth/views.py:24-89`
- **Professional login template**: Standalone HTML template with Auth0 SDK integration
  - Loading states, error handling, graceful degradation
  - Uses Subresource Integrity (SRI) for CDN-hosted Auth0 SDK
  - **CSP Note**: Template uses inline JavaScript; if Content-Security-Policy is enabled,
    add `script-src 'unsafe-inline'` or implement CSP nonces
  - Files: `opencontractserver/templates/admin/auth0_login.html`
- **Comprehensive test coverage**: 50+ tests covering security edge cases
  - Open redirect prevention, boolean claim parsing, logout URL safety
  - Files: `opencontractserver/tests/test_admin_auth.py`

### Fixed

- **Admin token handling**: Admin login no longer accepts JWT tokens via query parameters (reduces CSRF/token leakage risk). Files: `config/admin_auth/views.py:146-179`
- **Admin claims demotion**: Missing or invalid admin claims now default to False to avoid privilege retention. Files: `config/graphql_auth0_auth/utils.py:331-411`
- **Token storage scope**: Admin Auth0 SPA client now uses in-memory token storage instead of localStorage. Files: `opencontractserver/templates/admin/auth0_login.html:249-257`

#### Runtime-Configurable Pipeline Settings (Superuser Only)

- **PipelineSettings singleton model**: Database-backed configuration for document processing pipeline
  - Stores preferred parsers, embedders, and thumbnailers per MIME type
  - Stores parser-specific kwargs and component settings overrides
  - Database is the single source of truth at runtime (no Django settings fallback)
  - Singleton pattern: only one instance exists, cannot be deleted
  - Files: `opencontractserver/documents/models.py:734-1140`
- **Encrypted secrets storage**: Secure storage for API keys and sensitive credentials
  - Uses Fernet symmetric encryption (key derived from Django SECRET_KEY)
  - Secrets are never exposed via GraphQL responses
  - GraphQL only returns list of components that have secrets configured
  - Methods: `set_secrets()`, `get_secrets()`, `update_secrets()`, `get_full_component_settings()`
  - Files: `opencontractserver/documents/models.py:1012-1139`
- **GraphQL query `pipelineSettings`**: Any authenticated user can read current pipeline configuration
  - Returns preferred components, parser kwargs, component settings
  - Includes `componentsWithSecrets` field (list of paths, not actual secrets)
  - Includes audit fields (modified, modified_by)
  - Files: `config/graphql/queries.py:4214-4250`
- **GraphQL mutation `updatePipelineSettings`**: Superusers can modify pipeline configuration at runtime
  - Validates component class paths exist in the pipeline registry
  - Tracks who made changes (modified_by field)
  - Changes take effect immediately for new document processing tasks
  - Files: `config/graphql/pipeline_settings_mutations.py:20-220`
- **GraphQL mutation `resetPipelineSettings`**: Superusers can reset to Django settings defaults
  - Restores all values from PREFERRED_PARSERS, PREFERRED_EMBEDDERS, etc.
  - Files: `config/graphql/pipeline_settings_mutations.py:223-302`
- **GraphQL mutation `updateComponentSecrets`**: Superusers can securely store API keys per component
  - Accepts component path and secrets dict, encrypts and stores in database
  - Supports merge mode (add to existing) or replace mode
  - Files: `config/graphql/pipeline_settings_mutations.py:305-411`
- **GraphQL mutation `deleteComponentSecrets`**: Superusers can remove secrets for a component
  - Files: `config/graphql/pipeline_settings_mutations.py:414-481`
- **Migration 0031**: Creates PipelineSettings table with encrypted_secrets field
  - Files: `opencontractserver/documents/migrations/0031_add_pipeline_settings.py`
- **Migration 0032**: Adds database index to `PipelineSettings.modified` for audit query performance
  - Files: `opencontractserver/documents/migrations/0032_add_index_to_pipeline_settings_modified.py`
- **Management command `migrate_pipeline_settings`**: Self-documenting component discovery and settings migration
  - `--list-components`: Introspects pipeline registry to show all components with settings schemas, env vars, defaults, and descriptions
  - `--sync-preferences`: Syncs PREFERRED_PARSERS, PREFERRED_EMBEDDERS, etc. from Django settings to database
  - `--component <name>`: Filters output to a specific component
  - Files: `opencontractserver/documents/management/commands/migrate_pipeline_settings.py`
- **Pipeline Configuration Guide**: Documentation covering first-time setup, upgrades, runtime configuration, and troubleshooting
  - Files: `docs/pipelines/pipeline_configuration.md`
- **Integration with doc_tasks**: `ingest_doc` and `extract_thumbnail` now read from PipelineSettings
  - Files: `opencontractserver/tasks/doc_tasks.py:355-374`, `opencontractserver/tasks/doc_tasks.py:686-721`
- **Integration with pipeline/utils**: `get_preferred_embedder` and `get_default_embedder` use PipelineSettings
  - Files: `opencontractserver/pipeline/utils.py:303-361`

### Changed

- Pipeline settings getters (`get_preferred_parser`, `get_preferred_embedder`, `get_parser_kwargs`, `get_default_embedder`) no longer fall back to Django settings at runtime. Database is the sole source of truth; initial values are populated from Django settings via `get_instance()`.
  - Files: `opencontractserver/documents/models.py:977-1092`
- Pipeline component settings are now DB-only at runtime (Django settings fallback removed from `PipelineComponentBase.get_component_settings()`)
  - Files: `opencontractserver/pipeline/base/base_component.py:180-217`
- Pipeline configuration UI now reads component settings schema from GraphQL instead of hardcoding config requirements.
  - Files: `frontend/src/components/admin/SystemSettings.tsx:69-129`
- Pipeline configuration UI now centralizes pipeline UI constants in `PIPELINE_UI` for sizing and validation values.
  - Files: `frontend/src/assets/configurations/constants.ts:174-187`

### Fixed

- Secrets modal now validates component existence, required secret fields, and payload size before mutation.
  - Files: `frontend/src/components/admin/SystemSettings.tsx:1500-1558`
- MIME filter accessibility labels now include stage context.
  - Files: `frontend/src/components/admin/SystemSettings.tsx:1076-1089`
- GraphQL `updateComponentSecrets` mutation now validates payload size before encryption attempt.
  - Files: `config/graphql/pipeline_settings_mutations.py:104-135`
- Pipeline components query now requires authentication; non-superusers only see configured components without settings schema details.
  - Files: `config/graphql/queries.py:1815-1949`
- Settings schema `_coerce_value` now logs warnings on coercion failures instead of silently swallowing errors.
  - Files: `opencontractserver/pipeline/base/settings_schema.py:416-419`
- `PipelineSettings.save()` now invalidates cache via `transaction.on_commit()` to prevent stale cache when DB write rolls back.
  - Files: `opencontractserver/documents/models.py:896-906`
- `PipelineComponentCard` memo now uses custom comparison to avoid unnecessary re-renders from object prop references.
  - Files: `frontend/src/components/admin/SystemSettings.tsx:1134-1140`
- Pipeline mutation error messages now consistently include component path for debugging.
  - Files: `config/graphql/pipeline_settings_mutations.py:644-656, 720-729`

### Removed

#### ModernBERT Embedders

- **⚠️ Breaking Change**: ModernBERT embedders have been removed from the codebase
  - `opencontractserver/pipeline/embedders/modern_bert_embedder.py` - removed
  - `opencontractserver/pipeline/embedders/minn_modern_bert_embedder.py` - removed
  - `model_preloaders/download_modernbert_model.py` - removed
  - Tests removed: `opencontractserver/tests/test_modern_bert_embedder.py`, `opencontractserver/tests/test_minn_modern_bert_embedder.py`
  - Documentation removed: `docs/embedders/modernbert_embedder.md`, `docs/embedders/minn_modernbert_embedder.md`
  - **Migration path**: Users currently using ModernBERT embedders must switch to alternative embedders:
    - `SentenceTransformerEmbedder` - General purpose sentence transformer embeddings
    - `OpenAIEmbedder` - OpenAI API-based embeddings (requires API key)
    - `VoyageAIEmbedder` - Voyage AI embeddings (requires API key)
  - Update PipelineSettings via admin UI or management command before upgrading

#### Personal Corpus ("My Documents") Feature

- **Personal corpus auto-creation**: Each user now automatically receives a personal "My Documents" corpus
  - Created via signal handler when user account is created
  - Uses database constraint to ensure one personal corpus per user (`one_personal_corpus_per_user`)
  - Personal corpus is private (`is_public=False`) and grants full permissions to owner
  - Files: `opencontractserver/corpuses/models.py:378-432`, `opencontractserver/users/signals.py:16-48`
- **All uploads default to personal corpus**: Documents without a specified corpus go to "My Documents"
  - Single file uploads via GraphQL now route to personal corpus
  - Zip file bulk uploads also default to personal corpus when no corpus specified
  - Files: `config/graphql/mutations.py:1807-1908`, `opencontractserver/tasks/import_tasks.py:514-585`
- **`Corpus.get_or_create_personal_corpus()` class method**: Idempotent method to get/create personal corpus
  - Thread-safe using `get_or_create` with atomic transaction
  - Grants full permissions on creation
  - Files: `opencontractserver/corpuses/models.py:390-432`
- **Data migration for existing users**: Migration creates personal corpuses for existing users and moves standalone documents
  - Creates "My Documents" corpus for all active users
  - Moves documents without any DocumentPath to their creator's personal corpus
  - **⚠️ IRREVERSIBLE MIGRATION**: This migration cannot be rolled back automatically. Attempting to reverse will raise `NotImplementedError`. Rolling back would delete DocumentPath records and orphan user documents from their corpus organization. If rollback is required, a custom migration must be written to handle data preservation.
  - Files: `opencontractserver/corpuses/migrations/0038_create_personal_corpuses.py`

#### Shared StructuralAnnotationSet

- **Reuse structural sets instead of duplicating**: `add_document()` now reuses the source document's structural set
  - Previously, adding a document to a corpus duplicated the entire StructuralAnnotationSet
  - Now shares the set, reducing storage and maintaining single source of truth
  - Files: `opencontractserver/corpuses/models.py:528-535`
- **Incremental embedding creation**: New Celery task checks for missing embeddings when document is added
  - `ensure_embeddings_for_corpus()` checks if embeddings exist for corpus's required embedders
  - Only queues embedding generation for annotations missing embeddings
  - Supports both DEFAULT_EMBEDDER and corpus's preferred_embedder
  - Files: `opencontractserver/tasks/corpus_tasks.py:712-850`

#### Inline Thread View with Corpus Context Sidebar

- **Added inline thread viewing**: Users can now view thread details inline within the Discussions tab instead of navigating away
  - Click a thread to view details in-place with a "Back" button to return to the list
  - Thread state tracked via `inlineSelectedThreadIdAtom` Jotai atom
  - Files: `frontend/src/components/discussions/CorpusDiscussionsView.tsx`, `frontend/src/atoms/threadAtoms.ts`
- **Added corpus context sidebar**: Displays corpus context alongside thread details
  - About section with corpus description (markdown rendered)
  - Documents section with collapsible table of contents
  - Quick stats grid (documents, threads, annotations, comments)
  - Collapsible sections with smooth animations via Framer Motion
  - Responsive behavior: hidden < 1024px, collapsible 1024-1200px, always expanded > 1200px
  - Sidebar expanded state persisted to localStorage via `threadContextSidebarExpandedAtom`
  - Files: `frontend/src/components/threads/CorpusContextSidebar.tsx`, `frontend/src/components/threads/ThreadDetailWithContext.tsx`
  - New styled components: `frontend/src/components/threads/styles/contextSidebarStyles.ts`
- **Added modernized discussion thread UI**: Comprehensive redesign following OS-Legal-Style design system
  - Typography-first design: Serif headings (Georgia), sans-serif body (Inter)
  - Teal accent color scheme (#0f766e) for interactive elements
  - Improved message cards, vote buttons, badges, and metadata displays
  - Mobile-responsive with proper breakpoints
  - Files: `frontend/src/components/threads/styles/discussionStyles.ts` (950+ lines)
- **Added agent mention rendering**: Discussion messages render styled agent mentions with custom colors
  - Runtime validation of badge configuration from GenericScalar fields
  - Hex color validation with fallback to default agent color
  - Tooltip display for agent mentions
  - Files: `frontend/src/components/threads/MarkdownMessageRenderer.tsx`
- **Added component tests for new features**:
  - Mention badge rendering tests: `frontend/tests/MentionRendering.ct.tsx`
  - Compact vote button tests: `frontend/tests/VoteButtonsCompact.ct.tsx`

### Technical Details

- Added `is_personal` BooleanField to Corpus model with database constraint
- Added composite index on `(creator, is_personal)` for efficient lookups
- Schema migration: `opencontractserver/corpuses/migrations/0037_add_is_personal_corpus.py`
- Comprehensive test suite: `opencontractserver/tests/test_personal_corpus.py` (14 tests)

#### AnnotationsPanel Shared Component

- **Created `AnnotationsPanel` reusable component**: Extracts shared filtering/display logic from annotations views
  - Provides filter tabs for type (All/Doc/Text) and source (All/Human/Agent/Structural)
  - Includes SearchBox, grid display with ModernAnnotationCard, empty state, and pagination
  - Can be used by both standalone Annotations view and corpus annotations tab
  - Files: `frontend/src/components/annotations/AnnotationsPanel.tsx`
- **Added `AnnotationsPanel` unit tests**: Comprehensive tests for filters, search, grid, empty/loading states
  - Files: `frontend/src/components/annotations/__tests__/AnnotationsPanel.test.tsx`
- **Added semantic search to corpus annotations tab**: Search box now uses vector similarity search
  - Debounced search triggers semantic search as user types (500ms delay)
  - Displays similarity scores on annotation cards
  - Supports infinite scroll for semantic search results
  - Shows appropriate empty state and loading messages for search mode
  - Files: `frontend/src/components/annotations/CorpusAnnotationCards.tsx`
- **Fixed semantic search similarity score calculation**: Scores now correctly display as percentages
  - CosineDistance returns distance (0=identical), converted to similarity (1=identical)
  - Results are sorted by similarity (highest first) and scores display correctly (e.g., 85% for close matches)
  - Files: `opencontractserver/shared/mixins.py`
- **Created lightweight `GET_ANNOTATIONS_FOR_CARDS` query**: Fetches only fields needed for ModernAnnotationCard display
  - Excludes heavy fields: `tokensJsons`, `json`, `page`, and unnecessary nested objects
  - Reduces payload from ~340KB to ~30KB for 2130 annotations (estimated 90% reduction)
  - Files: `frontend/src/graphql/queries.ts`

### Fixed

#### Annotations Panel Scroll Issue

- **Fixed corpus annotations tab scroll behavior**: Restructured AnnotationsPanel to scroll only the cards grid
  - Container uses flex column layout with `overflow: hidden`
  - FiltersSection has `flex-shrink: 0` to stay fixed at top
  - AnnotationsListContainer has `flex: 1` and `overflow-y: auto` for card scrolling
  - Filters (search, type tabs, source tabs) stay visible while cards scroll below
  - Files: `frontend/src/components/annotations/AnnotationsPanel.tsx`

#### Annotations Query Missing Pagination

- **Added initial page limit to annotations queries**: Previously loaded all annotations at once
  - Added `limit` and `cursor` fields to `GetAnnotationsInputs` interface
  - Set initial page size to 20 annotations for both Annotations.tsx and CorpusAnnotationCards.tsx
  - Infinite scroll loads more as user scrolls down
  - Files: `frontend/src/graphql/queries.ts`, `frontend/src/views/Annotations.tsx`, `frontend/src/components/annotations/CorpusAnnotationCards.tsx`

#### Corpus Annotations Tab Source Filter

- **Fixed structural annotations not visible in corpus tab**: Annotations tab now shows filter controls even when empty
  - Added source filter (Human/Agent/Structural) to `CorpusAnnotationCards`
  - Source filter syncs to GraphQL query variables: "structural" → `structural: true`, "human" → `structural: false, analysis_Isnull: true`, "agent" → `structural: false, analysis_Isnull: false`
  - Users can now toggle to see structural annotations that were previously hidden
  - Files: `frontend/src/components/annotations/CorpusAnnotationCards.tsx`
- **Added missing `usesLabelFromLabelsetId` to GetAnnotationsInputs interface**: Interface was missing a field used by the query
  - Files: `frontend/src/graphql/queries.ts:752`

### Changed

#### BREAKING: Removed Corpus.documents M2M Relationship (PR #840)

- **Removed `Corpus.documents` ManyToManyField**: DocumentPath is now the sole source of truth for corpus-document relationships
  - Migration `0039_remove_corpus_documents_m2m` validates no orphaned M2M entries before removal
  - All code paths now use `corpus.add_document()`, `corpus.remove_document()`, `corpus.get_documents()`, `corpus.document_count()`
  - GraphQL `CorpusType.documents` field now resolves via explicit DocumentPath-based resolver
  - Frontend queries updated to use `documentCount` field instead of `documents { totalCount }`
  - Files: `opencontractserver/corpuses/models.py`, `config/graphql/graphene_types.py`, `config/graphql/queries.py`
- **Removed deprecated Corpus methods**: `_create_text_document_internal()` and `create_text_document()` removed (use `import_content()` instead)
  - Removed deprecated `content` parameter from `add_document()` (use `import_content()` for content-based imports)
  - Files: `opencontractserver/corpuses/models.py`
- **Removed `sync_m2m_to_documentpath` management command**: No longer needed after M2M removal
  - Files: `opencontractserver/documents/management/commands/sync_m2m_to_documentpath.py` (deleted)
- **Added request-level caching to DocumentPathType**: Visible corpus IDs now cached per-request to prevent N+1 queries
  - Follows same pattern as `ConversationQueryOptimizer` and `DocumentRelationshipQueryOptimizer`
  - Files: `config/graphql/graphene_types.py:620-636`
- **Fixed stale frontend GraphQL queries**: Two queries still referenced removed `documents { totalCount }` connection field
  - `GET_EDITABLE_CORPUSES` in `AddToCorpusModal.tsx` now uses `documentCount`
  - `GET_MY_CORPUSES` in `queries.ts` now uses `documentCount`
  - Files: `frontend/src/components/modals/AddToCorpusModal.tsx`, `frontend/src/graphql/queries.ts`

#### Pipeline Configuration UI Redesign

- **Replaced JSON-based configuration with visual pipeline flow**: System Settings page redesigned for intuitive configuration
  - Visual pipeline stages: Document Upload → Parser → Thumbnailer → Embedder → Ready for Search
  - Clickable component cards replace JSON text editing
  - Per-stage MIME type selectors (PDF, TXT, DOCX)
  - Auto-expanding advanced settings for components requiring API keys
  - Collapsible advanced settings to reduce visual clutter
  - Files: `frontend/src/components/admin/SystemSettings.tsx`
- **New component icon system**: Custom SVG icons for each pipeline component type
  - Docling, LlamaParse, ModernBERT, OpenAI, and more
  - Semantic icons that are visually distinctive
  - Generic fallback icon for unknown components
  - Files: `frontend/src/components/admin/PipelineIcons.tsx`
- **Added accessibility attributes**: ARIA support for screen readers
  - `aria-pressed` on MIME type and component selection buttons
  - `aria-expanded` on collapsible settings sections
  - `aria-label` on interactive elements
  - Files: `frontend/src/components/admin/SystemSettings.tsx`

#### Window Resize Performance

- **Added debounce to window resize handler**: Prevents excessive re-renders during window resize
  - 150ms debounce delay on resize events
  - Properly cleans up timeout on unmount
  - Files: `frontend/src/components/hooks/WindowDimensionHook.tsx`

#### Annotations View Refactoring

- **Updated Annotations.tsx to use AnnotationsPanel**: DRY refactoring, keeps hero section, stats, semantic search, advanced filters
  - Files: `frontend/src/views/Annotations.tsx`
- **Deleted superseded AnnotationCards.tsx**: Functionality absorbed into AnnotationsPanel
  - Files: `frontend/src/components/annotations/AnnotationCards.tsx` (deleted)

#### Annotations Query Optimization

- **Switched to lightweight query for annotation cards**: Both `Annotations.tsx` and `CorpusAnnotationCards.tsx` now use `GET_ANNOTATIONS_FOR_CARDS`
  - Previous query fetched `tokensJsons` (huge JSON), `json`, full document paths (pdfFile, txtExtractFile, pawlsParseFile), full corpus details
  - New query fetches only: id, created, creator (id, email, username), corpus (id, slug, labelSet.title), document (id, slug, title), annotationLabel (id, text, color, labelType), analysis (id, analyzer.analyzerId), annotationType, structural, rawText, isPublic, contentModalities
  - Expected improvement: ~90% payload reduction, significantly faster load times
  - Files: `frontend/src/views/Annotations.tsx`, `frontend/src/components/annotations/CorpusAnnotationCards.tsx`

### Added

#### GraphQL Corpus Query Optimization

- **Added `documentCount` field to CorpusType**: Efficient document count using annotated subquery instead of N+1 queries
  - For list queries (`corpuses`), the resolver annotates `_document_count` via `DocumentPath` subquery
  - For single corpus queries, falls back to model's `document_count()` method
  - Files: `config/graphql/graphene_types.py:2028-2038`, `config/graphql/queries.py:836-869`
- **Added `annotationCount` field to CorpusType**: Efficient annotation count using annotated subquery
  - For list queries, `resolve_corpuses` annotates `_annotation_count` via Document→DocumentPath join
  - For single corpus queries, falls back to counting via DocumentPath query
  - Files: `config/graphql/graphene_types.py`, `config/graphql/queries.py`
- **Optimized LabelSet count resolvers**: Label counts now use corpus-annotated values when available
  - `resolve_label_set` on CorpusType copies annotated counts to LabelSet instance
  - `resolve_doc_label_count`, `resolve_span_label_count`, `resolve_token_label_count` check for annotations before querying
  - Files: `config/graphql/graphene_types.py:680-699, 2040-2056`
- **Optimized leaderboard `reputationGlobal` resolution**: `resolve_global_leaderboard` now attaches `_reputation_global` to user objects, avoiding N+1 queries when resolving `reputationGlobal`
  - Files: `config/graphql/queries.py`, `config/graphql/graphene_types.py`
- **Added query optimization tests**: Comprehensive tests for `documentCount`, `annotationCount`, and label set optimization
  - Files: `opencontractserver/tests/test_corpus_query_optimization.py`

### Changed

#### DiscoveryLanding GraphQL Query Optimization

- **Removed unused fields from landing page queries**: Eliminates ~39 N+1 queries per landing page load
  - Removed from `GET_DISCOVERY_DATA`: `chatMessages { totalCount }` (unused by ActivitySection), `totalMessages`, `totalThreadsCreated`, `totalAnnotationsCreated` (unused by CompactLeaderboard)
  - Replaced `documents { totalCount }` and `annotations { totalCount }` with `documentCount` and `annotationCount` (efficient subquery-backed fields)
  - Files: `frontend/src/graphql/landing-queries.ts`
- **Updated FeaturedCollections to use optimized count fields**: Uses `documentCount`/`annotationCount` instead of connection `totalCount`
  - Files: `frontend/src/components/landing/FeaturedCollections.tsx`
- **Updated TrendingCorpuses to use optimized count fields**: Uses `documentCount` instead of `documents.totalCount`
  - Files: `frontend/src/components/landing/TrendingCorpuses.tsx`
- **Updated RecentDiscussions to remove chatMessages dependency**: Display "View thread" instead of reply count
  - Files: `frontend/src/components/landing/RecentDiscussions.tsx`

#### Frontend Corpus Query Cleanup

- **Removed unused fields from GET_CORPUSES query**: Reduces payload and eliminates N+1 queries
  - Removed: `preferredEmbedder`, `appliedAnalyzerIds`, `documents.edges`, `annotations.totalCount`
  - Added: `documentCount` (efficient server-side count)
  - Files: `frontend/src/graphql/queries.ts:603-673`
- **Updated CorpusItem to use documentCount**: Uses new field instead of `documents?.edges?.length`
  - Files: `frontend/src/components/corpuses/CorpusItem.tsx:602-605`
- **Updated CorpusListView formatStats function**: Uses `documentCount` and removes annotation count display
  - Files: `frontend/src/components/corpuses/CorpusListView.tsx:303-306`
- **Added documentCount and annotationCount to TypeScript types**: Updated `RawCorpusType` interface
  - Files: `frontend/src/types/graphql-api.ts`

### Technical Details

- **Query reduction**: DiscoveryLanding page goes from ~39 N+1 queries to ~0 extra queries (all counts resolved via subqueries or removed)
- **Backward compatibility**: All new fields (`documentCount`, `annotationCount`) gracefully fall back to model methods for single corpus queries
- **Pattern**: Label counts are passed from corpus to label_set via instance attribute injection in `resolve_label_set`
- **Pattern**: Leaderboard reputation score is pre-attached to user objects via `_reputation_global` attribute

### Added

#### 2048-Dimensional Embedding Support

- **Added vector_2048 field to Embedding model**: Support for 2048-dimensional embeddings used by newer embedding models
  - Migration 0061 adds nullable `vector_2048` column to `annotations_embedding` table
  - Files: `opencontractserver/annotations/models.py:470`, `opencontractserver/annotations/migrations/0061_add_vector_2048.py`
- **Updated dimension handling across codebase**:
  - `Managers.py:_get_vector_field_name` returns "vector_2048" for 2048-dim vectors (lines 363-364)
  - `mixins.py:_dimension_to_field` returns embedding relation for 2048-dim (lines 37-38)
  - `mixins.py:get_embedding` retrieves 2048-dim vectors (lines 144-145)
  - Vector stores validate 2048 as supported dimension
  - Files: `opencontractserver/shared/Managers.py`, `opencontractserver/shared/mixins.py`, `opencontractserver/llms/vector_stores/core_vector_stores.py`, `opencontractserver/llms/vector_stores/core_conversation_vector_stores.py`

#### Multimodal Embedder Refactoring

- **Refactored MultimodalMicroserviceEmbedder into inheritance hierarchy**:
  - `BaseMultimodalMicroserviceEmbedder`: Abstract base class with shared multimodal embedding logic
  - `CLIPMicroserviceEmbedder`: CLIP ViT-L-14 model (768 dimensions) with backwards-compatible legacy settings
  - `QwenMicroserviceEmbedder`: Qwen embedding model (1024 dimensions)
  - Files: `opencontractserver/pipeline/embedders/multimodal_microservice.py`
- **Added model-specific settings**: `CLIP_EMBEDDER_URL`, `CLIP_EMBEDDER_API_KEY`, `QWEN_EMBEDDER_URL`, `QWEN_EMBEDDER_API_KEY`
  - Files: `config/settings/base.py:666-669`
- **Deprecated legacy settings**: `MULTIMODAL_EMBEDDER_URL` and `MULTIMODAL_EMBEDDER_API_KEY` still work but emit deprecation warnings
  - Users should migrate to `CLIP_EMBEDDER_URL` / `CLIP_EMBEDDER_API_KEY`

### Fixed

#### MicroserviceEmbedder Reliability

- **Fixed MicroserviceEmbedder production failures**: Added Content-Type header and 30s timeout to prevent silent failures
  - Files: `opencontractserver/pipeline/embedders/sent_transformer_microservice.py:522, 530`

### Added

#### Bulk Document Selection and Removal

- **Bulk document selection in folder toolbar**: New Select All / Deselect All functionality for corpus documents
  - Selection count display showing "X of Y" documents selected
  - Clear selection button to deselect all
  - Selection state persists across folder navigation for building cross-folder selections
  - Files: `frontend/src/components/corpuses/folders/FolderToolbar.tsx:780-827`
- **Bulk remove from corpus action**: Remove multiple selected documents from corpus in one operation
  - Dedicated danger button with document count indicator
  - Proper confirmation modal (replaces browser `window.confirm`)
  - Files: `frontend/src/components/corpuses/folders/RemoveDocumentsModal.tsx`, `frontend/src/components/corpuses/folders/FolderDocumentBrowser.tsx:367-371`
- **Mobile-responsive bulk actions**: Kebab menu includes selection controls for tablet/mobile viewports
  - Files: `frontend/src/components/corpuses/folders/FolderToolbar.tsx:976-993`
- **Loading state handling**: Select All button disabled while documents are loading to prevent incomplete selections
  - New `documentsLoading` reactive var syncs loading state from CorpusDocumentCards to FolderToolbar
  - Files: `frontend/src/graphql/cache.ts:379-384`, `frontend/src/components/documents/CorpusDocumentCards.tsx:193-201`

### Fixed

#### Embedder Error Handling and Response Parsing (PR #828)

- **Fixed silent embedding failures**: Added `EmbeddingGenerationError` exception class that triggers Celery task retries when default embeddings fail
  - Default embedding failures now properly raise and retry (up to 3 times with 60s delay)
  - Corpus-specific embedding failures are logged but don't fail the task (non-fatal)
  - Files: `opencontractserver/tasks/embeddings_task.py:165-273`
- **Fixed 1D vs 2D array response parsing**: Embedders now handle both array formats from embedding services
  - Some services return `[0.1, 0.2, ...]` (1D), others return `[[0.1, 0.2, ...]]` (2D batch format)
  - Previously caused "object of type 'float' has no len()" errors
  - Files: `opencontractserver/pipeline/embedders/sent_transformer_microservice.py:113-119`, `opencontractserver/pipeline/embedders/multimodal_microservice.py:195-201`
- **Fixed bytes-to-string decoding**: Added workaround for storage backends that return bytes even in text mode
  - Affects django-storages S3Boto3Storage with certain configurations
  - Previously caused "bytes not JSON serializable" errors
  - Files: `opencontractserver/tasks/embeddings_task.py:306-314`
- **Aligned error handling across embedders**: MicroserviceEmbedder now distinguishes 4xx (client) vs 5xx (server) errors like MultimodalMicroserviceEmbedder
  - Files: `opencontractserver/pipeline/embedders/sent_transformer_microservice.py:120-133`
- **Added comprehensive test coverage**: 18 new tests for error handling, bytes decoding, and array format parsing
  - Files: `opencontractserver/tests/test_embeddings_task.py`
- **Added TestEmbedder for fast, deterministic test embeddings**: Tests now use a fast in-memory embedder by default instead of the HTTP-based MicroserviceEmbedder
  - Returns deterministic fake vectors based on text hash (same text = same embedding)
  - Eliminates HTTP round-trips to vector-embedder service during tests (faster test execution)
  - Integration tests that need real service connectivity should explicitly instantiate MicroserviceEmbedder
  - Files: `opencontractserver/pipeline/embedders/test_embedder.py`, `config/settings/test.py:120-134`

#### Cache Eviction Consistency

- **Fixed folder document counts not updating after bulk removal**: Added `corpusFolders` cache eviction to `REMOVE_DOCUMENTS_FROM_CORPUS` mutation to match the pattern used by `MOVE_DOCUMENT_TO_FOLDER`
  - Files: `frontend/src/components/corpuses/folders/RemoveDocumentsModal.tsx:109-112`

#### Duplicate Tool Registration and Caller Tool Precedence

- **Fixed duplicate tool registration error in PydanticAI agent**: Resolved `pydantic_ai.exceptions.UserError` when caller-provided tools have the same name as default tools
  - Files: `opencontractserver/llms/agents/pydantic_ai_agents.py:2063-2082`
- **Caller-provided tools now take precedence over defaults**: When a caller passes a tool with the same name as a built-in default, the caller's tool configuration (description, requires_approval, etc.) is now used instead of silently dropping it
  - Allows callers to override tool behavior and configurations
  - Applies to both `PydanticAIDocumentAgent.create()` and `structured_response()`
  - Added info-level logging when caller tools override defaults
  - Files: `opencontractserver/llms/agents/pydantic_ai_agents.py:961-992`, `opencontractserver/llms/agents/pydantic_ai_agents.py:2063-2082`
- **Added comprehensive test coverage**:
  - `test_caller_tool_overrides_default_configuration` verifies caller's tool is used (not default)
  - `test_config_tools_deduplicated_in_structured_response` covers the config.tools path
  - Files: `opencontractserver/tests/test_duplicate_tool_registration.py`
- **Fixed PydanticAICorpusAgent consistency**: Now uses same caller-precedence pattern as document agent
- **Extracted `deduplicate_tools()` utility**: DRY refactor moves repeated deduplication logic to reusable function
  - Checks both `__name__` and `name` attributes for tool identification
  - Filters out `None` values to handle tools without names
  - Includes security documentation in docstring
  - Files: `opencontractserver/utils/tools.py`
- **Added documentation for tool precedence**: New section in LLM docs explaining when conflicts occur, which configuration wins, and security considerations
  - Files: `docs/architecture/llms/README.md`

### Added

#### Document Processing Pipeline Hardening (PR #824)

- **Document processing status tracking**: New `DocumentProcessingStatus` enum with PENDING, PROCESSING, COMPLETED, FAILED states
  - `processing_status` field on Document model with database index
  - `processing_error` and `processing_error_traceback` fields for failure diagnostics
  - Files: `opencontractserver/documents/models.py:24-32`, `opencontractserver/documents/models.py:141-159`
- **Typed parsing exceptions**: New `DocumentParsingError` with `is_transient` flag
  - Transient errors (network timeouts, service unavailable) trigger automatic retry
  - Permanent errors (invalid file, no parser) fail immediately
  - Files: `opencontractserver/pipeline/base/exceptions.py`
- **Automatic retry with exponential backoff**: Up to 3 retries with 60-300s backoff and jitter
  - Failed documents remain locked (`backend_lock=True`) to prevent broken state
  - Files: `opencontractserver/tasks/doc_tasks.py:287-435`
- **Manual retry via GraphQL**: New `RetryDocumentProcessing` mutation
  - Allows users to retry failed documents after infrastructure issues are resolved
  - Atomic state reset prevents race conditions from multiple retry clicks
  - Files: `config/graphql/mutations.py:2244-2330`
- **Failure notifications**: New `DOCUMENT_PROCESSING_FAILED` notification type
  - Notifies document creator when processing fails
  - Files: `opencontractserver/tasks/doc_tasks.py:113-146`, `opencontractserver/notifications/models.py`
- **Processing status constants**: Centralized in `opencontractserver/constants/document_processing.py`
- **24 unit tests**: Comprehensive coverage of new functionality
  - Files: `opencontractserver/tests/test_pipeline_hardening.py`

#### Bifurcated Conversation Permissions (CHAT vs THREAD)

- **New `conversation_type` field on Conversation model**: Distinguishes between personal agent chats and collaborative discussions
  - `CHAT` type: Restrictive permissions (creator + explicit permissions + public only)
  - `THREAD` type: Context-based permissions (inherits visibility from linked corpus/document)
  - Files: `opencontractserver/conversations/models.py:51-53`, `opencontractserver/conversations/migrations/`
- **Bifurcated `visible_to_user()` queryset method**: Different visibility logic based on conversation type
  - CHAT: Only creator, explicit guardian permissions, or public flag
  - THREAD: CHAT rules + context inheritance (READ on corpus AND/OR document)
  - AND logic when both corpus and document are linked (must have READ on both)
  - Files: `opencontractserver/conversations/models.py:127-238`
- **ConversationQueryOptimizer helper class**: Request-level caching to avoid N+1 queries
  - Caches visible conversation IDs per request
  - IDOR-safe `check_conversation_visibility()` method for mutations
  - Convenience methods: `get_threads_for_corpus()`, `get_threads_for_document()`, `get_chats_for_user()`
  - Files: `opencontractserver/conversations/query_optimizer.py`
- **ChatMessage visibility inheritance**: Messages inherit bifurcated permissions from parent conversation
  - Moderator access retained for corpus/document owners
  - Files: `opencontractserver/conversations/models.py:299-398`
- **22 new permission tests**: Comprehensive coverage of CHAT vs THREAD behavior
  - Files: `opencontractserver/tests/test_conversation_permissions.py`
- **Updated permissioning guide**: Documentation for bifurcated model with examples
  - Files: `docs/permissioning/consolidated_permissioning_guide.md`

#### Corpus Forking: Folder and Relationship Preservation

- **Folder hierarchy preservation during fork**: Forked corpora now maintain the complete folder structure
  - Folders cloned in tree-depth order to preserve parent-child relationships
  - Documents retain their folder assignments in the forked corpus
  - Uses `tree_queries` CTE with `.with_tree_fields()` for proper ordering
  - Files: `opencontractserver/tasks/fork_tasks.py:126-159`, `config/graphql/mutations.py:1180-1187`
- **Relationship preservation during fork**: Annotation relationships are now copied
  - Source and target annotations remapped to forked annotation IDs
  - Relationship labels preserved via label_map
  - Uses `prefetch_related()` for efficient M2M loading
  - Files: `opencontractserver/tasks/fork_tasks.py:286-356`
- **Fork task signature extended**: Added `folder_ids` and `relationship_ids` parameters
  - Files: `opencontractserver/tasks/fork_tasks.py:29-38`
- **Round-trip test suite**: Comprehensive tests validating fork data integrity across generations
  - Files: `opencontractserver/tests/test_corpus_fork_round_trip.py`

#### Corpus-Scoped MCP Endpoints for Shareable Links

- **New `/mcp/corpus/{corpus_slug}/` endpoint**: Scoped MCP endpoint for single-corpus access
  - All tools automatically operate within the specified corpus context
  - No need for explicit `corpus_slug` parameters in tool calls
  - Validates corpus exists and is publicly accessible before accepting requests
  - Returns 404 with helpful message for private/nonexistent corpuses
  - Files: `opencontractserver/mcp/server.py:371-651`, `config/asgi.py`
- **New `get_corpus_info` tool**: Returns detailed corpus information for scoped endpoints
  - Replaces `list_public_corpuses` for scoped context
  - Includes label set information, document count, and metadata
  - Files: `opencontractserver/mcp/tools.py:401-447`
- **Scoped tool wrappers**: Auto-inject corpus_slug into existing tools
  - Creates corpus-specific tool handlers that wrap global tool implementations
  - Files: `opencontractserver/mcp/tools.py:450-498`
- **TTL-based cache with eviction**: Scoped session managers cached with 1-hour TTL
  - LRU eviction at 100 entries to prevent unbounded memory growth
  - Cache invalidation logging for monitoring
  - Files: `opencontractserver/mcp/server.py:583-630`
- **Comprehensive test coverage**: 15+ tests for scoped endpoint functionality
  - Tests for validation, tool execution, cache behavior, error handling
  - Files: `opencontractserver/mcp/tests/test_mcp.py`
- **Updated MCP documentation**: Usage examples for both global and scoped endpoints
  - Files: `docs/mcp/README.md`

#### Unified Upload Modal with @os-legal/ui Design System

- **Consolidated `BulkUploadModal` and `DocumentUploadModal`** into single `UploadModal` component
  - Auto-detects upload mode: ZIP files → bulk mode, PDFs → single mode
  - Multi-step wizard for single mode (Select → Details → Corpus)
  - Simplified single-step flow for bulk ZIP uploads
  - Files: `frontend/src/components/widgets/modals/UploadModal/`
- **Custom hooks for upload state management**:
  - `useUploadState` - file list and selection state
  - `useUploadMutations` - GraphQL mutations with consistent `makePublic` handling
  - `useCorpusSearch` - debounced corpus search with permission filtering
  - Files: `frontend/src/components/widgets/modals/UploadModal/hooks/`
- **Modular sub-components**: `FileDropZone`, `FileList`, `FileDetailsForm`, `CorpusSelectorCard`, `StepIndicator`, `UploadProgress`
- **File size validation**: 100MB limit with user feedback (configurable via `UPLOAD.MAX_FILE_SIZE_BYTES` constant)
- **16 Playwright component tests** covering both modes, validation, and mobile responsiveness
- **Manual test documentation**: `docs/manual-test-scripts/upload-modal.md`
- **Upload constants**: Added `UPLOAD.MAX_FILE_SIZE_BYTES` and `DEBOUNCE.CORPUS_SEARCH_MS` to `frontend/src/assets/configurations/constants.ts`

#### Pre-extracted Image Content for Annotations

- **`image_content_file` FileField on Annotation model**: Stores pre-extracted image data as JSON files
  - Eliminates need to reload full PAWLs file (~10MB) for each image embedding request
  - Performance improvement: ~10-20x faster for image annotation embeddings
  - Files: `opencontractserver/annotations/models.py:109-114`
  - Migration: `opencontractserver/annotations/migrations/0060_add_annotation_image_content_file.py`
- **Batch image extraction utilities**: Efficient batch processing during annotation creation
  - `extract_and_store_annotation_images()` - extracts images from PAWLs and stores as JSON
  - `batch_extract_annotation_images()` - batch processes multiple annotations sharing PAWLs data
  - Files: `opencontractserver/utils/multimodal_embeddings.py:351-502`
- **Unique constraints on Embedding model**: Database-level prevention of duplicate embeddings
  - Migration: `opencontractserver/annotations/migrations/0059_add_embedding_unique_constraints.py`

#### Corpus-Specific Embeddings

- **Dual embedding strategy**: Creates both default (global search) and corpus-specific embeddings
  - Default embedder for cross-corpus search compatibility
  - Corpus-preferred embedder for corpus-specific semantic search
  - Files: `opencontractserver/tasks/embeddings_task.py:88-160`
- **Corpus ID propagation through ingestion chain**: Parser now receives corpus context
  - Enables corpus-specific embeddings during document ingestion
  - Files: `opencontractserver/tasks/doc_tasks.py:203-248`, `opencontractserver/pipeline/base/parser.py:130-143`

### Fixed

- **Corpus title not getting [FORK] prefix**: Fixed f-string that did nothing (`f"{corpus.title}"` → `f"[FORK] {corpus.title}"`)
  - Files: `config/graphql/mutations.py:1199`
- **tree_depth ordering error**: Removed explicit `order_by("tree_depth", "pk")` and rely on default `tree_ordering` from `with_tree_fields()`. The `tree_depth` field is CTE-computed and only available at SQL execution time, not during Django's `order_by()` validation.
  - Files: `config/graphql/mutations.py:1184`, `opencontractserver/tasks/fork_tasks.py:133`, `opencontractserver/utils/corpus_forking.py:46`, `opencontractserver/tests/test_corpus_fork_round_trip.py:389`
- **Fork fails with corpuses without label_set**: Added conditional handling to skip label set cloning when corpus has no label_set
  - Files: `opencontractserver/tasks/fork_tasks.py:56-136`
- **Document slug uniqueness violation during fork**: Clear slug before saving forked document so save() generates a new unique slug
  - Files: `opencontractserver/tasks/fork_tasks.py:186`
- **Annotation label mapping error**: Gracefully handle annotations without labels or when label_map is empty
  - Files: `opencontractserver/tasks/fork_tasks.py:279-285`
- **Test assertion bug**: Fixed comparison of count to queryset (`forked_labelset_labels.count() == original_labelset_labels.all()` → `.count() == .count()`)
  - Files: `opencontractserver/tests/test_corpus_forking.py:99`
- **Incorrect CorpusFolder permission setting in tests**: Removed `set_permissions_for_obj_to_user()` call for folders - CorpusFolder inherits permissions from parent Corpus, not individual permissions per the consolidated permissioning guide
  - Files: `opencontractserver/tests/test_corpus_fork_round_trip.py:277`
- **Critical: Infinite loop in corpus document copies**: Fixed chain reaction where corpus copies triggered re-ingestion
  - **Root Cause**: `add_document()` created corpus copies without setting `processing_started`, causing the ingestion signal to fire on each copy
  - **Impact**: Uploading one document created infinite chain of copies (doc → copy → copy of copy → ...)
  - **Fix**: Set `processing_started=timezone.now()` on corpus copies to prevent signal from firing
  - **Files**: `opencontractserver/corpuses/models.py:478-481`
- **Multimodal embeddings for structural annotations**: Fixed PAWLs loading from `structural_set.pawls_parse_file`
  - Structural annotations now correctly load images for embedding generation
  - Files: `opencontractserver/utils/multimodal_embeddings.py:136-166`
- **Embedding duplicate constraint violations with race condition handling**
  - **Root Cause**: Parallel Celery workers could create duplicate embeddings due to race conditions between check and create
  - **Fix**: Added `IntegrityError` catch in `store_embedding()` to handle race conditions atomically
  - **Fix**: Migration 0059 now cleans up existing duplicates before adding unique constraints (keeps best embedding per group)
  - **Fix**: Migration uses `atomic=False` to avoid PostgreSQL "pending trigger events" error
  - **Fix**: Removed `visible_to_user()` filtering from existence checks (constraints apply globally)
  - **Files**: `opencontractserver/shared/Managers.py:369-442`, `opencontractserver/annotations/migrations/0059_add_embedding_unique_constraints.py`

### Changed

- **Permission consistency**: Utility function `build_fork_corpus_task()` now uses `PermissionTypes.CRUD` (was `ALL`) to match mutation
  - Files: `opencontractserver/utils/corpus_forking.py:69`
- **`BulkUploadModal`** is now a thin wrapper: `<UploadModal forceMode="bulk" />`
- **`DocumentUploadModal`** is now a thin wrapper: `<UploadModal forceMode="single" />`
- **Image retrieval uses fast path**: Both REST API and embedding tasks check `image_content_file` first
  - Falls back to PAWLs loading only for legacy annotations without pre-extracted images
  - Files: `opencontractserver/llms/tools/image_tools.py:281-349`, `opencontractserver/utils/multimodal_embeddings.py:101-127`
- **`import_annotations()` accepts `pawls_data` parameter**: Enables batch image extraction during import
  - Files: `opencontractserver/utils/importing.py:58-150`
- **`StructuralAnnotationSet.duplicate()` copies image files**: Preserves pre-extracted images during corpus isolation
  - Files: `opencontractserver/annotations/models.py:705-745`
-

### Technical Details

- Documentation consolidated from separate remediation/edit plan files into `docs/architecture/corpus_forking.md`
- Removed obsolete files: `corpus_forking_edit_plan.md`, `corpus_forking_remediation_plan.md`
- Migrated from Semantic UI to `@os-legal/ui` design system (Modal, Button, Input, Progress, etc.)
- Uses `--oc-*` CSS design tokens for consistent theming
- Debounce cleanup on unmount to prevent memory leaks
- Sequential uploads to avoid server overload (documented trade-off)

### Removed

- **Deleted unused components**: `DocumentUploadList.tsx`, `DocumentListItem.tsx`

### Security

- **JWT authentication error message hardening** (CWE-209: Information Exposure Through Error Messages)
  - JWT errors now return generic messages (`"Invalid token"`) instead of exposing exception details
  - Detailed errors logged server-side only for debugging
  - Files: `config/rest_jwt_auth.py:80-90`
- **Sensitive data redaction in logs** (CWE-532: Insertion of Sensitive Information into Log File)
  - New `redact_sensitive_kwargs()` utility recursively redacts API keys, secrets, passwords, tokens, credentials
  - Applied to parser, embedder, and post-processor kwargs logging
  - Files: `opencontractserver/utils/logging.py`, `opencontractserver/tasks/doc_tasks.py`,
    `opencontractserver/pipeline/base/embedder.py`, `opencontractserver/pipeline/base/post_processor.py`,
    `opencontractserver/pipeline/parsers/llamaparse_parser.py`, `opencontractserver/pipeline/post_processors/pdf_redactor.py`

### Added

#### Image Annotation Display in UnifiedContentFeed

- **Modality badges for annotations**: Visual indicators showing TEXT, IMAGE, or MIXED modalities
  - Color-coded badges: Blue (text), Orange (image), Purple (mixed)
  - Integrated inline with annotation labels in HighlightItem
  - Files: `frontend/src/components/annotator/sidebar/ModalityBadge.tsx`
- **Image thumbnail previews**: Display image content directly in annotation feed
  - 80x80px thumbnails with hover effects and lazy loading
  - Automatic fetching only when IMAGE modality is present
  - Files: `frontend/src/components/annotator/sidebar/AnnotationImagePreview.tsx`
- **REST API endpoint for annotation images**: `/api/annotations/<id>/images/`
  - Permission-checked image retrieval using existing `get_annotation_images_with_permission()`
  - IDOR protection: Returns empty array for unauthorized access
  - Files: `opencontractserver/annotations/views.py`, `config/urls.py`
- **Unified JWT authentication utility**: Single entry point for token validation across all API surfaces
  - Automatic handling of both Auth0 (RS256/JWKS) and standard graphql_jwt (HS256) tokens
  - DRY architecture eliminates conditional Auth0/non-Auth0 switching in multiple files
  - Files: `config/jwt_utils.py` (NEW)
- **GraphQL content_modalities field exposure**: Added to AnnotationType schema
  - Enables frontend to filter annotations by modality
  - Files: `config/graphql/graphene_types.py`

### Fixed

- **Image annotations now clearly visible**: Image and mixed-modality annotations display properly in UnifiedContentFeed
  - Previously showed as empty text with no indication of content
  - Files: `frontend/src/components/annotator/sidebar/HighlightItem.tsx:163-167,225,249`
  - Files: `frontend/src/components/knowledge_base/document/unified_feed/ContentItemRenderer.tsx:218`
- **Structural annotations now return images**: Fixed image retrieval for structural annotations without direct document references
  - **Root Cause**: `get_annotation_images_with_permission()` returned empty array for structural annotations (no `document` field)
  - **Fix**: Load PAWLs data from `structural_set.pawls_parse_file` when document is None
  - **Impact**: Structural image annotations (e.g., figures, charts) now display thumbnails in UI
  - **Files Modified**:
    - `opencontractserver/llms/tools/image_tools.py:220-305` - Added `_extract_image_from_pawls()` helper
    - `opencontractserver/llms/tools/image_tools.py:278-305` - Updated `get_annotation_images()` to check structural_set
    - `opencontractserver/llms/tools/image_tools.py:434-492` - Updated `get_annotation_images_with_permission()` for structural permissions
  - **Test Coverage**: Added test for structural annotation image retrieval
    - Files: `opencontractserver/tests/test_annotation_images_api.py:253-321`
    - All 6 tests passing including new structural annotation test
- **Parser pipeline now populates content_modalities**: Text parser now correctly sets content_modalities field
  - **Text Parser**: Sets content_modalities to `["TEXT"]` for all text-only annotations
    - Files: `opencontractserver/pipeline/parsers/oc_text_parser.py:108`
  - **Backfill Command**: Created management command to populate existing annotations with missing content_modalities
    - Analyzes token references in PAWLs data to determine modalities
    - Fallback: Uses annotation label text as hint (e.g., "image", "figure", "chart")
    - Files: `opencontractserver/annotations/management/commands/populate_content_modalities.py`
    - Usage: `python manage.py populate_content_modalities [--dry-run] [--force]`

### Changed

- **Unified JWT authentication architecture**: Refactored authentication to use single shared utility
  - **REST API**: `config/rest_jwt_auth.py` now uses `jwt_utils.get_user_from_jwt_token()`
  - **WebSocket**: Unified `JWTAuthMiddleware` replaces separate Auth0/non-Auth0 middlewares
    - Files: `config/websocket/middleware.py` - Single middleware handles both token types
    - Files: `config/websocket/middlewares/websocket_auth0_middleware.py` - Now alias to unified middleware (deprecated)
  - **ASGI**: Simplified `config/asgi.py` to use single middleware instead of conditional switching
  - **Benefit**: DRY architecture - token validation logic centralized in one place

### Removed

- **NLM Ingest Parser**: Removed legacy NLM-Ingest PDF parser in favor of Docling (default) and LlamaParse
  - **Rationale**: Docling provides superior ML-based parsing with better structure extraction; NLM parser was rarely used
  - **Migration**: Users with `PDF_PARSER=nlm` should switch to `PDF_PARSER=docling` (default) or `PDF_PARSER=llamaparse`
  - **Files Removed**:
    - `opencontractserver/pipeline/parsers/nlm_ingest_parser.py`
    - `opencontractserver/tests/test_doc_parser_nlm_ingest.py`
    - `docs/pipelines/nlm_ingest_parser.md`
  - **Settings Updated**: Removed `nlm` option from `_PDF_PARSER_MAP` in `config/settings/base.py`

### Technical Details

- **Backend**: REST endpoint leverages existing permission-checked `image_tools.py` functions
- **Frontend hook**: `useAnnotationImages` conditionally fetches images only for IMAGE modality (performance optimization)
- **TypeScript types**: Added `contentModalities?: string[]` to annotation types
  - Files: `frontend/src/types/graphql-api.ts:147`
  - Files: `frontend/src/components/annotator/types/annotations.ts:92,145`
- **Test coverage**: 5 backend tests for REST endpoint with authentication and permission checking
  - Files: `opencontractserver/tests/test_annotation_images_api.py`

### Added

#### Corpus-Isolated Structural Annotations

- **StructuralAnnotationSet duplication per corpus**: Each corpus now gets its own copy of structural annotations when documents are added
  - Enables multi-embedder support (each corpus can use different embedding models)
  - Maintains consistent per-corpus vector spaces for similarity search
  - Files: `opencontractserver/annotations/models.py`, `opencontractserver/corpuses/models.py`
- **Extended content_hash format**: Changed from `{sha256}` to `{sha256}_{corpus_id}` (max 128 chars)
  - Migration: `opencontractserver/annotations/migrations/0056_alter_structuralannotationset_content_hash.py`

#### Multimodal Embedding Support

- **Image token extraction from PDFs**: Extract images from PDFs via Docling parser and store as unified tokens in PAWLs format
  - Storage path convention: `document_images/{doc_id}/page_{page}_img_{idx}.{format}`
  - Image tokens include position, dimensions, format, and storage path
  - Files: `opencontractserver/utils/pdf_token_extraction.py`
- **CLIP ViT-L-14 multimodal embedder**: 768-dimensional vectors in shared text/image embedding space
  - Enables cross-modal similarity search (text queries find relevant images)
  - Files: `opencontractserver/pipeline/embedders/multimodal_microservice.py`
- **ContentModality enum**: Type-safe modality tracking for embedders and annotations
  - Single source of truth: `supported_modalities: set[ContentModality]`
  - Convenience properties: `is_multimodal`, `supports_text`, `supports_images`
  - Files: `opencontractserver/types/enums.py`, `opencontractserver/pipeline/base/embedder.py`
- **Multimodal embedding utilities**: Weighted averaging for mixed text+image content
  - Default weights: 30% text, 70% image (configurable via `MULTIMODAL_EMBEDDING_WEIGHTS`)
  - Files: `opencontractserver/utils/multimodal_embeddings.py`
- **content_modalities field on Annotation model**: ArrayField tracking `["TEXT"]`, `["IMAGE"]`, or `["TEXT", "IMAGE"]`
  - Computed from PAWLs token analysis during annotation creation
  - Files: `opencontractserver/annotations/models.py`, `opencontractserver/annotations/utils.py`
- **LLM image tools for agents**: `list_document_images`, `get_document_image`, `get_annotation_images`
  - Permission-checked variants prevent IDOR vulnerabilities
  - Files: `opencontractserver/llms/tools/image_tools.py`, `opencontractserver/llms/tools/tool_registry.py`
- **Modality filtering in vector search**: Filter annotations by content type in similarity search
  - Files: `opencontractserver/llms/vector_stores/core_vector_stores.py`
- **Comprehensive documentation**: Architecture docs for multimodal embeddings and PAWLs format
  - Files: `docs/architecture/multimodal-embeddings.md`, `docs/architecture/pawls-format.md`

### Changed

#### Corpus Isolation Architecture

- **Removed content-based deduplication**: Each upload creates independent documents regardless of content hash
- **Removed source_document provenance**: `source_document_id` no longer set when adding documents to corpus
- **Structural annotations no longer shared**: Each corpus gets duplicated structural annotation sets
- **Updated documentation**: Rewrote `structural_vs_non_structural_annotations.md`, updated `document_versioning.md`, `documents_and_annotations.md`

#### Multimodal Support

- Extended PAWLs token format to support unified image tokens (`is_image=True`)
- Updated `BaseEmbedder` to use `ContentModality` enum instead of boolean flags
- Updated `PipelineComponentDefinition` in registry to store `supported_modalities`
- Enhanced embedding task to detect multimodal content and generate appropriate embeddings

### Fixed

#### Android Share URL Missing Entity Prefix (PR #795)

- **Bug**: Android native share was dropping entity type prefixes (`/c/`, `/d/`, `/e/`) from shared URLs
- **Root Cause**: `frontend/src/components/seo/MetaTags.tsx:50-56` was generating canonical URLs as `/{userSlug}/{entitySlug}` instead of `/{prefix}/{userSlug}/{entitySlug}`
- **Impact**: Links shared via Android browser resulted in 404s (e.g., `/john/my-corpus` instead of `/c/john/my-corpus`)
- **Fix**: Refactored `MetaTags.tsx` to use existing `buildCanonicalPath()` utility from `navigationUtils.ts`
- **Added**: Unit tests for MetaTags canonical URL generation (`frontend/src/components/seo/__tests__/MetaTags.test.tsx`)
- **Added**: Development warning for unexpected `entityType` values
- **Added**: Enhanced Cloudflare worker request logging for crawler debugging (`cloudflare-og-worker/src/index.ts`)

### Security

#### WebSocket Agent Permission Vulnerability Fixed (PR #792)

- **CRITICAL**: Fixed permission bypass in legacy WebSocket consumers
  - `config/websocket/consumers/corpus_conversation.py` - No permission checks
  - `config/websocket/consumers/document_conversation.py` - No permission checks
  - `config/websocket/consumers/standalone_document_conversation.py` - No permission checks
  - **Impact**: Any authenticated user could access ANY document/corpus via WebSocket
- **Solution**: Migrated to `UnifiedAgentConsumer` with three-layer permission model:
  - Consumer layer validates READ permission at WebSocket connect time (`config/websocket/consumers/unified_agent_conversation.py:93-187`)
  - Tool filtering layer removes write tools for read-only users (`opencontractserver/llms/agents/agent_factory.py:178-210`)
  - Runtime layer validates permissions before every tool execution (`opencontractserver/llms/tools/pydantic_ai_tools.py:20-111`)
- **Defense-in-depth**: Added `_check_user_permissions()` function that validates user has READ permission on document/corpus before any tool execution
- **Tool permission flags**: Added `requires_write_permission` flag to `CoreTool` (`opencontractserver/llms/tools/tool_factory.py:45-50`)
- **Write tools protected**: `add_document_note`, `update_document_summary`, `update_corpus_description`, `duplicate_annotations`

### Added

#### WebSocket Permission Escalation Tests (49 tests)

- **Test file**: `opencontractserver/tests/websocket/test_agent_permission_escalation.py`
- **Consumer-level permission tests** (13 tests): Validates connection-time permission checks for corpus, document, and combined contexts
- **Tool filtering tests** (13 tests): Verifies write tools filtered for read-only users and anonymous access
- **Runtime permission validation tests** (6 tests): Defense-in-depth `_check_user_permissions()` function
- **Permission escalation scenarios** (9 tests): Cross-user access, mid-session permission changes, resource substitution attacks
- **Integration tests** (8 tests): Full conversation flows with permission verification

#### MCP Telemetry Tracking

- **PostHog telemetry for MCP usage** (`opencontractserver/mcp/telemetry.py`): Track MCP tool calls and resource reads when telemetry is enabled
  - Records tool usage (`mcp_tool_call`): tool name, success/failure, error type
  - Records resource access (`mcp_resource_read`): resource type (corpus, document, annotation, thread), success/failure
  - Records general requests (`mcp_request`): endpoint, method, transport type, success/failure
  - Privacy-preserving: Uses salted SHA-256 IP hashing for unique user counting (raw IPs are never sent to PostHog)
  - Support for all MCP transports: streamable_http, sse, stdio
  - No query content or outputs are captured - only usage metadata
- **Telemetry integration in MCP server** (`opencontractserver/mcp/server.py`):
  - Context-based telemetry with per-request isolation via ContextVar
  - Automatic client IP extraction from ASGI scope (supports X-Forwarded-For, X-Real-IP)
  - Error telemetry for failed requests with error type classification
  - Records both successful and failed requests for error rate calculations
- **Comprehensive test coverage** (`opencontractserver/mcp/tests/test_mcp.py`):
  - Unit tests for IP hashing, context management, event recording
  - Integration tests for telemetry recording in tool/resource handlers
  - Context manager for test isolation (`isolated_telemetry_context`)

### Removed

#### Legacy WebSocket Consumers (Security Cleanup)

- **Deleted**: `config/websocket/consumers/corpus_conversation.py` (~330 lines)
- **Deleted**: `config/websocket/consumers/document_conversation.py` (~610 lines)
- **Deleted**: `config/websocket/consumers/standalone_document_conversation.py` (~530 lines)
- **Deleted**: `opencontractserver/tests/test_websocket_corpus_consumer.py` - Obsolete tests for deleted consumer
- **Deleted**: `opencontractserver/tests/test_websocket_document_consumer.py` - Obsolete tests for deleted consumer
- **Deleted**: `opencontractserver/tests/websocket/test_standalone_document_consumer.py` - Obsolete tests for deleted consumer
- **Updated**: `config/asgi.py` - Removed legacy WebSocket routes

### Changed

#### Frontend WebSocket Migration

- **Updated**: `frontend/src/components/chat/get_websockets.ts` - All WebSocket URLs now use unified endpoint
- **Updated**: `frontend/src/components/knowledge_base/document/utils.ts` - Document chat uses unified endpoint

### Technical Details

- Uses existing PostHog infrastructure from `config/telemetry.py`
- Respects `TELEMETRY_ENABLED` setting and TEST mode disable
- IP hashing uses `TELEMETRY_IP_SALT` setting to prevent rainbow table attacks
- ContextVar ensures proper isolation in concurrent async requests

### Changed

#### NavMenu Refactoring (PR #779)

- **Migrated to @os-legal/ui NavBar** (`frontend/src/components/layout/NavMenu.tsx`): Complete refactor from Semantic UI Menu to unified NavBar component
  - Single responsive component replaces separate NavMenu and MobileNavMenu
  - Built-in hamburger menu at 1100px breakpoint eliminates conditional rendering in App.tsx
  - Modern styling consistent with os-legal-style design system
- **Deleted obsolete files**: Removed `MobileNavMenu.tsx` and `MobileNavMenu.css` (~370 lines)
- **Simplified App.tsx** (`frontend/src/App.tsx:320-325`): Removed conditional menu rendering and `useWindowDimensions` dependency
- **Improved code quality** (`frontend/src/components/layout/NavMenu.tsx`):
  - Replaced inline SVG icons with lucide-react imports (Download, User, Settings, LogOut)
  - Refactored login button to use styled-components instead of inline styles
  - Added type-safe `getUserProps` helper to replace `as any` casts for user properties

### Added

#### NavMenu Component Tests

- **Playwright component tests** (`frontend/tests/NavMenu.ct.tsx`): 18 comprehensive tests covering:
  - Navigation items and active state highlighting
  - Authentication states (login button vs user menu)
  - User menu items (Exports, Profile, Admin Settings, Logout)
  - Superuser-only features (Badge Management nav item, Admin Settings menu)
  - Branding elements (logo, version badge, brand name)
  - Responsive behavior (hamburger menu, mobile navigation)
- **Test wrapper** (`frontend/tests/NavMenuTestWrapper.tsx`): Provides Auth0Provider, MockedProvider, MemoryRouter, and JotaiProvider context

### Fixed

#### Superuser Features in Non-Auth0 Mode

- **LOGIN_MUTATION missing isSuperuser** (`frontend/src/graphql/mutations.ts:49-65`): Added `isSuperuser` field to login query
  - Previously, superuser features (Badge Management, Admin Settings) were broken in non-Auth0 mode
  - Updated `LoginOutputs` interface to include `username`, `isUsageCapped`, and `isSuperuser` fields

### Fixed

#### WebSocket Connection Performance (Issue: Chat "Reconnecting" delay)

- **Auth0 JWKS caching** (`config/graphql_auth0_auth/utils.py:17-38`): Added in-memory cache for Auth0 JWKS with 10-minute TTL
  - Previously fetched JWKS from Auth0 on every token validation, causing 6-10 second delays
  - Now caches JWKS keys, reducing subsequent WebSocket auth to near-instant
- **CorpusChat double connection fix** (`frontend/src/components/corpuses/CorpusChat.tsx:1043-1056`): Skip forceNewChat useEffect on initial mount
  - `isNewChat` state already initialized from `forceNewChat` prop
  - Prevents redundant `startNewChat()` call that caused socket close/reconnect cycle
- **Notification WebSocket auth guard** (`frontend/src/hooks/useNotificationWebSocket.ts:312-318`): Skip connection attempt without auth token
  - Prevents 403 Access Denied errors when connecting before auth token is available
  - Eliminates unnecessary connection attempts and error spam in console

### Added

#### Extract View Refactoring (PR #772)

- **Route-based extract detail view** (`frontend/src/views/ExtractDetail.tsx:439-1063`, `frontend/src/components/routes/ExtractDetailRoute.tsx:1-58`): Complete refactor from modal-based to route-based architecture
  - Modern full-page layout with tabbed interface (Data, Documents, Schema)
  - Stats grid showing document count, column count, rows, and success rate
  - WebSocket-based real-time updates for running extracts (replaced polling)
  - Responsive design following existing patterns
- **WebSocket notification hook** (`frontend/src/hooks/useExtractCompletionNotification.ts:1-86`): Real-time extract completion detection
  - Listens for `EXTRACT_COMPLETE` notifications via WebSocket
  - Filters for specific extract ID and triggers refetch on completion
  - Eliminates need for polling (previously every 5 seconds)
- **Extracts list page** (`frontend/src/views/Extracts.tsx:1-410`): New landing page for extract management
  - Filter tabs (All, My Extracts, Running, Completed)
  - Search with debounced input and cleanup on unmount
  - CollectionCard components with status indicators
- **Extract list card** (`frontend/src/components/extracts/ExtractListCard.tsx:1-228`): Card component for extract listing
  - Status-aware styling (Running, Completed, Failed, Not Started)
  - Context menu with view and delete actions
  - Keyboard accessibility (Escape to close, Enter/Space to activate)
- **Shared utilities** (`frontend/src/utils/extractUtils.ts:1-70`): DRY utility functions using centralized constants
- **Extract landing route** (`frontend/src/components/routes/ExtractLandingRoute.tsx:1-35`): Route component for /extracts

### Removed

- **EditExtractModal component**: Replaced by route-based ExtractDetail view - modal approach deprecated
- **Obsolete test files** (`frontend/tests/EditExtractModal.ct.tsx`, `frontend/tests/EditExtractModalTestWrapper.tsx`): Removed tests for deleted modal component
- **Polling constants** (`frontend/src/constants/extract.ts`): Removed `EXTRACT_POLLING_INTERVAL_MS` and `EXTRACT_POLLING_TIMEOUT_MS` - replaced by WebSocket notifications

### Changed

- **openedExtract reactive var documentation** (`frontend/src/graphql/cache.ts:364-388`): Clarified that route components (like ExtractDetailRoute) can set this var, not just CentralRouteManager
- **Consolidated constants** (`frontend/src/assets/configurations/constants.ts:47-51`): Moved `EXTRACT_SEARCH_DEBOUNCE_MS` to centralized `DEBOUNCE` object
- **extractUtils refactor** (`frontend/src/utils/extractUtils.ts:32-55`): Now uses `EXTRACT_STATUS` and `EXTRACT_STATUS_COLORS` constants instead of hardcoded values

### Added

#### Corpuses Page Redesign

- **CorpusListView component** (`frontend/src/components/corpuses/CorpusListView.tsx`): Modern corpus listing page using @os-legal/ui components
  - Hero section with search and filter tabs (All, My Corpuses, Shared, Public)
  - Stats grid showing corpus, document, annotation, and shared counts
  - CollectionCard components with category badges, visibility status, and labelset information
  - Context menu for edit, view, export, fork, and delete actions
  - Infinite scroll support for large corpus lists
- **PostHog Analytics Integration** (`frontend/src/utils/analytics.ts`): Consent-based analytics tracking
  - Cookie consent banner integration
  - Automatic test/CI environment detection to prevent analytics in non-production
  - User identification and event tracking functions
  - Page view tracking for SPA navigation
- **Component tests** (`frontend/tests/CorpusListView.ct.tsx`): 12 Playwright component tests for CorpusListView
- **Unit tests** (`frontend/src/utils/__tests__/analytics.test.ts`): 20 Vitest tests for analytics utility functions

### Fixed

#### Routing Audit Follow-ups

- **Notification navigation fallback** (`frontend/src/components/notifications/NotificationDropdown.tsx:182-188`, `NotificationCenter.tsx:207-213`): Added fallback navigation to `/discussions` when corpus slug data is missing. Previously, users clicking notifications with missing slug data would see no response.
- **Network-only fetch policy optimization** (`frontend/src/routing/CentralRouteManager.tsx:621-632`): Changed thread resolution corpus query from `network-only` to `cache-and-network` since `authInitComplete` now ensures `clearStore()` completes before route queries run. This improves navigation performance when corpus data is already cached.
- **Unit test coverage** (`frontend/src/utils/__tests__/navigationUtils.test.ts:497-503`): Added missing test for `parseRoute("/discussions")` to prevent regression of discussions route parsing.

#### Type Safety and Bug Fixes

- **User email detection** (`frontend/src/components/corpuses/CorpusListView.tsx:345-349`): Fixed currentUserEmail logic to use `userObj` reactive variable from Apollo cache instead of inferring from corpus permissions - prevents filter failures when no corpus has CAN_REMOVE permission
- **TypeScript type casts** (`frontend/src/components/corpuses/CorpusListView.tsx`, `CorpusModal.tsx`): Removed 7 `as any` type casts by correcting `CorpusType.categories` type from `CorpusCategoryTypeConnection` to `CorpusCategoryType[]` to match backend GraphQL schema
- **N+1 query prevention** (`config/graphql/queries.py:820-825`): Added `prefetch_related("categories")` to `resolve_corpuses` to avoid N+1 queries when fetching corpus categories

### Changed

- **Deleted CorpusCards component** (`frontend/src/components/corpuses/CorpusCards.tsx`): Replaced by CorpusListView with @os-legal/ui components

#### LabelSet Detail Page Refactoring

- **LabelSetDetailPage component split** (`frontend/src/components/labelsets/LabelSetDetailPage.tsx`): Reduced from 2,064 lines to ~1,100 lines
  - Extracted 16 SVG icons to `detail/LabelSetIcons.tsx`
  - Extracted 40+ styled-components to `detail/LabelSetDetailStyles.ts`
  - Added barrel exports in `detail/index.ts`
- **Color constants centralized** (`frontend/src/assets/configurations/constants.ts`): Added `DEFAULT_LABEL_COLOR` and `PRIMARY_LABEL_COLOR`

### Security

- **Frontend permission checks** (`frontend/src/components/labelsets/LabelSetDetailPage.tsx:1189-1344`): Added defensive permission checks to all mutation handlers
  - `handleDeleteLabel` now verifies `canRemove` before deletion
  - `handleSaveEdit` now verifies `canUpdate` before updating
  - `handleSaveCreate` now verifies `canUpdate` before creation
  - `handleDelete` (labelset) now verifies `canRemove` before deletion
- **Color input sanitization** (`frontend/src/components/labelsets/LabelSetDetailPage.tsx:125-143`): Added `isValidHexColor` and `sanitizeColor` utilities
  - Validates hex color format (3 or 6 character, with or without #)
  - Prevents potential XSS via CSS color injection

### Technical Details

- New test file: `frontend/tests/LabelSetDetailPage.ct.tsx` with comprehensive component tests
  - Rendering tests for all tabs (Overview, Text/Doc/Span/Relationship Labels)
  - Permission-based UI visibility tests
  - Search functionality tests
  - Mobile navigation tests

### Added

#### Secure Zip Import with Folder Structure Preservation

- **Zip security utilities** (`opencontractserver/utils/zip_security.py`): Comprehensive security validation for zip file imports
  - Path traversal protection: Sanitizes all paths, rejects `..` sequences, drive letters, absolute paths
  - Zip bomb detection: Monitors compression ratios, enforces size limits (500MB total, 100MB per file)
  - Symlink rejection: Detects and skips symbolic links in zip entries
  - Resource limits: Max 1000 files, 500 folders, 20 levels deep (all configurable)
  - Hidden file filtering: Skips `.DS_Store`, `__MACOSX`, `Thumbs.db`, etc.
- **Security constants** (`opencontractserver/constants/zip_import.py`): Configurable limits via Django settings
- **Folder structure creation** (`opencontractserver/corpuses/folder_service.py:1268-1411`): `create_folder_structure_from_paths()` efficiently creates folder hierarchies, reusing existing folders
- **Import Celery task** (`opencontractserver/tasks/import_tasks.py:580-912`): `import_zip_with_folder_structure` task with three-phase processing:
  - Phase 1: Security validation without extraction
  - Phase 2: Atomic folder structure creation
  - Phase 3: Batched document processing with per-file error handling
- **GraphQL mutation** (`config/graphql/mutations.py:1890-2040`): `importZipToCorpus` mutation with rate limiting
  - Accepts base64-encoded zip file
  - Optional target folder placement
  - Returns job_id for async tracking
  - Requires corpus EDIT permission
- **Document upversioning on collision**: When importing a document to a path that already has a document, the new document becomes version 2 (or higher), with the previous version preserved in history
- **Comprehensive test suites**:
  - Security tests (`opencontractserver/tests/test_zip_security.py`): 49 tests for path sanitization, validation, edge cases
  - Integration tests (`opencontractserver/tests/test_zip_import_integration.py`): 17 tests for task and folder service
- **Design documentation** (`docs/features/zip_import_with_folders_design.md`): Complete specification including security model, API, error handling

#### Corpus Categories and Landing Page Redesign

- **CorpusCategory model** (`opencontractserver/corpuses/models.py`): New model for organizing corpuses by type (Legislation, Contracts, Case Law, Knowledge)
  - Admin-provisioned structural data - managed via Django Admin only
  - ManyToMany relationship with Corpus for flexible categorization
  - Default categories seeded via migration (`0035_seed_default_categories.py`)
- **CorpusCategoryType GraphQL type** (`config/graphql/graphene_types.py:1589-1633`):
  - Globally visible to all users (no individual permissions)
  - `corpusCount` field with N+1 query optimization via annotation
- **Landing page redesign** using @os-legal/ui component library:
  - `CompactLeaderboard` component - clean list-based leaderboard replacing grid cards
  - `CategorySelector` component for corpus categorization
  - Skeleton loading states and error handling throughout
- **TypeScript types** (`frontend/src/types/graphql-api.ts`): Added `CorpusCategoryType`, `CorpusCategoryTypeConnection`, `CorpusCategoryTypeEdge`
- **Array utilities** (`frontend/src/utils/arrayUtils.ts`): `arraysEqualUnordered` and `arraysEqualOrdered` for DRY comparison logic

### Fixed

#### Security and Performance

- **System user security** (`opencontractserver/corpuses/migrations/0035_seed_default_categories.py`): Defense-in-depth with unusable password for system user
- **N+1 query in corpusCount** (`config/graphql/queries.py:835-866`): Pre-annotated counts in `resolve_corpus_categories` resolver
- **Type safety** (`frontend/src/components/corpuses/CorpusModal.tsx`, `CorpusSettings.tsx`): Removed `as any` casts for categories field

### Changed

- **Permission model** (`config/graphql/graphene_types.py`): `CorpusCategoryType` no longer uses `AnnotatePermissionsForReadMixin` - categories are globally visible structural data
- **Documentation** (`docs/permissioning/consolidated_permissioning_guide.md`): Added section on CorpusCategory permissions

### Technical Details

- Categories are created by a `system` user with `is_active=False` and unusable password
- `corpusCount` respects user visibility: anonymous sees public corpuses only, authenticated users see corpuses they have access to
- Removed 632-line `TopContributors.tsx` component, replaced with ~280-line `CompactLeaderboard.tsx`

### Added

#### Moderation Dashboard and Rollback Features (Issue #742)

- **ModerationActionType GraphQL type** (`config/graphql/graphene_types.py:3071-3121`): Exposes ModerationAction audit records with computed fields:
  - `corpusId`: Links to parent corpus for filtering
  - `isAutomated`: Identifies agent vs. human moderation
  - `canRollback`: Indicates whether action can be undone
- **ModerationMetricsType** (`config/graphql/graphene_types.py:3109-3121`): Aggregated metrics for monitoring moderation activity:
  - Total/automated/manual action counts
  - Hourly action rate with threshold alerting
  - Actions grouped by type
- **New GraphQL queries** (`config/graphql/queries.py:1875-2043`):
  - `moderationActions`: Filterable query for audit logs (corpus, thread, moderator, action type)
  - `moderationAction`: Single action lookup by ID
  - `moderationMetrics`: Aggregated stats with threshold violations
- **RollbackModerationActionMutation** (`config/graphql/moderation_mutations.py:594-707`): Undo automated moderation actions:
  - Supports rollback of delete_message, delete_thread, lock_thread, pin_thread
  - Creates new audit record for the rollback
  - Permission-gated to moderators
- **DeleteThreadMutation and RestoreThreadMutation** (`config/graphql/moderation_mutations.py:267-363`): Complete thread lifecycle management for frontend
- **ModerationDashboard component** (`frontend/src/components/moderation/ModerationDashboard.tsx`): Full-featured moderation UI:
  - Metrics display with threshold alerts
  - Filterable action table (action type, automated only)
  - Rollback confirmation modal
  - Time range selector (1h, 24h, 7d, 30d)
- **Dynamic tool fetching** (`frontend/src/components/corpuses/CreateCorpusActionModal.tsx`): Replaces hardcoded moderation tools with GraphQL query to `availableTools(category: "moderation")`

### Fixed

#### Race Condition in Agent Thread Actions (Issue #742)

- **Fixed TOCTOU vulnerability** (`opencontractserver/tasks/agent_tasks.py:859-898`): Added `select_for_update()` with `transaction.atomic()` to prevent duplicate agent execution claims

#### Tool Validation for Inline Agents (Issue #742)

- **Added tool category validation** (`config/graphql/mutations.py:3875-3897`): CreateCorpusAction now validates that inline agent tools are from the MODERATION category when using thread/message triggers

### Added

#### MCP (Model Context Protocol) Interface Proposal (Issue #387)

- **Comprehensive MCP interface design** (`docs/mcp/mcp_interface_proposal.md`): Read-only access to public OpenContracts resources for AI assistants
- **4 resource types**: corpus, document, annotation, thread - with hierarchical URI patterns
- **7 tools for discovery and retrieval**: `list_public_corpuses`, `list_documents`, `get_document_text`, `list_annotations`, `search_corpus`, `list_threads`, `get_thread_messages`
- **Anonymous user permission model**: Operates as AnonymousUser with automatic filtering to `is_public=True` resources
- **Synchronous Django ORM implementation**: Uses `sync_to_async` wrapper pattern for MCP server integration
- **Performance optimizations**: Uses existing `AnnotationQueryOptimizer`, `prefetch_related` for threaded messages, and proper pagination
- **Robust URI parsing**: Regex-based URI parsing with slug validation to prevent injection attacks
- **Helper function implementations**: Complete `format_*` functions for corpus, document, annotation, thread, and message formatting

#### Markdown Link Generation Tool for Agent Responses (Issue #530)

- **New `create_markdown_link` agent tool** (`opencontractserver/llms/tools/core_tools.py:1990-2174`): Agents can now generate properly formatted markdown links for annotations, corpus, documents, and conversations
- **Supported entity types**:
  - **Annotations**: `[Annotation text](/d/user/corpus/doc?ann=123)` - Links to annotation with document context
  - **Corpus**: `[Corpus Title](/c/user/corpus-slug)` - Direct links to corpus
  - **Documents**: `[Document Title](/d/user/corpus/doc-slug)` - Smart routing (standalone or corpus-based)
  - **Conversations/Threads**: `[Discussion Title](/c/user/corpus/discussions/123)` - Links to discussion threads
- **Intelligent link generation**:
  - Automatically detects if documents belong to a corpus for proper URL structure
  - Truncates long annotation text (>100 chars) for readability
  - Uses entity titles when available, falls back to generic labels (e.g., "Annotation 123")
  - Validates entity existence, creator, and slug availability before generating links
- **Async support**: Both sync (`create_markdown_link`) and async (`acreate_markdown_link`) versions available
- **Tool registry entry** (`opencontractserver/llms/tools/tool_registry.py:364-380`): Registered as COORDINATION category tool with full parameter documentation
- **Comprehensive test coverage** (`opencontractserver/tests/test_llm_tools.py:2031-2417`):
  - 35+ test cases covering all entity types, edge cases, and error conditions
  - Tests for both sync and async implementations
  - Validation of error messages for missing entities, creators, slugs, and invalid types

#### Real-Time Notification System via WebSocket (Issue #637)

- **WebSocket notification consumer** (`config/websocket/consumers/notification_updates.py`): New `NotificationUpdatesConsumer` provides real-time notification delivery for all notification types (BADGE, REPLY, MENTION, THREAD_REPLY, moderation actions)
- **Frontend WebSocket hook** (`frontend/src/hooks/useNotificationWebSocket.ts`): `useNotificationWebSocket` hook manages WebSocket connection lifecycle with auto-reconnection, heartbeat monitoring, and graceful error handling
- **Signal broadcasting** (`opencontractserver/notifications/signals.py:33-100`): All notification creation signals now broadcast via WebSocket channel layer for instant delivery
- **ASGI routing** (`config/asgi.py:88-94`): Registered `ws/notification-updates/` WebSocket endpoint with authentication middleware
- **WebSocket URL helper** (`frontend/src/components/chat/get_websockets.ts:226-259`): `getNotificationUpdatesWebSocket` function constructs WebSocket URLs with proper protocol handling

### Changed

#### Badge Notifications Migrated from Polling to WebSocket (Issue #637)

- **useBadgeNotifications hook** (`frontend/src/hooks/useBadgeNotifications.ts`): Completely refactored from Apollo Client polling (30s intervals) to WebSocket-based real-time updates
- **Zero latency**: Badge awards now appear instantly instead of 0-30 second delay
- **Reduced server load**: Eliminated continuous polling requests from all connected clients
- **Backward compatible**: Maintains same interface (`newBadges`, `clearNewBadges`) with added `connectionState` for debugging

### Fixed

#### WebSocket Token Expiration Close Code Handling (PR #746)

- **Updated all WebSocket consumers** to check `scope['auth_error']` from middleware and use specific close codes:
  - `config/websocket/consumers/document_conversation.py:77-91`: Uses auth_error codes for expired/invalid tokens
  - `config/websocket/consumers/corpus_conversation.py:67-79`: Uses auth_error codes for expired/invalid tokens
  - `config/websocket/consumers/standalone_document_conversation.py:97-106`: Checks auth_error before falling back to anonymous handling
  - `config/websocket/consumers/unified_agent_conversation.py:119-127`: Uses auth_error codes for expired/invalid tokens
  - `config/websocket/consumers/thread_updates.py:77-88`: Uses auth_error codes for expired/invalid tokens
- **Removed unused `Union` import** from `config/websocket/middleware.py:2`
- **Fixed lazy import issue** in `config/graphql_auth0_auth/utils.py:124`: Moved `sync_remote_user` import inside function to avoid import error when `USE_AUTH0=False`
- **Added Auth0 test settings** in `config/settings/test.py:120-133`: Default Auth0 settings for test environment to allow importing Auth0 modules during testing

#### Impact

- Frontend can now distinguish between expired tokens (4001) and invalid tokens (4002) via WebSocket close codes
- Enables targeted token refresh vs full re-authentication based on close code
- Fixes issue #744 where token expiration wasn't properly signaled to clients

### Added

#### LlamaParse Document Parser Integration (Issue #692)

- **New LlamaParseParser** (`opencontractserver/pipeline/parsers/llamaparse_parser.py`): Full integration with LlamaParse API for document parsing with layout extraction
  - Supports PDF and DOCX file types
  - Extracts structural annotations (Title, Heading, Paragraph, Table, Figure, List, etc.) with bounding boxes
  - Generates PAWLS tokens from LlamaParse layout data for PDF annotation display
  - Supports multiple bounding box formats (fractional 0-1, absolute coordinates, array format)
  - Configurable via environment variables or Django settings
- **Environment variable configuration**:
  - `LLAMAPARSE_API_KEY` / `LLAMA_CLOUD_API_KEY`: API key for LlamaParse authentication
  - `LLAMAPARSE_RESULT_TYPE`: Output type ("json", "markdown", "text") - default: "json"
  - `LLAMAPARSE_EXTRACT_LAYOUT`: Enable layout extraction with bounding boxes - default: True
  - `LLAMAPARSE_NUM_WORKERS`: Parallel processing workers - default: 4
  - `LLAMAPARSE_LANGUAGE`: Document language - default: "en"
  - `LLAMAPARSE_VERBOSE`: Enable verbose logging - default: False
- **Parser selection via environment variable**:
  - `PDF_PARSER`: Set to "llamaparse" or "docling" (default) to select default PDF parser
  - Location: `config/settings/base.py:740-765`
- **Comprehensive test suite** (`opencontractserver/tests/test_doc_parser_llamaparse.py`):
  - Tests for successful parsing with layout extraction
  - Tests for markdown mode without layout
  - Tests for bounding box format conversion (fractional, absolute, array)
  - Tests for annotation creation and token generation
  - Tests for error handling (missing API key, API errors, empty results)
  - Tests for configuration via settings and kwargs override

#### Thread/Message Triggered Corpus Actions for Automated Moderation

- **Extended CorpusActionTrigger enum** with `NEW_THREAD` and `NEW_MESSAGE` triggers (`opencontractserver/corpuses/models.py:849-854`) to enable automated moderation of discussion threads
- **New moderation tools** (`opencontractserver/llms/tools/moderation_tools.py`): 9 tools for thread moderation including:
  - `get_thread_context`: Retrieve thread metadata (title, creator, lock/pin status)
  - `get_thread_messages`: Get recent messages for context
  - `get_message_content`: Get full content of a specific message
  - `delete_message`: Soft delete a message with audit logging
  - `lock_thread`/`unlock_thread`: Control thread access
  - `add_thread_message`: Post agent messages to threads
  - `pin_thread`/`unpin_thread`: Feature important threads
- **New MODERATION tool category** (`opencontractserver/llms/tools/tool_registry.py:42`) with 9 registered tools and proper approval requirements
- **Signal handlers** for thread/message creation (`opencontractserver/corpuses/signals.py`) using `transaction.on_commit` pattern to trigger corpus actions
- **New Celery tasks** (`opencontractserver/tasks/corpus_tasks.py`):
  - `process_thread_corpus_action`: Processes actions when threads are created
  - `process_message_corpus_action`: Processes actions when messages are posted
- **Agent thread action task** (`opencontractserver/tasks/agent_tasks.py:run_agent_thread_action`): Runs AI agents with thread context and moderation tools
- **Updated CorpusActionExecution model** (`opencontractserver/corpuses/models.py`) with optional `conversation` and `message` FKs for audit trail
- **Updated AgentActionResult model** (`opencontractserver/agents/models.py`) with nullable document FK and new `triggering_conversation`/`triggering_message` FKs
- **Frontend updates** (`frontend/src/components/corpuses/CreateCorpusActionModal.tsx`):
  - Added "On New Thread" and "On New Message" trigger options
  - Thread/message triggers automatically select agent action type
  - Info message explaining available moderation tools
- **Comprehensive test coverage**:
  - Backend tests: `opencontractserver/tests/test_thread_corpus_actions.py`
  - Frontend tests: `frontend/tests/create-corpus-action-modal.ct.tsx`
- **Database migrations**:
  - `opencontractserver/agents/migrations/0008_add_thread_message_triggers.py`: Adds nullable `triggering_conversation` and `triggering_message` FKs to AgentActionResult, makes `document` nullable
  - `opencontractserver/corpuses/migrations/0032_add_thread_message_triggers.py`: Adds nullable `conversation` and `message` FKs to CorpusActionExecution

#### Use Cases Enabled

- Automated content moderation (e.g., auto-delete messages with prohibited content)
- Thread management (e.g., auto-lock threads discussing prohibited topics)
- Automated responses (e.g., welcome messages for new threads)
- Content classification (e.g., auto-pin important announcements)

#### Proactive Apollo Cache Management System (PR #725)

- **New `CacheManager` service** (`frontend/src/services/cacheManager.ts`): Centralized Apollo cache management with debouncing, targeted invalidation, and auth-aware cache operations
  - `resetOnAuthChange()`: Full cache clear with optional refetch for login/logout transitions
  - `refreshActiveQueries()`: Soft refresh without clearing cache
  - `invalidateEntityQueries()`: Targeted invalidation for document/corpus/annotation CRUD operations
  - Debouncing: 1000ms for full resets, 500ms for entity invalidations
  - Debug utilities: `logCacheSize()`, `extractCacheForDebug()`
- **New `useCacheManager` hook** (`frontend/src/hooks/useCacheManager.ts`): React hook with memoized CacheManager instance and stable callback references
- **Comprehensive test suite** (`frontend/src/services/__tests__/cacheManager.test.ts`, `frontend/src/hooks/__tests__/useCacheManager.test.tsx`): 30+ tests covering debouncing, error handling, lifecycle, singleton management, and auth scenarios

### Technical Details

#### LlamaParse Parser Architecture

- Uses `llama-parse` library for API communication
- JSON mode with `extract_layout=True` provides bounding boxes as fractions of page dimensions (0-1)
- Converts LlamaParse layout elements to OpenContracts structural annotations
- Generates PAWLS tokens by splitting text into words and distributing across bounding box
- Element type mapping converts LlamaParse labels (title, paragraph, table, etc.) to OpenContracts annotation labels
- Falls back to text extraction mode when layout extraction is disabled

#### Markdown Link Tool Implementation

- Follows OpenContracts routing patterns from `docs/frontend/routing_system.md`
- Uses `select_related()` to minimize database queries (single query per entity)
- Handles both standalone documents and corpus-based document contexts
- Entity validation with clear error messages for IDOR prevention
- URL patterns match frontend `navigationUtils.ts` for consistency

### Fixed

#### Token Expiration Signal to Frontend (Issue #744)

- **Fixed `Auth0RemoteUserJSONWebTokenBackend.authenticate()` swallowing `JSONWebTokenExpired` exceptions** (`config/graphql_auth0_auth/backends.py:44-52`):
  - Previously, when a JWT token expired, the authentication backend caught all exceptions and returned `None`
  - The GraphQL layer then returned a generic "User is not authenticated" error
  - Frontend's `errorLink.ts` could not detect token expiration and trigger automatic refresh
  - Fix: Re-raise `JSONWebTokenExpired` so the GraphQL layer returns "Signature has expired"
  - Frontend now correctly detects expiration and triggers page reload for silent token refresh
- **Enhanced WebSocket middleware with auth error signaling** (`config/websocket/middleware.py:44-124`):
  - Added `scope["auth_error"]` dict with `code` and `message` fields
  - New close codes: `WS_CLOSE_TOKEN_EXPIRED` (4001), `WS_CLOSE_TOKEN_INVALID` (4002)
  - Consumers can now close connections with specific codes for frontend handling
- **Enhanced Auth0 WebSocket middleware** (`config/websocket/middlewares/websocket_auth0_middleware.py:52-130`):
  - Added consistent `scope["auth_error"]` handling for Auth0 tokens
  - Matches close code behavior with non-Auth0 middleware
- **New test coverage** (`opencontractserver/tests/test_token_expiration.py`):
  - Tests for `Auth0RemoteUserJSONWebTokenBackend` token expiration re-raising
  - Tests for WebSocket middleware auth error handling
  - Tests for WebSocket close code consistency

#### Independent Structural Annotation and Show Selected Controls (Issue #735)

- **Removed forced coupling between structural and showSelectedOnly controls** (`frontend/src/components/annotator/controls/AnnotationControls.tsx:200-207`):
  - Previously, enabling "Show Structural" would force "Show Only Selected" to be checked and disabled
  - Users can now toggle "Show Only Selected" independently when structural annotations are visible
  - All combinations now work:
    - Show all structural annotations: structural ON, selectedOnly OFF
    - Show only selected structural annotation: structural ON, selectedOnly ON
    - Hide all structural annotations: structural OFF
- **Updated checkbox onChange handler** (`frontend/src/components/annotator/controls/AnnotationControls.tsx:268`): Now correctly extracts `data?.checked ?? false` for consistency with other toggle handlers
- **Updated component tests** (`frontend/tests/FloatingDocumentControls.ct.tsx:263-371`):
  - Renamed test to reflect new independent behavior
  - Added new test verifying controls can be toggled independently
- **Note**: Users who previously had `showStructural: true` will notice different behavior: the "Show Only Selected" control now respects their actual preference instead of being forced to true

#### Cache Management Race Condition Fix (PR #725)

- **Auth state now set BEFORE cache clear** (`frontend/src/components/auth/AuthGate.tsx:69-92`, `frontend/src/views/Login.tsx:106-117`, `frontend/src/components/layout/useNavMenu.ts:64-90`):
  - Previously, cache was cleared before updating auth state, creating a window where queries could fetch with wrong auth context
  - Fixed by setting auth token/user/status first, then clearing cache
  - Refetched queries now correctly use the new auth context
- **AuthGate uses useCacheManager hook** (`frontend/src/components/auth/AuthGate.tsx:7,27`): Replaced direct `new CacheManager()` instantiation with proper hook usage, eliminating `as any` type assertion and ensuring memoization
- **Fire-and-forget logout cache clear** (`frontend/src/components/layout/useNavMenu.ts:69-79`): Logout no longer blocks on cache clear operation, improving perceived performance

### Technical Details

#### Cache Management Architecture

- **Race condition prevention**: Auth state updates are synchronous; cache clear is async. By setting auth first, any queries triggered during cache clear use the correct credentials.
- **Singleton pattern preserved for non-React contexts**: The singleton functions (`initializeCacheManager`, `getCacheManager`, etc.) remain exported for testing and non-React usage, with documentation clarifying when to use hooks vs singleton.
- **Dependency management**: `useCacheManager` hook returns stable callback references via `useCallback`, safe to include in effect dependencies.

### Fixed

#### Mobile Responsive Styling for Settings and Badge Widgets (Issue #690)

- **Badge component z-index optimization** (`frontend/src/components/badges/Badge.tsx:47,107`): Lowered z-index values from 9999/10000 to 200/201 to avoid conflicts with other UI elements while maintaining proper layering
- **Unified mobile behavior detection** (`frontend/src/components/badges/Badge.tsx:148-152`): Combined touch device detection with viewport width check to ensure mobile UX works consistently across real devices and test environments
- **Test wrapper extraction** (`frontend/tests/UserBadgesTestWrapper.tsx`, `frontend/tests/GlobalSettingsPanelTestWrapper.tsx`): Moved test wrappers to separate files following Playwright component testing best practices
- **Improved test reliability** (`frontend/tests/mobile-responsive.ct.tsx`): Fixed element disambiguation issues using proper locator strategies

#### Agent Chat Processing Indicator (PR #687)

- **Added visual feedback for agent processing** (`frontend/src/components/widgets/chat/ChatMessage.tsx:1342-1405`): When an agent starts processing a response, an animated "Agent is thinking..." indicator now displays instead of an empty message bubble
- **Processing indicator conditions**: Shows when assistant message is incomplete with no content and no timeline entries
- **Accessibility improvements**: Added ARIA attributes (`role="status"`, `aria-live="polite"`, `aria-label`) for screen reader support
- **Animation performance**: Added `will-change: transform, opacity` to animated dots for smoother rendering
- **Component tests**: Added comprehensive Playwright component tests (`frontend/tests/chat-message-processing-indicator.ct.tsx`) covering indicator visibility, accessibility, and state transitions

### Fixed

#### Trash View Error Prevention (Issue #691)

- **State synchronization fix** (`frontend/src/components/corpuses/folders/FolderTreeSidebar.tsx:363-369`):
  - Fixed trash folder click handler to use consistent state update pattern matching other folder navigation
  - Added `handleTrashClick` callback that properly delegates to `onFolderSelect` when provided (URL-driven state)
  - Removed direct Jotai atom manipulation that caused race conditions with CentralRouteManager
- **Defensive null handling** (`frontend/src/components/corpuses/folders/TrashFolderView.tsx`):
  - Added `safeFormatDistanceToNow()` and `safeFormat()` helper functions for robust date formatting
  - Added optional chaining for `creator`, `document`, and nested properties to prevent runtime errors
  - Added validation in `handleRestoreSingle()` and `handleRestoreSelected()` to check for valid document data
- **Type safety improvements** (`frontend/src/graphql/queries/folders.ts:92-104`):
  - Updated `DeletedDocumentPathType` interface to mark `creator` and `document` as potentially null
  - Ensures TypeScript catches potential null access issues at compile time

### Added

#### Agent Message Visual Differentiation (Issue #688)

- **Enhanced MessageItem component** (`frontend/src/components/threads/MessageItem.tsx:27-50, 59-66, 68-191, 211-245, 461-466, 530-550`):
  - Agent detection logic using `getAgentDisplayData()` helper function
  - `hexToRgba()` utility for generating color-tinted backgrounds from agent badge colors
  - Distinct visual styling for agent messages vs user messages:
    - **Background**: Subtle gradient using agent's badge color with low opacity (8% to 3%)
    - **Border**: Colored border matching agent's badge color instead of default gray
    - **Accent strip**: 4px colored left border (like highlighted messages) using agent color
    - **Avatar**: Bot icon instead of User icon, with agent-colored gradient background
    - **Box shadow**: Agent-colored shadow on avatar for visual consistency
- **Accessibility improvements**:
  - Updated `aria-label` to include "(AI Agent)" suffix for screen readers
  - Avatar `title` attribute identifies agent name and type
- Agent color sourced from `AgentConfiguration.badgeConfig.color` field (falls back to default blue #4A90E2)

#### Network Recovery on Screen Unlock (Issue #697)

- **New `useNetworkStatus` hook** (`frontend/src/hooks/useNetworkStatus.ts`): Monitors page visibility and network status changes to detect when the app resumes from background (e.g., screen unlock on mobile)
- **New `NetworkStatusHandler` component** (`frontend/src/components/network/NetworkStatusHandler.tsx`): Automatically refetches active Apollo Client queries when:
  - The page becomes visible after being hidden (screen unlock on mobile)
  - The network comes back online after being offline
- **WebSocket reconnection on resume**: Updated `useThreadWebSocket` and `useAgentChat` hooks to reconnect WebSockets when the page becomes visible
- **Toast notifications**: Informs users of connectivity changes ("Reconnecting...", "Connection restored", "You appear to be offline")

### Technical Details

#### Real-Time Notification System Architecture (Issue #637)

- **Security**: User-specific channel groups (`notification_user_{user_id}`) prevent IDOR and cross-user data leakage
- **Performance optimizations**:
  - User-specific WebSocket channels (not global broadcast)
  - Efficient `bulk_create()` for thread participant notifications
  - Exponential backoff on reconnection failures (2s → 4s → 8s → 16s, max 8x)
  - Heartbeat monitoring every 30s to detect stale connections
- **Signal integration**: All notification types (BADGE, REPLY, MENTION, THREAD_REPLY, moderation) automatically broadcast via `broadcast_notification_via_websocket()` helper
- **Error handling**: WebSocket broadcast failures don't break signal handlers - notifications still save to database
- **Connection lifecycle**: Auto-reconnection on network recovery, page visibility change, and authentication token refresh
- **Testing**: Comprehensive test suite (`opencontractserver/tests/test_notification_websocket.py`) covering authentication, IDOR prevention, concurrent connections, and signal integration
- **Network monitoring**: Integrated with `useNetworkStatus` hook for automatic reconnection on mobile screen unlock and network recovery

#### Network Recovery Implementation

- Uses `visibilitychange` event to detect page visibility changes
- Uses `online`/`offline` events to detect network status changes
- Configurable resume threshold (default 2s for NetworkStatusHandler, 1s for WebSocket hooks)
- Debounced refetch to prevent rapid repeated calls
- Graceful degradation: continues to work if events are not supported

#### Upload Modal Styling Improvements (Issue #696)

- **New styled components for upload modals** (`frontend/src/components/widgets/modals/UploadModalStyles.ts`): Comprehensive styled-components library with 25+ responsive components including `StyledUploadModal`, `DropZone`, `StepIndicator`, `FileListItem`, and more
- **Step indicator UI** for DocumentUploadModal showing progress through upload workflow (Select → Details → Corpus)
- **Modern gradient header** with icon and subtitle for both upload modals
- **Progress bar integration** showing real-time upload progress with success/error states

#### Mobile UI Improvements for Picker and Edit Message Modal (Issue #686)

- **Backend UpdateMessage mutation** (`config/graphql/conversation_mutations.py:455-619`):
  - New `UpdateMessageMutation` for editing existing thread messages
  - Validates CRUD permission on message or moderator status
  - Re-parses mentions when content is updated (with race condition protection - parsing happens before DB modifications)
  - Triggers agent responses for newly mentioned agents
  - Documented behavior: agents respond to ALL mentions, including re-mentions in edited messages
- **Frontend UPDATE_MESSAGE mutation** (`frontend/src/graphql/mutations.ts:2726-2760`): GraphQL mutation with TypeScript types
- **EditMessageModal component** (`frontend/src/components/threads/EditMessageModal.tsx`):
  - Full-screen modal on mobile for better touch interaction
  - Uses MessageComposer for consistent editing experience
  - Safe area insets for notched devices
  - Loading states and error handling
  - Custom unsaved changes confirmation modal (replaces browser `window.confirm()`)
  - Debounced content updates (150ms) for improved performance during typing
  - XSS protection documented: uses MarkdownMessageRenderer with `rehype-sanitize`
- **Message actions dropdown in MessageItem** (`frontend/src/components/threads/MessageItem.tsx:219-432`):
  - Desktop: Standard dropdown menu with Edit/Delete options
  - Mobile: Bottom sheet style for thumb-friendly interaction
  - Inline delete confirmation with mobile-optimized buttons
  - Backdrop overlay on mobile for visual focus

#### Improved Inline Reference Cards for Mentions (Issue #689)

- **Annotation mentions** now display the first ~24 characters of annotation text instead of cryptic IDs
  - Full annotation text accessible via hover tooltip
  - Falls back to label type if no raw text available
  - Location: `frontend/src/components/threads/MentionChip.tsx:212-229`
- **Document mentions** show document title with corpus context (e.g., "Document Title (in Corpus Name)")
  - Location: `frontend/src/components/threads/MessageComposer.tsx:361-375`
- **Corpus mentions** show corpus name instead of `@corpus:slug` format
  - Location: `frontend/src/components/threads/MessageComposer.tsx:351-359`
- **Shared constant** `MENTION_PREVIEW_LENGTH = 24` for consistent truncation across components
  - Location: `frontend/src/assets/configurations/constants.ts:6-8`
- **Text sanitization utility** for user-generated content to prevent XSS
  - Location: `frontend/src/utils/textSanitization.ts`
  - Unit tests: `frontend/src/utils/textSanitization.test.ts`
- **Component tests** for MentionChip covering all resource types and text truncation
  - Location: `frontend/tests/mention-chip.spec.tsx`

### Changed

#### Upload Modal Mobile Responsiveness (Issue #696)

- **DocumentUploadModal** (`frontend/src/components/widgets/modals/DocumentUploadModal.tsx`): Refactored to use new styled components with responsive grid layout for edit step
- **BulkUploadModal** (`frontend/src/components/widgets/modals/BulkUploadModal.tsx`): Complete visual overhaul with styled drop zone, file size display, and responsive layout
- **DocumentUploadList** (`frontend/src/components/documents/DocumentUploadList.tsx`): New drop zone styling with drag-active feedback and pulse animation
- **DocumentListItem** (`frontend/src/components/documents/DocumentListItem.tsx`): Improved file list items with proper touch targets (56px min-height, 64px on mobile), status icons, and delete button styling
- **Mobile-first breakpoints**: All upload modal components now have explicit breakpoints at 480px (mobile) and 768px (tablet)
- **Touch target compliance**: All interactive elements meet 44px minimum touch target size for mobile accessibility
- **Responsive action buttons**: Modal actions stack vertically on mobile for full-width tappable buttons
- **Custom scrollbar styling**: File list has styled scrollbars for visual polish

#### MentionChip Component Improvements (Issue #689)

- Extended `MentionChip` to support ANNOTATION type with green gradient styling
- Added default cases to all switch statements for TypeScript exhaustiveness checking
- Refactored `handleClick` to `handleActivation` accepting `React.MouseEvent | React.KeyboardEvent` union type (fixes unsafe `as any` assertion)
- Sanitized user-generated annotation text before display to prevent XSS

### Fixed

#### Mobile Layout for Picker Components (Issue #686)

- **Picker keyboard handling** (`MentionPicker.tsx:22-54`, `UnifiedMentionPicker.tsx:25-57`):
  - Added CSS environment variables (`env(safe-area-inset-bottom)`) for keyboard-aware positioning
  - Smooth slide-up animation for picker appearance
  - Max-height constraints using `min()` to prevent overflow on small screens
- **Touch targets** (`MentionPicker.tsx:83-108`, `UnifiedMentionPicker.tsx:96-108`):
  - Increased touch target size (52-60px min-height) for easier selection
  - Larger font size (15px) on mobile for readability
  - Mobile-specific border radius for rounded corners
- **MessageComposer mobile improvements** (`MessageComposer.tsx:48-93`):
  - Larger toolbar button touch targets (40x40px) on mobile
  - Increased gap between buttons for easier tapping

### Technical Details

#### Message Editing Tests (Issue #686)

- **New test for parent relationship preservation** (`opencontractserver/tests/test_conversation_mutations_graphql.py:1071-1168`):
  - Verifies that editing a reply message preserves its `parent_message` field
  - Ensures thread structure integrity when users edit replies
  - Part of comprehensive UpdateMessage mutation test suite

#### Upload Modal Architecture

- Styled-components with transient props (`$active`, `$selected`, `$status`) to prevent DOM attribute warnings
- CSS keyframe animations for drag-active pulse effect and fade-in modal transitions
- Gradient backgrounds using `linear-gradient(135deg, #667eea 0%, #764ba2 100%)` for visual consistency
- Semantic UI React components wrapped with styled-components for enhanced styling while preserving functionality

#### Permanent Deletion (Empty Trash) Functionality (PR #707)

- **Core deletion logic** (`opencontractserver/documents/versioning.py:617-760`):
  - `permanently_delete_document()`: Irreversible deletion with cascade cleanup
  - `permanently_delete_all_in_trash()`: Bulk deletion (empty trash) with partial success support
- **Cascade cleanup** deletes:
  - All DocumentPath records for the document in the corpus (entire history)
  - User annotations (non-structural) on the document
  - Relationships involving those annotations (uses Q objects to avoid duplicate counting)
  - DocumentSummaryRevision records for the document+corpus
  - The Document itself if no other corpus references it (Rule Q1)
- **Service layer** (`opencontractserver/corpuses/folder_service.py:1096-1181`): Permission-checked wrappers
- **GraphQL mutations** (`config/graphql/mutations.py:4069-4187`):
  - `PermanentlyDeleteDocument`: Delete single soft-deleted document
  - `EmptyTrash`: Delete all soft-deleted documents in corpus
  - Both enforce DELETE permission via django-guardian
- **Frontend UI** (`frontend/src/components/corpuses/folders/TrashFolderView.tsx`):
  - "Empty Trash" button with confirmation modal
  - Warning message explaining what will be permanently deleted
  - Auto-dismiss success/error messages with configurable durations
  - TypeScript type safety for all mutation responses
- **Comprehensive test suite** (`opencontractserver/tests/test_permanent_deletion.py`): 34 tests covering core logic, cascade cleanup, Rule Q1, permissions, GraphQL mutations, and edge cases

### Technical Details

- Partial deletions are allowed in bulk operations (each document deletion is atomic)
- Structural annotations are preserved (shared via StructuralAnnotationSet)
- Corpus-isolated deletion: Only affects target corpus, other corpus references preserved
- Composite index `[corpus, is_current, is_deleted]` on DocumentPath for efficient trash queries

#### Mobile-Friendly Corpus Modal

- **New CorpusModal component** (`frontend/src/components/corpuses/CorpusModal.tsx`): Purpose-built modal replacing CRUDModal for corpus create/edit/view operations with mobile-first design
- **13 comprehensive component tests** (`frontend/tests/corpus-modal.ct.tsx`): Full test coverage for all modal modes and interactions
- **Smart change detection for EDIT mode**: Only sends changed fields to backend using original value comparison (`CorpusModal.tsx:498-519`)
- **ARIA accessibility**: CloseButton includes `aria-label="Close modal"` for screen reader users

### Changed

#### Corpus Modal Architecture

- **Replaced CRUDModal with CorpusModal**: Simplified form handling with controlled inputs instead of complex JSON Schema Form library
- **Removed debug console.log statements** (`Corpuses.tsx`): Cleaned up 4 debug logging statements

### Technical Details

#### Corpus Modal Implementation

- Mobile-first responsive design: 16px input font prevents iOS auto-zoom, 48px min touch targets
- Proper TypeScript types: Icon type is `string | null` (not ArrayBuffer), slug field uses existing type from RawCorpusType
- isDirty computed by comparing current values against stored original values (not just tracking changes)

#### Social Media Preview (OG Metadata) System (PR #701)

- **Cloudflare Worker for social media previews** (`cloudflare-og-worker/`): Intercepts requests from social media crawlers (Facebook, Twitter, LinkedIn, Discord, Slack, etc.) and returns HTML with Open Graph meta tags for rich link previews
- **Public OG metadata GraphQL queries** (`config/graphql/queries.py:3235-3403`): New unauthenticated queries for fetching public corpus, document, thread, and extract metadata
  - `ogCorpusMetadata`: Returns title, description, icon, document count for public corpuses
  - `ogDocumentMetadata`: Returns title, description, icon for public standalone documents
  - `ogDocumentInCorpusMetadata`: Returns document metadata with corpus context
  - `ogThreadMetadata`: Returns discussion thread metadata (title, corpus, message count)
  - `ogExtractMetadata`: Returns data extract metadata
- **Worker architecture**: Modular TypeScript implementation with crawler detection, URL parsing, metadata fetching, and HTML generation
- **Comprehensive documentation** (`docs/architecture/social-media-previews.md`): Architecture overview, deployment guide, and testing instructions

### Fixed

#### New Corpus Modal Mobile Issues (Issue #702)

- **Mobile form data loss in CorpusModal** (`frontend/src/components/corpuses/CorpusModal.tsx:406-418`):
  - Fixed fields clearing when typing on mobile by tracking modal open transitions with `prevOpenRef` instead of resetting form on every render
  - The original `useEffect` was running on every `corpus` or `open` change, causing form state to reset during keyboard/focus events on mobile
- **Slow embedder loading** (`frontend/src/components/widgets/CRUD/EmbedderSelector.tsx:43-46`):
  - Changed Apollo query to `cache-first` policy since embedders rarely change
  - Prevents unnecessary network requests when reopening CorpusModal
- **Cramped mobile layout** (`frontend/src/components/corpuses/CorpusModal.tsx:327-333`, `frontend/src/components/widgets/file-controls/FilePreviewAndUpload.tsx:54-57,129-135`):
  - Reduced icon upload area max-width from 200px to 150px on mobile
  - Reduced ImagePreview height from 150px to 100px on mobile
  - Made EditBadge smaller and better positioned on mobile viewports

#### Production Deployment

- **Missing COLLECTFAST_STRATEGY for GCP storage backend** (`config/settings/base.py:436`): Added `collectfast.strategies.gcloud.GoogleCloudStrategy` for GCP deployments. Previously, `collectfast` was installed in production but `COLLECTFAST_STRATEGY` was only configured for AWS, causing `collectstatic` to fail with `ImproperlyConfigured: No strategy configured` error when using `STORAGE_BACKEND=GCP`.
- **GCS static files ACL incompatible with uniform bucket-level access** (`opencontractserver/utils/storages.py:38`): Changed `StaticRootGoogleCloudStorage.default_acl` from `"publicRead"` to `None`. GCS buckets with uniform bucket-level access enabled cannot use per-object ACLs; access must be controlled via IAM policies at the bucket level instead.

#### Social Media Preview Security & Performance Fixes (PR #701 remediation)

- **Prevented potential infinite loop in worker passthrough** (`cloudflare-og-worker/src/index.ts:23-42`): Added `passToOrigin()` helper function with `X-OG-Worker-Pass` header to prevent Cloudflare Worker from re-invoking itself on route-based deployments
- **Added rate limiting to public OG queries** (`config/graphql/queries.py`): All five OG metadata resolvers now have `@graphql_ratelimit(key="ip", rate="60/m", group="og_metadata")` to prevent abuse and DoS attacks
- **Fixed N+1 query in corpus document count** (`config/graphql/queries.py:3250-3255`): Changed from `corpus.documents.count()` to `Corpus.objects.annotate(doc_count=Count("documents"))` for single-query optimization
- **Fixed N+1 query in thread message count** (`config/graphql/queries.py:3359-3364`): Changed from `thread.messages.count()` to `Conversation.objects.annotate(msg_count=Count("messages"))` for single-query optimization
- **Added input validation for decodeURIComponent** (`cloudflare-og-worker/src/parser.ts:88-95`): Wrapped `decodeURIComponent()` in try-catch to handle malformed URLs gracefully instead of crashing the worker
- **Unified description truncation** (`config/graphql/queries.py`): Removed redundant Python-side `[:500]` truncation; description truncation now handled solely by the worker at 200 characters for consistency

### Added

#### Mobile UI/UX Improvements for Corpus Navigation

- **Mobile-first folder sidebar defaults**: Sidebar now collapses by default on mobile/tablet devices (≤768px) to maximize document viewing area
- **Mobile bottom-sheet mention pickers**: User, resource, and unified mention pickers now display as bottom sheets on mobile (≤600px) for thumb-friendly interaction
- **Discussions and Analytics quick access**: Added icon buttons to CorpusHome stat cards for direct navigation to Discussions and Analytics tabs
- **Sidebar auto-close behavior**: Folder sidebar automatically closes on mobile/tablet after folder selection for seamless navigation
- **Mobile sidebar backdrop overlay**: Semi-transparent backdrop behind mobile sidebar for visual focus and easy dismissal
- **Escape key accessibility**: Mobile sidebar can now be dismissed with Escape key for keyboard accessibility
- **TABLET_BREAKPOINT constant**: Added to `constants.ts` for consistent responsive breakpoint management across components

### Fixed

#### Mobile UI/UX Fixes

- **Settings button variable name bug** (`frontend/src/components/corpuses/CorpusHome.tsx:780`): Fixed `canUpdate` → `canEdit` reference error that prevented Settings button from displaying for users with update permissions
- **FAB z-index layering** (`frontend/src/views/Corpuses.tsx:1320`): Raised FAB z-index from 100 to 150 to ensure visibility above folder sidebar toggle (z-index: 101)
- **Explicit z-index layering**: Made mobile sidebar z-index layering explicit (backdrop: 98, toggle button: 99) to prevent fragile DOM-order-dependent behavior

#### Mobile Responsive Styling for Settings and Badge Widgets (PR #690)

- **UserSettingsModal responsive styling** (`frontend/src/components/modals/UserSettingsModal.tsx:14-80`):
  - Modal takes 95% width on mobile (≤768px) with reduced padding
  - Form groups stack vertically on small screens (≤480px) for single-column layout
  - Action buttons display full-width and stack vertically (Save above Close) on mobile
  - Added `styled-components` import and styled wrapper components
- **Badge component touch support** (`frontend/src/components/badges/Badge.tsx:23-41, 96-112, 145-199`):
  - Added tap-to-toggle tooltip on touch devices (detects via `ontouchstart`)
  - Created `MobileOverlay` backdrop for dismissing badge popups by tapping outside
  - Popup centers on mobile screens using fixed positioning instead of floating-ui
  - Increased touch target size (min-height 36px, larger padding)
  - Disabled hover transforms on touch devices using `@media (hover: none)`
- **UserBadges container responsive layout** (`frontend/src/components/badges/UserBadges.tsx:18-27, 37-48, 58-61`):
  - Reduced padding and gap on mobile viewports
  - Badges center-aligned on mobile for better visual balance
  - Empty state and header text sizes reduced on mobile
- **GlobalSettingsPanel responsive grid** (`frontend/src/components/admin/GlobalSettingsPanel.tsx:11-67, 82-104, 119-123, 137-139, 148-150, 163-168`):
  - Container padding reduced on mobile (2rem → 1rem → 0.75rem)
  - Settings grid switches to single column on small mobile (≤480px)
  - Card content padding reduced progressively on smaller screens
  - Touch-friendly card interactions with active state feedback (scale 0.98)
  - "Coming Soon" badge displays on its own line on very small screens

### Changed

#### Mobile UI/UX Refactoring

- **Hardcoded breakpoints replaced with constants**: Updated all hardcoded `768px` references in `FolderDocumentBrowser.tsx` and `folderAtoms.ts` to use `TABLET_BREAKPOINT` constant for maintainability
- **Improved breakpoint documentation**: Added detailed JSDoc comment in `folderAtoms.ts` explaining why `TABLET_BREAKPOINT` (768px) is used for sidebar collapse rather than `MOBILE_VIEW_BREAKPOINT` (600px)

## [3.0.0.b3] - 2025-12-11

### Added

#### v3.0.0.b3 Migration Tools (Issue #654)

- **New management command: `validate_v3_migration`**

  - Pre-flight and post-migration validation for dual-tree versioning and structural annotations
  - Checks: version_tree_id, is_current, DocumentPath records, XOR constraints, structural set uniqueness
  - Reports structural migration candidates
  - Options: `--verbose`, `--fix`
  - Location: `opencontractserver/documents/management/commands/validate_v3_migration.py`

- **New management command: `migrate_structural_annotations`**

  - Optional command to migrate structural annotations to shared StructuralAnnotationSet objects
  - Creates StructuralAnnotationSet by content hash (pdf_file_hash) for storage efficiency
  - Moves structural annotations/relationships from document FK to structural_set FK
  - Documents with same hash share StructuralAnnotationSet (O(1) storage vs O(n))
  - Options: `--dry-run`, `--document-id`, `--corpus-id`, `--batch-size`, `--verbose`, `--force`
  - Location: `opencontractserver/annotations/management/commands/migrate_structural_annotations.py`

- **Comprehensive migration test suite** (`opencontractserver/tests/test_v3_migration.py`)

  - DocumentVersioningMigrationTests: version_tree_id, is_current, DocumentPath creation
  - XORConstraintTests: Annotation/Relationship XOR constraint validation
  - StructuralMigrationCommandTests: Management command functionality, idempotency
  - RollbackAndEdgeCaseTests: Edge cases, error handling, data integrity
  - ValidationCommandTests: validate_v3_migration command testing
  - 25+ human-readable tests covering all migration scenarios

- **Migration documentation** (`docs/migrations/v3_upgrade_guide.md`)
  - Pre-upgrade checklist with backup recommendations
  - Step-by-step migration instructions for production and development
  - Optional structural annotation migration guide
  - Rollback procedure documentation
  - FAQ addressing common concerns (XOR constraint safety, storage savings, incremental migration)

#### Discovery Landing Page (New)

- **Beautiful, modern landing page** as the main entry point for the application

  - Replaces direct redirect to /corpuses with a unified discovery experience
  - Different content for anonymous vs authenticated users
  - Responsive design with mobile-first approach
  - Location: `frontend/src/views/DiscoveryLanding.tsx`

- **New landing page components** (`frontend/src/components/landing/`)

  - `HeroSection.tsx`: Animated hero with gradient backgrounds, floating icons, and global search
  - `StatsBar.tsx`: Community metrics display with animated counters (users, collections, documents, threads, annotations, weekly active)
  - `TrendingCorpuses.tsx`: Card grid of popular document collections with engagement metrics
  - `RecentDiscussions.tsx`: List of recent public discussions with badges for pinned/locked threads
  - `TopContributors.tsx`: Leaderboard-style display of top community contributors with reputation scores
  - `CallToAction.tsx`: Conversion section for anonymous users with feature highlights
  - All components feature modern UI/UX: glass morphism, smooth Framer Motion animations, skeleton loaders

- **GraphQL queries for discovery data** (`frontend/src/graphql/landing-queries.ts`)

  - `GET_DISCOVERY_DATA`: Unified query fetching corpuses, conversations, community stats, and leaderboard
  - `GET_TRENDING_CORPUSES`: Public corpuses with engagement metrics
  - `GET_RECENT_DISCUSSIONS`: Recent threads with pagination
  - `GET_COMMUNITY_STATS`: Platform-wide statistics
  - `GET_GLOBAL_LEADERBOARD`: Top contributors with badges

- **Route integration**

  - Root path (`/`) now displays DiscoveryLanding instead of redirecting to /corpuses
  - Location: `frontend/src/App.tsx:377-382`

- **Component tests** (`frontend/tests/landing-components.spec.tsx`)
  - HeroSection tests: rendering, authenticated/anonymous variants, search submission
  - StatsBar tests: stats rendering, loading state, null handling
  - TrendingCorpuses tests: corpus cards, loading skeletons, empty state
  - RecentDiscussions tests: discussion items, pinned badges, reply counts
  - TopContributors tests: contributor cards, reputation scores, leaderboard button
  - CallToAction tests: anonymous visibility, authenticated hiding
  - DiscoveryLanding integration tests: full page rendering, section visibility

#### Permission Audit Remediation - Query Optimizers

- **New `UserQueryOptimizer`** for centralized user profile visibility logic

  - Respects `is_profile_public` privacy setting
  - Private profiles visible via corpus membership with > READ permission
  - Inactive users filtered out (except for superusers)
  - IDOR-safe visibility checks
  - Location: `opencontractserver/users/query_optimizer.py`

- **New `BadgeQueryOptimizer`** for centralized badge visibility logic

  - Badge visibility follows recipient's profile privacy rules
  - Corpus-specific badges visible only to corpus members
  - Own badges always visible regardless of privacy
  - IDOR-safe visibility checks
  - Location: `opencontractserver/badges/query_optimizer.py`

- **New `DocumentActionsQueryOptimizer`** for document-related actions

  - Centralized permission logic for corpus actions, extracts, and analysis rows
  - Follows least-privilege model: `Effective Permission = MIN(document_permission, corpus_permission)`
  - Integrates with ExtractQueryOptimizer and AnalysisQueryOptimizer
  - Location: `opencontractserver/documents/query_optimizer.py`

- **Comprehensive permission test suites** (40 tests total)

  - `opencontractserver/tests/permissioning/test_user_visibility.py` - 16 tests for user profile visibility
  - `opencontractserver/tests/permissioning/test_badge_visibility.py` - 13 tests for badge visibility
  - `opencontractserver/tests/permissioning/test_document_actions_permissions.py` - 11 tests for document actions

- **Updated permissioning documentation**
  - Added Section 8: User Profile and Badge Visibility
  - Added Section 9: Document Actions Permissions
  - Added callouts for new privacy features
  - Updated Key Changes table with new optimizer rows
  - Location: `docs/permissioning/consolidated_permissioning_guide.md`

#### Corpus Engagement Analytics Dashboard (Issue #579)

- **New CorpusEngagementDashboard component** displaying comprehensive engagement metrics

  - Thread metrics: total threads, active threads, average messages per thread
  - Message activity: total messages, 7-day and 30-day message counts with bar chart visualization
  - Community engagement: unique contributors, active contributors (30d), total upvotes
  - Auto-refresh every 5 minutes with last updated timestamp
  - Mobile-responsive design with conditional layouts and grid systems
  - Location: `frontend/src/components/analytics/CorpusEngagementDashboard.tsx`

- **GraphQL integration for engagement metrics**

  - New query: `GET_CORPUS_ENGAGEMENT_METRICS` with TypeScript interfaces
  - Leverages existing backend `CorpusEngagementMetrics` model (already tested)
  - Location: `frontend/src/graphql/queries.ts:3873-3979`

- **Analytics tab in Corpus view**

  - New tab with BarChart3 icon next to Discussions tab
  - Conditionally rendered based on corpus ID availability
  - Location: `frontend/src/views/Corpuses.tsx:2209-2216`

- **Dependencies**
  - Added recharts@3.4.1 for data visualization (BarChart, ResponsiveContainer, Tooltip, Legend)
  - Added react-countup for animated number counters

#### Thread Search UI (Issue #580)

- **Backend pagination support for conversation search**

  - Updated `searchConversations` resolver to use `relay.ConnectionField` with cursor-based pagination
  - Supports `first`, `after`, `last`, `before` parameters for efficient result pagination
  - Returns paginated structure with `edges`, `pageInfo`, and `totalCount`
  - Location: `config/graphql/queries.py:1659-1748`

- **GraphQL queries and TypeScript types with pagination**

  - Updated `SEARCH_CONVERSATIONS` query to support paginated results
  - Added pagination parameters: `first`, `after`, `last`, `before`
  - Enhanced TypeScript interfaces with connection structure (edges, nodes, cursors, pageInfo)
  - Includes full thread metadata: chatMessages count, isPinned, isLocked, corpus/document references
  - Location: `frontend/src/graphql/queries.ts:3923-4059`

- **New search components** (`frontend/src/components/search/`)

  - `SearchBar.tsx`: Search input with clear button and Enter key support
  - `SearchFilters.tsx`: Filter by conversation type with clear filters button
  - `SearchResults.tsx`: Results display with pagination, reuses ThreadListItem component
  - `ThreadSearch.tsx`: Main search container with debounced query (300ms) and pagination
  - All components follow existing design patterns and are mobile-responsive

- **Embedded search in Corpus Discussions view**

  - Added tab navigation to switch between "All Threads" and "Search"
  - Search scoped to current corpus when embedded
  - Location: `frontend/src/components/discussions/CorpusDiscussionsView.tsx`

- **Standalone /threads route**

  - New dedicated search page accessible at `/threads`
  - Global search across all accessible discussions
  - Location: `frontend/src/views/ThreadSearchRoute.tsx`, `frontend/src/App.tsx:421`

- **Backend tests for paginated search**

  - Tests verify pagination structure (edges, pageInfo, totalCount)
  - Tests verify cursor-based pagination with multiple pages
  - Location: `opencontractserver/tests/test_conversation_search.py:609-743`

- **Frontend component tests** (18 tests, 100% passing)

  - SearchBar component tests (5 tests): input rendering, search icon, clear button, Enter key submission
  - SearchFilters component tests (5 tests): filter rendering, option counting, selected state, clear filters button
  - SearchResults component tests (4 tests): loading state, empty state, no results state, results rendering
  - ThreadSearch component tests (4 tests): search bar integration, filters toggle, corpus-scoped search
  - Location: `frontend/tests/search-components.ct.tsx`

- **Enhanced backend test coverage for conversation search** (Issue #580 - Coverage Improvement)
  - Added `GraphQLResolverEdgeCasesTest` class with 8 new comprehensive tests
  - Tests cover GraphQL resolver edge cases including:
    - Default embedder path fallback when no corpus/document ID provided
    - Error handling when DEFAULT_EMBEDDER_PATH is not configured
    - Reverse pagination with `last` and `before` parameters
    - Multiple result handling and pagination behavior
    - Message search with various filter combinations
  - Coverage improvements target previously untested code paths in `config/graphql/queries.py:1711-1722, 1797-1808`
  - Location: `opencontractserver/tests/test_conversation_search.py:2666-3050`

#### Structural Annotation Sets (Phase 2.5)

- **New `StructuralAnnotationSet` model** for shared, immutable structural annotations

  - Content-hash based uniqueness (`content_hash` field)
  - Stores parser metadata (`parser_name`, `parser_version`, `page_count`, `token_count`)
  - Stores shared parsing artifacts (`pawls_parse_file`, `txt_extract_file`)
  - Location: `opencontractserver/annotations/models.py`

- **Document → StructuralAnnotationSet FK** with PROTECT on delete

  - Multiple corpus-isolated documents can share the same structural annotation set
  - Eliminates duplication of structural annotations across corpus copies
  - Location: `opencontractserver/documents/models.py:119-127`

- **Annotation.structural_set FK** with XOR constraint

  - Annotations now belong to EITHER a document OR a structural_set (not both, not neither)
  - Database constraint: `annotation_has_single_parent`
  - Location: `opencontractserver/annotations/models.py`

- **Relationship.structural_set FK** with XOR constraint

  - Same pattern as Annotation for relationships
  - Database constraint: `relationship_has_single_parent`
  - Location: `opencontractserver/annotations/models.py`

- **Database migrations**

  - `opencontractserver/annotations/migrations/0048_add_structural_annotation_set.py`
  - `opencontractserver/documents/migrations/0026_add_structural_annotation_set.py`

- **Comprehensive test suite** (32 tests)
  - `opencontractserver/tests/test_structural_annotation_sets.py` (22 tests)
  - `opencontractserver/tests/test_structural_annotation_portability.py` (10 tests)

### Fixed

#### Permission Audit Remediation - GraphQL Resolver Fixes

1. **User profile visibility not respecting privacy settings**

   - **File**: `config/graphql/queries.py` - `resolve_user_by_slug`, `resolve_search_users_for_mention`
   - **Issue**: Resolvers returned users without checking `is_profile_public` or corpus membership
   - **Fixed**: Now uses `UserQueryOptimizer` for proper privacy filtering
   - **Impact**: Private user profiles no longer visible to unauthorized users

2. **Badge visibility not respecting recipient privacy**

   - **File**: `config/graphql/queries.py` - `resolve_user_badges`, `resolve_user_badge`
   - **Issue**: Badge awards were visible regardless of recipient's profile privacy
   - **Fixed**: Now uses `BadgeQueryOptimizer` which filters by recipient visibility
   - **Impact**: Badges of private users no longer leaked to unauthorized viewers

3. **Document actions missing permission checks**

   - **File**: `config/graphql/queries.py` - `resolve_document_corpus_actions`
   - **Issue**: Inline permission checks were inconsistent with least-privilege model
   - **Fixed**: Now uses `DocumentActionsQueryOptimizer` for centralized permission logic
   - **Impact**: Document-related data properly filtered by document AND corpus permissions

4. **Assignment resolver using incorrect visible_to_user signature**

   - **File**: `config/graphql/queries.py` - `resolve_assignments`, `resolve_assignment`
   - **Issue**: Called `Assignment.objects.visible_to_user(info.context.user)` but manager expected different signature
   - **Fixed**: Updated to use correct manager method call pattern
   - **Impact**: Assignment queries now properly filter by user visibility

5. **Unused local imports shadowing top-level imports**
   - **File**: `config/graphql/queries.py` - lines 2810, 2990
   - **Issue**: Local `UserBadge` imports inside resolvers were redundant and caused flake8 warnings
   - **Fixed**: Removed redundant local imports, using top-level import
   - **Impact**: Cleaner code, no shadowing warnings

#### Thread Search (Issue #580)

6. **Anonymous user null reference in searchConversations resolver**
   - **File**: `config/graphql/queries.py:1725`
   - **Issue**: Resolver accessed `info.context.user.is_anonymous` without checking if user was `None`, causing AttributeError in tests with anonymous users
   - **Fixed**: Added null check before accessing `is_anonymous` attribute
   - **Impact**: Anonymous user search queries now work correctly without AttributeError

#### Critical Production Code Fixes

2. **Missing parsing artifacts in corpus copies**

   - **Files**: `opencontractserver/corpuses/models.py:445-451`, `opencontractserver/documents/versioning.py:238-244`
   - **Issue**: When creating corpus-isolated document copies, essential parsing artifacts were not being copied
   - **Fixed**: Added copying of `pawls_parse_file`, `txt_extract_file`, `icon`, `md_summary_file`, `page_count`
   - **Impact**: Corpus copies now have all parsing data needed for annotation, search, and display

3. **Missing `is_public` inheritance in corpus copies**

   - **Files**: `opencontractserver/corpuses/models.py:451`, `opencontractserver/documents/versioning.py:244`
   - **Issue**: Public documents became private when added to a corpus (copy didn't inherit `is_public`)
   - **Fixed**: Added `is_public=document.is_public` to corpus copy creation
   - **Impact**: Document visibility is now correctly preserved across corpus isolation

4. **NULL hash deduplication bug**

   - **File**: `opencontractserver/corpuses/models.py:414-425`
   - **Issue**: All documents without PDF content hashes were incorrectly treated as duplicates
   - **Fixed**: Added null check: `if document.pdf_file_hash is not None:` before hash-based deduplication
   - **Impact**: Documents without hashes are now correctly treated as distinct documents

5. **Structural annotation portability**

   - **Files**: `opencontractserver/corpuses/models.py:456`, `opencontractserver/documents/versioning.py:248`
   - **Issue**: Structural annotations were not traveling with documents when added to multiple corpuses
   - **Fixed**: Corpus copies now inherit `structural_annotation_set` from source document
   - **Impact**: Structural annotations are shared (not duplicated) across corpus-isolated copies

6. **GraphQL corpus.documents field missing**

   - **Files**: `config/graphql/graphene_types.py:1179-1184`, `config/graphql/graphene_types.py:1297-1302`
   - **Issue**: After corpus isolation migration (removing M2M documents field), GraphQL queries for `corpus.documents` returned empty because no explicit field declaration existed
   - **Fixed**: Added explicit `DocumentTypeConnection` class and `documents = relay.ConnectionField()` declaration to CorpusType
   - **Impact**: GraphQL queries now correctly resolve documents via DocumentPath-based relationships

7. **Parser `save_parsed_data()` using old M2M relationship**

   - **File**: `opencontractserver/pipeline/base/parser.py:126-133`
   - **Issue**: `save_parsed_data()` used deprecated `corpus.documents.add()` M2M method which no longer exists
   - **Fixed**: Updated to use `corpus.add_document(document=document, user=user)` for corpus isolation
   - **Impact**: Parsers can now correctly associate documents with corpuses during processing

8. **Document mention resolver using old M2M relationship**

   - **File**: `config/graphql/queries.py:976-1015`
   - **Issue**: `resolve_search_documents_for_mention()` queried via `corpus__in` M2M relationship which no longer exists
   - **Fixed**: Updated to query via `DocumentPath` with `is_current=True, is_deleted=False` filters
   - **Impact**: Document mention autocomplete now correctly finds documents in corpuses

9. **BaseFixtureTestCase not adding documents to corpus**
   - **File**: `opencontractserver/tests/base.py:385-399`
   - **Issue**: Test setup created corpus but didn't add fixture documents to it via DocumentPath
   - **Fixed**: Added loop to call `corpus.add_document()` for each fixture document and update references to corpus copies
   - **Impact**: WebSocket and other tests now properly test with documents in corpus context

### Changed

#### Test Suite Updates for Corpus Isolation Architecture

- **Removed deprecated legacy manager tests**

  - **File**: `opencontractserver/tests/test_document_path_migration.py`
  - **Removed**: Test classes for deprecated `DocumentCorpusRelationshipManager` (20+ tests)
  - **Reason**: The backward compatibility M2M manager was removed in Issue #654 Phase 2
  - **Note**: `DocumentCorpusRelationshipManager` in `opencontractserver/documents/managers.py` remains as documentation but is unused
  - **Impact**: Improved test clarity by removing tests for code that never executes

- **Permission assignment order** in test setups

  - Moved permission assignment AFTER `add_document()` calls
  - Ensures permissions are assigned to corpus copies, not originals
  - Files: `test_visibility_managers.py`, `test_resolvers.py`, `test_permissioning.py`, `test_version_aware_query_optimizer.py`

- **Document count expectations**

  - Updated tests to account for both originals and corpus copies existing
  - Example: Owner sees 6 documents (3 originals + 3 corpus copies) instead of 3
  - Files: `test_visibility_managers.py`, `test_resolvers.py`

- **Document-to-corpus linking**

  - Changed from M2M `corpus.documents.add()` to `corpus.add_document()`
  - File: `test_custom_permission_filters.py:211-213`

- **Corpus document queries**
  - Updated tests to query corpus documents via DocumentPath, not M2M
  - File: `test_bulk_document_upload.py:305-313`

### Technical Details

#### Architectural Changes

The structural annotation set feature implements Phase 2.5 of the dual-tree versioning architecture:

1. **Content-based deduplication**: Structural annotations are tied to content hash, not individual documents
2. **Corpus isolation compatibility**: When a document is copied to multiple corpuses, all copies share the same structural annotation set
3. **Immutability guarantee**: Structural annotations in shared sets cannot be modified (protected by PROTECT on delete)
4. **XOR constraints**: Database-level enforcement that annotations belong to either a document or a structural set

#### File Changes Summary

**New Files:**

- `opencontractserver/tests/test_structural_annotation_sets.py`
- `opencontractserver/tests/test_structural_annotation_portability.py`
- `opencontractserver/annotations/migrations/0048_add_structural_annotation_set.py`
- `opencontractserver/documents/migrations/0026_add_structural_annotation_set.py`
- `docs/architecture/STRUCTURAL_ANNOTATION_SETS.md`
- `CHANGELOG.md`

**Modified Files:**

- `opencontractserver/annotations/models.py` - Added StructuralAnnotationSet model, updated Annotation/Relationship models
- `opencontractserver/documents/models.py` - Added structural_annotation_set FK
- `opencontractserver/corpuses/models.py` - Fixed add_document() to copy all artifacts + structural set
- `opencontractserver/documents/versioning.py` - Fixed import_document() to copy all artifacts + structural set
- `config/graphql/graphene_types.py` - Added DocumentTypeConnection and explicit documents field for CorpusType
- `config/graphql/queries.py` - Updated document mention resolver to use DocumentPath
- `opencontractserver/pipeline/base/parser.py` - Updated save_parsed_data() to use add_document()
- `opencontractserver/tests/base.py` - Updated BaseFixtureTestCase to add documents to corpus
- `opencontractserver/tests/test_visibility_managers.py` - Updated for corpus isolation
- `opencontractserver/tests/test_resolvers.py` - Updated for corpus isolation
- `opencontractserver/tests/test_bulk_document_upload.py` - Updated for corpus isolation
- `opencontractserver/tests/permissioning/test_permissioning.py` - Updated for corpus isolation
- `opencontractserver/tests/permissioning/test_custom_permission_filters.py` - Updated for corpus isolation
- `opencontractserver/tests/permissioning/test_version_aware_query_optimizer.py` - Updated for corpus isolation
- `CLAUDE.md` - Added Changelog Maintenance section

### Fixed (Continued)

10. **Query optimizer missing structural_set annotations**

- **Files**: `opencontractserver/annotations/query_optimizer.py:189-212, 273-301, 541-564, 624-643`
- **Issue**: `AnnotationQueryOptimizer.get_document_annotations()` and `RelationshipQueryOptimizer.get_document_relationships()` only queried by `document_id`, missing annotations/relationships stored in `structural_set` (which have `document_id=NULL`)
- **Impact**: GraphQL queries using query optimizer (most annotation/relationship queries) did NOT return structural annotations from structural sets - only vector store had the dual-query logic
- **Fixed**:
  - Added document fetch with `select_related("structural_annotation_set")` for efficiency
  - Built OR filter: `Q(document_id=X) | Q(structural_set_id=Y, structural=True)` to query BOTH sources
  - Updated corpus filtering to preserve structural_set items (which have `corpus_id=NULL`)
  - Applied same fix to both AnnotationQueryOptimizer and RelationshipQueryOptimizer
- **Tests Added**: `opencontractserver/tests/test_query_optimizer_structural_sets.py` (10 comprehensive integration tests)
- **Test Results**: All 42 structural annotation tests pass (10 new + 32 existing)

11. **Vector store returning duplicate results**

- **File**: `opencontractserver/shared/mixins.py:40-89`
- **Issue**: `search_by_embedding()` method returned duplicate results (2x, 4x, 6x expected counts) when annotations had multiple Embedding rows with the same `embedder_path`
- **Root Cause**: JOIN to Embedding table created cartesian product - if annotation had 2 Embedding rows, JOIN produced 2 result rows
- **Investigation**: Confirmed annotations have multiple Embedding rows due to dual FK relationship:
  1.  `Embedding.annotation` FK (one-to-many): annotation can have multiple embeddings
  2.  `Annotation.embeddings` FK (many-to-one): annotation points to single "primary" embedding
- **Fixed**: Hybrid deduplication approach in `search_by_embedding()`:
  1.  Order by `id, similarity_score` and apply PostgreSQL `DISTINCT ON (id)`
  2.  Materialize query to list
  3.  Sort in Python by `similarity_score`
  4.  Return top_k results
- **Rationale**: PostgreSQL `DISTINCT ON` requires the distinct field to be first in ORDER BY, conflicting with need to order by similarity_score. Hybrid approach ensures correctness.
- **Test Results**: All 9 version-aware vector store tests now pass (previously all 8 failing)

12. **Vector store excluding structural annotations from StructuralAnnotationSet**

- **File**: `opencontractserver/llms/vector_stores/core_vector_stores.py:168-196, 221-270`
- **Issue**: Version filtering excluded ALL structural annotations from structural sets, causing vector search to return 0 results
- **Root Cause - Filter Ordering Bug**:
  1.  `only_current_versions` filter applied `Q(document__is_current=True)` (line 170)
  2.  This creates `INNER JOIN` on document table
  3.  Structural annotations have `document_id=NULL` (stored in StructuralAnnotationSet)
  4.  NULL document_id fails the JOIN → structural annotations excluded
  5.  This happened BEFORE document/corpus scoping (lines 221-270)
  6.  Result: Scoping logic tried to include structural annotations, but they were already filtered out
- **Symptoms**:
  - Initial queryset: 1344 annotations
  - After version filter: 0 results (all structural annotations excluded)
  - WebSocket tests failed with no ASYNC_CONTENT (agent had no context)
- **Fixed**:
  - Modified version filter to preserve structural annotations:
    ```python
    active_filters &= Q(document__is_current=True) | Q(
        document_id__isnull=True, structural=True
    )
    ```
  - Logic: Annotations with document FK must have `is_current=True`, structural annotations (no document FK) pass through
  - Later scoping filters by `structural_set_id` to ensure only relevant structural annotations included
- **Comments Added**: Comprehensive inline documentation explaining:
  - Why structural annotations have `document_id=NULL`
  - Filter ordering and interaction between version filter and scoping
  - Two-phase filtering approach (version → scoping)
- **Test Results**:
  - Vector store now finds 336 annotations (was 0)
  - SQL shows correct filter: `(document.is_current OR (annotation.document_id IS NULL AND structural))`

13. **Agent tool execution failing due to list/QuerySet type mismatch**

- **Files**: `opencontractserver/llms/vector_stores/core_vector_stores.py:30-90`
- **Issue**: After deduplication fix (#10), `search_by_embedding()` returns list instead of QuerySet, breaking agent tool execution
- **Root Cause - Type Assumption**:
  1.  Deduplication fix materialized QuerySet to list for DISTINCT ON + Python sorting
  2.  Helper functions `_safe_queryset_info()` and `_safe_execute_queryset()` assumed QuerySet
  3.  Called `.count()` method on lists (which don't have `.count()` for length)
  4.  Agent's `similarity_search` tool failed silently
  5.  LLM called tool → tool execution broke → no second LLM call → no ASYNC_CONTENT
- **Symptoms**:
  - Only 1 LLM API call in cassettes (should be 2: tool call + final answer)
  - Agent produced ASYNC_START and ASYNC_FINISH but no ASYNC_CONTENT
  - Cassette files abnormally small (27KB vs expected 50-70KB)
- **Fixed**: Updated helper functions to handle both QuerySets and lists:

  ```python
  async def _safe_queryset_info(queryset, description: str) -> str:
      if isinstance(queryset, list):
          return f"{description}: {len(queryset)} results"
      # ... handle QuerySet

  async def _safe_execute_queryset(queryset) -> list:
      if isinstance(queryset, list):
          return queryset  # Already materialized
      # ... execute QuerySet
  ```

- **Test Results**:
  - Tool execution now succeeds ✅
  - Cassettes show 2 LLM calls (tool call + response) ✅
  - Cassette size increased to 55KB (proper content) ✅
  - WebSocket tests still fail (different issue: agent streaming layer - not tool execution)

### Known Issues

1. **Pre-existing annotation visibility limitation**: `AnnotationQuerySet.visible_to_user()` doesn't check object-level permissions (only checks `is_public` or `creator`). This was not introduced by these changes but is more apparent with corpus isolation.

2. **WebSocket conversation tests** (`ConversationSourceLoggingTestCase`): Tests fail with no ASYNC_CONTENT messages.
   - **Current Status**: Tests fail with `AssertionError: [] is not true : At least one ASYNC_CONTENT expected`
   - **Vector Store Issues RESOLVED**:
     1. ✅ Vector store deduplication (issue #10 above) - All 9 vector store tests pass
     2. ✅ Query optimizer structural_set support (issue #9 above) - All 42 structural annotation tests pass
     3. ✅ Vector store version filtering (issue #11 above) - Now finds 336 annotations (was 0)
   - **Remaining Issue**: Agent produces no streaming content despite finding annotations
     - Vector store successfully returns 336 annotations to agent
     - Agent runs but produces no ASYNC_CONTENT messages (only ASYNC_START and ASYNC_FINISH)
     - Likely cause: VCR cassette mocking issue or LLM API configuration
     - **NOT a vector store or structural annotation architecture issue**
   - **Next Steps**: Investigate VCR cassette recordings and LLM mocking configuration
   - **Impact**: Isolated to WebSocket tests - production vector search and retrieval works correctly

### Migration Notes

- Run migrations in order: annotations/0048 before documents/0026
- No data migration required - new fields are nullable
- Existing documents will have `structural_annotation_set=None` until parsed

### Performance Considerations

- Structural annotations are now shared (O(1) storage) instead of duplicated per corpus copy
- DocumentPath queries are indexed for efficient corpus document lookups
- Content-hash based deduplication prevents redundant parsing
