# Reliability

The reliability system is a cross-domain product/system that turns live agent traffic plus human judgment into:

- canonical scores
- discovered issues
- evaluation monitors
- long-term reliability analytics
- simulation-driven pre-ship validation

## Documentation Contract

These docs describe the intended implemented reliability system, not just a temporary snapshot of the code that has already landed.

While the reliability spec is active, some duplication between spec and docs is intentional.

The docs should remain precise enough to stand on their own after the spec is deleted.

## Core Principles

1. Canonical mutable score rows live in Postgres from day one.
2. ClickHouse stores only immutable score analytics rows used for analytics.
3. Issue hybrid search runs against canonical Postgres rows using pgvector cosine similarity and a generated `search_document` tsvector; there is no external search projection.
4. Evaluation artifacts are stored as scripts from day one, even before the full portable runtime exists.
5. The Latitude reliability platform should be equally accessible to humans through the web app UI and to other LLM Agents through MCP/API.
6. Build reliability in `apps/web` first when that improves iteration speed, but design schemas/use-cases and public capabilities so the product does not dead-end into UI-only flows.
7. Canonical reliability domain data contracts are defined as shared Zod schemas first, with TypeScript types inferred from those schemas or the matching Drizzle schemas where appropriate.
8. Shared domain schemas validate data crossing from app/platform boundaries into domain use-cases, while external request/response contracts stay boundary-specific and may reuse or narrow those schemas.
9. Enum-like domain contracts use literal-string unions or `as const` objects, not TypeScript enums.
10. Configurable thresholds, weights, debounce windows, sentinel values, and similar tunables live in named constants inside the owning domain package rather than as scattered inline literals.
11. New reliability tables follow repository conventions: no foreign keys and RLS for organization-scoped Postgres tables.
12. Reliability background work uses the existing queue stack in `@domain/queue`, `@platform/queue-bullmq`, and `apps/workers`, with durable coordination in Postgres rather than in BullMQ job history.

## Legacy V1 References

This documentation keeps explicit legacy references to the old v1 repository.

Those references are paths relative to the repository root of branch `latitude-v1`.

Future coding agents that want to inspect the old implementation should first checkout branch `latitude-v1` in the old repository, then read the cited relative paths from that repository root.

For UI/product-surface work, old v1 components and patterns should also be treated as a reference:

- when this documentation refers to reusable v1 components, it means the old design-system components in `packages/web-ui/src/ds` from the old repository on branch `latitude-v1`
- do not reuse route-specific components from the old `apps/web/src` tree; the reliability entities, route structure, and product workflows have changed too much
- reuse as much as possible when the old design-system implementation is still solid
- do not copy v1 UI blindly; review it critically and improve it to meet v2 conventions, architecture, and quality standards when needed

## Storage Split

| Store      | Responsibility                                                                                                                                   |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| Postgres   | canonical `scores`, `evaluations`, `issues` (including the derived `centroid_embedding` pgvector and generated `search_document` tsvector used for hybrid search), `annotation_queues`, `simulations`, and embedded settings on `organization`, `projects`, and `user` |
| ClickHouse | immutable score analytics rows, score rollups, and score-aware span/trace/session analytics                                                         |

## Postgres Indexing

Postgres index design is part of the core schema work from day one, not a later optimization pass.

Rules:

- start from real query shapes: tenant scope first, then lifecycle/filter columns, then the dominant sort column
- define the needed secondary indexes in the relevant domain foundation phase together with the table definitions
- express those indexes directly in the Drizzle table model definitions, not only as surrounding prose; single-column `.unique()` constraints may stay inline on the column when that is the clearest shape
- do not add speculative JSONB, array, or text-search indexes when those queries are already served by owner-row primary keys, the issues `search_document` GIN index, or ClickHouse

## System Loop

```mermaid
flowchart LR
  annotations[Annotations]
  custom[Custom Scores]
  monitors[Issue Monitors]
  scores[Canonical Scores]
  issues[Issues]
  alignment[Evaluation Alignment]
  simulations[Simulations]

  annotations --> scores
  custom --> scores
  monitors --> scores
  scores --> issues
  issues --> alignment
  alignment --> monitors
  monitors --> simulations
  simulations --> scores
```

## Domain Layout

Reliability should not become a single `@domain/reliability` package.

Instead, the implementation should be split into direct domains such as:

- `packages/domain/scores`
- `packages/domain/annotations`
- `packages/domain/issues`
- `packages/domain/evaluations`
- `packages/domain/annotation-queues`
- `packages/domain/simulations`

Settings remain attached to their owner domains:

- organization settings stay with organizations/auth
- project settings stay with projects
- user settings stay with users/auth

Provider/model configuration and provider credentials stay embedded inside those owner `settings` JSONB payloads.

The first phase that needs each external provider capability introduces it as a platform package:

- `@platform/ai-vercel` for calling LLMs
- `@platform/ai-voyage` for embeddings and reranking

The canonical cross-domain `AI.generate` feature catalog (purpose, telemetry tags, and call sites) lives in `./ai-generation-features.md`.

Optimizer abstractions live in domain packages, while concrete optimizer implementations live in platform packages:

- `@domain/optimizations` for the optimizer interface/abstraction
- `@platform/op-gepa` for the first GEPA implementation, including the Python engine bridge and GEPA-specific runtime details

## Product Surface Implementation Pattern

Use the repository product pattern for reliability surfaces:

- human-facing reliability product pages live in `apps/web`
- server-side reads and writes for those pages live in `apps/web/src/domains/<domain>/*.functions.ts` via `createServerFn`
- reactive client state and optimistic sync live in `apps/web/src/domains/<domain>/*.collection.ts`
- route-specific reliability UI components should live in the route directory's dedicated `-components/` subfolder so route files stay separate from their supporting UI
- only rarely, when a component is genuinely shared across multiple routes, it may live in the shared `apps/web/src/components` folder
- stable public or machine-facing reliability capabilities live in `apps/api/src/routes/*` modules under the existing versioned organization-scoped path shape `/v1/organizations/{organizationId}/...`
- `apps/api` must not become the internal backend for `apps/web`; the web product should compose domain use-cases directly
- MCP clients consume that public REST API surface; reliability does not need a separate MCP-only backend contract

## Background Task Pattern

Reliability background work uses the existing async stack:

- `@domain/queue`, `@platform/queue-bullmq`, and `apps/workers` for single-step topic tasks
- the workflow abstraction plus the existing `apps/workflows` worker service for durable multi-step orchestration; Temporal is the first configured provider behind that abstraction
- the existing `domain-events` topic as the domain-event rail, with direct publication through `createEventsPublisher(queuePublisher)` for the reliability events introduced here

Rules:

- queue topics are the stable routing identity; BullMQ's per-job `name` is transport detail only
- each queue topic may define several lower-kebab-case task names, and the topic worker dispatches by that task name
- queue topic names should be added to `packages/domain/queue/src/index.ts`, while transport-facing topic constants or aliases belong in `packages/platform/queue-bullmq/src/topics.ts`
- when a topic needs payload parsing or validation for one worker only, keep that parser/validator next to the worker, for example `apps/workers/src/workers/<topic>-payload.ts`; only promote a payload contract into a domain package when several apps or phases truly share it
- domain events use PascalCase names such as `TracesIngested` and `ScoreCreated`
- queue topics use lower-kebab-case names such as `live-evaluations`
- queue payloads carry ids or opaque storage keys, not full mutable rows, and workers re-fetch current state before acting
- each queue topic owns one worker module under `apps/workers/src/workers/`
- the `domain-events` worker is a dispatcher only: it publishes downstream tasks or starts workflows and never runs synchronous business logic inline
- the domain-event routing map belongs in `apps/workers/src/workers/domain-events.ts`; later phases add new event-to-topic or event-to-workflow routes there rather than creating parallel event-consumer rails
- the new reliability domain events publish directly through `createEventsPublisher(queuePublisher)` into `domain-events` only after their upstream writes are durable
- BullMQ is transport, not lifecycle storage; durable progress, dedupe semantics, ownership, and visible progress stay in Postgres/domain state even when BullMQ provides dedupe, debounce, throttle, or latest-throttle coalescing primitives
- topic publication may request logical coalescing keyed by entity identity so BullMQ can reuse delayed jobs or job ids without leaking transport detail to callers
- use `debounceMs` when repeated publishes should reschedule the fire time and preserve the newest payload; use `throttleMs` when the first publish should schedule a fixed fire time and later publishes in the window should be dropped; use `latestThrottleMs` when the first publish should schedule a fixed fire time but later publishes in the window should replace the pending payload without extending the window
- the `TracesIngested` debounced trace-end runtime is the canonical queue-backed debounce example: each new ingest batch fan-outs one publish per deduped trace id, and each publish reschedules the same logical delayed `trace-end:run` task for that trace; the worker composition root is `runTraceEndJob` in `apps/workers/src/workers/trace-end.ts`, with domain rules split across `@domain/spans`, `@domain/evaluations`, and `@domain/annotation-queues` (see `./spans.md`)
- user-triggered async work that needs frontend progress feedback runs as a Temporal workflow with a deterministic, per-resource workflow id; a server-side status endpoint asks Temporal directly via `workflow.describe()` and workflow queries, and the UI polls that endpoint — there is no Redis-backed job-status mirror, because Temporal is the single source of truth for workflow state
- queues stay single-step; long-running or multi-step orchestration belongs in workflows whose activities own retries, timers, and progress
- reliability work extends the existing `apps/workflows` Temporal service and the existing `domain-events` dispatcher rail rather than introducing a second workflow runner or ad-hoc event-execution path
- workflow orchestration definitions should live in `apps/workflows/src/workflows/*.ts`, activity implementations should live in `apps/workflows/src/activities/*.ts`, and both should be re-exported from the local `index.ts` files so the worker bootstrap stays infrastructure-only
- the phase that first introduces a workflow should also introduce that workflow's concrete input schema and activity contracts; if several later phases truly need the same shared workflow contract before any one phase can own it cleanly, introduce a tiny intermediate phase then rather than speculating in the async-foundation work

Initial reliability async contracts:

- domain events: `TracesIngested`, `ScoreCreated`, `ScoreAssignedToIssue`
- topic tasks: `annotation-scores:publishHumanAnnotation`, `issues:discovery`, `issues:refresh`, `trace-end:run`, `live-evaluations:execute`, `evaluations:automaticRefreshAlignment`, `evaluations:automaticOptimization`
- workflows: `issue-discovery`, `refresh-evaluation-alignment`, `optimize-evaluation`, `annotation-publication`, `system-queue-flagger`

These workflow names map to concrete Temporal workflows registered in the existing `apps/workflows` service.

For the initial reliability events, `TracesIngested` publishes directly through `createEventsPublisher(queuePublisher)` because it comes from a high-volume append-only flow whose upstream write is already durable before publication.

`ScoreCreated` and `ScoreAssignedToIssue` are intentionally different from the direct-publication telemetry events: they are recorded through the transactional outbox rail so score-row persistence stays atomic with downstream fan-out. `ScoreCreated` is emitted for every canonical score write (including drafts); the `domain-events` dispatcher always publishes a deduped `issues:discovery` task plus a debounced `annotation-scores:publishHumanAnnotation` task (five-minute debounce window keyed by score id), while `issues:discovery` itself loads the score and skips when no issue work applies. The debounced annotation task only starts `annotation-publication` when the score is still a human-authored draft; otherwise it returns immediately. `ScoreAssignedToIssue` carries throttled `issues:refresh` intent after a score is linked into an existing issue — the first assignment schedules the refresh for `now + 8h`, later assignments within that window are dropped so a constant annotation stream cannot starve the refresh.

## Spec Governance

- the local task list in `specs/reliability.md` is the source of truth for reliability planning and execution state
- the roadmap is intentionally organized as domain-vertical phases rather than horizontal system-wide layers
- each phase is intended to become one GitHub PR and one Linear issue in the `Reliability` project
- before Linear sync, phase headings should use the placeholder id `LAT-XXX`; after sync, replace them with the created Linear issue ids
- task bullets inside a phase are the local implementation checklist for that phase; they are not independently synced planning units
- synchronization is one-way from the local spec/task list to Linear
- future coding agents must update the local spec/task list first, then reconcile Linear from it
- future coding agents must not sync or reconcile Linear until the user explicitly approves the spec and explicitly instructs them to do so

## Canonical Flows

### Score ingestion

- annotation, evaluation, and custom flows all write the same canonical score model
- machine-facing score uploads use `POST /v1/organizations/:organizationId/projects/:projectId/scores`, defaulting to custom-score semantics unless `_evaluation: true` is set for a locally executed Latitude evaluation upload
- annotation ingestion remains on `POST /v1/organizations/:organizationId/projects/:projectId/annotations` even though it still writes canonical `source = "annotation"` score rows
- all score writes land in Postgres first
- ClickHouse only receives immutable score analytics rows after the score lifecycle is ready for analytics save
- every score write records `ScoreCreated` transactionally with the row; the `domain-events` dispatcher publishes a deduped `issues:discovery` task keyed by score id, and the `issues:discovery` worker runs centralized issue handling only when the loaded score is eligible (non-draft, failed, non-errored, not already linked to an issue)
- immutable-score analytics sync runs directly after the owning Postgres transaction commits through the shared `syncScoreAnalyticsUseCase`, which re-checks current Postgres immutability and dedupes by score id
- annotation-originated feedback can be enriched before issue discovery through the `annotation-publication` workflow, which runs after the debounced `annotation-scores:publishHumanAnnotation` task coalesces rapid draft edits
- draft annotations use the same annotation score model, but stay drafts through `draftedAt` instead of a fake error value
- the canonical `feedback` text must stay human/LLM-friendly and intentionally clusterable, because discovery uses it for the primary embedding and lexical matching pass; annotation scores also preserve `metadata.rawFeedback` so discovery can run a raw-feedback fallback pass before creating a new issue
- scores can optionally attach to spans, traces, sessions, simulations, and issues

### Annotation queues

- annotation queues are trace-based review backlogs; session context is derived from related traces that share `session_id`
- each project starts with default system-created manual queues such as `Jailbreaking`, `Refusal`, `Frustration`, `Forgetting`, `Laziness`, `NSFW`, and `Trashing`; deterministic telemetry signals (`tool-call-errors`, `output-schema-validation`, `empty-response`) skip the queue path and publish annotation scores directly so `issues:discovery` can cluster them
- user-managed manual queues are populated from the trace dashboard table and the sessions dashboard table; session selection resolves to the newest trace and still creates `annotation_queue_items` with `trace_id` only and `completedAt = null`
- system-created manual queues are marked with `system = true`, provision `settings.sampling` from a named default constant, and let users tune that sampling later without changing the canonical queue definitions
- system-created manual queues are populated asynchronously from debounced `TracesIngested`: the `domain-events` dispatcher publishes one `trace-end:run` per deduped trace id, that runtime samples system queues first, and it starts one `system-queue-flagger` workflow per selected queue; the workflow performs deterministic routing or a limited-context flagger pass and, when it confirms a match, writes the queue item and pending-review draft annotation with full context
- queues are conceptually live when they store the shared `FilterSet` used by evaluation triggers inside queue settings, and `trace-end:run` incrementally materializes new matching traces by batching live-queue filters together with live-evaluation filters after sample-first pruning
- newly created live annotation queues initialize `settings.sampling` from a named constant, with an initial default of `10%`
- queue review is the focused in-product annotation workflow for fast human feedback

### Issue discovery

- failed non-errored scores are embedded and searched against issue centroids/text
- drafted scores stay out of discovery until `draftedAt` is cleared
- eligible non-draft failed non-errored scores are reached when `issues:discovery` runs after `ScoreCreated`; draft rows still skip discovery because the worker re-checks eligibility against canonical Postgres state instead of running issue routing inline in request paths
- the `issues:discovery` task rechecks canonical eligibility and centralizes three branches: selected annotation issue intent (including a `ScoreCreated` payload `issueId` captured while drafting when the published row no longer stores that link), issue-linked evaluation routing for evaluation-originated failed scores that still arrive unowned, or the fallback Temporal `issue-discovery` workflow when similarity search is still needed
- annotations are primary, but unlinked failed evaluation scores and failed custom scores may also create new issues
- when discovery creates a brand-new issue, the workflow goes through a dedicated create-from-score step that generates issue details from the initial occurrence before the first issue row is persisted
- issue name/description regeneration for existing issues runs through the throttled `issues:refresh` task keyed by the canonical issue id (at most once per 8h per issue; first publish wins)
- new issues are named from occurrences; users can later generate evaluations from those issues when they want active monitoring
- ignoring an issue archives its linked evaluations immediately, while resolution still uses `keepMonitoring`
- issue search uses canonical Postgres issue rows: `centroid_embedding vector(2048)` for exact cosine relevance plus generated full-text `search_document` for a bounded lexical boost, then Voyage reranking
- in the `issue-discovery` workflow, feedback embedding/normalization runs first and the search/rerank/create-or-assign decision runs inside locked serialization; annotation scores search with enriched canonical feedback first and raw annotation feedback second when the enriched pass has no match
- the shared issue-details generation use case serves both paths: synchronous first-details generation before a new issue is persisted, and later debounced refresh generation for existing issues from the last `25` assigned occurrences plus the current details as the stabilization baseline
- rerank candidates carry canonical issue ids from Postgres search, so no projection UUID resolution step is needed before assignment
- the create-from-score and assign-to-issue activities re-check canonical Postgres state, conditionally claim `scores.issue_id`, and persist the canonical issue row, centroid update, and derived pgvector state; after commit the workflow runs direct `syncScoreAnalyticsUseCase` so immutable scores reach ClickHouse; only assignments into an existing issue write `ScoreAssignedToIssue` transactionally for later debounced issue-details regeneration
- the `issues:refresh` worker must re-lock and re-read the canonical issue row before saving generated details so it cannot overwrite a newer centroid or lifecycle update; the generated `search_document` updates from canonical text without a separate projection sync
- existing-issue assignment must lock the canonical issue row before recomputing and saving centroid state so concurrent discovery workflows targeting the same issue serialize their centroid updates instead of losing one occurrence
- resolved and ignored issues remain valid match targets in discovery; lifecycle semantics are handled after assignment (including regression and ignored ownership behavior), not by pre-filtering them out of retrieval
- fuzzy no-match races are still guarded by feedback-scoped and project-scoped Redis locks because a worker cannot row-lock a similar issue that does not exist yet

### Evaluation alignment

- annotation-derived ground truth is split into positive and negative examples
- drafts and errored scores are excluded from evaluation alignment entirely
- evaluations generated from issues are created from the issue surfaces when the user asks for them, rather than as an automatic issue-discovery side effect
- issue-generated evaluation creation starts the `optimize-evaluation` Temporal workflow under a deterministic `evaluations:generate:${issueId}` id (initial generation) or `evaluations:optimize:${evaluationId}` id (manual realignment) and returns immediately; the frontend polls a status endpoint that asks Temporal directly via `workflow.describe()` on the three relevant workflow ids (generate + optimize + refreshAlignment), with no Redis-backed status mirror
- issues may have several linked evaluations; explicit generation is not limited to a single linked monitor
- live evaluation triggering is incremental on debounced `TracesIngested`; the `domain-events` dispatcher publishes one `trace-end:run` per deduped trace id, that runtime checks active evaluations project-wide, applies deterministic sampling first, batches the remaining shared `FilterSet` checks together with live queues, then applies evaluation turn/debounce rules and publishes `live-evaluations:execute` tasks for matches; the downstream execute path persists passed, failed, and errored evaluation-originated scores through the canonical score writer
- initial issue-linked evaluation generation requires at least one failed, non-errored, non-draft human annotation linked to that issue and does not require any negative examples
- sparse first-pass monitors may be weakly aligned at first, but annotation-driven realignment should improve them as more evidence accumulates
- evaluations are script-native, GEPA-backed artifacts that run through a portable runtime shared with simulations
- newly created issue-linked evaluations initialize `trigger.sampling` from a named constant, with an initial default of `10%`
- GEPA is the first optimizer, but the optimizer interface must stay replaceable
- the optimizer objective is the scalar trajectory score derived from whether `predictedPositive` equals `expectedPositive`
- only the confusion matrix is persisted; the headline alignment metric (currently balanced accuracy), plus recall, specificity, precision, F1, MCC, and accuracy are all derived from it on read. Decision and display code refer to the headline value generically as the "alignment metric" so the underlying formula can change without touching callers
- unchanged scripts can refresh alignment incrementally before a full re-optimization run; annotation-driven automatic refresh flows through the throttled `evaluations:automaticRefreshAlignment` queue task (1h, first-publish-wins) into `refresh-evaluation-alignment`, which on a `full-reoptimization` outcome publishes `evaluations:automaticOptimization` (8h, first-publish-wins) to start `optimize-evaluation`; the throttle (not debounce) semantics bound worst-case latency at 1h / 8h and cap fires at once per window per evaluation even under a constant annotation stream. Initial generation and manual realignment also run through `optimize-evaluation` directly under `evaluations:generate:${issueId}` / `evaluations:optimize:${evaluationId}` ids, so a manual run and a pending automatic optimize collapse into one in-flight run via Temporal's workflow-id dedupe
- v1's useful architecture split remains: TypeScript owns orchestration and candidate execution, Node workers remain the primary runtime, and Python can remain just the search engine behind a stdio JSON-RPC boundary packaged into the workers image

### Simulations

- simulations reuse the same evaluation artifact and portable runtime used by backend monitoring
- the same evaluation artifact used by the backend must run locally in the CLI
- the CLI is intentionally local-first and should remain useful as a standalone runner even without the hosted Latitude platform
- simulations include a JavaScript/TypeScript CLI and SDK, and the system can also provide additional lightweight SDKs for other languages such as Python, Ruby, PHP, and Go
- simulations can run with or without instrumentation
- results can remain local or be uploaded back to Latitude
