# Auto-labeling

Heuristic detectors that stamp advisory **suggested_labels** on
memories at compile time. Introduced in **v0.9 (issue #158)** as the
ingest-time half of Statewave's governance story; the runtime half is
the authoritative `sensitivity_labels` column the policy evaluator
reads on every assembly call.

## The two columns

Statewave tracks **two** label columns on `memories`:

| Column                | Source              | Read by policy? | Mutable by detectors? |
|-----------------------|---------------------|-----------------|------------------------|
| `sensitivity_labels`  | Tenant (explicit)   | ✅ Yes          | ❌ Never               |
| `suggested_labels`    | Auto-labeling       | ❌ Never        | ✅ At compile time     |

The split is deliberate. Suggestions are *advisory* — surfaced for
operator review but never used to gate retrieval, refuse a request,
or change retention. A noisy detector cannot tighten policy on real
traffic; the worst it can do is produce a noisy suggestion in the
admin UI. Promotion into the authoritative column is a deliberate,
audited operator action (admin UI / SDK call); v0.9 ships review
only and the promotion endpoint lands in a follow-up.

## Enabling

Auto-labeling is **off by default** so a v0.9 upgrade is a no-op for
existing tenants. To enable it process-wide:

```bash
STATEWAVE_AUTO_LABELING_ENABLED=true
STATEWAVE_AUTO_LABELING_PROVIDER=heuristic   # the only supported provider (introduced in v0.9)
```

With the flag on, both compilers (`heuristic` and `llm`) run the
pipeline after MemoryRow construction. The detector pass is in-process
and adds a sub-millisecond hop per memory; there is no network call.

## Label schema

Suggested labels follow `<category>.<specific>`:

| Label              | What it flags                                              |
|--------------------|-------------------------------------------------------------|
| `pii.email`        | RFC-5322-ish email addresses                                |
| `pii.phone`        | E.164 international numbers or grouped national US/EU forms |
| `financial.card`   | 13–19 digit runs that pass the Luhn checksum                |
| `secret.token`     | Known-provider API keys (AWS, GitHub, OpenAI, Google,        |
|                    | Slack) or bearer JWTs                                       |

The catalogue is also returned by
`GET /admin/memories/with-suggested-labels` so an admin UI can build
filter dropdowns without hard-coding the list.

## Example: pipeline output

Given an episode containing

> "Reach me at alice@example.com or +1 415 555 0199. Card on file is
> 4111-1111-1111-1111."

a `MemoryRow` derived from that text will be stamped with:

```json
{
  "suggested_labels": ["financial.card", "pii.email", "pii.phone"],
  "sensitivity_labels": []
}
```

`sensitivity_labels` stays empty unless an operator promotes the
suggestions or the tenant set them explicitly.

## Reviewing and promoting suggestions

### Review queue

```http
GET /admin/memories/with-suggested-labels
    ?subject_id=<id>          # optional
    &tenant_id=<id>           # optional
    &label=pii.email          # optional — narrows to one detector
    &limit=50&offset=0
```

Response shape:

```json
{
  "memories": [
    {
      "id": "...",
      "subject_id": "...",
      "tenant_id": null,
      "kind": "profile_fact",
      "content": "alice@example.com",
      "summary": "alice email",
      "suggested_labels": ["pii.email"],
      "sensitivity_labels": [],
      "created_at": "..."
    }
  ],
  "total": 1,
  "limit": 50,
  "offset": 0,
  "catalogue": [
    {"label": "pii.email",      "description": "Email address ..."},
    {"label": "pii.phone",      "description": "Phone number ..."},
    {"label": "financial.card", "description": "Credit-card-shaped ..."},
    {"label": "secret.token",   "description": "Known-provider API key ..."}
  ]
}
```

The filter is GIN-indexed (migration 0022), so the `label=` overlap
path is cheap even on millions of memories.

### Promote a suggestion (v0.9 #160)

```http
POST /admin/memories/{memory_id}/promote-labels
    ?tenant_id=<id>           # optional, defence-in-depth
Content-Type: application/json

{"labels": ["pii.email"]}
```

Closes the auto-labeling loop. Every label in the request body MUST
currently be on the memory's `suggested_labels` — promotion is
strictly review-driven; the endpoint is not a backdoor for ad-hoc
tenant-side label writes. (Use the SDK for direct writes.)

On success (200) the response carries the new state:

```json
{
  "memory_id": "...",
  "promoted":            ["pii.email"],
  "sensitivity_labels":  ["legal.contract", "pii.email"],  // pre-existing preserved
  "suggested_labels":    ["pii.phone"]                     // promoted labels dropped
}
```

The promoted labels are:

1. Appended to `sensitivity_labels` (deduped + sorted; pre-existing
   tenant-set labels survive).
2. Removed from `suggested_labels` so the review queue does not
   re-surface them.
3. Recorded in an append-only audit entry on
   `memory.metadata.label_promotions`:

```json
{
  "labels":       ["pii.email"],
  "promoted_at":  "2026-05-25T21:30:42.123Z",
  "promoted_by":  null
}
```

`promoted_by` is `null` (v0.9+, still true in v1.0.0) — no admin
identity layer exists yet. The TODO is tracked at the field level so when admin identity
lands the column populates without an audit schema break. Time and
what-was-promoted are captured today.

**Refusal codes (HTTP 422, standard error envelope):**

| `error.code`                         | When                                                          |
|--------------------------------------|---------------------------------------------------------------|
| `promote_labels.empty`               | Request `labels` is empty                                     |
| `promote_labels.duplicate_labels`    | Request `labels` contains duplicates                          |
| `promote_labels.not_suggested`       | One or more labels are not on the memory's `suggested_labels` |

A 404 means the memory ID is unknown OR (with `tenant_id` set) belongs
to a different tenant — the two cases are indistinguishable on the
wire so a misconfigured caller cannot probe cross-tenant.

## What auto-labeling does **NOT** do (v0.9+)

* **Does not refuse or filter retrieval.** The policy evaluator
  ignores `suggested_labels`.
* **Does not change retention.** Memory TTL and the receipt
  retention worker do not key off suggestions.
* **Does not redact content.** Detectors are read-only; the memory
  body is preserved verbatim.
* **Does not promote suggestions automatically.** Promotion into
  `sensitivity_labels` is operator-driven.

These constraints are tested in
`tests/integration/test_auto_labeling.py::test_auto_labeling_never_writes_sensitivity_labels`
and are load-bearing for the governance story.

## Adding a detector

1. Add a pure `detect(text) -> bool` predicate in
   `server/services/auto_labeling/detectors.py`.
2. Wrap it in a `Detector` dataclass with a `<category>.<specific>`
   label and a one-line description.
3. Append the dataclass to the `DETECTORS` tuple.
4. Add positive + negative unit tests in
   `tests/test_auto_labeling.py` and update the registry assertion
   if you intend the new label to be part of the current contract.

Detectors should bias toward **precision over recall**: a false
positive is a noisy admin row; a flood of false positives undermines
operator trust in the column. Real low-recall gaps are recoverable
by an operator labelling the row by hand.
