# `@/lib/pii`

Internal PII detection, masking, and tokenization engine for the Tale
platform. Used by the chat guardrails dispatcher (Convex governance) and
by the admin PII configuration panel.

Detects personally identifiable information — email addresses, phone
numbers, credit cards, IBANs, IP addresses, dates of birth, postal
addresses, national identity numbers, and more — in free-form text and
either masks it with replacement tokens (`[EMAIL]`, `[PHONE]`, …),
tokenizes it round-trip for AI flows (`[EMAIL_1]` → AI → original
restored), or blocks the input outright. Detection covers 43 locales
spanning Latin, Cyrillic, Greek, Arabic, Hebrew, CJK, and Indic scripts.

## Design principles

1. **Offline-only.** Every detector is a regex, a checksum validator, or
   a call into bundled `libphonenumber-js` metadata. No network, no
   filesystem at runtime.
2. **Convex-V8-safe.** Pure regex + string ops. The chat sanitize action
   imports this directly; the engine must run inside Convex's V8
   isolate.
3. **Never throws.** `scrubPii` and `createScrubber().scrub()` always
   return a `FilterOutcome` (`pass` / `modified` / `blocked`). Validator
   exceptions are caught and logged with the pattern name only — never
   the matched text (GDPR).
4. **ReDoS-bounded.** Every regex runs under `execWithBudget` (50 ms
   wall clock per pattern). Input is clamped to 50 KB before scanning.
   Custom user-supplied regexes are statically analyzed by `safe-regex2`
   at the Zod-schema layer before they ever reach the detector.
5. **Locale-aware.** Address and national-ID detection compose
   per-locale bundles; consumers opt in to the locales they care about,
   keeping the active pattern set small and the false-positive rate
   low.

## Quick start

```ts
import { createScrubber } from '@/lib/pii';

const scrubber = createScrubber({
  mode: 'mask',
  patterns: {
    email: true,
    phone: true,
    creditCard: true,
    iban: true,
    address: { locales: ['en', 'de', 'fr'] },
  },
});

scrubber.scrub('Reach me at john@example.com or +1 (415) 555-0123.');
// { outcome: 'modified', text: 'Reach me at [EMAIL] or [PHONE].', … }
```

## Layout

```
services/platform/lib/pii/
├── core/        — outcome types, NFC normalization, ReDoS-safe regex exec
├── engine/      — detector, masker, scrubber, tokenizer, PatternRegistry
├── patterns/    — built-in pattern implementations (email, phone, …)
│   ├── address/locales/    — locale-specific address pattern bundles
│   └── national-ids/       — country-specific ID detectors + checksums
├── locales/     — typed per-locale data modules (one .ts per locale)
└── schemas/     — Zod schemas for runtime config validation

services/platform/test/pii/
├── fixtures/    — generated JSON test data (one dir per locale)
└── *.test.ts    — data-driven, ReDoS, Unicode, throughput suites

services/platform/scripts/pii-fixtures/
├── gen/         — fixture generator
├── refresh/     — dataset refresh scripts (offline)
└── datasets/    — committed dataset snapshots
```

See [patterns/address/locales/README.md](./patterns/address/locales/README.md)
for the recipe to add a new locale.

## Test fixtures

Fixtures are generated by `services/platform/scripts/pii-fixtures/gen/`
from committed dataset snapshots and frozen as JSON in
[`services/platform/test/pii/fixtures/<locale>/`](../../test/pii/fixtures/).
Each locale carries 2,000+ cases — a mix of positives (PII the
detector must find) and negatives (prose the detector must leave
alone). Regenerate with:

```bash
bun run --filter @tale/platform fixtures:gen
```

The generator runs fully offline.

## License attribution

The engine code is part of the Tale platform. The committed dataset
snapshots and generated fixtures are derived from public datasets under
ODbL 1.0, CC-BY 4.0, Apache 2.0, CC-BY 2.0 FR, and public-domain
sources. Per-source attribution is in [LICENSES.md](./LICENSES.md).
