# vaara-bench-v0.36

Methodology delta against [vaara-bench-v0.35](vaara-bench-v0.35.md).
v0.36 is a methodology release with three contributions:

1. **Cross-model held-out corpus.** 4,176 adversarial entries generated
   by Mixtral-8x7B-Instruct (FP16 on AMD-backed MI300X) and Claude
   Sonnet 4.6 (Anthropic API) on the three weakest v0.35 categories
   (`tool_misuse`, `privilege_escalation`, `data_exfil`). Pure eval fold
   under `tests/adversarial/v036_holdout.json`, never folded into TRAIN.
2. **v6 to v7 retrain.** 18 destination-aware features added to
   `build_features`. Same v035 TRAIN fold, same XGBoost hparams. v7
   becomes production. No regression on v035 TEST.
3. **First public cross-model held-out for an OSS runtime governance
   scanner.** External-validation gap published openly, mechanism
   identified, training-corpus constraint named as v0.37 scope.

## Why cross-model held-out is the right next move

v0.35 named limits explicitly flagged that the v035_split TEST fold
reflects Qwen-72B distribution rather than real-world attacker
distribution. v0.36 publishes what happens against a never-seen
generator family. This is the external-validation gap documented across
the broader ML literature: one recent example
(arxiv:2605.21566, CKD risk prediction, 2026-05-20) trained five
classifiers to AUROC 1.00 on the internal UCI test set, then recorded
AUROC 0.48-0.58 on the MIMIC-IV external cohort with conformal coverage
dropping from 0.80-0.98 internal to 0.21-0.25 against a 90% target.
Domain incomparable, lesson methodological: internal test performance
is a ceiling, the gap is visible only against held-out generators.

## Production headline (v7 on v035 TEST)

| metric | v7 | v6 (prior production) | Δ |
|---|---|---|---|
| TEST recall at T=0.9008 | **85.3% [83.1, 87.3]** | 85.0% [82.8, 87.1] | +0.3pp |
| TEST FPR at same calibration | 4.6% [3.3, 6.3] | 4.6% [3.3, 6.3] | flat |
| n_features | 638 (620 base + 18 dst) | 620 | +18 |
| Calibrated on v035 VAL FPR target 5% | T=0.9008 | T=0.9006 | noise |

v7 is the new production bundle
(`src/vaara/data/adversarial_classifier_v7.joblib`). v6 stays on disk
for cross-eval reproducibility. No regression. Weak-Pareto win.

## Cross-model held-out result (v7 on v036_holdout)

| cut | n | recall at T=0.9006 |
|---|---|---|
| **overall** | 4,176 | **59.2% [57.7, 60.7]** |
| Claude leg (closed-weight) | 2,099 | **46.1% [44.0, 48.3]** |
| Mixtral leg (open-weight) | 2,077 | **72.4% [70.4, 74.3]** |
| data_exfil | 1,390 | 47.5% [44.9, 50.1] |
| privilege_escalation | 1,400 | 67.1% [64.6, 69.6] |
| tool_misuse | 1,386 | 62.9% [60.3, 65.4] |
| data_exfil__claude | 700 | **26.0% [22.9, 29.4]** |
| data_exfil__mixtral | 690 | 69.3% [65.7, 72.6] |
| privilege_escalation__claude | 700 | 66.1% [62.6, 69.6] |
| privilege_escalation__mixtral | 700 | 68.1% [64.6, 71.5] |
| tool_misuse__claude | 699 | 46.2% [42.5, 49.9] |
| tool_misuse__mixtral | 687 | 79.9% [76.8, 82.7] |

In-distribution to external gap is 26 pp (85.3% v035 TEST to 59.2% v036
held-out). The Claude leg drops more than the Mixtral leg. The gap is
closed-weight-vs-open-weight, not Qwen-specific. Worst cell is
data_exfil × Claude (26.0%).

## Mechanism: destination-signal axis

Score distribution on data_exfil × Claude is bimodal: 36% of those
entries score below 0.40 (the classifier is confident-benign, not
borderline). The equivalent Mixtral cell has 7% in that range.
Tool-name overlap with the training corpus is 100% on both legs. Novel
tool surface is not the explanatory variable.

The difference is which tools each model picked. Mixtral data_exfil top
tools are `gsutil`, `scp`, `curl`, `wget`, `rclone`, `aws_s3_cp`,
command-line transfer tools strongly correlated with exfil in the
training corpus. Claude data_exfil top tools are `send_email`,
`s3_copy`, `http_post`, `azure_blob_copy`, `dropbox_upload`, `git_push`,
`pastebin_create`, higher-level APIs with both legitimate and exfil
uses. Claude's adversarial signal sits in destinations: external URLs,
personal email domains, public-read buckets, no-expiry pastebins. v6
was tool-name-heavy. The destination axis was not in feature space.

Sample missed Claude entries (v6 score, v7 in parens):

- `send_email` to personal Gmail to payroll xlsx attached to 0.18 (0.21)
- `sql_query_execute` SELECT ssn/dob/credit_score to webhook.site to 0.01 (0.05)
- `pastebin_create` visibility=public expiry=never to 0.09 (0.17)
- `s3_copy` corp-sensitive-docs to public-temp-storage to 0.39 (0.48)

## Engineering attempt: 18 destination-aware features

`scripts/train_adversarial_classifier.py:_DST_PATTERNS` adds 18 regex
features prefixed `dst__`: personal-email-domain set, webhook/relay
services, paste services, free file-host services, public-bucket
markers, no-expiry markers, non-internal share schemes
(smb/ftp/sftp/webdav), external-bucket-name patterns, PII column
patterns, bulk-SELECT-LIMIT, SELECT-star-FROM, export/dump destination
keys, suspicious TLDs, webhook parameter keys, public-path markers,
share platforms, recipient-to-personal-email, attachment-with-external-
recipient. Inference path mirrored in `src/vaara/adversarial_classifier.py`
with backward compatibility for v6-style bundles (detect `dst__`
presence in `feature_names`).

v7 retrained on v035 TRAIN with the new feature set. Features fire
correctly on previously-missed Claude entries (every sample above hits
at least one `dst__` feature). But v7 lifts overall held-out recall
only +0.8pp (58.4% to 59.2%) and data_exfil__claude only +4.0pp
(22.0% to 26.0%).

## Honest diagnosis: training-corpus constraint, not feature space

Destination features fire. XGBoost weights them weakly because v035
TRAIN does not contain enough adversarial examples where these features
are the discriminating signal. The constraint is training-corpus
distribution, not feature space. This is the v0.34 matched-benign
lesson revisited: features that exist in feature space but not in
training signal carry little weight.

v0.37 scope shifts accordingly. The fix is corpus augmentation, not
more feature engineering: fold a subset of the v036 entries (Mixtral
TM/PE + Claude TM/PE) into TRAIN, retain a held-out fold (data_exfil
from both legs + fresh small generation from a third model family),
retrain v8 on the union, re-evaluate on the new held-out.

## Ship gate

v0.36 ships under a methodology-release gate, not a recall-target gate:

| gate | result |
|---|---|
| v035 TEST recall does not regress | PASS, 85.0% to 85.3% within CI |
| v035 TEST FPR does not regress | PASS, 4.6% to 4.6% |
| Held-out gap published openly with mechanism | PASS |
| Training-corpus constraint named honestly | PASS |
| v0.37 scope written with concrete corpus plan | PASS |

The 59.2% held-out is not a ship-gate failure because v0.36 is not a
recall release. v6 held 85% on its in-distribution test set across five
releases. v7 holds the same number while opening the methodology surface.

## Chain of custody

| anchor | path | pins |
|---|---|---|
| corpus manifest | `tests/adversarial/MANIFEST.sha256` (302 lines) | SHA-256 of every JSONL including v036 |
| v035 split | `tests/adversarial/v035_split.json` | TRAIN/VAL/TEST for v7 calibration |
| v036 held-out | `tests/adversarial/v036_holdout.json` | 4,176 keys to "holdout", never in TRAIN |
| production bundle | `src/vaara/data/adversarial_classifier_v7.joblib` | trained on v035 TRAIN with dst features |
| prior production | `src/vaara/data/adversarial_classifier_v6.joblib` | retained for cross-eval |
| Mixtral generator | `scripts/generate_targeted_v036.py` | vLLM HTTP, FP16 on MI300X |
| Claude generator | `scripts/generate_targeted_v036_claude.py` | Anthropic SDK, Sonnet 4.6 |
| held-out eval | `scripts/eval_v036_holdout.py` | per-category and per-leg cuts |

## Reproduction recipe

```
cd tests/adversarial && sha256sum -c MANIFEST.sha256
.venv/bin/python scripts/eval_v032.py \
    --bundle src/vaara/data/adversarial_classifier_v7.joblib \
    --split-manifest tests/adversarial/v035_split.json \
    --target-fpr 0.05 --json-out bench/v036_eval_v7_v035test.json
.venv/bin/python scripts/eval_v036_holdout.py \
    --bundle src/vaara/data/adversarial_classifier_v7.joblib \
    --split-manifest tests/adversarial/v036_holdout.json \
    --threshold 0.9006 --json-out bench/v036_eval_v7_holdout.json
```

## Compute provenance

Mixtral generation ran on an AMD-backed MI300X DigitalOcean SR-IOV
droplet under `rocm/vllm:latest` serving
`mistralai/Mixtral-8x7B-Instruct-v0.1` in FP16
(`--max-model-len 8192 --enforce-eager --gpu-memory-utilization 0.92`).
Three parallel category generators, steady-state ~28 entries/min/cat.
Droplet shutdown issued post-rsync. Claude generation ran locally via
Anthropic SDK against `claude-sonnet-4-6`, rate-limited at the org's
8k-output-tokens-per-minute tier. Total Claude API spend under $3.

## Named limits

1. **Two attacker families, not three.** Mixtral-8x22B was the original
   target. AWQ quantization is not supported on AMD ROCm (verified
   against vLLM docs), FP8 bring-up was longer than the v0.36 cycle
   allowed. Mixtral-8x7B FP16 was the no-friction substitution. Adding
   DeepSeek-V3 or Llama-3-Instruct as a third family is v0.37+ scope.
2. **No public-benchmark eval (PINT, BIPIA, INJECT) yet.** v0.37+ scope.
3. **PAIR multi-attacker scale-up not performed.** The FPR-bounded
   three-stage combiner depending on it (rules-veto in uncertain band,
   arxiv:2605.22004 cited as methodology pointer) is v0.37+ scope.
4. **18 destination features designed pre-eval, not from importance
   regression.** They cover patterns visible in v6's missed Claude
   entries but were not selected by training-side feature importance.
   v0.37 corpus augmentation work will let XGBoost reweight or expand
   them.
