# Detecting Language Model Attacks with Perplexity -- Research Summary

**Date:** 2026-02-08
**Paper:** Detecting Language Model Attacks with Perplexity
**Authors:** Gabriel Alon (University of Michigan), Michael Kamfonas (Independent Researcher)
**arXiv:** [2308.14132](https://arxiv.org/abs/2308.14132)
**Published:** 7 Nov 2023 (v3)
**Source PDF:** `docs/research/papers/2308.14132.pdf`

## LLMTrace Application Notes

**Required signals/features:**: Token-level log-probabilities from a reference LM (GPT-2 or equivalent), raw prompt text, token sequence length (GPT-2 tokenizer count). Two features total: perplexity score and token count.


**Runtime characteristics:**: GPT-2 perplexity computation is a single forward pass (~5-15ms on GPU for typical prompt lengths). LightGBM inference is sub-millisecond. Total latency overhead is dominated by the GPT-2 forward pass. Fully compatible with pre-request blocking (synchronous gate). Not inherently streaming -- requires the full prompt text before scoring.


**Integration surface (proxy):**: Compute perplexity on each inbound prompt before forwarding to the target LLM. Attach perplexity + token-length as metadata fields. Run the LightGBM classifier on these two features and apply a configurable threshold. Block or flag requests exceeding the threshold. Can also run on tool outputs in agent pipelines to detect adversarial string propagation (IS-052).


**Productizable vs research-only:**: The two-feature LightGBM classifier (perplexity + token length) is fully productizable as a lightweight pre-filter. The perplexity computation via GPT-2 is straightforward to deploy. The limitation around human-crafted jailbreaks is a known research gap -- this approach must be layered with other defences, not used standalone.


## Paper Summary

### Problem Statement and Motivation

Zou et al. (2023) demonstrated that adversarial suffixes generated by the Greedy Coordinate Gradient (GCG) algorithm can reliably jailbreak aligned LLMs (ChatGPT, BARD, LLaMA-2-Chat, Claude). These suffixes are trained on an offline model (e.g., Vicuna-7B) and transfer to closed-source models. The methodology and code are publicly available, creating an urgent need for detection mechanisms.

The core insight is that GCG-generated adversarial suffixes produce bizarre, non-natural-language token sequences. Since perplexity measures how "surprised" a language model is by a given text, and language models are trained to assign low perplexity to natural text, adversarial suffixes should exhibit anomalously high perplexity.

### Proposed Approach

**Perplexity computation with GPT-2:**: Each prompt is scored using GPT-2 perplexity: `PPL(x) = exp(-1/t * sum(log p(xi|x<i)))` where `t` is the token count and `p` is the GPT-2 probability.


**Observation: plain perplexity threshold is insufficient.**: While adversarial prompts have very high perplexity (mean = 3525, sigma = 2585), many benign prompts also exhibit high perplexity -- particularly short phrases, non-English text, mathematical expressions, programming code, misspellings, and irregular symbols. A simple threshold causes unacceptable false positive rates.


**Two-feature LightGBM classifier:**: The authors train a LightGBM on two features: (a) perplexity and (b) token sequence length. The key insight is that adversarial suffix attacks are naturally lengthy (they append a long suffix to a complete prompt, mean seq-len ~38.72 tokens) while many high-perplexity benign prompts are very short. This interaction between perplexity and length provides strong separation.


**Training details:**: Adversarial/non-adversarial split: 50:25:25 (train/val/test) for adversarial, 70:15:15 for non-adversarial, to boost adversarial representation.

   - Less than 1% of the combined dataset is adversarial.
   - Threshold tuned on validation set to maximize F2 score (beta=2, favoring recall over precision).

### Data

**Adversarial datasets:**
- 1407 machine-generated adversarial prompts using GCG on Vicuna-7b-1.5 (67 objectives, 20 suffixes each). Generated over 50 hours on an A100 GPU.
- 79 human-crafted GPT-4 jailbreak prompts from Jaramillo (2023).

**Non-adversarial datasets (174,923 total):**

| Dataset | Prompts | Type |
|---------|---------|------|
| Puffin (GPT-4 conversations) | 6,994 | Conversational |
| DocRED | 998 | Document RE |
| SuperGLUE (boolq) | 3,270 | Yes/No QA |
| SQuAD-v2 | 11,873 | Span-based QA |
| Open Platypus | 24,926 | Reasoning/instruction |
| Tapir | 116,862 | Instruction-following |
| Code-search-net-python | 10,000 | Code instructions |

### Key Results

**Perplexity distributions:**

| Prompt Type | Mean PPL | Std PPL | Notes |
|-------------|----------|---------|-------|
| GCG adversarial | 3525.00 | 2585.57 | ~90% above PPL 1000, all relevant above 200 |
| Human-crafted jailbreak | 42.83 | 24.86 | Indistinguishable from benign text |
| DocRED (benign) | 38.06 | 13.13 | Low perplexity, long sequences |
| SQuAD-v2 (benign) | 32.70 | 10.12 | Low perplexity |
| Platypus (benign) | 44.78 | 135.11 | Highly variable, includes math/code |
| Puffin (benign) | 971.12 | 11463.44 | Extreme outliers from short/non-English inputs |
| Tapir (benign) | 106.22 | 29.66 | Moderate perplexity |
| Code prompts (benign) | 128.31 | 200.47 | Code content drives up PPL |

**Simple PPL threshold vs. GBM classifier (Table 2):**

| Method | F2 (test) | F2 (test, no human-crafted) |
|--------|-----------|----------------------------|
| PPL threshold = 400 | 83.3% | 87.0% |
| PPL threshold = 1000 | 87.2% | 91.6% |
| LightGBM (PPL + token length) | **94.2%** | **99.1%** |

**Test set confusion matrix (26,571 examples):**

| | Predicted Negative | Predicted Positive |
|---|---|---|
| Non-Attack (True) | 26,188 | 1 |
| Attack (True) | 27 | 355 |

**Breakdown by attack type (Table 1):**

| Metric | Human-Crafted | Machine-Generated (GCG) |
|--------|---------------|------------------------|
| True Positive | 0 (0%) | 355 (96.2%) |
| False Negative | 23 (100%) | 4 (1.1%) |
| False Positive | 0 | 0 |
| True Negative | 0 | 10 (2.7%) |

The classifier achieves 96.2% true positive rate on machine-generated GCG attacks with only 1 false positive out of 26,189 benign prompts. It completely fails on human-crafted jailbreaks (0% detection).

### Limitations

**Human-crafted jailbreaks are undetectable**: by this method. All 23 false negatives come from human-designed prompts, which have perplexity distributions indistinguishable from benign text.


**GPT-2 only.**: The paper uses exclusively GPT-2 for perplexity. Other reference models may yield different distributions.


**Default GCG budget.**: The adversarial suffixes were generated with the default GCG configuration. Adaptive attacks that minimize perplexity (as in Jain et al., 2023) can reduce perplexity of adversarial suffixes, though Jain et al. report their windowed perplexity filter still blocks 80% of such adaptive attacks.


**Limited real-world prompt distribution.**: The non-adversarial datasets, while diverse, may not fully represent production LLM traffic.


**Perplexity-minimizing attacks exist.**: Han et al. (2020) demonstrated adversarial examples with minimized perplexity for NLP tasks. Attackers who incorporate perplexity minimization into the GCG objective could evade this filter.


## Feature Delta with LLMTrace

| Paper Technique | LLMTrace Coverage | Gap | Planned Feature |
|---|---|---|---|
| GPT-2 perplexity scoring on inbound prompts | No perplexity computation in pipeline | Need reference-LM perplexity module as a pre-request signal | IS-050 (perplexity-based anomaly detection) |
| Token sequence length as a discriminative feature | Token counting available implicitly | Not used as a classifier feature alongside perplexity | IS-050 (include token length as second feature) |
| LightGBM two-feature classifier (PPL + seq-len) | No lightweight ML classifier for anomaly gating | Need a fast binary classifier on (perplexity, token_length) | IS-050 / ML-001 (can be a sub-classifier in the fusion pipeline) |
| F2 score optimisation (recall-biased) | No configurable beta for detection threshold | Configurable F-beta threshold per deployment policy | IS-050 (threshold policy configuration) |
| Perplexity scoring on tool outputs (agent pipeline) | Monitors tool outputs but no perplexity scoring | Apply perplexity gate to tool-output text to catch propagated GCG strings | IS-052 (adversarial string propagation blocking) |
| Detection of human-crafted jailbreaks | DeBERTa-based semantic classifier (ML-001) handles these | Perplexity alone cannot detect human-crafted attacks; layered defence needed | ML-001 (binary classifier covers this gap) |
| Interaction of perplexity and token entropy | Token-wise bias detection exists (IS-001) | IS-001 targets vocabulary-level bias, not per-prompt entropy gating | IS-001 could be extended with per-prompt entropy as a complementary signal |

### Relationship to Tracked Features

**IS-050 (perplexity-based anomaly detection):**: This paper is the primary evidence base. It validates that GPT-2 perplexity + token length achieves 99.1% F2 on GCG attacks. The LightGBM classifier approach directly maps to IS-050 implementation.


**IS-001 (MOF training / token-wise bias detection):**: Orthogonal. IS-001 addresses vocabulary-level false positive bias in classifiers. The perplexity approach addresses a different signal entirely (statistical naturalness of token sequences). Both are needed.


**ML-001 (binary classifier / fusion training):**: The perplexity+length classifier can serve as an input feature or a parallel gate alongside the DeBERTa fusion classifier. The paper confirms that perplexity alone cannot catch human-crafted jailbreaks, which is exactly what ML-001's semantic classifier covers. Together they form complementary layers.


## Actionable Recommendations

**[P0] Implement GPT-2 perplexity scoring as a pre-request gate (IS-050).**: Deploy GPT-2 (or a distilled variant) as a sidecar model. Compute perplexity on every inbound prompt. Store both `ppl_score` and `token_length` as trace metadata. This is the lowest-latency defence against GCG-style attacks (~10ms overhead). The paper demonstrates 99.1% F2 when human-crafted attacks are excluded, which is the correct scope for this specific defence layer.


**[P0] Apply perplexity scoring to tool outputs in agent pipelines (IS-052).**: GCG-optimised strings propagate through agent tool outputs. Run the same perplexity gate on tool-output text before the agent incorporates it into subsequent traces. This directly addresses the Agent-as-a-Proxy attack vector.


**[P1] Train and ship a LightGBM classifier on (perplexity, token_length).**: A plain threshold of PPL > 1000 achieves 87.2% F2. The two-feature LightGBM lifts this to 94.2% (or 99.1% excluding human-crafted). The classifier is trivial to train (two features, LightGBM) and adds negligible inference cost. Use F2 (beta=2) during training to favor recall.


**[P1] Make the detection threshold configurable by deployment policy.**: The optimal F-beta and threshold depend on the cost tradeoff between blocking benign users vs. missing attacks. Expose `beta` and `threshold` as configuration parameters. Default to beta=2 (recall-biased) per the paper's recommendation.


**[P2] Evaluate modern reference models beyond GPT-2.**: The paper uses GPT-2 exclusively. Smaller, more efficient models (distilled GPT-2, or purpose-built perplexity estimators) may provide better latency. Larger models may provide sharper perplexity separation. Benchmark against the paper's distributions before switching.


## Key Takeaways

- GCG-generated adversarial suffixes have dramatically high perplexity (mean 3525 vs. ~30-45 for natural English text), making perplexity a strong signal for detecting machine-generated suffix attacks.
- A simple perplexity threshold is insufficient in production because benign prompts containing code, non-English text, math, or short phrases can exceed the perplexity of adversarial prompts by an order of magnitude. The two-feature classifier (perplexity + token length) resolves this by exploiting the fact that adversarial suffixes are long while high-perplexity benign prompts are typically short.
- Perplexity-based detection is fundamentally blind to human-crafted jailbreaks, which have perplexity distributions indistinguishable from benign text. This defence must be layered with semantic classifiers (ML-001) and structural defences (AS-001/AS-002), never deployed as a sole defence.
- The LightGBM classifier achieves 99.1% F2 on machine-generated GCG attacks with only 1 false positive out of 26,189 benign prompts -- suitable as a high-precision, low-overhead first gate in a multi-layer defence pipeline.
- Adaptive attackers who add perplexity minimization to the GCG objective can partially evade this defence, but Jain et al. (2023) found windowed perplexity still blocks 80% of such adaptive attacks at same computational budget. The defence raises the attacker's cost.
