---
title: "The Anvil: Why I Built a 6-Tier Quality Gate for AI-Generated Code"
published: false
description: "AI writes code with an 86% XSS failure rate. Here's how 6 automated tiers catch what AI misses — before it reaches production."
tags: ai, security, showdev, programming
cover_image:
---

# The Anvil: Why I Built a 6-Tier Quality Gate for AI-Generated Code

**86% of AI-generated code contains XSS vulnerabilities.**

That's not a scare number. That's from peer-reviewed research (Pearce et al., Perry et al., Sandoval et al.) studying code generated by GitHub Copilot, ChatGPT, and Claude.

Other hits:
- **40%** of AI code suggestions contain security vulnerabilities
- **5-21%** of AI-suggested packages don't exist (package hallucination)
- AI consistently fails at: input validation, authentication checks, error handling, and secret management

I've been using AI to build production software for months. The code is fast and often clever. It's also frequently wrong in ways that are hard to spot during manual review.

So I built The Anvil — a 6-tier quality gate that runs automatically between every agent handoff. No human review needed for routine issues.

## The Problem: Manual Review Doesn't Scale

When AI generates a feature, the typical workflow is:

```
AI generates code → You review it → You find issues → You fix them → Repeat
```

This works for small changes. It falls apart when you're generating entire features across multiple files, with database schemas, API endpoints, frontend components, tests, and docs.

I was spending more time reviewing AI code than I would have spent writing it myself.

## The Anvil: 6 Tiers

Every piece of code passes through all 6 tiers before it's accepted:

### Tier 1: Shell Pre-Flight (Pure Bash, No LLM)

Fast, deterministic, zero cost.

```bash
# What it checks:
- Syntax errors (bash -n, python -c, tsc --noEmit)
- Banned patterns: TODO, FIXME, HACK, PLACEHOLDER, NotImplementedError
- Hardcoded secrets (API keys, passwords, tokens)
- Import validation (no hallucinated packages)
```

**Why it matters:** This catches the 30% of issues that are purely mechanical. No LLM needed, runs in milliseconds. If your code has a TODO in it, it doesn't deserve a more expensive check.

### Tier 2: Canary Smoke Test

```
Can the module import without crashing?
```

Simple but devastating. If the Coder agent generates a module that can't even be imported, there's no point running the Tester on it. FAIL here → skip downstream agents → route to Fixer.

### Tier 3: Self-Adversarial Review

This is where it gets interesting. Before handing off, the generating agent must list **3+ ways its own code could fail**.

```
Coder: "I implemented the payment endpoint. Here are 3 failure modes:
1. Race condition if two payments hit simultaneously
2. No retry logic for gateway timeouts
3. Decimal precision loss in currency conversion"
```

If the agent can't think of failure modes → BLOCK. If the failure modes are real → they become test cases for the Tester agent.

**Why it works:** Forcing the agent to adversarially review its own work catches a surprising number of issues that wouldn't surface in a separate review. It's like asking a developer to rubber-duck their own PR.

### Tier 4: Scope Validation

```
Expected: auth.py, auth_test.py, auth_routes.py
Actual:   auth.py, auth_test.py, auth_routes.py, settings.py, main.py
                                                  ↑ UNEXPECTED
```

If the story says "implement authentication," and the agent also modifies `settings.py` and `main.py` — that's scope creep. Missing expected files → BLOCK. Unexpected files → WARN.

### Tier 5: Contract Enforcement

The story spec says:
```
POST /auth/login → 200 {token, user} | 401 {error}
POST /auth/register → 201 {user} | 400 {errors}
```

Tier 5 checks that the actual implementation matches the contract. Missing endpoint → BLOCK. Wrong response shape → BLOCK.

### Tier 6: Shadow Tester

A read-only parallel agent that independently assesses risk and prioritizes what the Tester should focus on. It doesn't block — it informs. Think of it as a second opinion that runs in the background.

## Self-Healing: The Fixer

When a tier fails, the code doesn't come back to you. It goes to the **Fixer Orchestrator** — an agent that:

1. Routes the violation to the right specialist (security issue → Security agent, missing test → Tester)
2. Retries up to 3 times with exponential backoff
3. Only escalates to the human for **critical decisions** (architecture choices, business logic ambiguity)

Result: 90%+ of violations are fixed automatically.

## Zero Tolerance

The Anvil enforces a zero-tolerance policy. These patterns trigger **immediate rejection**:

| Pattern | Why |
|---------|-----|
| `TODO`, `FIXME`, `HACK` | Incomplete work markers |
| `PLACEHOLDER`, `STUB`, `MOCK` (in prod) | Fake implementations |
| `COMING SOON`, `NOT IMPLEMENTED` | Unfinished features |
| `NotImplementedError` / empty function bodies | No-op code |
| Hardcoded credentials | Security violation |
| `@ts-ignore` without justification | Type evasion |

A mock is a lie. A TODO is a promise to fail.

## Results

After deploying The Anvil across 20+ projects:

- **Routine violations (missing tests, security headers, dead code):** Fixed automatically, no human needed
- **Security vulnerabilities:** Caught at Tier 1 (secrets) and Tier 3 (self-adversarial review)
- **Scope creep:** Caught at Tier 4 before it compounds
- **Contract drift:** Caught at Tier 5 before integration testing

The quality of AI-generated code isn't the AI's problem — it's an **infrastructure problem**. The AI is a powerful but undisciplined generator. The Anvil is the discipline.

## Try It

The Anvil is part of [Claude AS](https://github.com/samibs/claude_as), an open-source framework for structured AI-assisted development.

```bash
git clone https://github.com/samibs/claude_as.git ~/tools/claude_as
cd your-project
~/tools/claude_as/install.sh --platform=claude

# Run The Anvil manually:
> /anvil

# Or let it run automatically during /go:
> /go
```

MIT license. Works with Claude Code, GitHub Copilot CLI, Cursor, and OpenAI Codex.

---

*References:*
- *Pearce et al. — "Asleep at the Keyboard? Assessing the Security of Code Generated by GitHub Copilot"*
- *Perry et al. — "Do Users Write More Insecure Code with AI Assistants?"*
- *Sandoval et al. — "Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants"*
