Augustus

Judge configuration (required for judge.Judge, judge.Refusal, and multi-turn probes)

judge: generator_type: openai.OpenAI model: gpt-4o-mini config: api_key: "${OPENAI_API_KEY}"

Installation

Requires Go 1.25.3 or later.

go install github.com/praetorian-inc/augustus/cmd/augustus@latest

Or build from source:

git clone https://github.com/praetorian-inc/augustus.git
cd augustus
make build

Build binary

make build

Install to $GOPATH/bin

make install ```

Quick Start

Basic Usage

export OPENAI_API_KEY="your-api-key"
augustus scan openai.OpenAI \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --verbose

Example Output

+--------------+-------------+--------+-------+--------+
| PROBE        | DETECTOR    | PASSED | SCORE | STATUS |
+--------------+-------------+--------+-------+--------+
| dan.Dan_11_0 | dan.DAN     | false  | 0.85  | VULN   |
| dan.STAN     | dan.STAN    | true   | 0.10  | SAFE   |
| dan.AntiDAN  | dan.AntiDAN | true   | 0.05  | SAFE   |
+--------------+-------------+--------+-------+--------+

Usage

Advanced Options

```bash

Configuration

YAML Configuration File

Create a config.yaml file:

```yaml

Runtime configuration

run: max_attempts: 3 timeout: "30s"

Generator configurations

generators: openai.OpenAI: model: "gpt-4" temperature: 0.7 api_key: "${OPENAI_API_KEY}" # Environment variable interpolation

anthropic.Anthropic: model: "claude-3-opus-20240229" temperature: 0.5 api_key: "${ANTHROPIC_API_KEY}"

ollama.OllamaChat: model: "llama3.2:3b" temperature: 0.8

Output configuration

output: format: "jsonl" path: "./results.jsonl"

Environment Variables

```bash

Proxy Configuration

Route HTTP traffic through a proxy (e.g., Burp Suite) for inspection:

```bash

Method 1: Via config parameter

augustus scan rest.Rest \ --probe dan.Dan_11_0 \ --detector dan.DAN \ --config '{"uri":"https://api.example.com","proxy":"http://127.0.0.1:8080"}' \ --output results.jsonl

Method 2: Via environment variables

export HTTP_PROXY=http://127.0.0.1:8080 export HTTPS_PROXY=http://127.0.0.1:8080 augustus scan rest.Rest --probe dan.Dan_11_0 --config '{"uri":"https://api.example.com"}' ```

• TLS verification automatically disabled for proxy inspection
• HTTP/2 support enabled for modern APIs
• Server-Sent Events (SSE) responses automatically detected and parsed

Is Augustus suitable for production environments?

Yes, Augustus is designed for production use with: - Concurrent scanning with configurable limits - Rate limiting to respect API quotas - Timeout handling for long-running probes - Retry logic for transient failures - Structured logging for observability

Benchmark Environment (DevPod)

A ready-to-go cloud development environment for benchmarking LLMs is available via DevPod. It provisions a remote container with Augustus, Ollama, Go, and all dependencies pre-installed.

```bash cd devpod

Custom REST Endpoints

```bash

Test proprietary LLM endpoint (OpenAI-compatible API)

augustus scan rest.Rest \ --probe dan.Dan_11_0 \ --detector dan.DAN \ --config '{ "uri": "https://api.example.com/v1/chat/completions", "method": "POST", "headers": {"Authorization": "Bearer YOUR_API_KEY"}, "req_template_json_object": { "model": "custom-model", "messages": [{"role": "user", "content": "$INPUT"}] }, "response_json": true, "response_json_field": "$.choices[0].message.content" }'

Test local model with Ollama (no API key needed)

augustus scan ollama.OllamaChat \ --probe dan.Dan_11_0 \ --config '{"model":"llama3.2:3b"}' ```

API Keys

export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." export COHERE_API_KEY="..."

CLI Reference

Usage: augustus scan <generator> [flags]

Arguments:
  <generator>                 Generator name (e.g., openai.OpenAI, anthropic.Anthropic)

Probe Selection (choose one):
  --probe, -p                 Probe name (repeatable)
  --probes-glob               Comma-separated glob patterns (e.g., "dan.*,goodside.*")
  --all                       Run all registered probes

Detector Selection:
  --detector                  Detector name (repeatable)
  --detectors-glob            Comma-separated glob patterns

Buff Selection:
  --buff, -b                  Buff names to apply (repeatable)
  --buffs-glob                Comma-separated buff glob patterns (e.g., "encoding.*")

Configuration:
  --config-file               Path to YAML config file
  --config, -c                JSON config for generator

Execution:
  --harness                   Harness name (default: probewise.Probewise)
  --timeout                   Overall scan timeout (default: 30m)
  --probe-timeout             Per-probe timeout (default: 5m)
  --concurrency               Max concurrent probes (default: 10, env: AUGUSTUS_CONCURRENCY)

Output:
  --format, -f                Output format: table, json, jsonl (default: table)
  --output, -o                JSONL output file path
  --html                      HTML report file path
  --verbose, -v               Verbose output

Global:
  --debug, -d                 Enable debug mode

Commands:

augustus version              # Print version information
augustus list                 # List available probes, detectors, generators, harnesses, buffs
augustus scan <generator>     # Run vulnerability scan
augustus completion <shell>   # Generate shell completion (bash, zsh, fish)

Exit Codes:

Can I test local models without API keys?

Yes! Use the Ollama integration for local model testing:

```bash

No API key needed

augustus scan ollama.OllamaChat \ --probe dan.Dan_11_0 \ --config '{"model":"llama3.2:3b"}' ```

Error: "API rate limit exceeded"

Cause: Too many concurrent requests or requests per minute.

Solutions: 1. Reduce concurrency: --concurrency 5 2. Use provider-specific rate limit settings in YAML config:

   generators:
     openai.OpenAI:
       rate_limit: 10  # requests per minute
   

Error: "invalid API key" or "authentication failed"

Cause: Missing or invalid API credentials.

Solutions: 1. Verify environment variable is set: echo $OPENAI_API_KEY 2. Check for typos in config file 3. Ensure API key has required permissions 4. For Ollama, ensure the service is running: ollama serve

CPU-only instance (~$0.08/hr) - cloud APIs only

make devpod-up-cpu

Scan Pipeline

1. Probe Selection: Choose probes by name, glob pattern, or --all
2. Buff Transformation: Optionally transform prompts (encode, paraphrase, translate, poeticize)
3. Generator Call: Send adversarial prompts to the target LLM via its provider integration
4. Detector Analysis: Analyze responses using pattern matching, LLM-as-a-judge, or specialized detectors
5. Result Recording: Score each attempt and produce output in the requested format
6. Attack Engine: For iterative probes (PAIR, TAP), the single-turn attack engine refines prompts across iterations with candidate pruning and judge-based scoring
7. Multi-Turn Engine: For conversational probes (Crescendo, GOAT), the multi-turn engine maintains full conversation history with the target across turns, with refusal detection and dynamic adaptation

Run specific package tests

go test ./pkg/scanner -v

How does Augustus compare to garak?

Augustus is a Go-native reimplementation inspired by garak (NVIDIA's Python-based LLM vulnerability scanner). Key differences: - Performance: Go binary vs Python interpreter — faster execution and lower memory usage - Distribution: Single binary with no runtime dependencies vs Python package with pip install - Concurrency: Go goroutine pools (cross-probe parallelism) vs Python multiprocessing pools (within-probe parallelism) - Probe coverage: Augustus has 210+ probes; garak has 160+ probes with a longer research pedigree and published paper (arXiv:2406.11036) - Provider coverage: Augustus has 28 providers; garak has 35+ generator variants across 22 provider modules

Run equivalence tests (compare Go vs Python implementations)

go test ./tests/equivalence -v

FAQ

Troubleshooting