Local execution — your API key is sent only to the relay URL you choose

AI API Relay Security Audit

API Relay Audit is a local security audit tool for AI API relays and LLM proxies. It detects prompt injection, model substitution, tool rewriting, SSE anomalies, error leakage, and Web3 wallet risks. Your API key is sent only to the relay URL you choose.

14
Audit Steps
6D
Risk Matrix
708
Unit Tests
0
Dependencies (standalone)
Start Audit See Demo Report

What is API Relay Audit?

API Relay Audit is a local security audit tool for AI API relays and LLM proxies. It checks whether a third-party relay injects prompts, substitutes models, rewrites tool output, leaks credentials in error responses, or produces stream integrity anomalies.

When to use it

  • Audit third-party AI API relays, mirrors, gateways, and LLM proxies.
  • Check Claude-compatible or OpenAI-compatible proxies before production traffic.
  • Test relay behavior before coding-agent automation or wallet-related actions.

What it does not claim

  • It does not certify that a relay is safe.
  • It does not replace manual security review.
  • It does not treat inconclusive as clean.
How your Key flows — the critical difference
You
→
Your Terminal
→
Your Relay
→
Anthropic / OpenAI
api-relay-audit: no extra checker server sees your key vs Web tools: Key goes to a 3rd-party server first
The base URL of the relay you want to audit
Stays in your browser to generate the command. When you run it, the key is sent only to your chosen relay URL.
1
Open your terminal (macOS: Terminal.app / Windows: PowerShell)
2
Paste this command and press Enter — takes ~2 minutes:
3
Done! A Markdown audit report will be saved to your current directory.
Terminal

Install as an OpenClaw or Hermes skill — let an agent run the local 14-step audit from a secure environment.

1
Install the Skill

Use Hermes direct install now, or OpenClaw after ClawHub publication:

hermes skills install toby-bridges/api-relay-audit/skills/api-relay-audit
2
Trigger with one sentence

Set the key through the agent's secure environment, then say:

"Audit this relay: https://your-relay.com/v1 using API_RELAY_AUDIT_KEY"
3
Agent does everything

The agent downloads the pinned script, runs the 14-step audit locally, and presents the findings — you read the report.

Same security model — the agent runs locally on your machine, and the key is sent only to your chosen relay URL.

Each audit costs ~$0.2-0.5 in API usage (billed by your provider)

Your API key is only sent to the relay URL you specify; it is not sent to API Relay Audit or an extra web checker.

Audit Report Demo

Real audit results from three relay services — click tabs to compare

Domain names redacted. Data from actual audits run with api-relay-audit.

Detect Prompt Injection and Model Substitution

Threat taxonomy based on Liu et al., "Your Agent Is Mine" (arXiv:2604.08407)

Step 1-2

Infrastructure Recon

DNS, CDN, SSL certificate, management panel fingerprint, model list enumeration — understand what's behind the relay.

Step 3

Token Injection (AC-1)

Compares actual token usage against expected values. Hidden system prompt injection adds extra tokens — the delta reveals it.

Step 4 & 6

Prompt Extraction

3 attack vectors attempt to extract hidden system prompts: verbatim recall, translation trick, JSON continuation. Plus jailbreak resistance tests.

Step 5

Identity Substitution

An identity keyword set detects if "Claude" is actually GPT, DeepSeek, GLM, Qwen, or other models in disguise. Anchor phrases confirm true identity.

Step 7

Context Truncation

5 canary markers + binary search pinpoint the real context window boundary. Is your 200K context really 200K?

Step 8 (AC-1.a)

Tool-Call Rewriting

Checks if the relay silently modifies package install commands in responses — typosquatting supply-chain attacks at the proxy layer.

Step 9 (AC-2)

Error Response Leakage

7 deliberately broken requests probe for API key, env vars, file paths, and LiteLLM internals leaking in error responses.

Step 10-11

Stream Integrity & Web3

SSE event whitelist, usage monotonicity, thinking signature validity, model identity check. Plus Web3 signature-isolation probes (profile-gated).

How We Compare

Three tools, three approaches — pick the right one for your needs

Dimension api-relay-audit hvoy.ai cctest.ai
Token Injection✓✕✓
Prompt Extraction✓✕✕
Identity Substitution✓✓✕
Jailbreak Resistance✓✕✕
Context Truncation✓✕✕
Tool-Call Rewriting (AC-1.a)✓✕✕
Error Response Leakage (AC-2)✓✕✕
Stream Integrity (SSE)✓✓✕
Web3 Injection✓✕✕
Upstream Channel Classifier✓✕✓
Local Execution (No extra checker server)✓✕✕
Fully Open Source✓Partial✕
Public Leaderboard✕✓✕
Structured Audit Report✓✕✕

Guides for AI API Relay Security

Short, citation-friendly pages for GitHub, Google, and AI summaries.

What is an AI API relay or LLM proxy?

Define the trust boundary and what an intermediary can change.

How to audit a Claude API relay safely

Run a local audit without adding another API-key trust hop.

api-relay-audit vs hvoy.ai vs cctest.ai

Compare local audits, relay lookup, and web-based checks.

Detecting prompt injection in LLM API proxies

Understand token deltas, extraction probes, and identity signals.

Web3 wallet prompt injection through AI agents

Check transfer guidance, signed-transaction refusal, and private-key refusal.

OpenClaw and Hermes skill for relay audits

Run local AI API relay audits from agent workflows without adding a web checker.

FAQ

What is an API relay or LLM proxy?
An API relay or LLM proxy is a third-party service between you and an AI provider such as Anthropic or OpenAI. It forwards your requests upstream, but it can also inject hidden instructions, swap models, truncate context, rewrite tool output, or leak credentials in error responses.
Is it safe to enter my API Key?
API Relay Audit runs locally, so your API key is sent only to the relay URL you specify. No data is sent to API Relay Audit servers, and the standalone version has zero Python package dependencies.
What does prompt injection mean here?
Prompt injection means the relay may prepend or insert hidden instructions into your request. API Relay Audit compares token usage, runs extraction probes, and records evidence when hidden prompt content appears.
What is model substitution?
Model substitution means the relay claims to provide one model but routes you to another model or leaks a different model identity. API Relay Audit checks identity patterns and stream model identity where available.
What is tool-call rewriting?
Tool-call rewriting means the relay modifies package-install commands or tool-like output in the model response. API Relay Audit compares pinned package commands against returned text to detect proxy-layer supply-chain tampering.
What are SSE anomalies?
SSE anomalies are stream-level integrity issues in Anthropic-style streaming responses. API Relay Audit checks event types, usage monotonicity, thinking signatures, and stream model identity when supported.
What Web3 wallet risks does it check?
With the web3 or full profile, API Relay Audit checks transfer guidance, signed-transaction refusal, and private-key refusal behavior before wallet-related traffic is trusted.
What does inconclusive mean?
Inconclusive means the tool could not determine a clean or anomalous result for that step. Blocked probes, unsupported formats, and ambiguous responses are not treated as clean.