# Control 2.20: Adversarial Testing and Red Team Framework

**Control ID:** 2.20<br>
**Pillar:** Management<br>
**Regulatory Reference:** OCC Bulletin 2026-13 (formerly OCC Bulletin 2011-12), Federal Reserve SR 26-2 (formerly SR 11-7), FINRA Rule 3110, FINRA RN 24-09, NIST AI RMF (with Generative AI Profile, NIST AI 600-1), MITRE ATLAS, OWASP Top 10 for LLM Applications (2025)<br>
**Last UI Verified:** April 2026<br>
**Governance Levels:** Baseline / Recommended / Regulated

---

## Objective

Establish a proactive adversarial testing program to identify vulnerabilities in AI agents before deployment through red team exercises, prompt injection testing, jailbreak simulations, and robustness validation using golden datasets.

---

## Why This Matters for FSI

- **OCC Bulletin 2026-13 (formerly OCC Bulletin 2011-12) / Federal Reserve SR 26-2 (formerly SR 11-7):** Effective challenge of model design helps meet supervisory expectations for independent stress and robustness testing of AI agents used in business decisions.
- **FINRA Rule 3110 / Regulatory Notice 24-09 (Generative AI / LLM Guidance):** Documented adversarial testing supports the firm's supervisory system and Written Supervisory Procedures (WSPs) for AI tools used by associated persons.
- **SEC Rule 17a-4(b)(4) / 18a-6:** Red-team test artifacts (prompts, responses, severity, remediation) are business records that must be preserved when they evidence supervision of an AI system.
- **GLBA 501(b) Safeguards Rule:** Periodic adversarial assessment helps demonstrate ongoing evaluation of safeguards on systems handling customer NPI.
- **NIST AI RMF (MEASURE 2.6, MANAGE 4.1):** Structured red-team programs support the "Measure" and "Manage" functions for generative AI risk.

---

!!! info "No companion solution by design"
    Not all controls have a companion solution in [FSI-AgentGov-Solutions](https://github.com/judeper/FSI-AgentGov-Solutions); solution mapping is selective by design. This control is operated via native Microsoft admin surfaces and verified by the framework's assessment-engine collectors. See the [Solutions Index](../../reference/solutions-index.md#coverage-scope) for the catalog and coverage scope.


## Control Description

This control establishes a proactive adversarial testing program — distinct from the **detection** posture in Control 1.21 (Adversarial Input Logging) and the general QA posture in Control 2.5 (Testing & Validation). The program is built on six elements:

1. **Red Team Program Charter** — Documented scope, rules of engagement, authorization, separation of duties, and out-of-scope assets. Aligned with the firm's WSPs and the Microsoft AI Red Team operating model.
2. **Attack Library** — Curated, version-controlled test cases covering OWASP Top 10 for LLM Applications (2025), MITRE ATLAS techniques, and FSI-specific abuse cases (NPI exfiltration, MNPI elicitation, unsuitable-recommendation prompting, suitability bypass).
3. **Golden Dataset Validation** — Reference datasets of representative domain-specific Q&A pairs used to detect regressions in agent accuracy and refusal behaviour after configuration or model changes.
4. **Tooling and Automation** — Integration with Microsoft PyRIT (Python Risk Identification Toolkit) for orchestrated probes against Azure OpenAI / Microsoft Foundry (formerly Azure AI Foundry) -backed agents, and Microsoft Foundry built-in **Risk and Safety Evaluations** for content-safety scoring.
5. **Evidence and Remediation Tracking** — SHA-256-hashed, WORM-retained test packs with severity classification (Critical / High / Medium / Low) and SLA-based remediation tracked to closure.
6. **Cadence and Independence** — Pre-deployment gates plus quarterly (Zone 2) or continuous + annual third-party (Zone 3) cycles, with operator role separation enforced for model-validation independence per OCC Bulletin 2026-13 (formerly OCC 2011-12).

Adversarial testing is preventive; Control 1.21 is detective. Both should be deployed together for Zone 2 and Zone 3 agents.

---

## Key Configuration Points

- Charter the red team program with named operators, rules of engagement, authorization, and out-of-scope assets; review annually.
- Maintain an attack library that maps to **OWASP Top 10 for LLM Applications (2025)** and **MITRE ATLAS** technique IDs; version-control under change management.
- Build and maintain a golden dataset of at least 100 domain-specific Q&A pairs per Zone 3 agent; refresh after any material model, prompt, or grounding-source change.
- Integrate adversarial probes into the pre-deployment pipeline (gate releases on a defined defense-rate threshold per zone).
- Use **Microsoft PyRIT** for orchestrated probing of Azure OpenAI / Azure AI Foundry-backed agents; use Azure AI Foundry **Risk and Safety Evaluations** for content-safety scoring.
- Define remediation SLAs: Critical (24 hours to fix or compensating control), High (7 days), Medium (30 days), Low (next release).
- Schedule quarterly red-team exercises for Zone 2 agents; continuous testing plus annual independent third-party assessment for Zone 3.
- Preserve test artefacts (prompts, responses, severity, remediation, SHA-256 manifest) to immutable storage for the firm's records-retention horizon (commonly 6+ years for FINRA 4511 / SEC 17a-4(b)(4)).
- Enforce operator role separation — the red-team operator may not also be the agent's primary developer or owner without a co-signer.

---

## Zone-Specific Requirements

| Zone | Requirement | Rationale |
|------|-------------|-----------|
| **Zone 1** (Personal) | Basic prompt injection tests; pre-deployment only | Low risk, minimal adversarial exposure |
| **Zone 2** (Team) | Comprehensive test suite; quarterly testing; golden dataset validation | Shared agents warrant structured testing |
| **Zone 3** (Enterprise) | Full red team program; continuous testing; third-party assessment annually; immediate remediation | Customer-facing requires maximum adversarial validation |

---

## Roles & Responsibilities

| Role | Responsibility |
|------|----------------|
| AI Governance Lead | Charter, scope, rules of engagement, remediation oversight, and reporting to the AI governance committee |
| Security Architect | Maintain attack library; align test coverage with MITRE ATLAS and OWASP LLM Top 10 (2025) |
| Cloud Security Architect | Execute adversarial probes; tune detection patterns; reconcile findings with Control 1.21 detection telemetry |
| Model Risk Manager | Independent challenge per OCC Bulletin 2026-13 (formerly OCC 2011-12) / Fed SR 26-2 (formerly SR 11-7); sign off on golden dataset adequacy |
| Agent Owner | Remediate identified vulnerabilities within SLA; provide configuration and prompt-design fixes |
| Microsoft Copilot Studio Agent Author | Implement remediations (topic, instruction, grounding-scope, content-filter changes) |
| Compliance Officer | Confirm WSP language reflects documented testing cadence and findings retention |

---

## Related Controls

| Control | Relationship |
|---------|--------------|
| [1.21 - Adversarial Input Logging](../pillar-1-security/1.21-adversarial-input-logging.md) | Detection complements proactive testing |
| [2.5 - Testing & Validation](2.5-testing-validation-and-quality-assurance.md) | Adversarial testing is part of validation |
| [2.6 - Model Risk Management](2.6-model-risk-management-sr-26-2.md) | Red team supports model validation |
| [2.11 - Bias Testing](2.11-bias-testing-and-fairness-assessment.md) | Complementary testing approach |

---

## Implementation Playbooks

!!! info "Step-by-Step Implementation"

    This control has detailed playbooks for implementation, automation, testing, and troubleshooting:

    - [Portal Walkthrough](../../playbooks/control-implementations/2.20/portal-walkthrough.md) — Step-by-step portal configuration
    - [PowerShell Setup](../../playbooks/control-implementations/2.20/powershell-setup.md) — Automation scripts
    - [Verification & Testing](../../playbooks/control-implementations/2.20/verification-testing.md) — Test cases and evidence collection
    - [Troubleshooting](../../playbooks/control-implementations/2.20/troubleshooting.md) — Common issues and resolutions

---

## Verification Criteria

Confirm control effectiveness by verifying:

1. Red team program charter exists, is approved, and documents scope, authorization, rules of engagement, and operator role separation.
2. Attack library covers all OWASP Top 10 for LLM Applications (2025) categories and at least the high-relevance MITRE ATLAS techniques for the agent's deployment surface.
3. Golden dataset for each Zone 3 agent contains ≥100 representative Q&A pairs and shows a documented refresh date within the last quarter.
4. Pre-deployment adversarial probes run automatically and gate releases against a defined defense-rate threshold; evidence pack (prompts, responses, severity, manifest with SHA-256 hashes) is generated per run.
5. Remediation tracking shows SLA conformance for identified vulnerabilities; overdue items have a documented compensating control and risk-acceptance signature.
6. For Zone 3 agents, an annual independent third-party adversarial assessment report is on file and findings are tracked to closure.
7. Test artefacts are stored on immutable / WORM-capable media for the firm's records-retention horizon and are reconcilable with Control 1.21 detection events by ConversationId / AgentId.

---

## Additional Resources

- [MITRE ATLAS: Adversarial Threat Landscape for AI Systems](https://atlas.mitre.org/) (tactics and techniques catalog)
- [OWASP Top 10 for LLM Applications (2025)](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
- [Microsoft AI Red Team](https://learn.microsoft.com/en-us/security/ai-red-team/)
- [Microsoft PyRIT — Python Risk Identification Toolkit for generative AI](https://learn.microsoft.com/en-us/security/ai-red-team/)
- [Azure AI Foundry — Risk and Safety Evaluations for generative AI](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-metrics-built-in)
- [Azure AI Content Safety — Prompt Shields](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection)
- [NIST AI Risk Management Framework (with Generative AI Profile, NIST AI 600-1)](https://www.nist.gov/itl/ai-risk-management-framework)
- [FINRA Regulatory Notice 24-09 — Generative AI guidance (June 2024)](https://www.finra.org/rules-guidance/notices/24-09)

---

*Updated: May 2026 | Version: v1.6.2 | UI Verification Status: Current*
