# Control 2.18: Automated Conflict of Interest Testing

**Control ID:** 2.18<br>
**Pillar:** Management<br>
**Regulatory Reference:** SEC Reg BI, SEC Rule 10b-5, FINRA 2111, FINRA RN 24-09, FINRA Rule 3110<br>
**Last UI Verified:** April 2026<br>
**Governance Levels:** Baseline / Recommended / Regulated<br>

---

## Objective

Implement automated testing to detect potential conflicts of interest in AI agent recommendations, particularly for agents providing product recommendations, investment guidance, or financial advice, supporting compliance with Regulation Best Interest.

---

## Why This Matters for FSI

- **SEC Regulation Best Interest (Reg BI) / FINRA Rule 2111 (Suitability):** Broker-dealers must act in the best interest of retail customers when making recommendations. The FINRA 2026 Annual Regulatory Oversight Report's dedicated GenAI section reinforces that suitability and best-interest obligations are technology-neutral — firms cannot outsource these obligations to AI systems, and human supervisors must validate that AI-generated recommendations are suitable for the specific customer.
- **SEC Rule 10b-5 (Anti-Fraud):** Prohibits material misstatements, omissions, and self-serving recommendations; helps ensure AI agents do not steer clients toward proprietary or higher-fee products without a documented best-interest basis.
- **FINRA Rule 3110 (Supervision):** Requires written supervisory procedures (WSPs) and reasonable supervision of recommendations. Pre-deployment and ongoing conflict testing supports the supervisory diligence FINRA expects for GenAI-assisted recommendations.
- **FINRA RN 24-09 / Rule 3110 (AI Supervision):** Reinforces that firms must implement testing, monitoring, and documentation of AI tools used in recommendations to retail customers.

!!! info "Updated April 2026"
    The FINRA 2026 Annual Regulatory Oversight Report (December 2025) introduced a standalone GenAI section emphasizing pre-deployment testing, output monitoring, model risk management, and that obligations under Reg BI, Rule 2111, and Rule 3110 apply to AI-assisted recommendations on the same basis as human recommendations.

---

## Control Description

This control establishes conflict testing through:

1. **Proprietary Bias Detection** - Test for preferential recommendation of firm's own products
2. **Commission Bias Testing** - Detect bias toward higher-compensation products
3. **Cross-Selling Analysis** - Identify inappropriate bundling or upselling patterns
4. **Competitor Fairness** - Ensure competitor products aren't unfairly excluded
5. **Prompt Audit** - Review system prompts for prohibited bias instructions
6. **Scoring Validation** - Verify product scoring algorithms are unbiased

### Conflict Types to Test

| Conflict Type | Description | Example |
|---------------|-------------|---------|
| **Employee vs Customer** | Agent recommendations favor firm over client | Recommending proprietary products over better alternatives |
| **Customer vs Customer** | Agent treats different customers unfairly | Allocating limited resources to preferred customers |
| **Related-Party** | Agent involved in transactions with affiliated entities | Recommendations involving parent/subsidiary companies |
| **Cross-Business Unit** | Information barriers not properly enforced | Research influencing investment banking recommendations |

---

## Key Configuration Points

- Define conflict-of-interest test scenarios relevant to agent use case
- Create test datasets with comparable proprietary vs. competitor products
- Establish bias thresholds (e.g., proprietary recommendations should not exceed market share)
- Configure automated testing in pre-deployment pipeline
- Schedule recurring conflict testing (quarterly minimum)
- Audit system prompts for prohibited bias language
- Document testing methodology and results for examination

!!! tip "Automation Available"
    See [COI Testing Framework](https://github.com/judeper/FSI-AgentGov-Solutions/tree/main/coi-testing) in FSI-AgentGov-Solutions for automated conflict of interest testing with 10 predefined scenarios covering proprietary bias, suitability, fee transparency, and cross-selling.

!!! tip "Copilot Studio Evaluation Framework"
    Microsoft Copilot Studio's agent evaluation framework can complement automated COI testing with an 8-step methodology for evidence-based agent validation. Key capabilities include:

    - **Set-level grading** — Evaluate COI test scenarios across entire test sets with aggregate scoring, enabling statistically meaningful bias detection rather than individual response review
    - **Multi-dimensional graders** — Score responses across accuracy, groundedness, coherence, and relevance dimensions to assess recommendation quality and appropriateness
    - **Classification grading** — Supports detection of proprietary bias, commission bias, and suitability issues by classifying agent responses against expected categories
    - **Capability verification** — Helps validate that agents invoke the correct topics and tools for different recommendation scenarios
    - **Import/export test sets** — Import standardized COI test scenarios and export results for version control, supporting reproducible conflict testing across environments
    - **Production data import** — Import real-world recommendation interactions as test inputs to validate against actual customer scenarios
    - **Enterprise audit trail** — Purview integration captures evaluation activities and grading results, supporting examination readiness under FINRA Rule 3110 and Reg BI evidence retention requirements
    - **Comparative monitoring** — Enables sequential evaluation runs to track quality trends and detect regressions over time

    The evaluation framework supports observable, repeatable, and explainable testing—key attributes for demonstrating supervisory diligence under FINRA Rule 3110. See the [Verification & Testing playbook](../../playbooks/control-implementations/2.18/verification-testing.md) for evaluation methodology guidance and [Agent Evaluation in Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-intro) for platform documentation.

---

## Zone-Specific Requirements

| Zone | Requirement | Rationale |
|------|-------------|-----------|
| **Zone 1** (Personal) | Awareness only; no recommendation agents | Personal agents typically don't make recommendations |
| **Zone 2** (Team) | Basic conflict testing for recommendation agents; documented methodology | Team recommendation agents warrant validation |
| **Zone 3** (Enterprise) | Comprehensive Reg BI testing; automated monitoring; independent validation; quarterly review | Customer-facing recommendations require rigorous conflict controls |

---

## Roles & Responsibilities

| Role | Responsibility |
|------|----------------|
| Compliance Officer | Define testing requirements, validate Reg BI alignment |
| AI Governance Lead | Configure testing, oversee methodology |
| Model Risk Manager | Independent validation of testing approach |
| Agent Owner | Remediate identified conflicts, update prompts |

---

## Related Controls

| Control | Relationship |
|---------|--------------|
| [2.11 - Bias Testing](2.11-bias-testing-and-fairness-assessment.md) | Complementary bias testing for fairness |
| [2.6 - Model Risk Management](2.6-model-risk-management-sr-26-2.md) | Conflict testing is MRM component |
| [2.12 - Supervision](2.12-supervision-and-oversight-finra-rule-3110.md) | Supervisory review of recommendations |
| [2.20 - Adversarial Testing](2.20-adversarial-testing-and-red-team-framework.md) | Red team testing for hidden biases |

---

## Implementation Playbooks

!!! info "Step-by-Step Implementation"

    This control has detailed playbooks for implementation, automation, testing, and troubleshooting:

    - [Portal Walkthrough](../../playbooks/control-implementations/2.18/portal-walkthrough.md) — Step-by-step portal configuration
    - [PowerShell Setup](../../playbooks/control-implementations/2.18/powershell-setup.md) — Automation scripts
    - [Verification & Testing](../../playbooks/control-implementations/2.18/verification-testing.md) — Test cases and evidence collection
    - [Troubleshooting](../../playbooks/control-implementations/2.18/troubleshooting.md) — Common issues and resolutions

---

## Verification Criteria

Confirm control effectiveness by verifying:

1. Test scenarios defined for proprietary, commission, and cross-selling bias
2. Test datasets include comparable proprietary and competitor products
3. Automated conflict testing executes in pre-deployment pipeline
4. System prompts audited and free of prohibited bias instructions
5. Conflict testing report generated with statistical analysis

---

## Additional Resources

- [SEC Regulation Best Interest](https://www.sec.gov/about/divisions-offices/division-trading-markets/regulation-best-interest-form-crs-related-interpretations)
- [FINRA Rule 2111 — Suitability](https://www.finra.org/rules-guidance/rulebooks/finra-rules/2111)
- [FINRA Rule 3110 — Supervision](https://www.finra.org/rules-guidance/rulebooks/finra-rules/3110)
- [FINRA 2026 Annual Regulatory Oversight Report](https://www.finra.org/rules-guidance/guidance/reports/2026-finra-annual-regulatory-oversight-report)
- [SEC Examination Priorities](https://www.sec.gov/compliance/examination-priorities)
- [Microsoft Learn: About agent evaluation in Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-intro)
- [Microsoft Learn: Choose evaluation methods](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-overview)
- [Microsoft Learn: Iterative evaluation framework (4 stages)](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/evaluation-iterative-framework)
- [Microsoft Learn: Evaluation-driven triage and remediation](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/evaluation-triage-overview)

---

*Updated: May 2026 | Version: v1.6.2 | UI Verification Status: Current*
