# Control 3.10: Hallucination Feedback Loop

**Control ID:** 3.10<br>
**Pillar:** Reporting<br>
**Regulatory Reference:** CFPB UDAAP, SOX 302, FINRA 4511, FINRA 3110, FINRA RN 24-09, SEC 17a-4<br>
**Last UI Verified:** April 2026<br>
**Governance Levels:** Baseline / Recommended / Regulated<br>

---

## Objective

Establish a systematic process for capturing, categorizing, and remediating AI agent hallucinations (factually incorrect, fabricated, or misleading outputs) to enable continuous improvement of agent accuracy and provide quality management evidence for regulatory purposes.

---

## Why This Matters for FSI

- **CFPB UDAAP:** Tracks and remediates misleading outputs that could constitute unfair or deceptive practices
- **SOX 302:** Helps support accuracy of financial information delivered by AI agents
- **FINRA Rule 3110 (Supervision) and Regulatory Notice 24-09:** Supports supervisory review of generative AI outputs; FINRA's Gen AI guidance applies existing supervision rules to AI on a technology-neutral basis and expects firms to monitor for hallucinations, bias, and accuracy issues in agent outputs
- **FINRA Rule 4511:** Documents quality management activity as part of books and records obligations
- **SEC 17a-4:** Preserves hallucination evidence and remediation records under broker-dealer record retention requirements (typically 6 years, first two readily accessible)

---

## Control Description

!!! warning "Detection Limitations (January 2026)"
    **No automated hallucination detection exists in Microsoft Copilot Studio.** All hallucination identification relies on manual user feedback (CSAT thumbs up/down, explicit flagging) and human review. Organizations must implement structured feedback collection and manual review workflows rather than expecting automatic detection of inaccurate outputs. Published research on LLM accuracy varies significantly by model, domain, and evaluation methodology; organizations should establish their own baseline accuracy metrics for each agent use case rather than relying on generalized industry estimates.

This control establishes a hallucination feedback loop through:

1. **User Feedback Collection** - Thumbs down, flag, and report mechanisms
2. **Hallucination Categorization** - Taxonomy for classifying inaccuracy types (factual errors, fabrications, outdated info, calculation errors)
3. **Remediation Tracking** - Workflow with defined SLAs by severity (Critical: 4hrs, High: 24hrs, Medium: 72hrs)
4. **Trend Analysis** - Pattern identification and dashboards to detect systemic problems
5. **Continuous Improvement** - Integration with knowledge source updates and prompt refinement

---

## Key Configuration Points

- Enable user feedback (thumbs up/down) in Copilot Studio agent settings
- Define hallucination taxonomy: Factual Error, Fabrication, Outdated, Misattribution, Calculation Error, Conflation, Overconfidence, Misleading
- Create SharePoint tracking list or integrate with ServiceNow/Jira
- Configure Power Automate workflows for automated routing and escalation
- Set up trend reporting dashboards in Power BI
- Establish severity-based SLAs and escalation paths

!!! tip "Automation Available"
    See [Hallucination Tracker](https://github.com/judeper/FSI-AgentGov-Solutions/tree/main/hallucination-tracker) in FSI-AgentGov-Solutions for multi-source feedback collection, pattern detection with clustering, and integration with FINRA Supervision Workflow for high-severity hallucinations.

### Mitigation Strategies

Since automated detection is not available, organizations should implement proactive mitigation:

| Strategy | Description | FSI Application |
|----------|-------------|-----------------|
| **Explicit Fallbacks** | Configure "I don't know" responses for low-confidence queries | Helps prevent fabrication in compliance-sensitive contexts |
| **Grounding Requirements** | Require citation of source documents for factual claims | RAG-based agents surface source attribution |
| **Human-in-the-Loop** | Require human approval for high-stakes outputs | Investment advice, regulatory filings |
| **Response Confidence Thresholds** | Filter responses below confidence threshold | Reduce low-quality outputs reaching users |
| **Source Restriction** | Limit knowledge sources to verified content | Reduce reliance on potentially inaccurate data |

### Feedback Capture Mechanisms

| Mechanism | Native Support | Data Location | FSI Use Case |
|-----------|---------------|---------------|--------------|
| **CSAT (Thumbs)** | Yes | Copilot Studio Analytics | Basic quality signal |
| **Custom Feedback Form** | Via Topics | SharePoint/Dataverse | Structured categorization |
| **Application Insights** | Yes | Custom telemetry | Detailed conversation analysis |
| **Conversation Transcript** | Yes | Dataverse | Full context for RCA |

!!! info "Copilot Studio Analytics Retention Windows (May 2026)"
    Analytics data (including CSAT feedback signals) is available for up to **180 days**; session details and transcript information is available for the last **28 days** (per Microsoft Learn). Hallucination tracking records, flagged-session transcripts, and remediation evidence required beyond 28 days must be exported to SharePoint/Dataverse or a retention-bound store before the session window closes. The 28-day limit is especially relevant for root-cause analysis workflows that depend on transcript review.

---

## Zone-Specific Requirements

| Zone | Requirement | Rationale |
|------|-------------|-----------|
| **Zone 1** (Personal) | Basic thumbs up/down; quarterly review of patterns | Low risk, minimal tracking needed |
| **Zone 2** (Team) | Structured feedback with categorization; weekly review; integration with team issue tracking | Shared agents need quality monitoring |
| **Zone 3** (Enterprise) | Comprehensive tracking with full workflow; real-time alerting; formal RCA for critical issues; regulatory-ready documentation | Customer-facing agents require rigorous quality control |

---

## Roles & Responsibilities

| Role | Responsibility |
|------|----------------|
| AI Governance Lead | Feedback process ownership, remediation oversight, trend analysis |
| Power Platform Admin | Configure feedback mechanisms, workflow automation |
| QA Lead | Validate fixes, verify remediation effectiveness |
| Content Owner | Update knowledge sources based on root cause findings |

!!! note "Read-only Analytics Access for QA Reviewers"
    Quality reviewers and content owners often need visibility into CSAT thumbs-down trends and hallucination patterns surfaced on the agent's Analytics page without holding edit rights on the agent. The Copilot Studio **Analytics Viewer** sharing role helps meet this need by granting read-only access to the Analytics page; pair it with the **Bot Transcript Viewer** role to also expose conversation transcripts for root-cause analysis. The role is shared by the agent owner via the agent's three-dots menu → **Share** → **Analytics viewer** and **must be assigned to individual users — security groups are not supported**, so maintain a named-individual attestation log to support FINRA 3110 supervisory evidence. See [Share an agent](https://learn.microsoft.com/en-us/microsoft-copilot-studio/admin-share-bots).

---

## Related Controls

| Control | Relationship |
|---------|--------------|
| [2.9 - Performance Monitoring](../pillar-2-management/2.9-agent-performance-monitoring-and-optimization.md) | Baseline quality metrics for comparison |
| [3.4 - Incident Reporting](3.4-incident-reporting-and-root-cause-analysis.md) | Critical issue escalation path |
| [2.16 - RAG Source Integrity](../pillar-2-management/2.16-rag-source-integrity-validation.md) | Knowledge source updates when root cause identified |

---

## Implementation Playbooks

!!! info "Step-by-Step Implementation"

    This control has detailed playbooks for implementation, automation, testing, and troubleshooting:

    - [Portal Walkthrough](../../playbooks/control-implementations/3.10/portal-walkthrough.md) — Step-by-step portal configuration
    - [PowerShell Setup](../../playbooks/control-implementations/3.10/powershell-setup.md) — Automation scripts
    - [Verification & Testing](../../playbooks/control-implementations/3.10/verification-testing.md) — Test cases and evidence collection
    - [Troubleshooting](../../playbooks/control-implementations/3.10/troubleshooting.md) — Common issues and resolutions

---

## Verification Criteria

Confirm control effectiveness by verifying:

1. Test feedback via thumbs down creates tracking item with correct categorization
2. Critical hallucinations trigger incident creation within 4-hour SLA
3. Remediation workflow progresses through all status stages correctly
4. Trend reports generate and display metrics (hallucination rate, MTTR, category distribution)
5. Root cause findings integrate with knowledge source updates (Control 2.16)

---

## Additional Resources

- [Microsoft Learn: Copilot Studio Analytics](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-overview)
- [Microsoft Learn: Customer Satisfaction (CSAT) Analytics](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-csat)
- [Microsoft Learn: Sessions Analytics](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-sessions)
- [Microsoft Learn: Power Automate Approval Workflows](https://learn.microsoft.com/en-us/power-automate/get-started-approvals)
- [FINRA Regulatory Notice 24-09 — Generative AI guidance (June 2024)](https://www.finra.org/rules-guidance/notices/24-09)
- [FINRA Rule 3110 — Supervision](https://www.finra.org/rules-guidance/rulebooks/finra-rules/3110)
- [SEC Rule 17a-4 — Records to be Preserved by Broker-Dealers](https://www.ecfr.gov/current/title-17/chapter-II/part-240/subject-group-ECFRdcfdde6c8732c0a/section-240.17a-4)

---

*Updated: May 2026 | Version: v1.6.2 | UI Verification Status: Current*
