# Control 3.2: Usage Analytics and Activity Monitoring

**Control ID:** 3.2<br>
**Pillar:** Reporting<br>
**Regulatory Reference:** FINRA 4511, FINRA RN 24-09, FINRA 3110, SEC 17a-3/4 (incl. 17a-4(f) electronic records), SOX 404, GLBA 501(b)<br>
**Last UI Verified:** April 2026<br>
**Governance Levels:** Baseline / Recommended / Regulated<br>

---

!!! note "Agent 365 Architecture Update"
    Agent 365 Observability provides rich usage analytics via Application Insights, offering consolidated activity monitoring across all agent types through a single telemetry pipeline. See [Unified Agent Governance](../../framework/agent-identity-architecture.md) for observability and usage analytics details.

## Objective

Establish comprehensive monitoring of AI agent usage, performance, and activity patterns through Power Platform Admin Center dashboards, alerts, and audit integration to support governance, compliance, and operational optimization.

---

## Why This Matters for FSI

- **FINRA Rule 4511 / RN 24-09:** Books-and-records and AI-specific recordkeeping require monitored evidence of agent activity, prompt/response capture, and supervisory review trails. Native PPAC retention (28 days) does not meet FINRA's six-year requirement on its own — pair with extended retention (Purview, Application Insights, or custom pipeline).
- **SEC 17a-3 / 17a-4(f):** Examination readiness depends on time-indexed, tamper-evident records of customer-data processing. SEC 17a-4(f) accepts electronic records when stored in non-rewriteable, non-erasable form (or with audit-trail equivalents under the 2022 amendments) for the prescribed retention period (generally 3–6 years).
- **SOX 404:** Monitoring of agents that touch financial-reporting data produces the operating-effectiveness evidence required for management's assertion and external-auditor walkthroughs.
- **GLBA 501(b):** Tracks access to customer NPI inside agent conversations and downstream connector calls; supports the "monitor" element of the Safeguards Rule risk-assessment lifecycle.
- **OCC Bulletin 2026-13 (formerly OCC 2011-12) / Fed SR 26-2 (formerly SR 11-7):** Model-risk management expects ongoing monitoring of model performance, drift, and operational issues — usage analytics, success rate, and deny-event monitoring contribute to that evidence base.

---

!!! tip "Automation Available"
    See [Compliance Dashboard](https://github.com/judeper/FSI-AgentGov-Solutions/tree/main/compliance-dashboard) in FSI-AgentGov-Solutions for aggregated compliance reporting across the framework control catalog with zone-based filtering.


## Control Description

Usage analytics and activity monitoring form the backbone of effective AI governance. The Power Platform Admin Center provides a comprehensive **Monitor** section with:

| Capability | Description | FSI Relevance |
|------------|-------------|---------------|
| **Monitor Overview** | Centralized view of platform health | Quick governance status |
| **Alerts (Preview)** | Pre-built and custom alert rules | Proactive issue detection |
| **Logs** | Activity and error logging | Audit trail and troubleshooting |
| **Microsoft Copilot Studio Dashboard** | Agent-specific success metrics | Agent performance monitoring |

For environments with Managed Environments enabled, additional usage insights including weekly digests and adoption trends are available.

### New Analytics Features (2025-2026)

Microsoft has expanded analytics capabilities significantly:

| Feature | Status | Release | Description |
|---------|--------|---------|-------------|
| **Agent Dashboard** | GA | Ignite 2025 | Centralized agent adoption measurement across M365 |
| **Action Usage Analytics** | GA | November 2025 | Connector and API call tracking per agent |
| **Copilot Benchmarks** | GA | Ignite 2025 | Peer comparison metrics across industry verticals |
| **Copilot Chat Insights** | Expanded | February 2026 | 50-license minimum removed; available to all tenants |
| **New Usage Page (PPAC)** | Preview | January 2026 | Unified usage dashboard with drill-down capabilities |

!!! info "Data Availability"
    - **DAU/MAU metrics** require user authentication; anonymous usage is not tracked
    - **Native retention** for PPAC analytics is 28 days; export to Log Analytics for extended retention
    - **Copilot Studio Analytics retention (May 2026):** analytics data is available for up to **180 days**; session details and transcript information is available for the last **28 days** (per [Microsoft Learn — analytics overview](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-overview)). Export conversation transcripts and session-level evidence to Log Analytics or a retention-bound store before the 28-day session window closes for any sessions requiring regulatory preservation.
    - **Settings changes** may take up to 8 hours to reflect in dashboards

### Custom Power BI Analytics Infrastructure

For organizations requiring analytics beyond native PPAC capabilities (extended retention, cross-environment correlation, custom metrics), implement a custom Power BI pipeline:

```
Dataverse (Agent Metadata)
         ↓
    Synapse Link
         ↓
  Azure Data Lake
         ↓
     Power BI
```

**Infrastructure Components:**

| Component | Purpose | FSI Consideration |
|-----------|---------|-------------------|
| **Dataverse** | Source for agent metadata, conversation logs, custom telemetry | Already included with Copilot Studio |
| **Synapse Link for Dataverse** | Real-time data export to Data Lake | Requires Azure subscription; Premium capacity recommended |
| **Azure Data Lake Storage Gen2** | Long-term storage for compliance retention | 7-year retention for SEC 17a-4; immutable storage optional |
| **Power BI Premium** | Enterprise reporting with large dataset support | Recommended for datasets >1GB |

**When to Implement Custom Pipeline:**

| Requirement | Native PPAC | Custom Pipeline |
|-------------|-------------|-----------------|
| 28-day retention | ✅ | ✅ |
| 1+ year retention | ❌ | ✅ |
| Cross-environment correlation | ❌ | ✅ |
| Custom KPIs/metrics | Limited | ✅ |
| Regulatory evidence export | Manual | Automated |
| Real-time dashboards | Limited | ✅ |

!!! warning "Licensing Requirements"
    Synapse Link requires Azure subscription. Power BI Premium or Premium Per User licensing recommended for large-scale deployments. Confirm costs before implementation.

---

## Key Configuration Points

- Enable access to PPAC Monitor section for governance personnel
- Configure pre-built Microsoft alert rules (e.g., "High-use agents have success rate under 90%")
- Create custom alerts for organization-specific thresholds
- Set up weekly digest notifications for Managed Environments
- Configure audit log integration for compliance reporting
- Establish dashboard review cadence by governance tier

---

## Zone-Specific Requirements

| Zone | Requirement | Rationale |
|------|-------------|-----------|
| **Zone 1** (Personal) | Monthly dashboard review; >80% success-rate target; rely on native PPAC + Purview Audit (Standard) retention (≥180 days where licensed). | Baseline visibility for low-risk personal productivity use; FSI books-and-records reach for these agents is limited. |
| **Zone 2** (Team) | Weekly dashboard review; >90% success rate; pre-built alerts enabled with named recipients; Purview Audit retention extended to ≥1 year; daily deny-event report (Policy Block, DLP Match, RAI ContentFiltered). | Shared agents create supervisory obligations under FINRA 3110/4511; weekly cadence and alerting support timely supervisory review. |
| **Zone 3** (Enterprise) | Daily dashboard review; >95% success rate; real-time alerting with documented response SLAs; executive monthly reporting; conversation-level telemetry (Application Insights "Log sensitive activity properties" enabled with PII governance); 6–7 year retention via Purview long-term retention or custom Synapse Link → ADLS pipeline aligned with SEC 17a-4(f). | Customer-facing and regulated workloads require examination-ready evidence with the longest retention horizon; near-real-time detection supports incident-response SLAs and OCC Bulletin 2026-13 (formerly OCC 2011-12) ongoing monitoring expectations. |

---

## Roles & Responsibilities

| Role | Responsibility |
|------|----------------|
| Power Platform Admin | Configure PPAC Monitor dashboards, alert rules, Managed Environment usage insights, and Copilot Studio analytics access. |
| Purview Audit Admin | Configure audit log retention policies (Standard / Premium), validate Copilot/PowerApps RecordType ingestion, and grant Purview Audit Reader access for evidence collection. |
| Purview Compliance Admin | Validate audit log search access, configure long-term retention policies aligned with FINRA/SEC requirements. |
| AI Governance Lead | Define monitoring requirements, dashboard cadence, and KPI thresholds by zone; review executive summaries. |
| Compliance Officer | Review compliance reports, validate evidence completeness, attest to monitoring posture for examinations. |
| Operations Team | Respond to alerts within zone-specific SLA; investigate performance and deny-event spikes; document incident handoffs. |

!!! note "Analytics Viewer Sharing Role (Copilot Studio)"
    Microsoft introduced an **Analytics Viewer** sharing role on Copilot Studio agents that grants read-only access to the agent's Analytics page without conferring edit rights. This supports separation-of-duties by allowing analysts, compliance officers, and business stakeholders to consume usage metrics without modify permissions on the agent itself. Pair with the **Bot Transcript Viewer** role to also surface conversation transcripts. The role is assigned by the agent owner via the agent's three-dots menu → **Share** → select **Analytics viewer**, and **can only be assigned to individual users — Microsoft Entra security groups are not supported**. Maintain an attestation log of named individuals granted the role to support audit traceability. See [Share an agent](https://learn.microsoft.com/en-us/microsoft-copilot-studio/admin-share-bots).

---

## Related Controls

| Control | Relationship |
|---------|--------------|
| [2.1 - Managed Environments](../pillar-2-management/2.1-managed-environments.md) | Required for usage insights |
| [1.7 - Audit Logging](../pillar-1-security/1.7-comprehensive-audit-logging-and-compliance.md) | Provides activity tracking foundation |
| [3.1 - Agent Inventory](./3.1-agent-inventory-and-metadata-management.md) | Correlates metrics to agent inventory |
| [3.3 - Compliance Reporting](./3.3-compliance-and-regulatory-reporting.md) | Compliance Dashboard integrates usage metrics ([Compliance Dashboard](https://github.com/judeper/FSI-AgentGov-Solutions/tree/main/compliance-dashboard)) |
| [3.4 - Incident Reporting](./3.4-incident-reporting-and-root-cause-analysis.md) | Alert-driven incident response |

---

## Implementation Playbooks

!!! info "Step-by-Step Implementation"

    This control has detailed playbooks for implementation, automation, testing, and troubleshooting:

    - [Portal Walkthrough](../../playbooks/control-implementations/3.2/portal-walkthrough.md) — Step-by-step portal configuration
    - [PowerShell Setup](../../playbooks/control-implementations/3.2/powershell-setup.md) — Automation scripts
    - [Verification & Testing](../../playbooks/control-implementations/3.2/verification-testing.md) — Test cases and evidence collection
    - [Troubleshooting](../../playbooks/control-implementations/3.2/troubleshooting.md) — Common issues and resolutions

!!! tip "Agent Usage & Performance Workbook"
    For organizations where ALM separation-of-duties policies restrict direct access to the Copilot Studio Analytics tab, the [Agent Usage & Performance Workbook](../../playbooks/advanced-implementations/agent-usage-workbook/index.md) provides Azure Monitor-based usage analytics, adoption metrics, and business value estimation — without requiring Copilot Studio portal access. See the [Deployment Guide](../../playbooks/advanced-implementations/agent-usage-workbook/deployment-guide.md) for setup instructions.

---

## Verification Criteria

Confirm control effectiveness by verifying:

1. PPAC Monitor section is accessible to designated administrators (Power Platform Admin role).
2. Pre-built alert rules are enabled and routed to named recipients with documented response SLAs.
3. Custom alerts are created for Zone 3 success-rate thresholds (<95%), after-hours activity, and customer-data access anomalies.
4. Weekly digest emails are received for Managed Environments and reach the compliance distribution.
5. Purview Audit captures Copilot Studio / PowerApps RecordType events and retention is configured to meet zone requirements (Zone 2 ≥1 year; Zone 3 ≥6 years aligned with FINRA 4511 / SEC 17a-4).
6. For Zone 2/3, daily deny-event reporting (Policy Block, DLP Match, RAI ContentFiltered, XPIA, Jailbreak) is operational and reviewed within SLA.
7. Where Application Insights is used for conversation-level monitoring (Zone 2/3), both the agent-level "Log sensitive activity properties" and the environment-level "Allow conversation transcripts" settings are enabled, and PII governance controls are applied to the App Insights resource.
8. Dashboard review cadence is documented per zone, and review attestations are retained as audit evidence.

---

## Daily Operational Monitoring

While weekly and monthly reporting address strategic oversight, Zone 2/3 environments require **daily operational monitoring** to detect and respond to deny events, content filtering, and policy blocks in near-real-time.

### Deny Event Categories

| Category | Source | Description | FSI Relevance |
|----------|--------|-------------|---------------|
| **Policy Block** | Purview Audit (CopilotInteraction) | DLP policy prevented resource access | NPI (Non-Public Information) protection evidence |
| **XPIA Detection** | Purview Audit (CopilotInteraction) | Cross-prompt injection attempt detected | Security incident tracking |
| **Jailbreak Attempt** | Purview Audit (CopilotInteraction) | User attempted to bypass agent guardrails | Adversarial input logging |
| **RAI Content Filter** | Application Insights (ContentFiltered) | Responsible AI filter blocked response | Model risk control evidence |
| **DLP Match** | Purview Audit (DlpRuleMatch) | Sensitivity-based blocking in Copilot location | Data governance enforcement |

### Daily Monitoring Cadence

| Zone | Monitoring Requirement | Response SLA |
|------|------------------------|--------------|
| **Zone 1** | Weekly summary (optional) | Best effort |
| **Zone 2** | Daily automated report | 4-hour response |
| **Zone 3** | Daily automated report + real-time alerts | 15-minute response |

### Implementation

For organizations requiring daily deny event correlation across Purview Audit, DLP, and Application Insights:

- [Deny Event Correlation Report Playbook](../../playbooks/advanced-implementations/deny-event-correlation-report/index.md) - Multi-source correlation with Power BI dashboard

### Environment Provisioning Monitoring

For baseline configuration that enables usage insights from environment creation:

- [Environment Lifecycle Management](../../playbooks/advanced-implementations/environment-lifecycle-management/index.md) - Environments created with usage insights enabled automatically

---

## Additional Resources

- [Power Platform Admin Center Monitoring](https://learn.microsoft.com/en-us/power-platform/admin/admin-activity-logging)
- [Set Up Alerts in PPAC](https://learn.microsoft.com/en-us/power-platform/admin/monitoring/monitor-copilot-studio)
- [Managed Environment Usage Insights](https://learn.microsoft.com/en-us/power-platform/admin/managed-environment-usage-insights)
- [Copilot Studio Analytics](https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-overview)
- [Unified Audit Log](https://learn.microsoft.com/en-us/purview/audit-search)

### Microsoft Audit Reporting Tools

For enhanced Copilot/AI reporting beyond native M365 Admin Center capabilities, see:

- [Microsoft Audit Reporting Tools Playbook](../../playbooks/advanced-implementations/microsoft-audit-reporting-tools.md) - AI-in-One Dashboard and PAX (Portable Audit eXporter) for enterprise-scale analytics

### Agent 365 Observability SDK (Preview)

!!! info "Agent 365 GA — May 2026"
    Microsoft Agent 365 reached general availability on May 1, 2026 (bundled in Microsoft 365 E7 or available as a standalone Microsoft Agent 365 per-user license). Agent Essentials category definitions and SDK feature scope continue to mature post-GA — verify current feature availability against Microsoft Learn before implementing production controls dependent on specific Agent 365 capabilities.

Agent 365 SDK introduces OpenTelemetry-based observability for Blueprint-registered agents, providing FSI-compliant telemetry capture and export capabilities.

**Observability SDK Architecture:**

```
Agent 365 SDK Application
         ↓
    Observability SDK
         ↓
  OpenTelemetry Collector
    ↓           ↓
Azure Monitor  Third-Party SIEM
    ↓
Application Insights → DSPM Activity Explorer
```

**Key Capabilities for FSI Compliance:**

| Capability | Description | Regulatory Alignment |
|------------|-------------|---------------------|
| **Structured Telemetry** | Captures agent interactions with consistent schema | FINRA 4511, FINRA RN 24-09 records requirements |
| **Prompt/Response Logging** | Full conversation capture for audit trail | FINRA 4511, FINRA RN 24-09 recordkeeping requirements |
| **Correlation IDs** | Links multi-turn conversations and agent handoffs | SEC 17a-3 transaction tracing |
| **Export Formats** | OTLP, JSON, compliance-ready formats | SOX 404 evidence requirements |

**Developer Instrumentation Guidance:**

For Zone 2-3 agents built with Agent 365 SDK, implement the following instrumentation:

```python
# Example: Agent 365 SDK Observability Configuration
from agent365.observability import TelemetryClient

telemetry = TelemetryClient(
    connection_string="<Application-Insights-Connection-String>",
    enable_prompt_logging=True,  # Required for Zone 3
    enable_pii_scrubbing=True,   # Recommended for all zones
    retention_days=365           # Align with regulatory requirements
)

# Instrument agent interactions
@telemetry.trace_interaction
async def process_request(request):
    # Agent logic here
    pass
```

**OpenTelemetry Integration Patterns:**

| Pattern | Use Case | Implementation |
|---------|----------|----------------|
| **Direct Export** | Single Application Insights instance | Configure SDK connection string |
| **Collector Sidecar** | Multi-destination routing | Deploy OTEL Collector with exporters |
| **Dapr Integration** | Kubernetes deployments | Use Dapr observability building block |

**Zone-Specific Observability Requirements:**

| Zone | Telemetry Requirement | Retention |
|------|----------------------|-----------|
| **Zone 1** | Basic metrics (optional) | 90 days |
| **Zone 2** | Full telemetry with prompt logging | 1 year |
| **Zone 3** | Full telemetry + real-time alerting + compliance export | 7–10 years |

**Integration with DSPM for AI:**

Observability SDK telemetry flows into DSPM Activity Explorer when:
1. Application Insights is configured as the telemetry destination
2. DSPM Extended Insights is enabled (see Control 1.6)
3. Agent 365 workload is included in DSPM scope

This enables sensitive data detection in agent conversations and supports oversharing assessments.

**Additional Resources:**

- [Microsoft Learn: Agent 365 SDK Overview (Preview)](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/) - Observability and telemetry capabilities
- [Microsoft Learn: OpenTelemetry with Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview) - OpenTelemetry integration patterns

### Observability by Agent Type

**(A) Copilot Studio Agents:**

- Power Platform Admin Center analytics
- Managed Environment insights
- Microsoft Purview Audit logs
- Application Insights (requires explicit enablement — see below)

!!! warning "Copilot Studio Application Insights: Sensitive Properties Required"
    When configuring Application Insights for Copilot Studio agents, two settings control whether conversation content is included in telemetry:

    | Setting | Location | Effect |
    |---------|----------|--------|
    | **Log activities** | Copilot Studio > Agent > Settings > Advanced > Application Insights | Enables basic (sanitized) telemetry flow |
    | **Log sensitive activity properties** | Copilot Studio > Agent > Settings > Advanced > Application Insights | Includes conversation text, user IDs, and node details in telemetry |
    | **Allow conversation transcripts** | PPAC > Environment > Settings > Product > Features | Tenant-level prerequisite for transcript storage |

    Without "Log sensitive activity properties" enabled, queries against the `customEvents` table (e.g., `BotMessage`, `UserMessage` events) will return records with **empty `text` fields** — creating a false sense of audit coverage.

    For FSI organizations relying on Application Insights for conversation-level monitoring (Zone 2-3), both the agent-level and environment-level settings must be enabled. PII governance controls should be applied to the Application Insights resource.

**(B) Agent 365 SDK Agents (Preview):**

- OpenTelemetry SDK integration
- Application Insights workbooks
- Custom telemetry configuration

---

*Updated: May 2026 | Version: v1.6.2 | UI Verification Status: Current*
