<!-- AUTO-GENERATED by export-skills.py — DO NOT EDIT -->
---
name: cloudwatch-observability
description: Amazon CloudWatch patterns for AI agent observability. Use when monitoring Bedrock agent invocations, tracking token usage, setting up alarms for agent failures, or analyzing agent performance via CloudWatch Logs Insights.
---

# Amazon CloudWatch for AI Agent Observability

Monitor AI agent performance, costs, and reliability using CloudWatch metrics, logs, and alarms.

## When to Use

- Monitoring Bedrock agent invocation latency, token usage, and error rates
- Setting up alarms for agent failures or cost spikes
- Analyzing agent reasoning traces via CloudWatch Logs Insights
- Building dashboards for AI operations

## Patterns

### 1. Bedrock Invocation Metrics

Key CloudWatch metrics emitted by Amazon Bedrock:

| Metric | Namespace | Description |
|---|---|---|
| `Invocations` | `AWS/Bedrock` | Number of model invocations |
| `InvocationLatency` | `AWS/Bedrock` | End-to-end invocation time (ms) |
| `InvocationClientErrors` | `AWS/Bedrock` | 4xx errors (throttling, validation) |
| `InvocationServerErrors` | `AWS/Bedrock` | 5xx errors |
| `InputTokenCount` | `AWS/Bedrock` | Input tokens consumed |
| `OutputTokenCount` | `AWS/Bedrock` | Output tokens generated |
| `InvocationThrottles` | `AWS/Bedrock` | Throttled requests |

### 2. CloudWatch Logs Insights for Agent Traces

```sql
-- Find slowest agent invocations in the last 24h
fields @timestamp, @message
| filter @message like /agentId/
| parse @message '"invocationLatencyMs":*,' as latency
| sort latency desc
| limit 20

-- Token usage by model over time
fields @timestamp
| filter @message like /inputTokenCount/
| parse @message '"modelId":"*"' as model
| parse @message '"inputTokenCount":*,' as input_tokens
| parse @message '"outputTokenCount":*,' as output_tokens
| stats sum(input_tokens) as total_input, sum(output_tokens) as total_output by model, bin(1h)

-- Agent errors with reasoning trace
fields @timestamp, @message
| filter @message like /ERROR/ or @message like /ThrottlingException/
| sort @timestamp desc
| limit 50
```

### 3. Cost Tracking Alarm

```python
import boto3

cloudwatch = boto3.client("cloudwatch")

# Alarm when daily token usage exceeds threshold
cloudwatch.put_metric_alarm(
    AlarmName="bedrock-daily-token-budget",
    Namespace="AWS/Bedrock",
    MetricName="InputTokenCount",
    Statistic="Sum",
    Period=86400,  # 24 hours
    EvaluationPeriods=1,
    Threshold=10_000_000,  # 10M tokens
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:ai-ops-alerts"],
    Dimensions=[{"Name": "ModelId", "Value": "anthropic.claude-3-5-sonnet-20241022-v2:0"}],
)

# Alarm for high error rate
cloudwatch.put_metric_alarm(
    AlarmName="bedrock-agent-error-rate",
    Namespace="AWS/Bedrock",
    MetricName="InvocationServerErrors",
    Statistic="Sum",
    Period=300,  # 5 minutes
    EvaluationPeriods=2,
    Threshold=10,
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:ai-ops-alerts"],
)
```

### 4. Custom Agent Metrics

```python
import boto3

cloudwatch = boto3.client("cloudwatch")

def publish_agent_metrics(agent_name: str, metrics: dict):
    """Publish custom agent metrics to CloudWatch."""
    cloudwatch.put_metric_data(
        Namespace="FAOS/AgentOps",
        MetricData=[
            {
                "MetricName": "ToolCallCount",
                "Value": metrics["tool_calls"],
                "Unit": "Count",
                "Dimensions": [{"Name": "AgentName", "Value": agent_name}],
            },
            {
                "MetricName": "ResolutionRate",
                "Value": metrics["resolved_pct"],
                "Unit": "Percent",
                "Dimensions": [{"Name": "AgentName", "Value": agent_name}],
            },
            {
                "MetricName": "SessionDuration",
                "Value": metrics["duration_ms"],
                "Unit": "Milliseconds",
                "Dimensions": [{"Name": "AgentName", "Value": agent_name}],
            },
        ],
    )
```

## Anti-Patterns

- Not setting cost alarms -- AI token usage can spike unexpectedly
- Using `@message` full-text search instead of structured filters -- parse fields first
- Retaining agent logs indefinitely -- set log group retention policies (30-90 days)
- Missing CloudTrail integration -- always enable for Bedrock API audit

## References

- [Amazon CloudWatch Documentation](https://docs.aws.amazon.com/cloudwatch/)
- [Bedrock CloudWatch Metrics](https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-cw.html)
- [CloudWatch Logs Insights Query Syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
- [AWS CloudWatch MCP Server](https://github.com/awslabs/mcp)

<!-- Source: .faos/custom/skills/cloud/aws/cloudwatch-observability/SKILL.md -->
