---
name: cloudwatch_logs
display_name: CloudWatch Logs
description: Query and search AWS CloudWatch Logs and run Logs Insights queries
emoji: "☁️"
homepage: https://aws.amazon.com/cloudwatch/

api:
  base_url: ""
  domain: amazonaws.com
  type: aws
  aws_service: logs
  auth:
    type: aws

credentials:
  - key: aws_access_key_id
    label: Access Key ID
    placeholder: "AKIA..."
    help: "AWS Console → IAM → Users → [your user] → Security credentials → Create access key. Starts with 'AKIA'."
  - key: aws_secret_access_key
    label: Secret Access Key
    placeholder: "wJalr..."
    help: "Shown once when creating the access key. If lost, create a new key pair in IAM → Security credentials."
  - key: aws_region
    label: Region
    placeholder: "us-east-1"
    help: "Region where your log groups live. Find in AWS Console top-right dropdown (e.g., us-east-1, eu-west-1, ap-southeast-1)."
  - key: log_group_names
    label: Log Groups (optional)
    placeholder: "/aws/lambda/my-function, /aws/rds/my-db"
    help: "Comma-separated log group names to use as defaults. The agent can still explore other groups if needed. Example: /aws/lambda/auth, /aws/ecs/api-service"
    optional: true
---

# CloudWatch Logs API

Use the AWS CloudWatch Logs API to query log groups, search log events, and run CloudWatch Logs Insights queries.

## Setup Guide

This skill requires **read-only** access to AWS CloudWatch Logs. No write or delete permissions are needed — Byaan only reads your logs, it never modifies them.

### What you need

An **IAM user** with programmatic access (Access Key ID + Secret Access Key). This user only needs read permissions on CloudWatch Logs. We recommend creating a dedicated IAM user for Byaan rather than reusing an admin or broad-access account.

### Step-by-step

1. In the **AWS Console → IAM → Users**, create a new user (e.g., `byaan-cloudwatch-reader`)
2. Attach the **AWS managed policy** `CloudWatchLogsReadOnlyAccess`, or create a custom policy with these specific actions:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:FilterLogEvents",
        "logs:GetLogEvents",
        "logs:GetLogRecord",
        "logs:StartQuery",
        "logs:GetQueryResults",
        "logs:StopQuery"
      ],
      "Resource": "*"
    }
  ]
}
```

You can restrict `Resource` to specific log groups if you want tighter scoping (e.g., `"arn:aws:logs:us-east-1:123456789:log-group:/aws/lambda/*"`).

3. Under **Security credentials**, create an **Access Key** (select "Third-party service" as the use case)
4. Copy the **Access Key ID** and **Secret Access Key**
5. In Byaan, go to **Settings > Skills > CloudWatch Logs** and paste:
   - **Access Key ID** — starts with `AKIA...`
   - **Secret Access Key** — the paired secret
   - **Region** — the AWS region where your logs live (e.g., `us-east-1`). CloudWatch Logs are regional, so pick the region matching your services

## How Authentication Works

**You do not need to handle authentication manually.** When you call `execute_skill_api()`, the system automatically:

1. Retrieves the AWS credentials from the configured skill
2. Creates a boto3 CloudWatch Logs client with those credentials
3. Calls the specified action with the provided parameters

Just call the API with skill name, action name, and parameters — authentication is handled for you.

## How to Call the API

CloudWatch Logs uses **action-based invocation**. Set `endpoint_path` to the action name (PascalCase) and `body` to a JSON string of parameters.

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="DescribeLogGroups",
  method="POST",
  body="{}"
)
```

**Parameters:**

- `skill_name`: Always `"cloudwatch_logs"`
- `endpoint_path`: The action name in PascalCase (e.g., `"FilterLogEvents"`, `"DescribeLogGroups"`)
- `method`: Always `"POST"`
- `body`: JSON string of parameters for the action

## Key Operations

### DescribeLogGroups — List log groups

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="DescribeLogGroups",
  method="POST",
  body="{}"
)
```

With prefix filter:

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="DescribeLogGroups",
  method="POST",
  body="{\"logGroupNamePrefix\": \"/aws/lambda/\"}"
)
```

### DescribeLogStreams — List streams in a log group

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="DescribeLogStreams",
  method="POST",
  body="{\"logGroupName\": \"/aws/lambda/my-function\", \"orderBy\": \"LastEventTime\", \"descending\": true, \"limit\": 10}"
)
```

### FilterLogEvents — Search logs by pattern

This is the primary way to search for log entries. Supports filter patterns.

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="FilterLogEvents",
  method="POST",
  body="{\"logGroupName\": \"/aws/lambda/my-function\", \"filterPattern\": \"ERROR\", \"startTime\": 1700000000000, \"endTime\": 1700086400000, \"limit\": 100}"
)
```

**Important:** `startTime` and `endTime` are **epoch milliseconds** (not seconds). Multiply Unix timestamps by 1000.

### GetLogEvents — Read a specific stream

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="GetLogEvents",
  method="POST",
  body="{\"logGroupName\": \"/aws/lambda/my-function\", \"logStreamName\": \"2024/01/15/[$LATEST]abc123\", \"startFromHead\": true, \"limit\": 50}"
)
```

### GetLogRecord — Get full log record by pointer

When Insights queries return results, each record includes a `@ptr` field. Use `GetLogRecord` to retrieve the complete log event from that pointer.

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="GetLogRecord",
  method="POST",
  body="{\"logRecordPointer\": \"CpMBCgpBY2...(ptr value from Insights)\"}"
)
```

This is useful for drill-down: run an Insights query to find interesting records, then use `@ptr` to get the full untruncated log event.

### StartQuery + GetQueryResults — CloudWatch Logs Insights

Insights queries are **asynchronous**. You must start the query, then poll for results.

**Step 1: Start the query**

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="StartQuery",
  method="POST",
  body="{\"logGroupNames\": [\"/aws/lambda/my-function\"], \"startTime\": 1700000000, \"endTime\": 1700086400, \"queryString\": \"fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50\"}"
)
```

**Note:** For `StartQuery`, `startTime` and `endTime` are **epoch seconds** (not milliseconds, unlike FilterLogEvents).

This returns a `queryId`.

**Step 2: Get results (poll until status is "Complete")**

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="GetQueryResults",
  method="POST",
  body="{\"queryId\": \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\"}"
)
```

The response includes a `status` field. Poll until it is `"Complete"`. Possible values: `Scheduled`, `Running`, `Complete`, `Failed`, `Cancelled`, `Timeout`.

### StopQuery — Cancel a running Insights query

If a query is taking too long or the user wants to abort, cancel it with `StopQuery`:

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="StopQuery",
  method="POST",
  body="{\"queryId\": \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\"}"
)
```

Returns `true` if the query was successfully stopped. Use this when `GetQueryResults` keeps returning `Running` after multiple polls.

## CloudWatch Logs Insights Query Syntax

Insights uses a pipe-delimited query language:

```
fields @timestamp, @message, @logStream
| filter @message like /ERROR/
| sort @timestamp desc
| limit 100
```

### Commands

- **fields**: Select fields to display — `fields @timestamp, @message`
- **filter**: Filter results — `filter @message like /pattern/` or `filter statusCode >= 400`
- **sort**: Order results — `sort @timestamp desc`
- **limit**: Cap results — `limit 100`
- **stats**: Aggregation — `stats count(*) by bin(1h)` or `stats avg(duration) by functionName`
- **parse**: Extract fields from messages — `parse @message "Duration: * ms" as duration`
- **display**: Choose which fields appear in output (after stats)

### Built-in Fields

- `@timestamp` — Event timestamp
- `@message` — Full log message
- `@logStream` — Log stream name
- `@log` — Log group identifier
- `@ingestionTime` — When CloudWatch received the event

### Example Insights Queries

**Count errors per hour:**

```
fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errorCount by bin(1h)
| sort errorCount desc
```

**Average Lambda duration:**

```
filter @type = "REPORT"
| parse @message "Duration: * ms" as duration
| stats avg(duration) as avgDuration, max(duration) as maxDuration by bin(1h)
```

**Top 10 most frequent error messages:**

```
filter @message like /ERROR/
| parse @message "ERROR *" as errorMsg
| stats count(*) as cnt by errorMsg
| sort cnt desc
| limit 10
```

**Request tracing by ID:**

```
fields @timestamp, @message, @logStream
| filter @message like /request-id-here/
| sort @timestamp asc
```

**P99 latency:**

```
filter @type = "REPORT"
| parse @message "Duration: * ms" as duration
| stats pct(duration, 99) as p99, pct(duration, 95) as p95, avg(duration) as avg by bin(1h)
```

## Log Analysis Patterns

Use these Insights query recipes for operational analysis. These replicate the kind of analysis that dedicated log analysis tools perform.

### Error Pattern Detection

Identify distinct error types and their frequency:

```
filter @message like /(?i)(error|exception|fatal|fail)/
| parse @message /(?<errorType>[\w.]+Exception|[\w.]+Error|FATAL|FAIL\w*)/
| stats count(*) as occurrences by errorType
| sort occurrences desc
| limit 20
```

### Spike Detection

Compare current error rate against the previous period to detect anomalies:

```
filter @message like /ERROR/
| stats count(*) as errors by bin(5m)
| sort @timestamp desc
```

After retrieving results, compare the most recent bins against the average of earlier bins. A bin with >2x the average signals a spike.

### Log Volume Analysis

Detect unusual log volume changes (potential indicator of issues):

```
stats count(*) as logCount by bin(5m)
| sort @timestamp desc
```

### Message Pattern Extraction

Extract structured data from unstructured log messages:

```
parse @message /\[(?<level>\w+)\]\s+(?<component>[\w.]+)\s+-\s+(?<msg>.+)/
| stats count(*) as cnt by level, component
| sort cnt desc
```

### Lambda Cold Start Analysis

```
filter @type = "REPORT"
| parse @message "Init Duration: * ms" as initDuration
| filter ispresent(initDuration)
| stats count(*) as coldStarts, avg(initDuration) as avgInitMs, max(initDuration) as maxInitMs, pct(initDuration, 99) as p99InitMs by bin(1h)
```

### API Gateway Latency Breakdown

```
filter @message like /requestId/
| parse @message "integrationLatency: *" as integrationLatency
| parse @message "latency: *" as totalLatency
| stats avg(totalLatency) as avgTotal, avg(integrationLatency) as avgBackend, pct(totalLatency, 99) as p99Total by bin(15m)
```

### Error Rate as Percentage

```
stats count(*) as total, sum(@message like /ERROR/) as errors by bin(1h)
| sort @timestamp desc
```

### Cross-Log-Group Analysis

Query multiple log groups simultaneously by passing an array to `logGroupNames` in `StartQuery`:

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="StartQuery",
  method="POST",
  body="{\"logGroupNames\": [\"/aws/lambda/auth-service\", \"/aws/lambda/api-gateway\", \"/aws/lambda/order-service\"], \"startTime\": 1700000000, \"endTime\": 1700086400, \"queryString\": \"filter @message like /ERROR/ | stats count(*) as errors by @logStream | sort errors desc | limit 20\"}"
)
```

## Operational Troubleshooting Workflow

When a user reports an incident or asks to investigate an issue, follow this structured approach:

### Step 1: Gather Context

Ask the user for:
- Which service or log group is affected?
- When did the issue start? (Time range)
- Any known symptoms? (Error messages, status codes, affected endpoints)
- Are there active CloudWatch Alarms? (User may know from AWS Console)

### Step 2: Assess Scope

Run a broad error scan across relevant log groups:

```
filter @message like /(?i)(error|exception|fatal|timeout|refused)/
| stats count(*) as cnt by bin(5m)
| sort @timestamp desc
```

This reveals whether errors are spiking, steady, or resolved.

### Step 3: Identify Root Cause

Drill into the error types found in Step 2:

```
filter @message like /ERROR/
| parse @message /(?<errorType>[\w.]+Exception|[\w.]+Error)/
| stats count(*) as cnt by errorType
| sort cnt desc
| limit 10
```

### Step 4: Trace Specific Requests

Once you identify the error pattern, trace individual affected requests:

```
fields @timestamp, @message, @logStream
| filter @message like /the-specific-error-or-request-id/
| sort @timestamp asc
| limit 50
```

### Step 5: Correlate with Database Data

If the user has database connections configured, cross-reference:
- Find affected records in the database using timestamps or request IDs from logs
- Check for database-side errors in the same time window
- Present a unified timeline: log events + database records

### Step 6: Summarize and Recommend

Present findings as:
1. **What happened** — the error type, frequency, and affected scope
2. **When** — exact time window with spike detection
3. **Impact** — which services/endpoints/users were affected
4. **Likely cause** — based on error patterns and log context
5. **Next steps** — suggested remediation or further investigation

## Future: CloudWatch Metrics & Alarms

CloudWatch Metrics (`GetMetricData`, `ListMetrics`) and Alarms (`DescribeAlarms`, `DescribeAlarmHistory`) require the `cloudwatch` AWS service, not `logs`. These are planned as a separate skill (`cloudwatch_metrics`) to enable:
- Querying metric time-series data (CPU, memory, custom metrics)
- Checking active alarm states
- Analyzing alarm history and threshold recommendations
- Full alarm→metric→log correlation workflows

Until then, ask the user to share alarm/metric context from the AWS Console when investigating incidents.

## FilterLogEvents Filter Patterns

The `filterPattern` in `FilterLogEvents` supports:

- **Simple text match**: `"ERROR"` — matches any event containing "ERROR"
- **Multiple terms (AND)**: `"ERROR timeout"` — matches events containing both
- **Quoted phrase**: `"\"Connection refused\""` — matches exact phrase
- **JSON field match**: `{ $.statusCode = 500 }` — matches JSON log entries
- **JSON comparison**: `{ $.duration > 1000 }` — numeric comparison
- **JSON compound**: `{ $.statusCode = 500 && $.method = "POST" }` — multiple conditions

## Pagination

Many operations return a `nextToken` when there are more results:

```
execute_skill_api(
  skill_name="cloudwatch_logs",
  endpoint_path="FilterLogEvents",
  method="POST",
  body="{\"logGroupName\": \"/aws/lambda/my-function\", \"filterPattern\": \"ERROR\", \"nextToken\": \"token-from-previous-response\"}"
)
```

Pass the `nextToken` from the previous response to get the next page.

## Common Pitfalls

1. **Timestamps differ by API**: `FilterLogEvents` and `GetLogEvents` use **epoch milliseconds**. `StartQuery` uses **epoch seconds**. Mixing them up returns empty results.

2. **Insights queries are async**: `StartQuery` returns immediately with a `queryId`. You must poll `GetQueryResults` until `status` is `"Complete"`.

3. **Log group names include path**: Always include the full path, e.g., `/aws/lambda/my-function`, not just `my-function`.

4. **FilterLogEvents has a scan limit**: It scans up to 1MB of data per call. For large time ranges, use Insights queries instead.

5. **Results ordering**: `FilterLogEvents` returns events in chronological order by default. Use `startTime`/`endTime` to narrow the window.

6. **Insights query timeout**: Queries can time out after 60 minutes. For large datasets, narrow the time range.

7. **Insights time range**: If no results return, verify `startTime`/`endTime` cover the period when logs were generated.

## Agent Behavior & Workflow

When handling CloudWatch-related requests, follow this workflow:

1. **Check configured log groups first.** Before calling `DescribeLogGroups`, check if the user has pre-configured log groups via the `log_group_names` credential field. If present (comma-separated string), parse it into a list and use those as the default scope for queries. If the user asks to explore beyond those groups or the field is empty, call `DescribeLogGroups` to discover available options.

   Accessing configured log groups at runtime:
   ```
   credentials["log_group_names"]  # e.g., "/aws/lambda/auth, /aws/ecs/api"
   # Parse: [g.strip() for g in value.split(",") if g.strip()]
   ```

2. **Clarify time range.** If the user doesn't specify a time range, ask them — or default to the last 1 hour. Use the current datetime from the system prompt to compute relative times.

3. **Choose the right API:**
   - Use `FilterLogEvents` for simple keyword/pattern searches (e.g., "find ERROR logs", "show me logs containing timeout").
   - Use `StartQuery` (Insights) for aggregations, stats, parsing, or complex multi-field analysis (e.g., "count errors per hour", "average Lambda duration").

4. **Poll Insights queries efficiently.** After calling `StartQuery`, poll `GetQueryResults` with brief pauses between calls. Stop after 10 retries max — if still not complete, use `StopQuery` to cancel the query, inform the user, and suggest narrowing the time range.

5. **Paginate when needed.** If the response contains a `nextToken`, decide whether to fetch more pages based on the user's request. For exploratory queries, the first page is usually sufficient. For exhaustive searches, continue paginating.

6. **Handle errors gracefully.** If an API call fails, read the error message carefully and provide actionable guidance (see Troubleshooting Guide below).

## Time Range Handling

CloudWatch APIs require epoch timestamps. Convert user-friendly inputs as follows:

- **"last 1 hour"** → `endTime = now`, `startTime = now - 3600`
- **"last 24 hours"** → `endTime = now`, `startTime = now - 86400`
- **"last 7 days"** → `endTime = now`, `startTime = now - 604800`
- **"yesterday"** → `startTime = start of yesterday (00:00:00)`, `endTime = end of yesterday (23:59:59)`
- **Specific date/time** → Parse and convert to epoch

**Critical: ms vs seconds distinction.**

| API | Timestamp unit | Multiply Unix epoch by |
|-----|---------------|----------------------|
| `FilterLogEvents` | Milliseconds | × 1000 |
| `GetLogEvents` | Milliseconds | × 1000 |
| `StartQuery` | Seconds | × 1 (use as-is) |

Getting this wrong is the most common cause of empty results.

## Result Presentation

When displaying CloudWatch results to the user:

1. **Lead with a summary.** State the total log count, time range covered, and key patterns found (e.g., "Found 247 log events in the last hour, 38 contain ERROR").

2. **Show representative entries.** Display 5-10 representative log lines, not the full result set. Prioritize entries that illustrate the user's query (errors, warnings, relevant events).

3. **Highlight important details.** Call out timestamps, error levels (ERROR, WARN, FATAL), exception types, and stack traces in your response.

4. **Format aggregations as tables.** For Insights `stats` results, present data in a table. For time-bucketed stats, suggest a time series chart via the dashboard.

5. **Provide next steps.** After presenting results, suggest follow-up actions: narrowing the search, checking related log groups, correlating with other data sources, or saving the query to a dashboard.

## Dashboard Integration

CloudWatch query results can be saved for dashboard use:

1. **Save queries with `save_skill_query`.** After running a successful CloudWatch query, you can persist it for dashboard display. The saved query stores all parameters needed to re-execute via boto3 on each dashboard refresh.

2. **Map results to chart types:**
   - **Time series / line charts** — Best for time-bucketed stats (`stats count(*) by bin(1h)`, latency over time)
   - **Tables** — Best for raw log entries, filtered event lists
   - **Bar charts** — Best for categorical aggregations (`stats count(*) by functionName`, error counts by type)
   - **Single value / metric** — Best for a single aggregated number (total errors, p99 latency)

3. **Saved queries re-execute on refresh.** Each time the dashboard loads, saved CloudWatch queries run again via boto3 with the stored parameters. For relative time ranges (e.g., "last 1 hour"), the time window shifts forward automatically.

## Troubleshooting Guide

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| `AccessDeniedException` | IAM user lacks required permissions | Ensure the IAM policy includes `logs:DescribeLogGroups`, `logs:FilterLogEvents`, `logs:StartQuery`, `logs:GetQueryResults`, and other required actions |
| Empty results | Time range doesn't cover log activity, or wrong log group name | Verify `startTime`/`endTime` bracket the expected log period. Use `DescribeLogGroups` to confirm the exact log group name |
| Empty results (Insights) | `startTime`/`endTime` in wrong unit | `StartQuery` uses epoch **seconds**, not milliseconds |
| `ResourceNotFoundException` | Log group doesn't exist in this region | Check the region in skill settings. CloudWatch Logs are regional — logs in `us-east-1` won't appear when querying `eu-west-1` |
| Insights query stuck in `Running` | Query scanning too much data | Use `StopQuery` to cancel, then retry with a narrower time range or more specific filters |
| `InvalidParameterException` | Malformed query syntax or invalid parameter | Double-check Insights query syntax (pipe-delimited) and parameter names (PascalCase) |
| `LimitExceededException` | Too many concurrent queries | Wait a moment and retry. AWS limits concurrent Insights queries (typically 30 per account per region) |

## Combining with Database Data

When users have both database connections and CloudWatch configured, you can correlate data across sources:

1. **Match request IDs.** Query the database for a specific transaction or request ID, then search CloudWatch logs for the same ID to see the full request lifecycle.

2. **Correlate timestamps.** Find anomalies in database metrics (slow queries, error spikes) and check CloudWatch logs for the same time window to identify root causes.

3. **Enrich with context.** Use database records to identify affected users or resources, then search CloudWatch for related log entries to build a complete incident picture.

4. **Workflow example:**
   - Query the database: "Find all failed orders in the last hour"
   - Get the request IDs or timestamps from the results
   - Search CloudWatch: `FilterLogEvents` with those request IDs as the filter pattern
   - Present a unified view: database record + associated log trail
