SOP - Nudgebee EventBridge SQS Queue Backlog

Table of Contents

What is this alert?

CloudWatch Alarm SQS Messages Age - nudgebee-eventbridge-queue fires when the ApproximateAgeOfOldestMessage metric exceeds the configured threshold (60 seconds). This means the cloud-collector-server's SQS consumer is not processing incoming EventBridge events fast enough, causing messages to sit in the queue longer than expected.

The nudgebee-eventbridge-queue is the central ingestion point for all AWS EventBridge events across all connected cloud accounts. When this queue backs up, event processing is delayed — CloudWatch alarms, CloudTrail events, and service state changes will not appear in the Nudgebee events dashboard until the backlog clears.

What flows through this queue?

Event Category

Source

Approximate Volume

CloudWatch Alarms

CloudWatch Alarm State Changes (OK → ALARM, ALARM → OK)

~58% of events

CloudTrail Events

Resource creation, deletion, configuration changes

~30% of events

ECS Events

Task state changes, deployment failures

~8% of events

Other AWS Events

Auto-scaling, EC2, RDS, Lambda, etc.

~4% of events

Current throughput: ~100 events/day, ~712 events/week across all connected accounts.

Architecture

                    ┌─────────────────────────────────────────┐
                    │         AWS EventBridge Rules            │
                    │  (per-account, per-region)               │
                    └──────────────┬──────────────────────────┘
                                   │
                                   ▼
                    ┌─────────────────────────────────────────┐
                    │  SQS: nudgebee-eventbridge-queue         │
                    │  Region: us-east-1                       │
                    │  Visibility Timeout: 30s (default)       │
                    │  Long Polling: 20s                       │
                    │  Batch Size: 10 messages                 │
                    └──────────────┬──────────────────────────┘
                                   │
                                   ▼
                    ┌─────────────────────────────────────────┐
                    │  cloud-collector-server                   │
                    │  StartEventBridgeSQSConsumer()            │
                    │  (single goroutine, sequential)           │
                    └──────────────┬──────────────────────────┘
                                   │
                    ┌──────────────┼──────────────────────────┐
                    │              │                           │
                    ▼              ▼                           ▼
             Parse Event    Account Lookup            Match aws_runbook.yaml
             (JSON)         (DB query by               rules
                            external_id +
                            account_number)
                                   │
                    ┌──────────────┼──────────────────────────┐
                    │              │                           │
                    ▼              ▼                           ▼
             aws_get_metric  aws_get_log            aws_get_resource
             (CloudWatch     (CW Logs Insights      (Describe API calls)
              GetMetricData)  query, 10-30s)
                    │              │                           │
                    └──────────────┼──────────────────────────┘
                                   │
                                   ▼
                    ┌─────────────────────────────────────────┐
                    │  Store event in PostgreSQL               │
                    │  (events table, with evidences JSONB)    │
                    │  Dedup by finding_id                     │
                    └──────────────┬──────────────────────────┘
                                   │
                                   ▼
                    ┌─────────────────────────────────────────┐
                    │  AI Analysis (async)                     │
                    │  - Summary                               │
                    │  - Investigation (5-Whys)                │
                    │  - Log Analysis                          │
                    │  - Detailed Response                     │
                    └─────────────────────────────────────────┘

Key source file: collector-server/cloud-collector/providers/aws/event_eventbridge.go — StartEventBridgeSQSConsumer()

Config: cloud_collector_aws_eventbridge_sqs environment variable contains the SQS queue URL/ARN.

Why did I get paged?

The ApproximateAgeOfOldestMessage metric for the SQS queue exceeded the threshold of 60 seconds. This means one or more messages have been sitting in the queue without being processed for over a minute.

This typically indicates one of:

  1. The SQS consumer goroutine is slow (enrichment actions blocking)

  2. The SQS consumer goroutine has crashed (panic, no auto-restart)

  3. The cloud-collector pod is unhealthy (resource pressure, network issues)

  4. An upstream spike in EventBridge events overwhelmed the single-threaded consumer

Root Causes

Listed in order of likelihood based on code analysis and historical incidents:

#

Cause

Mechanism

Frequency

1

Slow enrichment actions

Each event triggers CloudWatch GetMetricData (2-5s) + CloudWatch Logs Insights (10-30s) + resource describe calls. These run sequentially per message. 10 events x 30s = 5 min per batch.

Most common

2

Single consumer goroutine

StartEventBridgeSQSConsumer runs as one goroutine with batch=10, no parallelism. Cannot scale horizontally within a single pod.

Structural

3

DB query slowness

Account lookup + dedup check + event insert hit PostgreSQL for every event. Slow DB = slow consumer.

Occasional

4

AWS API throttling

CloudWatch API rate limits (400 req/s/account) or Logs Insights concurrency limit (30 concurrent queries/account).

Rare

5

Pod resource constraints

cloud-collector pod hitting CPU/memory limits, causing GC pauses or OOM kills.

Rare

6

Consumer goroutine crash

If StartEventBridgeSQSConsumer panics (there is a defer recover), the goroutine exits without restart. Messages pile up until pod restarts.

Rare but impactful

Investigation Steps

Step 1: Check cloud-collector pod health

# Check pod status, restarts, age
kubectl -n nudgebee get pods -l app=cloud-collector-server

# Check resource usage
kubectl -n nudgebee top pods -l app=cloud-collector-server

# Check recent logs for errors/panics
kubectl -n nudgebee logs -l app=cloud-collector-server --tail=200 | grep -E "SQSConsumer|panic|ERROR|FATAL"

What to look for:

Symptom

Diagnosis

Pod in CrashLoopBackOff

Pod is crash-looping. Check logs for root cause, likely OOM or panic.

High restart count

Consumer keeps crashing. Check for panic traces in previous logs: kubectl logs <pod> --previous

CPU/Memory near limits

Resource starvation. Consider scaling resources.

No SQSConsumer log entries

Consumer goroutine has crashed silently. Pod is running but not consuming. Restart the pod.

SQSConsumer logs present but slow

Enrichment is the bottleneck. Proceed to Step 3.

Step 2: Check SQS queue metrics

# Message age (is consumer stuck?)
aws cloudwatch get-metric-statistics \
  --namespace AWS/SQS \
  --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=nudgebee-eventbridge-queue \
  --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Average \
  --region us-east-1

# Queue depth (are messages piling up?)
aws cloudwatch get-metric-statistics \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=nudgebee-eventbridge-queue \
  --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Average \
  --region us-east-1

# Throughput (is consumer processing at all?)
aws cloudwatch get-metric-statistics \
  --namespace AWS/SQS \
  --metric-name NumberOfMessagesDeleted \
  --dimensions Name=QueueName,Value=nudgebee-eventbridge-queue \
  --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Sum \
  --region us-east-1

Decision matrix:

Age Increasing?

Depth Increasing?

Deletes Happening?

Diagnosis

Yes

Yes

No

Consumer is dead. Restart pod.

Yes

Stable

Yes

Consumer is slow. Enrichment bottleneck.

Yes

Increasing

Yes

Consumer is working but can't keep up with event volume. Scale up.

Stable/Decreasing

Decreasing

Yes

Backlog is draining. Monitor, no action needed.

Step 3: Check for slow enrichment actions

kubectl -n nudgebee logs -l app=cloud-collector-server --since=1h \
  | grep -E "aws_get_metric|aws_get_log|aws_get_resource|QueryLogs|GetMetricData"

What to look for:

Step 4: Check for AWS API throttling

kubectl -n nudgebee logs -l app=cloud-collector-server --since=1h \
  | grep -iE "throttl|rate.exceed|TooManyRequest|LimitExceed"

If throttling is detected:

Step 5: Check DB connectivity

kubectl -n nudgebee logs -l app=cloud-collector-server --since=1h \
  | grep -iE "database|connection|timeout|pgx|SQLSTATE|deadlock"

If DB is the issue:

Resolution

Action

When to use

Command

Restart cloud-collector

Consumer goroutine crashed; pod running but not consuming; quick recovery needed

kubectl -n nudgebee rollout restart deployment/cloud-collector-server

Scale up replicas

Event volume spike overwhelming single consumer; queue depth increasing despite consumer being alive

kubectl -n nudgebee scale deployment/cloud-collector-server --replicas=2 (SQS supports multiple consumers safely)

Increase pod resources

Pod hitting CPU/memory limits (check kubectl top)

Patch deployment resource limits in Helm values

Check DLQ

Suspect messages are being dead-lettered

Check nudgebee-eventbridge-dlq in SQS console for messages. If populated, investigate why messages failed processing.

Escalate

Queue age >5 min AND not trending down after pod restart

Notify secondary on-call. Events may be silently lost (see Known Issues).

Emergency: Mass event loss suspected

If the consumer has been down for an extended period and the SQS message retention period (default 4 days) is approaching:

  1. Check SQS queue attributes for MessageRetentionPeriod

  2. If messages are close to expiry, temporarily increase retention: aws sqs set-queue-attributes --queue-url <url> --attributes MessageRetentionPeriod=1209600 (14 days max)

  3. Restart consumer and monitor drain rate

Confirm Resolution

Known Issues

1. Messages deleted even on processing failure

Severity: High

Code location: event_eventbridge.go, StartEventBridgeSQSConsumer()

The SQS consumer calls DeleteMessage after processing each message, regardless of whether processing succeeded or failed. This means:

Impact: Silent event loss during processing failures. No retry, no DLQ redrive.

Mitigation: Monitor for gaps in event timeline. Cross-reference SQS NumberOfMessagesReceived with events created in DB.

2. No consumer health check

Severity: Medium

The /health endpoint on cloud-collector does not verify the SQS consumer goroutine is alive. A crashed consumer goroutine produces no alerts until ApproximateAgeOfOldestMessage exceeds the CloudWatch alarm threshold (60 seconds).

Impact: Up to 60 seconds of detection delay when consumer crashes.

3. No parallelism in consumer

Severity: Medium

The consumer processes messages sequentially in a single goroutine. With enrichment actions taking 10-30 seconds per event, maximum throughput is approximately 2-6 events per minute.

Impact: Any burst of >6 events/minute causes backlog. During high-activity periods (e.g., infrastructure deployments triggering many CloudTrail events), the queue will back up.

4. Visibility timeout mismatch

Severity: Low

The consumer does not set a custom visibility timeout. SQS default is 30 seconds. If enrichment (especially aws_get_log Logs Insights queries) takes >30 seconds, the message becomes visible again and may be processed a second time by another consumer (if scaled) or the same consumer on next poll.

Impact: Potential duplicate events. Mitigated by finding_id deduplication in DB insert, but creates unnecessary processing overhead.

Runbook Automation

The following can be configured as a Nudgebee auto-playbook to handle this alert automatically:

Trigger

Source:  AWS_CloudWatch_Alarm
Title:   LIKE "%SQS Messages Age%nudgebee-eventbridge-queue%"
State:   OK → ALARM

Automated Steps

Step

Type

Action

Condition

1

Auto

Fetch cloud-collector pod status via K8s integration

Always

2

Auto

Check SQS queue depth (already enriched in event evidences)

Always

3

Auto

AI analysis — assess if queue is draining or worsening

Always

4

Condition

If pod restarts > 0 in last hour → k8s.workload_restart(deployment=cloud-collector-server)

Pod crash detected

5

Condition

If no SQSConsumer logs in last 5 min → k8s.workload_restart(deployment=cloud-collector-server)

Consumer goroutine dead

6

Condition

If queue age still rising 10 min after restart → Notify Slack #ops-alerts + create incident

Restart didn't fix it

7

Manual

Escalate to secondary on-call for deeper investigation

Auto-remediation failed

Alert History & Statistics

Based on production data (as of 2026-03-30):

Metric

Value

Total SQS age alerts (all queues, last 45 days)

340+ events

nudgebee-eventbridge-queue alerts (last 7 days)

32 events

karpenter-queue-creation-Queue alerts (since Feb 12)

124 events

Average events through EventBridge per day

~100

Alert sources on this queue

AWS_EventBridge (CloudWatch Alarm)

Auto-classification

Duplicates are auto-classified and deduplicated

Recurring pattern

The nudgebee-eventbridge-queue alerts tend to cluster during periods of high EventBridge activity (infrastructure changes, scaling events). The single-threaded consumer falls behind, triggers the alarm, and self-recovers once the burst subsides. Most alerts auto-resolve (ALARM → OK) within 10-20 minutes.

Related SOPs

Appendix: Key Files

File

Purpose

collector-server/cloud-collector/providers/aws/event_eventbridge.go

SQS consumer implementation

collector-server/cloud-collector/providers/aws/event_eventbridge_processor.go

Event rule matching and enrichment

collector-server/cloud-collector/providers/aws/aws_runbook.yaml

Event rules (which events to process, which enrichment actions to run)

collector-server/cloud-collector/config/config.go

Configuration (queue URL, timeouts, worker counts)

collector-server/cloud-collector/account/etl_events.go

Event storage in PostgreSQL

collector-server/cloud-collector/account/etl_events_async.go

Async event handler

collector-server/cloud-collector/cmd/main.go

Entry point, consumer startup (lines 142-152)

Last updated: 2026-03-30
Based on: Code review of cloud-collector-server + production event data analysis