# Event Reference and Troubleshooting Guide

This comprehensive guide covers all Kubernetes events generated by muster during CRD lifecycle operations, providing detailed explanations, troubleshooting steps, and operational best practices.

## Overview

> **Note:** Event emission is an **alpha feature** and is **disabled by default**.
> To enable it, use the `--enable-events` flag on `muster serve` or set `events: true` in your `config.yaml`.

When enabled, muster generates Kubernetes events for all major operations involving:
- **MCPServers**: Service lifecycle, tool discovery, health monitoring
- **Workflows**: Execution tracking, step monitoring, tool availability
- **Service Instances**: Instance lifecycle, health checks, tool execution

Events provide visibility into system behavior in both Kubernetes and filesystem modes, helping operators understand what's happening and troubleshoot issues effectively.

### Enabling Events

Via CLI flag:
```bash
muster serve --enable-events
```

Via `config.yaml`:
```yaml
events: true
```

## Event Structure

All muster events include:
- **Timestamp**: When the event occurred
- **Resource**: The object the event relates to (MCPServer (or Workflow))
- **Reason**: Short, machine-readable reason code
- **Message**: Human-readable description with context
- **Type**: Event severity (Normal or Warning)
- **Source**: Component that generated the event (`muster`)

## MCPServer Events

MCPServers are external MCP (Model Context Protocol) servers that provide tools to muster. Events track their lifecycle, health, and tool availability.

### CRD Management Events

#### MCPServerCreated
- **Type**: Normal
- **Meaning**: MCPServer CRD was successfully created in the cluster
- **Message Example**: "MCPServer 'github-server' successfully created"
- **Triggered When**: Running `muster create mcpserver` or applying YAML
- **Next Steps**: Check if server starts automatically if configured

#### MCPServerUpdated
- **Type**: Normal
- **Meaning**: MCPServer configuration was successfully updated
- **Message Example**: "MCPServer 'github-server' configuration updated"
- **Triggered When**: Modifying MCPServer spec (command, args, environment)
- **Next Steps**: Monitor for restart if changes affect running process

#### MCPServerDeleted
- **Type**: Normal
- **Meaning**: MCPServer CRD was successfully removed
- **Message Example**: "MCPServer 'github-server' successfully deleted"
- **Triggered When**: Running `muster delete mcpserver` or deleting YAML
- **Next Steps**: Verify associated service instances are handled gracefully

### Service Lifecycle Events

#### MCPServerStarting
- **Type**: Normal
- **Meaning**: MCPServer service is beginning startup process
- **Message Example**: "MCPServer 'github-server' service beginning startup"
- **Triggered When**: Service start requested or automatic startup
- **Next Steps**: Watch for `MCPServerStarted` or `MCPServerFailed` within reasonable time

#### MCPServerStarted
- **Type**: Normal
- **Meaning**: MCPServer service started successfully and is available
- **Message Example**: "MCPServer 'github-server' service started successfully"
- **Triggered When**: Process starts and initial health checks pass
- **Next Steps**: Tools should become available; check with `muster list tools`

#### MCPServerStopped
- **Type**: Normal
- **Meaning**: MCPServer service was stopped cleanly
- **Message Example**: "MCPServer 'github-server' service stopped"
- **Triggered When**: Running `muster stop mcpserver` or graceful shutdown
- **Next Steps**: Tools from this server become unavailable

#### MCPServerRestarting
- **Type**: Normal
- **Meaning**: MCPServer service is being restarted (stop + start)
- **Message Example**: "MCPServer 'github-server' service restarting"
- **Triggered When**: Configuration changes require restart or manual restart
- **Next Steps**: Brief tool unavailability expected during restart

#### MCPServerFailed
- **Type**: Warning
- **Meaning**: MCPServer operation failed (start, health check, or crash)
- **Message Example**: "MCPServer 'github-server' failed to start: connection refused"
- **Triggered When**: Process crashes, fails to start, or becomes unresponsive
- **Troubleshooting**:
  ```bash
  # Check server configuration
  muster get mcpserver github-server --output yaml

  # Verify command and arguments
  # Check if port is available
  # Review logs for startup errors

  # Test manual startup (filesystem mode)
  muster standalone
  ```

### Tool Discovery Events

#### MCPServerToolsDiscovered
- **Type**: Normal
- **Meaning**: Tools were successfully discovered from MCPServer
- **Message Example**: "MCPServer 'github-server' discovered 15 tools: github_create_issue, github_list_repos, ..."
- **Triggered When**: Server starts and advertises capabilities
- **Next Steps**: Tools are now available for Workflows

#### MCPServerToolsUnavailable
- **Type**: Warning
- **Meaning**: Tool discovery failed or previously available tools became unavailable
- **Message Example**: "MCPServer 'github-server' tools unavailable: server connection lost"
- **Triggered When**: Server crashes, network issues, or MCP protocol errors
- **Troubleshooting**:
  ```bash
  # Check server status
  muster get mcpserver github-server

  # Verify server is responsive
  muster check mcpserver github-server

  # Test connection manually
  muster agent --repl
  # > list tools  # Check if tools appear
  ```

#### MCPServerReconnected
- **Type**: Normal
- **Meaning**: Connection to MCPServer was restored after being lost
- **Message Example**: "MCPServer 'github-server' reconnected successfully"
- **Triggered When**: Network recovery, server restart, or connectivity restoration
- **Next Steps**: Tools should become available again

### Health and Recovery Events

#### MCPServerHealthCheckFailed
- **Type**: Warning
- **Meaning**: Health checks are consistently failing for MCPServer
- **Message Example**: "MCPServer 'github-server' health check failed: timeout after 30s"
- **Triggered When**: Health check endpoint non-responsive or returns errors
- **Troubleshooting**:
  ```bash
  # Check if server process is running
  # Verify health check endpoint configuration
  # Check server logs for errors
  # Test health endpoint manually if exposed
  ```

#### MCPServerRecoveryStarted
- **Type**: Normal
- **Meaning**: Automatic recovery process began for failed MCPServer
- **Message Example**: "MCPServer 'github-server' automatic recovery started"
- **Triggered When**: Health checks fail consistently and recovery is enabled
- **Next Steps**: Monitor for `MCPServerRecoverySucceeded` or `MCPServerRecoveryFailed`

#### MCPServerRecoverySucceeded
- **Type**: Normal
- **Meaning**: Automatic recovery successfully restored MCPServer
- **Message Example**: "MCPServer 'github-server' automatic recovery succeeded"
- **Triggered When**: Recovery process successfully restarts server
- **Next Steps**: Monitor stability; investigate root cause of failure

#### MCPServerRecoveryFailed
- **Type**: Warning
- **Meaning**: Automatic recovery failed to restore MCPServer
- **Message Example**: "MCPServer 'github-server' automatic recovery failed: max retries exceeded"
- **Triggered When**: Recovery attempts exceed limits or continue failing
- **Troubleshooting**:
  ```bash
  # Manual intervention required
  # Check underlying issues (ports, permissions, resources)
  # Consider restarting manually
  muster stop mcpserver github-server
  muster start mcpserver github-server
  ```

## Workflow Events

Workflows define sequences of tool executions. Events track configuration, execution, and step-level progress.

### Configuration Management Events

#### WorkflowCreated
- **Type**: Normal
- **Meaning**: Workflow CRD was successfully created
- **Message Example**: "Workflow 'deploy-app' successfully created"
- **Triggered When**: Creating workflow via CLI or YAML
- **Next Steps**: Check tool availability and test execution

#### WorkflowUpdated
- **Type**: Normal
- **Meaning**: Workflow configuration was updated
- **Message Example**: "Workflow 'deploy-app' configuration updated"
- **Triggered When**: Modifying workflow steps, tools, or parameters
- **Next Steps**: Tool availability may change; test updated workflow

#### WorkflowDeleted
- **Type**: Normal
- **Meaning**: Workflow was successfully removed
- **Message Example**: "Workflow 'deploy-app' successfully deleted"
- **Triggered When**: Deleting workflow via CLI or YAML removal
- **Next Steps**: Workflow tool is removed from aggregator

#### WorkflowValidationFailed
- **Type**: Warning
- **Meaning**: Workflow definition failed validation
- **Message Example**: "Workflow 'deploy-app' validation failed: step 2 references unknown tool 'invalid_tool'"
- **Triggered When**: Invalid workflow definition detected
- **Troubleshooting**:
  ```bash
  # Check workflow definition
  muster get workflow deploy-app --output yaml

  # Validate manually
  muster check workflow deploy-app

  # Common issues:
  # - Invalid tool references
  # - Malformed step definitions
  # - Invalid parameter mappings
  ```

#### WorkflowValidationSucceeded
- **Type**: Normal
- **Meaning**: Workflow definition passed validation
- **Message Example**: "Workflow 'deploy-app' validation successful"
- **Triggered When**: Successful validation check
- **Next Steps**: Workflow is ready for execution

### Execution Lifecycle Events

#### WorkflowExecutionStarted
- **Type**: Normal
- **Meaning**: Workflow execution began
- **Message Example**: "Workflow 'deploy-app' execution started with 5 steps (execution ID: abc123)"
- **Triggered When**: Workflow executed via CLI or tool call
- **Next Steps**: Monitor step completion events

#### WorkflowExecutionCompleted
- **Type**: Normal
- **Meaning**: Workflow execution completed successfully
- **Message Example**: "Workflow 'deploy-app' execution completed successfully in 2m30s (execution ID: abc123)"
- **Triggered When**: All steps complete successfully
- **Next Steps**: Check execution results if needed

#### WorkflowExecutionFailed
- **Type**: Warning
- **Meaning**: Workflow execution failed
- **Message Example**: "Workflow 'deploy-app' execution failed at step 3: tool 'docker_run' returned error (execution ID: abc123)"
- **Triggered When**: Step fails and doesn't allow failure, or critical error
- **Troubleshooting**:
  ```bash
  # Check execution details
  muster get workflow-execution abc123

  # Review failed step
  # Check tool availability
  # Verify step parameters

  # Test individual step if possible
  muster agent --repl
  # > call tool_name {"param": "value"}
  ```

#### WorkflowExecutionTracked
- **Type**: Normal
- **Meaning**: Execution state was persisted for tracking
- **Message Example**: "Workflow 'deploy-app' execution state tracked (execution ID: abc123)"
- **Triggered When**: Execution state is saved (start, steps, completion)
- **Next Steps**: Execution can be queried for status and results

### Step-Level Execution Events

#### WorkflowStepStarted
- **Type**: Normal
- **Meaning**: Individual workflow step began execution
- **Message Example**: "Workflow 'deploy-app' step 'build-image' started using tool 'docker_build'"
- **Triggered When**: Each step begins execution
- **Next Steps**: Watch for step completion

#### WorkflowStepCompleted
- **Type**: Normal
- **Meaning**: Individual workflow step completed successfully
- **Message Example**: "Workflow 'deploy-app' step 'build-image' completed successfully"
- **Triggered When**: Step execution succeeds
- **Next Steps**: Next step will begin (if any)

#### WorkflowStepFailed
- **Type**: Warning (unless allowFailure=true)
- **Meaning**: Individual workflow step failed
- **Message Example**: "Workflow 'deploy-app' step 'deploy-service' failed: service already exists (allowFailure: true)"
- **Triggered When**: Step execution fails
- **Troubleshooting**:
  ```bash
  # Check if failure was expected (allowFailure)
  # Review step configuration
  # Test tool manually
  # Check step parameters and context
  ```

#### WorkflowStepSkipped
- **Type**: Normal
- **Meaning**: Step was skipped due to condition evaluation
- **Message Example**: "Workflow 'deploy-app' step 'rollback' skipped: condition 'deployment_failed' evaluated to false"
- **Triggered When**: Step condition evaluates to false
- **Next Steps**: Normal behavior for conditional steps

#### WorkflowStepConditionEvaluated
- **Type**: Normal
- **Meaning**: Step condition was evaluated
- **Message Example**: "Workflow 'deploy-app' step 'notify-slack' condition evaluated: true (continue execution)"
- **Triggered When**: Conditional step evaluation occurs
- **Next Steps**: Step will execute or be skipped based on result

### Tool Availability Events

#### WorkflowAvailable
- **Type**: Normal
- **Meaning**: All required tools became available for workflow
- **Message Example**: "Workflow 'deploy-app' available: all 5 required tools found"
- **Triggered When**: All tools needed by workflow steps become available
- **Next Steps**: Workflow can be executed

#### WorkflowUnavailable
- **Type**: Warning
- **Meaning**: Required tools became unavailable for workflow
- **Message Example**: "Workflow 'deploy-app' unavailable: 2 required tools missing"
- **Triggered When**: MCPServers providing required tools become unavailable
- **Troubleshooting**:
  ```bash
  # Check workflow tool requirements
  muster check workflow deploy-app

  # Check MCPServer status
  muster list mcpserver
  ```

#### WorkflowToolsDiscovered
- **Type**: Normal
- **Meaning**: New required tools were discovered for workflow
- **Message Example**: "Workflow 'deploy-app' tools discovered: found tools from 'kubernetes-server'"
- **Triggered When**: MCPServer providing workflow tools becomes available
- **Next Steps**: Workflow may become available

#### WorkflowToolsMissing
- **Type**: Warning
- **Meaning**: Specific required tools became unavailable
- **Message Example**: "Workflow 'deploy-app' tools missing: kubectl_apply, kubectl_get from 'kubernetes-server'"
- **Triggered When**: MCPServer stops or tools become unavailable
- **Troubleshooting**: Check the source MCPServer status and the workflow tool dependencies

#### WorkflowToolRegistered
- **Type**: Normal
- **Meaning**: Workflow was registered as executable tool in aggregator
- **Message Example**: "Workflow 'deploy-app' registered as action_deploy-app tool"
- **Triggered When**: Workflow becomes available and is registered
- **Next Steps**: Workflow can be called as `action_deploy-app` tool

#### WorkflowToolUnregistered
- **Type**: Normal
- **Meaning**: Workflow tool was removed from aggregator
- **Message Example**: "Workflow 'deploy-app' tool action_deploy-app unregistered"
- **Triggered When**: Workflow deleted or becomes unavailable
- **Next Steps**: Tool no longer callable

#### WorkflowCapabilitiesRefreshed
- **Type**: Normal
- **Meaning**: Aggregator capabilities updated after workflow changes
- **Message Example**: "Workflow capabilities refreshed: 15 workflow tools available"
- **Triggered When**: Workflow registration/unregistration triggers capability update
- **Next Steps**: Updated capabilities available to MCP clients

## Event Querying and Analysis

### Using muster events Command

```bash
# View all recent events
muster events

# Filter by resource type
muster events --resource-type mcpserver
muster events --resource-type workflow

# Filter by specific resource
muster events --resource-type service --resource-name my-app

# Filter by event type
muster events --type Warning
muster events --type Normal

# Time-based filtering
muster events --since 1h
muster events --since 2024-01-15T10:00:00Z --until 2024-01-15T18:00:00Z

# Combined filtering
muster events --resource-type mcpserver --type Warning --since 2h --limit 10
```

### Using kubectl (Kubernetes Mode)

```bash
# View all muster events
kubectl get events --field-selector source=muster

# Filter by object
kubectl get events --field-selector involvedObject.kind=MCPServer,involvedObject.name=github-server

# Filter by namespace
kubectl get events -n muster-system --field-selector source=muster

# Filter by reason
kubectl get events --field-selector reason=MCPServerFailed

# Watch events in real-time
kubectl get events --watch --field-selector source=muster
```

### Event Analysis Patterns

#### Troubleshooting Workflow Issues
```bash
# 1. Check recent events for the workflow
muster events --resource-type workflow --resource-name my-workflow --since 1h

# 2. Check MCPServer events for required tools
muster events --resource-type mcpserver --type Warning --since 1h
```

#### Monitoring Workflow Executions
```bash
# 1. Check workflow execution events
muster events --resource-type workflow --resource-name deploy-app --since 30m

# 2. Filter for execution-specific events
muster events --resource-type workflow | grep "execution"

# 3. Look for step failures
muster events --resource-type workflow --type Warning
```

#### Health Monitoring
```bash
# 1. Check for health-related warnings
muster events --type Warning | grep -i health

# 2. Monitor recovery events
muster events | grep -i recovery

# 3. Check tool availability issues
muster events | grep -i "unavailable\|missing"
```

## Troubleshooting Common Scenarios

### Scenario 1: MCPServer Won't Start

**Symptoms:**
- `MCPServerFailed` events
- `MCPServerToolsUnavailable` events
- Tools missing from aggregator

**Investigation Steps:**
```bash
# 1. Check recent MCPServer events
muster events --resource-type mcpserver --resource-name <server> --since 1h

# 2. Check server configuration
muster get mcpserver <server> --output yaml

# 3. Test server availability
muster check mcpserver <server>

# 4. Try manual start
muster start mcpserver <server>
```

**Common Causes:**
- Port already in use
- Invalid command or arguments
- Missing dependencies
- Permission issues
- Network connectivity problems

### Scenario 2: Workflow Execution Fails

**Symptoms:**
- `WorkflowExecutionFailed` events
- `WorkflowStepFailed` events
- Incomplete workflow results

**Investigation Steps:**
```bash
# 1. Check workflow execution events
muster events --resource-type workflow --resource-name <workflow> --since 30m

# 2. Get execution details
muster get workflow-execution <execution-id>

# 3. Test failed tool manually
muster agent --repl
# > call <failed-tool> {...}

# 4. Check tool availability
muster check workflow <workflow>
```

### Scenario 3: Tools Suddenly Unavailable

**Symptoms:**
- Multiple `ToolsUnavailable` events
- MCPServer tools become unavailable
- Workflows fail with tool errors

**Investigation Steps:**
```bash
# 1. Check which MCPServers are affected
muster events --type Warning | grep -i "tools.*unavailable"

# 2. Check MCPServer status
muster list mcpserver

# 3. Check for connectivity issues
muster agent --repl
# > list tools

# 4. Restart affected servers
muster restart mcpserver <affected-server>
```

## Best Practices

### Event Monitoring

1. **Regular Health Checks**
   ```bash
   # Set up regular monitoring
   muster events --type Warning --since 1h
   ```

2. **Resource-Specific Monitoring**
   ```bash
   # Monitor critical resources
   muster events --resource-type mcpserver --resource-name critical-server
   ```

3. **Automated Alerting**
   ```bash
   # Example: Alert on warnings
   if muster events --type Warning --since 5m | grep -q .; then
     echo "Warning events detected!"
   fi
   ```

### Event Analysis

1. **Timeline Analysis**: Use `--since` and `--until` to understand event sequences
2. **Correlation**: Compare events across related resources (Workflow + MCPServer)
3. **Pattern Recognition**: Look for recurring issues or patterns

### Operational Guidelines

1. **Warning Events**: Always investigate Warning events promptly
2. **Recovery Events**: Monitor recovery success rates
3. **Tool Availability**: Track tool availability patterns for capacity planning
4. **Execution Monitoring**: Monitor workflow execution success rates

## Event Retention and Cleanup

### Kubernetes Mode
- Events follow Kubernetes TTL policies (typically 1 hour)
- Use external logging solutions for long-term retention
- Consider event aggregation for monitoring

### Filesystem Mode
- Events persist in `events.log` file
- Manual cleanup required
- Suitable for development and debugging

## Integration with Monitoring Tools

### Prometheus Integration
Events can be scraped and converted to metrics for dashboards and alerting.

### Log Aggregation
Events can be forwarded to log aggregation systems for analysis and retention.

### Custom Dashboards
Event data can power operational dashboards showing system health and activity.

## Conclusion

Muster's event system provides comprehensive visibility into system operations. By understanding event types, their meanings, and troubleshooting approaches, operators can effectively monitor, diagnose, and maintain muster deployments.

Regular event monitoring, combined with proper alerting and response procedures, ensures reliable operation of muster-managed services and workflows.
