# Sentry Monitoring for MCP Server

This guide explains how to set up and use Sentry monitoring for the MCP Server for Splunk. Sentry provides comprehensive error tracking, performance monitoring, and distributed tracing specifically optimized for MCP operations.

> **🔧 Sentry is Optional**: The MCP server works perfectly without Sentry. When `SENTRY_DSN` is not set or sentry-sdk is not installed, the server runs normally without any monitoring overhead.

## 🚀 Quick Start

### 1. Install Sentry SDK (Optional)

Sentry is an optional dependency. Install it with:

```bash
# If using pip
pip install mcp-server-for-splunk[sentry]

# Or install sentry-sdk directly
pip install sentry-sdk[starlette,httpx,asyncio]
```

> **Note**: The Docker image includes Sentry by default. For local development, you need to install it explicitly.

### 2. Get Your Sentry DSN

1. Create a Sentry account at [sentry.io](https://sentry.io)
2. Create a new Python project (select **ASGI** template for best results)
3. Copy your DSN from **Settings → Projects → [Your Project] → Client Keys (DSN)**

### 3. Configure Environment

Add to your `.env` file:

```bash
# Required: Your Sentry DSN
SENTRY_DSN=https://your-key@o123.ingest.sentry.io/project-id

# Recommended settings
SENTRY_ENVIRONMENT=development
SENTRY_RELEASE=mcp-server-splunk@0.5.0
SENTRY_TRACES_SAMPLE_RATE=1.0
```

### 4. Start the Server

```bash
# Docker (Sentry included automatically)
./scripts/build_and_run.sh --docker

# Local development (requires step 1)
uv run python src/server.py
```

You'll see in the logs:

```
INFO - Sentry initialized successfully (env=development, release=mcp-server-splunk@0.5.0, traces=100.0%)
INFO - Sentry MCP middleware added for request tracing
INFO - Sentry HTTP middleware added for request tracing
```

If Sentry is **not configured**, you'll see:

```
INFO - SENTRY_DSN not set, Sentry monitoring disabled
```

Or if the SDK is **not installed**:

```
INFO - sentry-sdk not installed - Sentry monitoring disabled
```

## 📊 What Gets Monitored

### Official Sentry MCP Integration

This server uses Sentry's official `MCPIntegration` which automatically captures:

- **Tool executions**: Tool name, arguments, results, and execution errors
- **Prompt requests**: Prompt name, arguments, message counts
- **Resource access**: Resource URI, protocol, access patterns
- **Request context**: Request IDs, session IDs, transport types (stdio/HTTP)
- **Execution spans**: Timing information for all handler invocations

### Automatic Instrumentation

The integration automatically monitors:

| Operation | Span Type | Attributes |
|-----------|-----------|------------|
| HTTP requests | `http.server` | method, path, status code, duration |
| Tool calls | `mcp.tool` | tool name, arguments, result status |
| Resource reads | `mcp.resource` | URI, content size |
| Prompt gets | `mcp.prompt` | name, arguments |
| Splunk API calls | `splunk.api` | operation, query preview, result count |

### MCP-Specific Attributes

Every span includes MCP context:

```python
{
    "mcp.session.id": "session_xyz789",
    "mcp.request.id": "req_abc123",
    "mcp.method.name": "tools/call",
    "mcp.tool.name": "splunk_search",
    "mcp.transport": "http"
}
```

## 🛠 Configuration Options

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `SENTRY_DSN` | Sentry project DSN (required to enable) | - |
| `SENTRY_ENVIRONMENT` | Environment name | `development` |
| `SENTRY_RELEASE` | Release/version identifier | `mcp-server-splunk@0.5.0` |
| `SENTRY_TRACES_SAMPLE_RATE` | Traces sampling rate (0.0-1.0) | `1.0` |
| `SENTRY_PROFILES_SAMPLE_RATE` | Profiling sampling rate (0.0-1.0) | `0.1` |
| `SENTRY_DEBUG` | Enable debug mode | `false` |
| `SENTRY_ENABLE_LOGS` | Enable Sentry Logs feature | `true` |
| `SENTRY_SEND_PII` | Include tool inputs/outputs in traces | `true` |

### Smart Sampling

The integration uses intelligent sampling:

- **MCP operations**: Always traced (100%)
- **Tool executions**: Always traced (100%)
- **Splunk operations**: Higher sampling rate (1.5x base rate)
- **Health checks**: Very low sampling (1%)

## 📝 Sentry Logs

The MCP server captures Python logs to Sentry's Logs feature, enabling centralized log search and analysis.

### How It Works

```python
import logging

# All your existing Python logging is captured automatically
logger = logging.getLogger(__name__)

# INFO+ logs → Sent to Sentry Logs (searchable/filterable)
logger.info("Tool executed", extra={"tool_name": "splunk_search", "duration_ms": 150})

# ERROR+ logs → Create error events AND sent to Sentry Logs
logger.error("Tool failed", extra={"tool_name": "splunk_search", "error": "timeout"})
```

### Log Levels

| Log Level | Sentry Logs | Error Events | Breadcrumbs |
|-----------|-------------|--------------|-------------|
| `DEBUG` | ❌ | ❌ | ❌ |
| `INFO` | ✅ | ❌ | ✅ |
| `WARNING` | ✅ | ❌ | ✅ |
| `ERROR` | ✅ | ✅ | ✅ |
| `CRITICAL` | ✅ | ✅ | ✅ |

### Searchable Attributes

Extra fields become searchable attributes in Sentry:

```python
logger.info("MCP tool call", extra={
    "mcp.tool.name": "splunk_search",
    "mcp.session.id": "session_123",
    "duration_ms": 250,
    "result_count": 100
})
```

Search in Sentry: `mcp.tool.name:splunk_search duration_ms:>200`

### Disable Logs

```bash
SENTRY_ENABLE_LOGS=false
```

## 📊 Session Tracking & Release Health

The MCP server automatically tracks sessions for Release Health monitoring.

### What's Tracked

- **Session per request**: Each MCP request creates a session
- **Session status**: `ok`, `crashed` (unhandled error), `errored` (captured exception)
- **Release association**: Sessions linked to `SENTRY_RELEASE` version

### Release Health Dashboard

View in Sentry at **Releases → [Your Release]**:

| Metric | Description |
|--------|-------------|
| **Crash Free Sessions** | Percentage of sessions without crashes |
| **Crash Free Users** | Percentage of users without crashes |
| **Adoption** | Rollout percentage of this release |
| **Sessions** | Total session count for this release |

### Example

```bash
# Set release version for tracking
SENTRY_RELEASE=mcp-server-splunk@0.4.1
```

Then in Sentry, you'll see:
- How many sessions for `v0.4.1`
- Crash rate comparison vs previous release
- Error trends per release

## 🔧 Custom Instrumentation

### Decorating Tool Functions

Use the `@trace_mcp_tool` decorator for automatic tool tracing:

```python
from src.core.sentry_integration import trace_mcp_tool

@trace_mcp_tool("my_custom_tool")
async def my_custom_tool(ctx: Context, param: str) -> dict:
    """This tool is automatically traced with Sentry spans."""
    result = await do_something(param)
    return {"status": "success", "data": result}
```

### Tracing Splunk Operations

Use `@trace_splunk_operation` for Splunk-specific operations:

```python
from src.core.sentry_integration import trace_splunk_operation

@trace_splunk_operation("search")
async def execute_search(query: str) -> dict:
    """Splunk search operations get specialized tracking."""
    results = await splunk_service.search(query)
    return results
```

### Manual Spans

Create custom spans for specific operations:

```python
from src.core.sentry_integration import mcp_span

async def complex_operation():
    with mcp_span(
        op="custom.operation",
        name="Process Data",
        step="validation",
        item_count=100
    ) as span:
        # Your operation here
        result = await process_data()
        
        # Add more context
        span.set_data("processed_count", len(result))
```

### Adding Breadcrumbs

Track the sequence of operations:

```python
from src.core.sentry_integration import add_breadcrumb

add_breadcrumb(
    message="User requested search",
    category="user.action",
    level="info",
    data={"query_type": "realtime", "index": "main"}
)
```

### Capturing Errors with Context

Manually capture exceptions with MCP context:

```python
from src.core.sentry_integration import capture_mcp_error

try:
    result = await risky_operation()
except Exception as e:
    capture_mcp_error(
        error=e,
        tool_name="risky_operation",
        extra={"retry_count": 3, "input_size": len(data)}
    )
    raise
```

## 📈 Viewing Data in Sentry

### Performance Dashboard

Navigate to **Performance** in Sentry to see:

- Transaction throughput
- Average response times
- Slowest operations
- Error rates by endpoint

### Trace View

Click any transaction to see:

- Full distributed trace
- Span waterfall with timing
- MCP-specific attributes
- Related errors

### Filtering by MCP Context

Use these search filters in Sentry:

```
# Find all tool calls
op:mcp.tool

# Find specific tool errors
mcp.tool.name:splunk_search mcp.tool.status:error

# Find session-specific issues
mcp.session.id:session_xyz789

# Find slow Splunk queries
op:splunk.api splunk.duration:>5000
```

## 🏗 Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        HTTP Request                              │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│  SentryHTTPMiddleware                                           │
│  • Creates transaction for HTTP request                         │
│  • Extracts session/request IDs                                 │
│  • Sets MCP context                                             │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│  SentryMCPMiddleware                                            │
│  • Creates spans for MCP methods                                │
│  • Captures tool/resource-specific data                         │
│  • Records method results                                       │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│  Tool Execution (with @trace_mcp_tool)                          │
│  • Individual tool spans                                        │
│  • Argument sanitization                                        │
│  • Result tracking                                              │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│  Splunk Operations (with @trace_splunk_operation)               │
│  • Query tracking                                               │
│  • Result count                                                 │
│  • Performance metrics                                          │
└─────────────────────────────────────────────────────────────────┘
```

## 🔒 Security Considerations

### Automatic Data Sanitization

The integration automatically:

- **Masks sensitive parameters**: Passwords, tokens, API keys
- **Truncates large values**: Queries > 200 chars shown as previews
- **Excludes content**: Only metadata tracked, not actual data

### PII Protection

By default:

```python
send_default_pii=False  # No automatic PII collection
```

Custom data you add should also be sanitized:

```python
from src.core.sentry_integration import _sanitize_kwargs

# Before logging parameters
safe_params = _sanitize_kwargs({"password": "secret", "query": "search *"})
# Result: {"password": "***REDACTED***", "query": "search *"}
```

## 🚨 Alerting

### Recommended Alerts

Set up alerts in Sentry for:

1. **Error spike**: More than 10 errors in 5 minutes
2. **Slow tools**: Tool execution > 10 seconds
3. **Splunk failures**: `splunk.status:error` rate > 5%
4. **Session issues**: Frequent session timeouts

### Example Alert Query

```
# Alert when Splunk searches are slow
transaction:splunk/* splunk.duration:>30000
```

## 🔍 Troubleshooting

### Sentry Not Initializing

Check logs for:

```
INFO - SENTRY_DSN not set, Sentry monitoring disabled
```

**Solution**: Ensure `SENTRY_DSN` is properly set in your environment.

### Traces Not Appearing

1. Check `SENTRY_TRACES_SAMPLE_RATE` is > 0
2. Verify network connectivity to Sentry
3. Enable debug mode: `SENTRY_DEBUG=true`

### Missing MCP Context

If session/request IDs are missing:

1. Ensure middleware order is correct (HeaderCapture before Sentry)
2. Check that clients send `MCP-Session-ID` or `X-Session-ID` headers

## 📚 Related Resources

- [Sentry Python SDK Docs](https://docs.sentry.io/platforms/python/)
- [Sentry MCP Instrumentation Guide](https://docs.sentry.io/platforms/python/tracing/instrumentation/custom-instrumentation/mcp-module/)
- [OpenTelemetry Support](https://docs.sentry.io/platforms/python/tracing/instrumentation/opentelemetry/)
- [MCP Protocol Specification](https://modelcontextprotocol.io/specification)

