## Deprecated: Agent Tracing Guide

This legacy document described tracing for a prior “dynamic agent” orchestration layer. The current system executes workflows via `workflow_runner`, which internally uses OpenAI Agents SDK tracing for workflow execution and optional summarization.

### How to trace now

- Ensure OpenAI env vars are set in `.env` (copy from `env.example`).
- Run workflows through `workflow_runner`; tracing is enabled internally when the OpenAI Agents SDK is available.
- See `workflow_runner_guide.md` for execution details.

## Architecture

The tracing follows a hierarchical structure:

```
🔍 Splunk Dynamic Troubleshooting Trace
├── 📋 Diagnostic Context Creation
├── 🤖 Workflow Type Detection
├── ⚡ Workflow Execution (Missing Data/Performance/Health Check)
│   ├── 🔍 Workflow Definition Lookup
│   ├── 📊 Dependency Analysis
│   ├── 🏃 Execution Phase 1
│   │   ├── 🤖 Micro Agent Task: license_verification
│   │   │   ├── 📝 Instruction Building
│   │   │   ├── 🔧 OpenAI Agent Execution
│   │   │   └── 🔍 Splunk Search Tool Calls
│   │   └── 🤖 Micro Agent Task: index_verification
│   │       ├── 📝 Instruction Building
│   │       ├── 🔧 OpenAI Agent Execution
│   │       └── 🔍 Splunk Tool Calls
│   ├── 🏃 Execution Phase 2 (dependent tasks)
│   └── 📊 Workflow Result Synthesis
└── 🧠 Orchestration Analysis
```

### Prerequisites

```bash
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o
OPENAI_TEMPERATURE=0.7
OPENAI_MAX_TOKENS=4000
```

## Using the Tracing System

### Example

```python
from src.tools.workflows.workflow_runner import WorkflowRunnerTool

runner = WorkflowRunnerTool("workflow_runner", "workflows")
result = await runner.execute(
    ctx=context,
    workflow_id="missing_data_troubleshooting",
    earliest_time="-24h",
    latest_time="now"
)
```

### Trace Information in Results

The enhanced results include comprehensive tracing metadata:

```python
{
    "status": "success",
    "tracing_info": {
        "trace_available": True,
        "workflow_traced": True,
        "orchestration_traced": True,
        "trace_name": "Splunk Dynamic Troubleshooting: My dashboard shows no data...",
        "trace_metadata": {
            "problem_description": "My dashboard shows no data for the last 2 hours",
            "time_range": "-24h to now",
            "workflow_type": "auto",
            "tool_name": "dynamic_troubleshoot_agent"
        }
    },
    "execution_metadata": {
        "tracing_enabled": True,
        "workflow_execution_time": 45.2,
        "orchestration_execution_time": 12.8
    }
}
```

## Trace Structure and Spans

### Top-Level Trace

Each troubleshooting session creates a main trace with:

- **Name**: `"Splunk Dynamic Troubleshooting: {problem_description}"`
- **Metadata**: Problem details, time range, workflow type, complexity level
- **Duration**: Full end-to-end execution time

### Workflow-Level Spans

#### Diagnostic Context Creation
- **Span**: `diagnostic_context_creation`
- **Attributes**: Time range, focus index/host, complexity level
- **Purpose**: Track context setup and validation

#### Workflow Type Detection
- **Span**: `workflow_type_detection`
- **Attributes**: Requested vs detected workflow, auto-detection usage
- **Purpose**: Track problem analysis and routing decisions

#### Workflow Execution
- **Span**: `workflow_execution_{workflow_type}`
- **Attributes**: Workflow type, problem description, status, task count
- **Purpose**: Track overall workflow orchestration

### Micro-Agent Level Spans

#### Task Execution
- **Span**: `micro_agent_task_{task_id}`
- **Attributes**: Agent name, task details, tools, dependencies, context
- **Purpose**: Track individual task execution within workflows

#### Instruction Building
- **Span**: `instruction_building`
- **Attributes**: Dependencies, context injection, instruction length
- **Purpose**: Track dynamic instruction generation

#### Agent Execution
- **Span**: `openai_agent_execution` or `fallback_execution`
- **Attributes**: Model, temperature, turns, execution method
- **Purpose**: Track AI agent reasoning and decision making

### Tool-Level Spans

#### Splunk Tool Calls
- **Spans**: `splunk_search_{task_id}`, `splunk_oneshot_search_{task_id}`, etc.
- **Attributes**: Tool name, query, parameters, success, results
- **Purpose**: Track individual Splunk API interactions

Example tool span attributes:
```python
{
    "tool_name": "run_splunk_search",
    "query": "index=_internal earliest=-24h | stats count",
    "earliest_time": "-24h",
    "latest_time": "now",
    "agent_name": "DynamicAgent_license_verification",
    "task_id": "license_verification",
    "tool_success": True,
    "result_length": 1024
}
```

### Viewing traces

- Use the OpenAI Platform “Traces” dashboard when the SDK is active.

### Custom Trace Processors

You can add custom trace processors to send traces to other observability platforms:

```python
from agents import add_trace_processor
from your_observability_platform import CustomTraceProcessor

# Add additional trace processor
add_trace_processor(CustomTraceProcessor(
    endpoint="https://your-platform.com/traces",
    api_key="your-platform-key"
))
```

### Supported External Platforms

The OpenAI Agents SDK supports many observability platforms:

- **Weights & Biases**
- **Arize Phoenix**
- **MLflow**
- **Braintrust**
- **LangSmith**
- **Langfuse**
- **Langtrace**
- **And many more...**

See the [external processors list](https://openai.github.io/openai-agents-python/tracing/#external-tracing-processors-list) for details.

### Troubleshooting

- Verify OpenAI dependencies are installed and `OPENAI_API_KEY` is set.

### Common Issues

#### No Traces Appearing
- Ensure `OPENAI_API_KEY` is set and valid
- Check that `OPENAI_AGENTS_DISABLE_TRACING` is not set to `1`
- Verify network connectivity to OpenAI's tracing endpoint

#### Sensitive Data in Traces
- Set `OPENAI_AGENTS_DONT_LOG_MODEL_DATA=1` to exclude LLM inputs/outputs
- Set `OPENAI_AGENTS_DONT_LOG_TOOL_DATA=1` to exclude tool inputs/outputs
- Review span attributes for any sensitive information

#### Performance Impact
- Tracing has minimal performance overhead (~1-5ms per span)
- For high-volume scenarios, consider sampling or selective tracing
- Use `trace_include_sensitive_data=False` in RunConfig to reduce payload size

## Advanced Configuration

### Custom Trace Metadata

You can add custom metadata to traces by modifying the trace creation:

```python
# In dynamic_troubleshoot_agent.py
trace_metadata = {
    "problem_description": problem_description,
    "user_id": ctx.user_id,  # Add user context
    "environment": "production",  # Add environment info
    "custom_field": "custom_value"
}
```

### Selective Tracing

Enable tracing only for specific scenarios:

```python
# Only trace complex workflows
if complexity_level == "advanced":
    with trace(workflow_name=trace_name, metadata=trace_metadata):
        return await self._execute_with_tracing(...)
else:
    return await self._execute_with_tracing(...)
```

### Trace Sampling

For high-volume environments, implement sampling:

```python
import random

# Sample 10% of traces
if random.random() < 0.1:
    with trace(workflow_name=trace_name, metadata=trace_metadata):
        return await self._execute_with_tracing(...)
```

## Best Practices

### Span Naming
- Use descriptive, hierarchical names: `workflow_execution_missing_data`
- Include identifiers for correlation: `micro_agent_task_{task_id}`
- Keep names consistent across similar operations

### Attribute Usage
- Add relevant business context: problem type, user info, environment
- Include performance metrics: execution time, result counts
- Capture error details: error messages, types, stack traces
- Use consistent attribute names across spans

### Security Considerations
- Never include passwords, API keys, or PII in span attributes
- Truncate long queries or data for readability: `query[:200]`
- Use environment variables to control sensitive data logging
- Review trace data before enabling in production

### Performance Optimization
- Minimize span creation overhead by checking availability first
- Use batch operations where possible
- Consider async span creation for high-throughput scenarios
- Monitor trace payload sizes and optimize as needed

## Integration Examples

### LangSmith Integration

```python
from langsmith.wrappers import OpenAIAgentsTracingProcessor
from agents import set_trace_processors

# Send traces to both OpenAI and LangSmith
set_trace_processors([
    OpenAIAgentsTracingProcessor(),  # Default OpenAI processor
    LangSmithTracingProcessor(
        project_name="splunk-troubleshooting",
        api_key=os.getenv("LANGSMITH_API_KEY")
    )
])
```

### Custom Analytics

```python
class CustomAnalyticsProcessor:
    def export(self, items):
        for item in items:
            if hasattr(item, 'metadata') and 'problem_description' in item.metadata:
                # Send to custom analytics platform
                self.analytics_client.track_troubleshooting_session(
                    problem=item.metadata['problem_description'],
                    duration=item.execution_time,
                    success=item.status == 'success'
                )

add_trace_processor(CustomAnalyticsProcessor())
```

For current docs, see `workflows-overview.md`, `README.md` in this folder, and `workflow_runner_guide.md`.
