# Enhanced Token Tracking Data Collection

## Current Implementation Status

### ✅ **Implemented Features**
- **Token Usage Tracking** - Basic token metrics fully operational
- **Model Usage Tracking** - Model performance and usage patterns
- **Star Reviews Analytics** - Comprehensive rating analytics with visualizations
- **Metrics Dashboard Integration** - Full web UI with charts and data tables
- **Database Schema** - Stable schema with fixed foreign key constraints

### Current Token Tracking

The existing system tracks basic token usage:
- **research_id** - Links to specific research sessions
- **model_name** - LLM model used
- **provider** - Model provider (OpenAI, Anthropic, Ollama, etc.)
- **prompt_tokens, completion_tokens, total_tokens** - Token usage counts
- **timestamp** - When the LLM call was made

**Note**: Database foreign key constraints were recently fixed to ensure all token tracking data is properly saved without constraint errors.

## Proposed Enhanced Data Collection

### ✅ **Already Available (Can Add Immediately)**

#### Research Context
- **research_query** - The original research question/query
- **research_mode** - `"quick"` or `"detailed"` research mode
- **research_phase** - Current stage of research:
  - `init`, `setup`, `iteration_1`, `iteration_2`, etc.
  - `question_generation`, `parallel_search`, `search_complete`
  - `output_generation`, `report_generation`, `complete`, `error`

#### Performance Metrics
- **response_time_ms** - Duration of LLM call (wrap with timing)
- **success_status** - `"success"`, `"error"`, `"timeout"`
- **error_type** - Specific error if call failed

#### Search Engine Context
- **search_engines_planned** - Which engines were planned (from progress messages)
- **search_engine_selected** - Which engine was actually used
- **search_iteration** - Which search iteration this relates to

### ⚠️ **Requires Minor Implementation Work**

#### Cost Analysis
- **estimated_cost_usd** - Calculated cost based on model pricing tables
- **model_tier** - Classification: `"free"`, `"paid"`, `"premium"`

#### Enhanced Performance
- **query_length** - Character count of original research query
- **tokens_per_second** - Processing speed metric (calculated from response_time_ms)

#### Session Context
- **session_id** - Unique session identifier for grouping related requests
- **request_source** - Origin of request: `"web_ui"`, `"api"`, `"cli"`, `"benchmark"`

### ❌ **Not Feasible with Current Architecture**

#### Complex Quality Metrics
- **response_quality_score** - Would require LLM evaluation system
- **search_results_used** - Not directly tracked in current system
- **result_relevance** - Would need sophisticated analysis

#### Advanced Context
- **retry_count** - Would require modifying LLM retry logic
- **cache_hit** - No caching system currently implemented
- **concurrent_requests** - Complex to track accurately

## Implementation Details

### Current Database Schema (Implemented)

```sql
-- TokenUsage table (current implementation)
CREATE TABLE token_usage (
    id INTEGER PRIMARY KEY,
    research_id INTEGER,  -- Foreign key constraint removed to fix tracking
    model_name TEXT,
    provider TEXT,
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    total_tokens INTEGER,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- ModelUsage table (current implementation)
CREATE TABLE model_usage (
    id INTEGER PRIMARY KEY,
    research_id INTEGER,  -- Foreign key constraint removed to fix tracking
    model_name TEXT,
    provider TEXT,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- ResearchRating table (fully implemented)
CREATE TABLE research_rating (
    id INTEGER PRIMARY KEY,
    research_id INTEGER NOT NULL,
    rating INTEGER NOT NULL CHECK (rating BETWEEN 1 AND 5),
    feedback TEXT,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
```

### Enhanced Database Schema (Proposed for Future)

```sql
-- Enhanced TokenUsage table
CREATE TABLE enhanced_token_usage (
    id INTEGER PRIMARY KEY,
    research_id INTEGER,
    model_name TEXT,
    provider TEXT,

    -- Existing token metrics
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    total_tokens INTEGER,

    -- NEW: Research context (available immediately)
    research_query TEXT,
    research_mode TEXT,
    research_phase TEXT,
    search_iteration INTEGER,

    -- NEW: Performance metrics (available immediately)
    response_time_ms INTEGER,
    success_status TEXT DEFAULT 'success',
    error_type TEXT,

    -- NEW: Search engine context (available immediately)
    search_engines_planned TEXT, -- JSON array
    search_engine_selected TEXT,

    -- NEW: Cost analysis (minor work required)
    estimated_cost_usd REAL,
    model_tier TEXT,

    -- NEW: Enhanced metrics (minor work required)
    query_length INTEGER,
    tokens_per_second REAL,
    session_id TEXT,
    request_source TEXT,

    -- Timing
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
```

### Integration Points

#### TokenCountingCallback Enhancement
The callback can be enhanced to capture additional context:

```python
class TokenCountingCallback:
    def __init__(self, research_id=None, research_context=None):
        self.research_id = research_id
        self.research_context = research_context or {}

    def on_llm_start(self, serialized, prompts, **kwargs):
        self.start_time = time.time()
        # Extract research phase from context
        # Extract search engine info from context

    def on_llm_end(self, response, **kwargs):
        self.response_time = (time.time() - self.start_time) * 1000
        # Calculate additional metrics
        # Save enhanced data to database
```

#### Progress Message Integration
Extract search engine information from existing progress messages:

```python
# From research progress system
if "SEARCH_PLAN:" in message:
    engines = message.split("SEARCH_PLAN:")[1].strip()
    research_context["planned_engines"] = engines

if "ENGINE_SELECTED:" in message:
    engine = message.split("ENGINE_SELECTED:")[1].strip()
    research_context["selected_engine"] = engine
```

#### Cost Calculation System
Add model pricing configuration:

```python
MODEL_PRICING = {
    "gpt-4": {"prompt": 0.03, "completion": 0.06},  # per 1K tokens
    "gpt-3.5-turbo": {"prompt": 0.001, "completion": 0.002},
    "claude-3-opus": {"prompt": 0.015, "completion": 0.075},
    # ... other models
}

def calculate_cost(model_name, prompt_tokens, completion_tokens):
    if model_name in MODEL_PRICING:
        pricing = MODEL_PRICING[model_name]
        prompt_cost = (prompt_tokens / 1000) * pricing["prompt"]
        completion_cost = (completion_tokens / 1000) * pricing["completion"]
        return prompt_cost + completion_cost
    return 0.0
```

## Benefits of Enhanced Tracking

### Immediate Value
- **Cost visibility** - See actual spending per research and model
- **Performance monitoring** - Track response times and success rates
- **Phase analysis** - Understand where tokens are consumed in research process
- **Search optimization** - Identify most effective search engines

### Analytics Possibilities
- **Cost optimization** - Identify expensive queries and suggest alternatives
- **Performance trends** - Monitor system performance over time
- **Usage patterns** - Understand how different research modes perform
- **Model comparison** - Compare efficiency across different models

### Future Dashboard Features
- **Cost breakdown charts** - Visual cost analysis per model/research
- **Performance timeline** - Response time trends
- **Phase efficiency** - Token usage by research stage
- **Search engine effectiveness** - Success rates by engine

## Implementation Priority

### ✅ Phase 1: Core Infrastructure (COMPLETED)
1. ✅ Fixed database foreign key constraints to enable token tracking
2. ✅ Implemented star reviews analytics system
3. ✅ Created comprehensive metrics dashboard with visualizations
4. ✅ Added SQLAlchemy ORM-based data access patterns
5. ✅ Integrated Chart.js for data visualization
6. ✅ Added responsive design with dark theme integration

### Phase 2: Enhanced Token Tracking (NEXT)
1. Add research context (query, mode, phase)
2. Add response timing
3. Add success/error tracking
4. Add search engine context from progress messages

### Phase 3: Cost Analysis
1. Implement model pricing configuration
2. Add cost calculation to token tracking
3. Add model tier classification

### Phase 4: Advanced Features
1. Add session tracking
2. Add request source detection
3. Implement calculated metrics (tokens_per_second, etc.)

## Data Sources

### Research Service Context
- Research query, mode, and ID available in research service
- Phase information from progress callback system
- Session context from Flask request

### Progress Message System
- Search engine planning and selection
- Research iteration tracking
- Error states and completion status

### LLM Integration Points
- Token counts from LLM responses
- Model and provider information
- Response timing from callback wrapper

## Recent Implementation Work

### Star Reviews Analytics (December 2024)
- **Complete implementation** of dedicated star reviews page at `/metrics/star-reviews`
- **API endpoints** for real-time data aggregation using SQLAlchemy ORM
- **Interactive visualizations** including:
  - LLM model performance bar charts
  - Search engine effectiveness charts
  - Rating distribution over time
  - Research analytics with clickable links to research details
- **Dark theme integration** with responsive design
- **Comprehensive testing** with Puppeteer UI automation

### Database Fixes (December 2024)
- **Critical fix**: Removed foreign key constraints from TokenUsage and ModelUsage tables
- **Problem**: Foreign keys referenced non-existent "research_history" table causing silent failures
- **Solution**: Modified `src/local_deep_research/metrics/db_models.py` to use simple integer fields
- **Result**: Token tracking now works correctly without constraint errors

### Files Modified
- `src/local_deep_research/web/routes/metrics_routes.py` - Star reviews routes and API
- `src/local_deep_research/web/templates/pages/star_reviews.html` - Complete analytics UI
- `src/local_deep_research/web/templates/pages/metrics.html` - Navigation integration
- `src/local_deep_research/metrics/db_models.py` - Database schema fixes
- `tests/ui_tests/test_star_reviews_debug.js` - Comprehensive UI testing

### Current Status
- ✅ Token tracking fully operational
- ✅ Star reviews analytics complete and deployed
- ✅ Database schema stable and constraint-free
- ✅ Web UI responsive and functional
- ✅ Testing infrastructure in place

---

*This enhanced tracking system provides significantly more insight into research performance, costs, and patterns while remaining feasible to implement with the current architecture. The core infrastructure is now complete and ready for Phase 2 enhancements.*
