# Analysis Tiers Reference

Tiered approach to semantic analysis of terminal recordings.

---

## Tier Overview

| Tier | Tool    | Speed (4MB) | When to Use                      | Accuracy |
| ---- | ------- | ----------- | -------------------------------- | -------- |
| 1    | ripgrep | 50-200ms    | Always start here                | High     |
| 2    | YAKE    | 1-5s        | Auto-discover unexpected terms   | Medium   |
| 3    | TF-IDF  | 5-30s       | Topic modeling                   | Variable |
| 4    | keyBERT | N/A         | **REJECTED** - overkill for logs | N/A      |

---

## Tier 1: ripgrep + Curated Keywords (Primary)

**Always start here.** Fastest and most reliable for known domains.

### Characteristics

- **Speed**: 50-200ms for 4MB file
- **Accuracy**: High (exact matches)
- **Dependencies**: System ripgrep only
- **Best for**: Known keyword domains, quick scans

### Implementation

```bash
/usr/bin/env bash << 'TIER1_EOF'
INPUT_FILE="${1:?}"
KEYWORDS="${2:-sharpe sortino backtest}"

echo "=== Tier 1: Curated Keywords ==="
start=$(date +%s.%N)

for kw in $KEYWORDS; do
  COUNT=$(rg -c -i "$kw" "$INPUT_FILE" 2>/dev/null || echo "0")
  [[ "$COUNT" -gt 0 ]] && echo "$kw: $COUNT"
done

end=$(date +%s.%N)
echo ""
echo "Time: $(echo "$end - $start" | bc)s"
TIER1_EOF
```

### When to Use

- First pass on any recording
- Known domain analysis (trading, ML, dev)
- Quick verification of session content
- Performance-critical workflows

---

## Tier 2: YAKE Unsupervised Extraction (Secondary)

**Use for auto-discovery.** Finds unexpected patterns without predefined keywords.

### Characteristics

- **Speed**: 1-5s for 4MB file
- **Accuracy**: Medium (statistical, may include noise)
- **Dependencies**: `uv run --with yake`
- **Best for**: Discovering new patterns, exploratory analysis

### Why YAKE

| Tool    | Pros                       | Cons                    | Decision |
| ------- | -------------------------- | ----------------------- | -------- |
| YAKE    | Unsupervised, no GPU, fast | Less semantic depth     | **Use**  |
| keyBERT | Semantic embeddings        | Requires GPU, slow      | Rejected |
| TF-IDF  | Well-understood, sklearn   | Needs corpus comparison | Optional |

### Implementation

```bash
/usr/bin/env bash << 'TIER2_EOF'
INPUT_FILE="${1:?}"

echo "=== Tier 2: YAKE Auto-Discovery ==="
start=$(date +%s.%N)

uv run --with yake python3 -c "
import yake

kw_extractor = yake.KeywordExtractor(
    lan='en',
    n=2,              # bi-grams
    dedupLim=0.9,     # deduplication threshold
    dedupFunc='seqm', # sequence matcher
    windowsSize=1,
    top=20
)

with open('$INPUT_FILE') as f:
    text = f.read()

keywords = kw_extractor.extract_keywords(text)
print('Top 20 keywords (lower score = more relevant):')
for score, keyword in keywords:
    print(f'  {score:.4f}  {keyword}')
"

end=$(date +%s.%N)
echo ""
echo "Time: $(echo "$end - $start" | bc)s"
TIER2_EOF
```

### When to Use

- After Tier 1 when looking for unexpected patterns
- New domain exploration
- Comprehensive analysis requests
- When curated keywords miss important content

---

## Tier 3: TF-IDF Topic Modeling (Optional)

**Use for document comparison.** Identifies distinguishing terms across segments.

### Characteristics

- **Speed**: 5-30s for 4MB file
- **Accuracy**: Variable (depends on segmentation)
- **Dependencies**: `uv run --with scikit-learn`
- **Best for**: Comparing recording segments, finding unique terms

### Implementation

```bash
/usr/bin/env bash << 'TIER3_EOF'
INPUT_FILE="${1:?}"
CHUNK_SIZE=1000  # lines per chunk

echo "=== Tier 3: TF-IDF Topic Modeling ==="

uv run --with scikit-learn python3 -c "
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

# Read and chunk file
with open('$INPUT_FILE') as f:
    lines = f.readlines()

chunk_size = $CHUNK_SIZE
chunks = []
for i in range(0, len(lines), chunk_size):
    chunk = ' '.join(lines[i:i+chunk_size])
    chunks.append(chunk)

print(f'Analyzing {len(chunks)} chunks of {chunk_size} lines each')

# TF-IDF vectorization
vectorizer = TfidfVectorizer(
    max_features=50,
    stop_words='english',
    ngram_range=(1, 2)
)
tfidf_matrix = vectorizer.fit_transform(chunks)
feature_names = vectorizer.get_feature_names_out()

# Top terms per chunk
print('')
for i, chunk in enumerate(chunks[:5]):  # First 5 chunks
    scores = tfidf_matrix[i].toarray().flatten()
    top_indices = scores.argsort()[-5:][::-1]
    terms = [feature_names[idx] for idx in top_indices]
    print(f'Chunk {i+1}: {terms}')
"
TIER3_EOF
```

### When to Use

- Comparing different sessions
- Finding distinguishing characteristics
- Long recordings with distinct phases
- Research and exploratory analysis

---

## Tier 4: keyBERT (Rejected)

**Not recommended for terminal recordings.**

### Why Rejected

| Factor       | Issue                                    |
| ------------ | ---------------------------------------- |
| Dependencies | Requires GPU for reasonable speed        |
| Overkill     | Semantic embeddings unnecessary for logs |
| Complexity   | Heavy ML stack (transformers, torch)     |
| Speed        | 30s+ for 4MB without GPU                 |

### Alternative

Use YAKE (Tier 2) for unsupervised extraction. It provides 80% of keyBERT's value at 10% of the cost.

---

## Tier Selection Guide

```
START
  │
  ├─> Known keywords? ─── YES ──> Tier 1 (ripgrep)
  │                                    │
  │                                    v
  │                              Found enough? ── YES ──> DONE
  │                                    │
  │                                    NO
  │                                    │
  │                                    v
  └─> Explore unknown? ─── YES ──> Tier 2 (YAKE)
                                       │
                                       v
                                 Compare segments? ── YES ──> Tier 3 (TF-IDF)
                                       │
                                       NO
                                       │
                                       v
                                     DONE
```

---

## Performance Benchmarks

Based on 4MB converted .txt file (from 3.8GB .cast):

| Tier | Tool    | Time  | Memory | Keywords Found |
| ---- | ------- | ----- | ------ | -------------- |
| 1    | ripgrep | 127ms | 12MB   | Exact matches  |
| 2    | YAKE    | 2.3s  | 180MB  | 20 bi-grams    |
| 3    | TF-IDF  | 8.7s  | 420MB  | 50 per chunk   |

---

## Combined Workflow

For comprehensive analysis:

```bash
/usr/bin/env bash << 'COMBINED_EOF'
INPUT_FILE="${1:?}"

echo "=== Combined Analysis ==="
echo ""

# Tier 1: Quick scan
echo "--- Tier 1: Curated Keywords ---"
for kw in sharpe backtest epoch training iteration commit; do
  COUNT=$(rg -c -i "$kw" "$INPUT_FILE" 2>/dev/null || echo "0")
  [[ "$COUNT" -gt 0 ]] && echo "$kw: $COUNT"
done

echo ""
echo "--- Tier 2: YAKE Discovery ---"
uv run --with yake python3 -c "
import yake
kw = yake.KeywordExtractor(lan='en', n=2, top=10)
with open('$INPUT_FILE') as f:
    for keyword, score in kw.extract_keywords(f.read()):
        print(f'{score:.4f}  {keyword}')
"

echo ""
echo "=== Analysis Complete ==="
COMBINED_EOF
```
