# State-of-the-Art Methods (2025-2026)

## Table of Contents

- [Regime Detection](#regime-detection)
  - [Why It Matters for Range Bars](#why-it-matters-for-range-bars)
  - [BOCPD (Bayesian Online Changepoint Detection)](#bocpd-bayesian-online-changepoint-detection)
  - [PELT (Pruned Exact Linear Time)](#pelt-pruned-exact-linear-time)
  - [Integration with Range Bar Metrics](#integration-with-range-bar-metrics)
  - [References](#references)
- [Probabilistic Forecasting](#probabilistic-forecasting)
  - [Why Point Estimates Are Insufficient](#why-point-estimates-are-insufficient)
  - [Conformal Prediction (Distribution-Free)](#conformal-prediction-distribution-free)
  - [Quantile Regression](#quantile-regression)
  - [References](#references-1)
- [Explainability (SHAP/LIME)](#explainability-shaplime)
  - [EU AI Act 2025 Compliance](#eu-ai-act-2025-compliance)
  - [SHAP (SHapley Additive exPlanations)](#shap-shapley-additive-explanations)
  - [LIME (Local Interpretable Model-agnostic Explanations)](#lime-local-interpretable-model-agnostic-explanations)
  - [Integration with Range Bar Metrics](#integration-with-range-bar-metrics-1)
  - [References](#references-2)
- [Transformers for Time Series (2025 SOTA)](#transformers-for-time-series-2025-sota)
  - [Why Transformers?](#why-transformers)
  - [Recommended: PatchTST for Range Bars](#recommended-patchtst-for-range-bars)
  - [References](#references-3)
- [Summary: SOTA Integration Checklist](#summary-sota-integration-checklist)
- [Installation](#installation)

Advanced techniques for range bar analysis beyond baseline metrics.

## Regime Detection

### Why It Matters for Range Bars

Range bars exhibit **regime-dependent behavior**:

- High volatility: More bars per day, shorter durations
- Low volatility: Fewer bars, longer durations
- Trend regimes: Directional clustering
- Mean-reversion regimes: Oscillatory patterns

A single model trained across all regimes **underperforms** regime-specific models.

### BOCPD (Bayesian Online Changepoint Detection)

**Adams & MacKay (2007)** - Gold standard for online detection.

```python
import numpy as np
from scipy import stats

def bocpd_online(
    data: np.ndarray,
    hazard_rate: float = 0.01,
    observation_model: str = "gaussian",
) -> np.ndarray:
    """Bayesian Online Changepoint Detection.

    Args:
        data: Time series observations (e.g., daily returns)
        hazard_rate: Prior probability of changepoint (1/expected_run_length)
        observation_model: "gaussian" or "student_t"

    Returns:
        Array of changepoint probabilities per timestep
    """
    n = len(data)
    R = np.zeros((n + 1, n + 1))  # Run length distribution
    R[0, 0] = 1.0

    # Sufficient statistics for Gaussian
    sum_x = np.zeros(n + 1)
    sum_x2 = np.zeros(n + 1)

    changepoint_probs = np.zeros(n)

    for t in range(n):
        x = data[t]

        # Update sufficient statistics
        sum_x[1:t+2] = sum_x[:t+1] + x
        sum_x2[1:t+2] = sum_x2[:t+1] + x**2

        # Predictive probability under each run length
        predprobs = np.zeros(t + 1)
        for r in range(t + 1):
            n_obs = r + 1
            mean_r = sum_x[r+1] / n_obs if n_obs > 0 else 0
            var_r = max(1e-10, sum_x2[r+1] / n_obs - mean_r**2 if n_obs > 1 else 1.0)
            predprobs[r] = stats.norm.pdf(x, mean_r, np.sqrt(var_r))

        # Growth probability
        R[1:t+2, t+1] = R[:t+1, t] * predprobs * (1 - hazard_rate)

        # Changepoint probability
        R[0, t+1] = np.sum(R[:t+1, t] * predprobs) * hazard_rate

        # Normalize
        R[:t+2, t+1] /= R[:t+2, t+1].sum()

        # Changepoint probability at t
        changepoint_probs[t] = R[0, t+1]

    return changepoint_probs


def detect_regimes(data: np.ndarray, threshold: float = 0.5) -> list[int]:
    """Get changepoint indices where probability > threshold."""
    cp_probs = bocpd_online(data)
    return list(np.where(cp_probs > threshold)[0])
```

### PELT (Pruned Exact Linear Time)

**Killick et al. (2012)** - Offline detection with optimal segmentation.

```python
# Using ruptures library (pip install ruptures)
import ruptures as rpt

def pelt_offline(
    data: np.ndarray,
    model: str = "rbf",  # "l1", "l2", "rbf", "normal"
    min_size: int = 10,
    penalty: float | None = None,
) -> list[int]:
    """PELT changepoint detection (offline).

    Args:
        data: Time series (1D or 2D)
        model: Cost function model
        min_size: Minimum segment length
        penalty: Penalty value (None = auto BIC)

    Returns:
        List of changepoint indices
    """
    if penalty is None:
        # BIC penalty
        penalty = np.log(len(data)) * data.ndim

    algo = rpt.Pelt(model=model, min_size=min_size).fit(data)
    changepoints = algo.predict(pen=penalty)

    # Remove last point (always included by ruptures)
    return changepoints[:-1] if changepoints else []
```

### Integration with Range Bar Metrics

```python
def regime_aware_metrics(
    predictions: np.ndarray,
    actuals: np.ndarray,
    timestamps: np.ndarray,
    regime_labels: np.ndarray,
) -> dict:
    """Compute metrics per regime for regime-aware evaluation."""
    results = {"overall": evaluate_fold(predictions, actuals, timestamps)}

    unique_regimes = np.unique(regime_labels)
    for regime in unique_regimes:
        mask = regime_labels == regime
        if mask.sum() < 10:
            continue

        results[f"regime_{regime}"] = evaluate_fold(
            predictions[mask],
            actuals[mask],
            timestamps[mask],
        )

    return results
```

### References

```bibtex
@article{adams2007bocpd,
  title={Bayesian Online Changepoint Detection},
  author={Adams, Ryan Prescott and MacKay, David JC},
  journal={arXiv preprint arXiv:0710.3742},
  year={2007}
}

@article{killick2012pelt,
  title={Optimal Detection of Changepoints with a Linear Computational Cost},
  author={Killick, Rebecca and Fearnhead, Paul and Eckley, Idris A},
  journal={Journal of the American Statistical Association},
  year={2012}
}
```

---

## Probabilistic Forecasting

### Why Point Estimates Are Insufficient

Point predictions (e.g., `prediction = 0.003`) provide no uncertainty quantification.

For risk management, you need:

- **Prediction intervals**: "95% CI: [-0.01, 0.02]"
- **Distribution forecasts**: Full predictive distribution
- **Calibration**: Stated coverage matches empirical coverage

### Conformal Prediction (Distribution-Free)

**Vovk et al. (2005), Angelopoulos et al. (2023)** - Finite-sample guarantees.

```python
def conformal_prediction_interval(
    residuals_cal: np.ndarray,  # Calibration set residuals
    predictions_test: np.ndarray,
    alpha: float = 0.05,
) -> tuple[np.ndarray, np.ndarray]:
    """Split conformal prediction interval.

    Guarantees: P(Y_test in interval) >= 1 - alpha (finite sample)

    Args:
        residuals_cal: |y_cal - pred_cal| from calibration set
        predictions_test: Point predictions for test set
        alpha: Miscoverage rate (0.05 = 95% interval)

    Returns:
        (lower_bounds, upper_bounds) for test predictions
    """
    n_cal = len(residuals_cal)

    # Quantile with finite-sample correction
    q_level = np.ceil((n_cal + 1) * (1 - alpha)) / n_cal
    q_hat = np.quantile(residuals_cal, min(q_level, 1.0))

    lower = predictions_test - q_hat
    upper = predictions_test + q_hat

    return lower, upper


def conformal_prediction_with_wfo(
    predictions: np.ndarray,
    actuals: np.ndarray,
    fold_indices: list[tuple[int, int]],  # (cal_start, test_start, test_end)
    alpha: float = 0.05,
) -> dict:
    """Conformal prediction integrated with WFO.

    Each fold uses previous OOS data as calibration set.
    """
    all_intervals = []
    coverages = []

    for i, (cal_start, test_start, test_end) in enumerate(fold_indices):
        if i == 0:
            # First fold: no calibration data, skip
            continue

        # Calibration: previous fold's OOS residuals
        cal_residuals = np.abs(
            actuals[cal_start:test_start] - predictions[cal_start:test_start]
        )

        # Test predictions
        test_preds = predictions[test_start:test_end]
        test_actuals = actuals[test_start:test_end]

        lower, upper = conformal_prediction_interval(
            cal_residuals, test_preds, alpha
        )

        # Check coverage
        covered = (test_actuals >= lower) & (test_actuals <= upper)
        coverage = np.mean(covered)

        all_intervals.append({
            "fold": i,
            "lower": lower,
            "upper": upper,
            "coverage": coverage,
            "target_coverage": 1 - alpha,
        })
        coverages.append(coverage)

    return {
        "intervals": all_intervals,
        "mean_coverage": np.mean(coverages),
        "coverage_std": np.std(coverages),
        "is_calibrated": np.mean(coverages) >= (1 - alpha - 0.05),  # Within 5%
    }
```

### Quantile Regression

Direct prediction of quantiles (e.g., 5th, 50th, 95th percentiles).

```python
import torch
import torch.nn as nn

class QuantileLoss(nn.Module):
    """Pinball loss for quantile regression."""

    def __init__(self, quantiles: list[float]):
        super().__init__()
        self.quantiles = torch.tensor(quantiles)

    def forward(self, predictions: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
        # predictions: (batch, n_quantiles)
        # targets: (batch, 1)

        errors = targets - predictions
        losses = torch.max(
            (self.quantiles - 1) * errors,
            self.quantiles * errors,
        )
        return losses.mean()


class QuantileBiLSTM(nn.Module):
    """BiLSTM with multiple quantile outputs."""

    def __init__(
        self,
        input_size: int,
        hidden_size: int = 48,
        quantiles: list[float] = [0.05, 0.25, 0.50, 0.75, 0.95],
    ):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size, hidden_size, bidirectional=True, batch_first=True
        )
        self.fc = nn.Linear(hidden_size * 2, len(quantiles))
        self.quantiles = quantiles

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        lstm_out, _ = self.lstm(x)
        last_hidden = lstm_out[:, -1, :]
        return self.fc(last_hidden)

    def predict_interval(self, x: torch.Tensor, alpha: float = 0.05) -> tuple:
        """Get prediction interval from quantile outputs."""
        quantiles = self.forward(x)
        # Assuming quantiles = [0.05, 0.25, 0.50, 0.75, 0.95]
        lower = quantiles[:, 0]  # 5th percentile
        median = quantiles[:, 2]  # 50th percentile
        upper = quantiles[:, 4]  # 95th percentile
        return lower, median, upper
```

### References

```bibtex
@book{vovk2005conformal,
  title={Algorithmic Learning in a Random World},
  author={Vovk, Vladimir and Gammerman, Alex and Shafer, Glenn},
  year={2005},
  publisher={Springer}
}

@article{angelopoulos2023conformal,
  title={Conformal Prediction: A Gentle Introduction},
  author={Angelopoulos, Anastasios N and Bates, Stephen},
  journal={Foundations and Trends in Machine Learning},
  year={2023}
}
```

---

## Explainability (SHAP/LIME)

### EU AI Act 2025 Compliance

The EU AI Act (effective 2025) requires explainability for "high-risk" AI systems, including financial trading algorithms with significant economic impact.

Key requirements:

- **Transparency**: Users must understand how predictions are made
- **Documentation**: Model behavior must be documented
- **Auditability**: Explanations must be reproducible

### SHAP (SHapley Additive exPlanations)

**Lundberg & Lee (2017)** - Game-theoretic feature attribution.

```python
import shap
import numpy as np

def explain_bilstm_predictions(
    model,  # Trained BiLSTM
    X_train: np.ndarray,
    X_explain: np.ndarray,
    feature_names: list[str],
    n_background: int = 100,
) -> dict:
    """SHAP explanations for BiLSTM predictions.

    Args:
        model: Trained model with predict() method
        X_train: Training data for background distribution
        X_explain: Samples to explain
        feature_names: Names of input features
        n_background: Number of background samples

    Returns:
        SHAP values and summary statistics
    """
    # Sample background data
    if len(X_train) > n_background:
        idx = np.random.choice(len(X_train), n_background, replace=False)
        background = X_train[idx]
    else:
        background = X_train

    # DeepExplainer for neural networks
    explainer = shap.DeepExplainer(model, background)
    shap_values = explainer.shap_values(X_explain)

    # Aggregate across time steps (for sequence data)
    # Shape: (n_samples, seq_len, n_features) -> (n_samples, n_features)
    if len(shap_values.shape) == 3:
        shap_aggregated = np.abs(shap_values).mean(axis=1)
    else:
        shap_aggregated = np.abs(shap_values)

    # Feature importance ranking
    feature_importance = shap_aggregated.mean(axis=0)
    ranked_features = sorted(
        zip(feature_names, feature_importance),
        key=lambda x: x[1],
        reverse=True,
    )

    return {
        "shap_values": shap_values,
        "shap_aggregated": shap_aggregated,
        "feature_importance": dict(ranked_features),
        "top_5_features": [f[0] for f in ranked_features[:5]],
    }


def generate_explanation_report(
    shap_results: dict,
    predictions: np.ndarray,
    actuals: np.ndarray,
    output_path: str,
) -> None:
    """Generate EU AI Act compliant explanation report."""
    report = f"""
# Model Explanation Report

## Feature Importance (SHAP)

Top contributing features to model predictions:

| Rank | Feature | Mean |SHAP| |
|------|---------|------------|
"""
    for i, (feat, importance) in enumerate(shap_results["feature_importance"].items()):
        if i >= 10:
            break
        report += f"| {i+1} | {feat} | {importance:.4f} |\n"

    report += f"""
## Model Performance Summary

- Total predictions: {len(predictions)}
- Mean prediction: {np.mean(predictions):.4f}
- Mean actual: {np.mean(actuals):.4f}
- Correlation: {np.corrcoef(predictions, actuals)[0,1]:.4f}

## Interpretation

The model primarily relies on the following features:
{', '.join(shap_results['top_5_features'])}

This report is generated for EU AI Act compliance documentation.
"""
    with open(output_path, "w") as f:
        f.write(report)
```

### LIME (Local Interpretable Model-agnostic Explanations)

**Ribeiro et al. (2016)** - Local surrogate explanations.

```python
from lime import lime_tabular

def lime_explain_prediction(
    model,
    X_train: np.ndarray,
    x_instance: np.ndarray,
    feature_names: list[str],
    num_features: int = 10,
) -> dict:
    """LIME explanation for a single prediction.

    Better for individual "why this prediction?" questions.
    """
    explainer = lime_tabular.LimeTabularExplainer(
        X_train,
        feature_names=feature_names,
        mode="regression",
    )

    explanation = explainer.explain_instance(
        x_instance,
        model.predict,
        num_features=num_features,
    )

    return {
        "feature_weights": dict(explanation.as_list()),
        "local_prediction": explanation.local_pred[0],
        "intercept": explanation.intercept[0],
        "r2_score": explanation.score,
    }
```

### Integration with Range Bar Metrics

```python
def evaluate_fold_with_explanations(
    predictions: np.ndarray,
    actuals: np.ndarray,
    timestamps: np.ndarray,
    model,
    X_train: np.ndarray,
    X_test: np.ndarray,
    feature_names: list[str],
) -> dict:
    """Full evaluation including explainability metrics."""
    # Standard metrics
    metrics = evaluate_fold(predictions, actuals, timestamps)

    # Explainability
    shap_results = explain_bilstm_predictions(
        model, X_train, X_test[:100], feature_names
    )

    metrics["explainability"] = {
        "top_5_features": shap_results["top_5_features"],
        "feature_importance": shap_results["feature_importance"],
        "explanation_method": "SHAP DeepExplainer",
    }

    return metrics
```

### References

```bibtex
@inproceedings{lundberg2017shap,
  title={A Unified Approach to Interpreting Model Predictions},
  author={Lundberg, Scott M and Lee, Su-In},
  booktitle={NeurIPS},
  year={2017}
}

@inproceedings{ribeiro2016lime,
  title={Why Should I Trust You?: Explaining the Predictions of Any Classifier},
  author={Ribeiro, Marco Tulio and Singh, Sameer and Guestrin, Carlos},
  booktitle={KDD},
  year={2016}
}
```

---

## Transformers for Time Series (2025 SOTA)

### Why Transformers?

BiLSTMs are being replaced by Transformer architectures for time series:

| Architecture | Pros                         | Cons                      |
| ------------ | ---------------------------- | ------------------------- |
| BiLSTM       | Well-understood, stable      | Sequential, slow training |
| **TFT**      | Interpretable, multi-horizon | Complex, heavy            |
| **Informer** | Long sequences efficient     | Less interpretable        |
| **PatchTST** | Simple, strong baseline      | Newer, less validated     |

### Recommended: PatchTST for Range Bars

```python
# Using pytorch-forecasting or huggingface
# pip install pytorch-forecasting

from pytorch_forecasting import TemporalFusionTransformer
from pytorch_forecasting.data import TimeSeriesDataSet

# Example setup (conceptual)
training = TimeSeriesDataSet(
    data,
    time_idx="time_idx",
    target="return",
    group_ids=["asset"],
    max_encoder_length=60,  # Look-back
    max_prediction_length=1,  # 1-step ahead
    # ... additional config
)

model = TemporalFusionTransformer.from_dataset(
    training,
    hidden_size=32,
    attention_head_size=4,
    dropout=0.1,
    hidden_continuous_size=16,
)
```

### References

```bibtex
@article{nie2023patchtst,
  title={A Time Series is Worth 64 Words: Long-term Forecasting with Transformers},
  author={Nie, Yuqi and others},
  journal={ICLR},
  year={2023}
}

@article{lim2021tft,
  title={Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting},
  author={Lim, Bryan and others},
  journal={International Journal of Forecasting},
  year={2021}
}
```

---

## Summary: SOTA Integration Checklist

When upgrading range bar analysis to 2025 SOTA:

- [ ] **Regime detection**: Implement BOCPD or PELT for volatility regimes
- [ ] **Uncertainty quantification**: Add conformal prediction intervals
- [ ] **Explainability**: Generate SHAP reports for EU AI Act compliance
- [ ] **Architecture**: Consider PatchTST or TFT over BiLSTM for new projects
- [ ] **Calibration**: Verify prediction interval coverage empirically

## Installation

```bash
# Regime detection
pip install ruptures

# Explainability
pip install shap lime

# Transformers (optional)
pip install pytorch-forecasting
```
