# Optimization — Technical Specification

**Version**: v1.0.0 | **Status**: Active | **Last Updated**: March 2026

## Overview

Inference optimization engine providing three capabilities: LRU result caching to avoid redundant model calls, request batching with configurable size and timeout to amortize per-request overhead, and a unified `InferenceOptimizer` facade that combines both with latency and cache-hit tracking.

## Architecture

Three-layer design: **models** (configuration and result dataclasses), **cache** (thread-safe LRU), and **batcher** (background-thread request collector). `InferenceOptimizer` composes `InferenceCache` and `RequestBatcher` to provide a single-call `infer()` interface that checks cache, calls the model, stores results, and tracks statistics.

## Key Classes

### `OptimizationConfig`

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `quantization` | `QuantizationType` | `FP32` | Model quantization level |
| `max_batch_size` | `int` | `32` | Maximum requests per batch |
| `batch_timeout_ms` | `float` | `100.0` | Maximum wait before dispatching incomplete batch |
| `enable_caching` | `bool` | `True` | Whether to use the inference cache |
| `cache_max_size` | `int` | `1000` | Maximum cached entries before LRU eviction |
| `num_workers` | `int` | `4` | Number of worker threads |

### `InferenceCache`

| Method | Parameters | Returns | Description |
|--------|-----------|---------|-------------|
| `get` | `key: str` | `Any or None` | Retrieve cached value; updates access order |
| `put` | `key: str, value: Any` | `None` | Store value; evicts LRU if at capacity |
| `contains` | `key: str` | `bool` | Check membership |
| `clear` | — | `None` | Remove all entries |
| `size` | — (property) | `int` | Current entry count |

### `RequestBatcher[T]`

| Method | Parameters | Returns | Description |
|--------|-----------|---------|-------------|
| `start` | — | `None` | Launch daemon processing thread |
| `stop` | — | `None` | Stop processing thread (2s join timeout) |
| `submit_sync` | `input_data: T, timeout: float` | `Any` | Submit and block for result |
| `submit_async` | `input_data: T` | `Future` | Submit and return `Future` |
| `stats` | — (property) | `dict` | Total requests, batches, average batch size |

### `InferenceOptimizer`

| Method | Parameters | Returns | Description |
|--------|-----------|---------|-------------|
| `infer` | `input_data: Any, use_cache: bool` | `InferenceResult` | Single inference with optional caching |
| `infer_batch` | `inputs: list[Any]` | `list[InferenceResult]` | Batch inference (direct model call) |
| `stats` | — (property) | `InferenceStats` | Requests, batches, cache hit rate, avg latency |
| `clear_cache` | — | `None` | Flush the inference cache |

## Dependencies

- **Internal**: None
- **External**: Standard library only (`threading`, `queue`, `time`, `concurrent.futures`, `dataclasses`, `enum`)

## Constraints

- LRU eviction in `InferenceCache` is O(n) for access-order removal (list-based).
- `RequestBatcher` daemon thread blocks on `queue.get()` with timeout.
- `infer_batch()` bypasses the batcher and calls the model function directly.
- Cache keys for non-string inputs use `str(hash(str(input_data)))`.
- Zero-mock: real data only, `NotImplementedError` for unimplemented paths.

## Error Handling

- `RequestBatcher._process_batch()` catches processor exceptions and sets them on all pending `Future` instances.
- `InferenceOptimizer.infer()` does not catch model exceptions; they propagate to the caller.
- All errors logged before propagation.

## Navigation

- **Self**: `SPEC.md`
- **Parent**: [../README.md](../README.md)
- **Readme**: [README.md](README.md)
- **Agents**: [AGENTS.md](AGENTS.md)
- **Repository Root**: [README.md](../../../../README.md)
