The Error-Handling Overhaul

01Why we did this

The state we started from, and the principle we settled on.

On feature/Temporal-merge-3, error handling was inconsistent. A tier review classified inference workers into three buckets — and a large share of them (Google, Mistral, Azure img-gen, FAL, HuggingFace, Docling, Linkup, pypdfium2) had no SDK exception handling at all: a raw provider error propagated up as-is, untyped, uncategorized. Other layers caught errors and re-wrapped them, sometimes dropping the __cause__. There was a PipeRouter retry loop that only ever ran on the direct path, re-ran at the wrong granularity, and carried a config bug. And except Exception appeared throughout business logic, silently swallowing bugs.

Three cross-cutting rules now hold everywhere and the tracks build on them:

• Single-rooted hierarchy. Every custom exception inherits from pipelex.base_exceptions.PipelexError.
• No broad catches in business logic. except Exception is allowed only at CLI entry points and async task roots — and ruff rule BLE001 is enabled to enforce it.
• Workers transform SDK errors into domain errors. Catch the SDK's typed exceptions, attach an InferenceErrorCategory, chain via from exc.

02The layer model

Errors flow up through six layers. Each one has exactly one job.

03The error's journey

From a raw SDK exception to four different rendered outputs — one schema throughout.

The tracks at a glance

04ErrorReport — the schema

One frozen dataclass. pipelex/base_exceptions.py.

ErrorReport is a frozen pydantic dataclass with extra="forbid". It is the single source of truth for error serialization — to_dict() drops None fields, from_dict() is its strict inverse (used to recover a report that crossed the Temporal boundary), and http_status maps it to an HTTP code for downstream APIs.

 @dataclass(frozen=True, config={"extra": "forbid"})
 class ErrorReport:
     error_type: str
     message: str
     error_category: str | None = None
+    error_domain: str | None = None          # input / config / runtime
     retryable: bool | None = None
-    user_action: str | None = None
+    user_action: UserAction | None = None    # typed: kind + detail
     model: str | None = None
     provider: str | None = None
+    provider_metadata: ProviderErrorMetadata | None = None

The most important behaviour change is on PipelexError itself. A wrapper exception (PipeRunError → PipeRouterError → PipelineExecutionError) carries no category of its own. to_error_report() now enriches the report from the __cause__ chain, so the inference classification survives every wrapping layer:

+def _enrich_error_report_from_cause(self, report: ErrorReport) -> ErrorReport:
+    """Fill the None classification fields of `report` from the __cause__ chain."""
+    cause = self.__cause__
+    if not isinstance(cause, PipelexError):
+        return report
+    # ... cyclic-chain guard omitted ...
+    cause_report = cause.to_error_report()
+    return ErrorReport(
+        error_type=report.error_type,                                  # keep own identity
+        message=report.message,
+        error_category=report.error_category or cause_report.error_category,
+        error_domain=report.error_domain or cause_report.error_domain,
+        retryable=report.retryable if report.retryable is not None else cause_report.retryable,
+        user_action=report.user_action or cause_report.user_action,
+        model=report.model or cause_report.model,
+        provider=report.provider or cause_report.provider,
+        provider_metadata=report.provider_metadata or cause_report.provider_metadata,
+    )

Two enums you'll see everywhere

05Worker classification — Extract / Classify / Render

Every inference worker collapses to one three-line shape.

After the ECR (refactor/ECR) sweep, every provider worker under pipelex/plugins/*/ handles SDK exceptions with the same three steps. Extract turns the provider's SDK exception into a provider-blind ProviderErrorMetadata envelope. Classify is a single shared, provider-blind function mapping that envelope to a ClassificationResult(category, user_action_kind, is_model_not_found). Render is a single shared function per worker family that picks the right CogtError subclass from an InferenceErrorFamily tag plus the is_model_not_found flag (e.g. LLMModelNotFoundError vs LLMCompletionError).

except (APIStatusError, APIConnectionError, APITimeoutError) as exc:
+   metadata = extract_openai_metadata(exc)                                       # provider-specific
+   classification = classify_inference_error(metadata)                           # shared, provider-blind
+   raise render_llm_error(                                                       # shared
+       classification=classification,
+       metadata=metadata,
+       family=InferenceErrorFamily.LLM,
+       model_desc=self.inference_model.desc,
+   ) from exc

Only the per-provider extract_*_metadata functions stay plugin-local — one per SDK family, registered against ProviderName. Classify and Render live once, in pipelex/cogt/inference/error_classify.py and error_render.py. A parity meta-test (test_provider_classification_parity.py) walks every ProviderName against the extract-fn registry and the worker-family map, so adding a new provider without wiring it fails fast.

This sweep lifted every worker kind — LLM, img-gen, extract, search, plus AWS Bedrock — to the same standard. The workers that previously had zero SDK handling (Google, Mistral, Azure img-gen, FAL, HuggingFace, Docling, Linkup, pypdfium2) now all classify. Refinements that landed alongside:

• Per-provider concern confined to Extract. Quota-vs-rate-limit, content-policy-vs-bad-request, model-not-found-vs-other-404 discriminators now live as @property accessors on ProviderErrorMetadata (is_quota_exhaustion, is_content_policy_violation, is_network_error) or as a flag the Extract fn sets. Classify reads them; the worker no longer branches on them.
• The instructor unwrap. On structured-generation paths, extract_underlying_sdk_exception() recovers the real SDK exception out of InstructorRetryException, so it routes through the same Extract → Classify → Render chain as the plain-text path.
• UNKNOWN instead of mis-labelling. An unrecognized inner exception (e.g. a pydantic.ValidationError from a schema mismatch) now lands in UNKNOWN rather than being mis-categorized as a real CONTENT-policy violation.
• ProviderErrorMetadata. Every raised inference error carries status_code, request_id, retry_after_seconds, provider_error_code, and the human message. The raw response body is held in-process but excluded from serialization — it can carry credentials.

06Retry & resilience

An honest model — and removing the loop that pretended otherwise.

The headline change: the PipeRouter transient-retry loop was removed. It only ever ran on the direct (non-Temporal) path, re-ran at the wrong granularity, and carried a per-run/global config bug. Rather than fix a retry loop for a path that is deliberately not the resilient one, it was deleted (PR #909).

Supporting changes that landed under this track:

• Tier 1 made explicit. cogt.transport_max_retries (default 2) is wired into every inference SDK client factory. The two families that defaulted to no transport retry (Mistral, Google) are brought up to the floor. The SDK-less azure_rest img-gen path gets a tenacity-based wrapper (transport_retry.py) that narrows retries for non-idempotent POSTs.
• instructor confined to schema re-ask. Its retry predicate now matches only validation failures — a transport error propagates raw instead of being wrapped, so Tier 1 stays the sole transport-retry layer.
• Bounded fan-out. PipeBatch uses gather_bounded() (max_concurrency, default 8) — admission control so a big batch doesn't trigger a self-inflicted rate-limit storm. This is not retry; it stays.

07The Temporal boundary

Keeping the classification intact across process boundaries.

When a pipe runs on a Temporal worker, the error has to survive serialization across the activity → workflow → submitter boundary. Temporal's default failure converter would auto-wrap a raw PipelexError — losing both our structured ErrorReport and the category-aware retry decision. The bridge closes that gap.

+@functools.wraps(func)
+async def wrapper(*args, **kwargs):
+    try:
+        return await func(*args, **kwargs)
+    except PipelexError as exc:                # ONLY PipelexError — never bare Exception
+        raise TemporalError.from_message_exception(exc=exc) from exc

Applied beneath @activity.defn on every in-scope activity. from_message_exception() derives non_retryable from InferenceErrorCategory.is_retryable (the class-name list is now only a fallback for category-less errors) and packs to_error_report().to_dict() into ApplicationError.details. On the way back, recover_error_report() walks the __cause__ chain, pulls the dict, and rebuilds the ErrorReport — tolerating version skew so the error path never crashes on a rolling deploy.

08CLI & HTTP delivery

Where the classified error finally gets rendered.

The agent JSON error payload is the structured shape downstream agents parse:

{
  "error": true,
  "error_type": "LLMCompletionError",
  "message": "...",
  "hint": "...",
  "retryable": true,
  "error_domain": "runtime",
  "error_category": "transient",
  "model": "gpt-4o",
  "provider": "openai",
  "error_source": ["LLMCompletionError @ .../worker.py:152 (in _gen_text)"]
}

HTTP-status mapping lives in the library. pipelex has no API server, but pipelex-relay and pipelex-back-office both render an ErrorReport as an HTTP response — and were reinventing the mapping. It now lives once in base_exceptions.py: error_domain_to_http_status() is the pure table, ErrorReport.http_status layers the provider-429 passthrough on top. Downstream FastAPI handlers become a trivial adapter and must not redefine the contract.

09Testing

What proves the system behaves — and what stops it rotting.

10What's still open

Deliberately scoped out — flagged here so review doesn't treat it as missing.

The bulk of the work has landed. One item remains, intentionally left for after this branch:

Everything else — the metadata model, worker classification, retry & resilience, CLI delivery, the Temporal bridge, and the test coverage — is landed and described in current-state terms in the track docs.