# PRD — ToolKit Distributed Gears

<!--
=============================================================================
PRODUCT REQUIREMENTS DOCUMENT (PRD)
=============================================================================
PURPOSE: Define WHAT the system must do and WHY — business requirements,
functional capabilities, and quality attributes.

STANDARDS ALIGNMENT:

- IEEE 830 / ISO/IEC/IEEE 29148:2018 (requirements specification)
- IEEE 1233 (system requirements)
- ISO/IEC 15288 / 12207 (requirements definition)
  =============================================================================
  -->

## Table of Contents

<!-- generated by `cypilot toc` -->

## 1. Overview

### 1.1 Purpose

This PRD defines how ToolKit gears run as separate OS processes (on-premise) or Kubernetes pods while maintaining full
developer transparency — the same gear code, the same ClientHub resolution, and the same OperationBuilder usage
regardless of the deployment target. The Platform Host process is managed by the **Flight Control** system gear, which
orchestrates gear lifecycle, discovery, and gateway registration.

### 1.2 Background / Problem Statement

Today all ToolKit gears run in a single process managed by `HostRuntime`. While this "embedded" model is simple and
performant, it creates hard limits: a single gear crash can take down the entire platform, horizontal scaling requires
replicating the full process, and language-locked gears cannot participate.

The existing OoP bootstrap (`libs/toolkit/src/bootstrap/oop.rs`) provides a starting point — it connects to
`DirectoryService`, sends heartbeats, and runs the gear lifecycle. However, several critical capabilities are missing:
OoP gears have no HTTP server of their own, `DirectoryService` has no REST endpoint resolution, `api-gateway` cannot
reverse-proxy to remote gears, and there is no mechanism to auto-generate typed REST clients for cross-process calls.
SecurityContext propagation also breaks across process boundaries because the `bearer_token` field is explicitly skipped
during serialization.

Industry platforms (Dapr, Spring Cloud, Quarkus, Akka) solve these problems through a combination of service discovery,
sidecar or client-based communication, and gateway integration — all hidden behind framework abstractions so that
application code remains deployment-agnostic. ToolKit Distributed Gears bring the same capability to the platform.

### 1.3 Goals (Business Outcomes)

- **G1**: Gear developers write code once and deploy it in-process or out-of-process without source changes.
- **G2**: Platform operators select a named deployment profile and get a tested, documented topology — no combinatorial
  explosion of configuration options.
- **G3**: OoP gear REST calls add less than 5 ms overhead at p95 on localhost (10 ms within a k8s cluster), making
  out-of-process deployment viable for latency-sensitive workloads.
- **G4**: The framework generates typed REST clients from annotated Rust traits (`#[toolkit::rest_contract]`),
  eliminating hand-written HTTP boilerplate and reducing integration bugs.

### 1.4 Glossary

| Term                            | Definition                                                                                                                                                                                                                                                                                                                                                           |
|---------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Gear                          | The minimal unit of independent deployment in ToolKit. In Profile 1 (Embedded), gears are composed into a single process. In Profile 3 (K8s Native), each gear is an independently deployable microservice (pod). A gear has its own lifecycle, configuration, REST API (via OperationBuilder), and DB schema.                                                  |
| Plugin                          | A gear that implements a specific plugin trait (e.g., `AuthNResolverPluginClient`) for a host gear. Plugins have no public REST API — they expose a private trait interface. Discovered via types-registry (GTS pattern), resolved via ClientHub scoped by GTS instance ID. Can be internal (in the host gear crate) or a separate gear (in-process or OoP). |
| Plugin Host Gear              | A gear that defines a plugin trait in its SDK and discovers/consumes plugin implementations at runtime (e.g., authn-resolver, authz-resolver, mini-chat).                                                                                                                                                                                                          |
| Flight Control                  | The Platform Host system gear responsible for OoP gear orchestration, lifecycle management, discovery coordination, and gateway registration. Replaces the informal term "master gear".                                                                                                                                                                        |
| Platform Host                   | The main process running the Flight Control gear along with other system gears (grpc-hub, optional api-gateway, types-registry). In K8s Native profile, this is just another pod providing infrastructure services — not a central authority.                                                                                                                    |
| OoP Worker                      | A separate OS process or k8s pod running one or more application gears.                                                                                                                                                                                                                                                                                            |
| System Gear                   | A gear providing platform infrastructure (orchestrator, gateway, auth, types-registry).                                                                                                                                                                                                                                                                            |
| Application Gear              | A business gear written by framework users.                                                                                                                                                                                                                                                                                                                        |
| Deployment Profile              | A named, tested configuration of how gears are deployed (Embedded, Host+Workers, K8s Native).                                                                                                                                                                                                                                                                      |
| ClientHub                       | A type-safe registry for inter-gear client resolution. Supports unscoped resolution (one impl per trait) and scoped resolution (keyed by GTS instance ID for plugins). The backing implementation may be in-process or a generated remote client.                                                                                                                  |
| OperationBuilder                | A ToolKit DSL for declaring REST routes, request/response schemas, and OpenAPI metadata.                                                                                                                                                                                                                                                                              |
| DirectoryService                | A gRPC-based service discovery registry. Gears register instances; consumers resolve endpoints.                                                                                                                                                                                                                                                                    |
| SecurityContext                 | An internal representation of the authenticated user, including roles, permissions, and the original bearer token (`SecretString`, `#[serde(skip)]`).                                                                                                                                                                                                                |
| GatewayProvider                 | An abstraction trait for registering and deregistering gear routes with a gateway (built-in or external).                                                                                                                                                                                                                                                          |
| Identity Broker / Login Service | A future platform subsystem (separate gear) responsible for login orchestration (OAuth2/OIDC flows, social login), identity binding (external ID → `subject_id`), tenant resolution, and platform token issuance. Distinct from authn-resolver (token validation) and api-gateway (HTTP routing).                                                                  |
| Embedded Edge (Mode A)          | Deployment where the built-in api-gateway, identity broker, and authn/authz resolvers handle all external traffic. Self-contained on-prem baseline.                                                                                                                                                                                                                  |
| External Edge (Mode B)          | Deployment where an external gateway (Tyk/Kong/Envoy) sits in front of the platform. Sub-modes: B1 (pass-through platform token) and B2 (token exchange via identity broker).                                                                                                                                                                                        |
| Token Exchange                  | A pattern (Mode B2) where an external gateway authenticates a user via its own IdP, then calls a platform endpoint to convert the external credential into a platform-internal token.                                                                                                                                                                                |

## 2. Actors

### 2.1 Human Actors

#### Gear Developer

**ID**: `cpt-cf-actor-gear-dev`

- **Role**: Writes application gears using ToolKit. Expects to write the same code for in-process and OoP targets.
- **Needs**: Transparent ClientHub resolution, automatic REST client generation, zero boilerplate for SecurityContext
  propagation.

#### Platform Operator

**ID**: `cpt-cf-actor-platform-op`

- **Role**: Deploys and operates the platform. Chooses a deployment profile, configures networking, monitors health.
- **Needs**: Named deployment profiles with clear documentation, straightforward configuration, observable gear
  health.

### 2.2 System Actors

#### Platform Host

**ID**: `cpt-cf-actor-platform-host`

- **Role**: The main orchestrating process. Runs system gears, spawns OoP workers (on-premise), provides
  DirectoryService, and optionally runs the built-in api-gateway.

#### External Gateway

**ID**: `cpt-cf-actor-external-gw`

- **Role**: An external API gateway (Kong, Tyk, Envoy) used in k8s deployments. Receives route registrations from the
  Flight Control gear and handles public API ingress, TLS termination, and auth.

#### OoP Worker

**ID**: `cpt-cf-actor-oop-worker`

- **Role**: A separate process or k8s pod running one or more application gears. Starts its own HTTP server, registers
  with DirectoryService, and serves REST and optionally gRPC APIs.

## 3. Operational Concept & Environment

### 3.1 Gear-Specific Environment Constraints

- OoP Workers MUST run on Linux, macOS, or Windows. K8s Native profile requires Linux containers.
- On-premise single-host communication uses UDS (Unix) or named pipes (Windows). Multi-host communication uses TCP and
  requires mTLS (P2).
- K8s Native profile requires Kubernetes 1.26+ with DNS-based service discovery.
- The built-in api-gateway is Axum-based and runs inside the Platform Host process. External gateways are configured via
  the GatewayProvider abstraction.

## 4. Scope

### 4.1 In Scope

**Priority 1:**

- Deployment Profile 1: Embedded (current behavior, no changes needed)
- Deployment Profile 3: K8s Native (each gear = pod, external gateway, k8s DNS discovery)
- Two-plane auth — gateway validates the JWT at the edge; OoP gears **re-validate it per hop** (tenant plane) and use a tenant-less identity for infra calls (platform plane). See [ADR-0008](ADR/0008-cpt-cf-adr-two-plane-auth.md)
- Direct gear-to-gear REST communication (not routed through gateway)
- Public vs. internal API distinction in route registration
- REST client generation from annotated Rust traits (trait-first, `#[toolkit::rest_contract]` proc-macro)
- Gateway abstraction trait with built-in ToolKit implementation
- SecurityContext propagation (tenant plane): `Authorization: Bearer <jwt>` forwarded and **re-validated at every hop**
  via AuthN Resolver; `x-secctx-bin` is not used over HTTP (see [ADR-0008](ADR/0008-cpt-cf-adr-two-plane-auth.md))
- Platform-plane authentication: tenant-less identity for infra calls (Directory/GTS registration, heartbeats) — K8s
  ServiceAccount tokens in the first phase; mTLS+SPIFFE (cert-manager) next phase
- K8s packaging: Helm library chart (`toolkit-common`), per-gear charts, umbrella chart (`toolkit-platform`) with preset
  values
- Edge architecture (Mode A): Embedded Edge with built-in api-gateway + identity broker baseline
- api-gateway scope constraint: must remain minimal HTTP edge (no login orchestration, no token issuance)

**Priority 2:**

- Deployment Profile 2 (Host + Workers): single-node on-premise (OoP Workers over UDS/pipes, bootstrap-token platform
  auth) and multi-node on-premise (mTLS). Deferred from P1 — the platform-plane `BootstrapToken` / `MtlsIdentity`
  credential variants are defined for forward-compatibility but are not validated/wired in P1 (which targets Profile 1 +
  Profile 3)
- Multi-gear OoP Workers (bundled binaries running several gears in one process)
- OoP plugins: plugin gears running out-of-process with transport-transparent ClientHub resolution (Path C — hybrid
  transport abstraction behind `ClientHub` scoped)
- `#[toolkit::plugin]` proc-macro for plugin gear definition, generating both server (trait → REST/gRPC endpoints) and
  client (remote trait impl) from the plugin trait
- Circuit breaker and advanced resilience patterns in generated REST clients
- Edge architecture (Mode B): External Edge with Tyk/Kong/external IdP integration, identity broker gear, token
  exchange endpoint

### 4.2 Out of Scope

- gRPC-only gears (existing pattern, unchanged)
- Streaming / SSE over OoP boundaries (future work)
- Non-Rust OoP gears (architecture MUST NOT block this, but no implementation now)
- Sidecar-based communication (Dapr-style)
- Custom load-balancing strategies beyond round-robin

## 5. Functional Requirements

### 5.1 Developer Transparency

#### Gear Code Portability

- [ ] `p1` - **ID**: `cpt-cf-fr-developer-transparency`

Gear code MUST work identically in-process and out-of-process without source changes. The same `impl Gear`,
`register_rest()`, and `register_clients()` calls MUST produce correct behavior in all deployment profiles.

- **Rationale**: Eliminates the cost of maintaining separate in-proc and OoP code paths. Gear developers focus on
  business logic, not deployment topology.
- **Actors**: `cpt-cf-actor-gear-dev`

#### Client Resolution Transparency

- [ ] `p1` - **ID**: `cpt-cf-fr-client-transparency`

Consumers MUST resolve gear clients via ClientHub without knowing whether the target gear is in-process or remote.
The ClientHub MUST return an in-process implementation when available and a REST client when the target is remote.

- **Rationale**: Decouples callers from deployment topology. A gear can be moved from embedded to OoP without changing
  any caller code.
- **Actors**: `cpt-cf-actor-gear-dev`

### 5.2 Communication

#### REST as Primary OoP Protocol

- [ ] `p1` - **ID**: `cpt-cf-fr-rest-primary`

OoP gears MUST expose REST as their primary public API surface. Each OoP Worker MUST run its own HTTP server using the
ToolKit middleware stack (same OperationBuilder routes, same error format, same OpenAPI generation).

- **Rationale**: REST is the universal integration protocol. It is debuggable (curl), tooling-rich (Postman, OpenAPI),
  and does not require proto compilation.
- **Actors**: `cpt-cf-actor-gear-dev`, `cpt-cf-actor-oop-worker`

#### Direct Inter-Gear Communication

- [ ] `p1` - **ID**: `cpt-cf-fr-direct-communication`

Inter-gear calls MUST go directly from caller to target (resolved via DirectoryService), not through the API gateway.
The gateway is for external traffic only.

- **Rationale**: Avoids an extra network hop and the gateway becoming a bottleneck for internal traffic. Aligns with
  industry patterns (Istio service mesh, Spring Cloud direct calls).
- **Actors**: `cpt-cf-actor-oop-worker`, `cpt-cf-actor-platform-host`

#### SecurityContext Propagation

- [ ] `p1` - **ID**: `cpt-cf-fr-secctx-propagation`

The tenant `SecurityContext` MUST be propagated across OoP boundaries by forwarding the original
`Authorization: Bearer <jwt>` header **as-is**. Each receiving OoP gear MUST **re-validate** the token via
`AuthNResolverClient::authenticate()` and reconstruct `SecurityContext` from the validated claims. `x-secctx-bin` MUST
NOT be used over HTTP. `encode_bin` / `decode_bin` remain only for in-process gRPC metadata (Profile 1).

- **Rationale**: The JWT is self-authenticating — the IdP signature is the trust root, so tenant integrity does not
  depend on transport trust being correctly deployed at every boundary. Re-validation is ~0.5 ms with a cached JWKS,
  well within the latency budget, and gives defense in depth: a compromised peer cannot forward a forged token
  undetected.
- **Actors**: `cpt-cf-actor-oop-worker`, `cpt-cf-actor-platform-host`

### 5.3 Gateway & Discovery

#### Gateway Route Registration

- [ ] `p1` - **ID**: `cpt-cf-fr-gateway-registration`

Gears with public API MUST register their routes in the gateway. Route registration MUST go through the
GatewayProvider abstraction, not directly to api-gateway internals.

- **Rationale**: In k8s, the gateway is external (Kong/Tyk). On-premise, it is the built-in api-gateway. A single
  registration path covers both.
- **Actors**: `cpt-cf-actor-oop-worker`, `cpt-cf-actor-external-gw`

#### Gateway Abstraction

- [ ] `p1` - **ID**: `cpt-cf-fr-gateway-abstraction`

Gateway integration MUST be abstracted behind a `GatewayProvider` trait supporting
`register_routes(gear, openapi, endpoint)` and `deregister_routes(gear)`. The framework MUST provide a built-in
implementation (`ToolKitGatewayProvider`) that adds reverse-proxy routes to the api-gateway gear.

- **Rationale**: Prevents vendor lock-in to the built-in gateway. Enables k8s deployments with external gateways without
  framework changes.
- **Actors**: `cpt-cf-actor-platform-host`, `cpt-cf-actor-external-gw`

#### Kubernetes Native Deployment

- [ ] `p1` - **ID**: `cpt-cf-fr-k8s-native`

Gears MUST be deployable as independent Kubernetes pods with k8s-native discovery (DNS:
`{gear}.{namespace}.svc.cluster.local`). DirectoryService MAY be used alongside k8s DNS but MUST NOT be required for
basic service resolution in k8s.

- **Rationale**: K8s provides built-in service discovery and load balancing. Requiring a custom DirectoryService adds
  unnecessary complexity in k8s environments.
- **Actors**: `cpt-cf-actor-platform-op`, `cpt-cf-actor-oop-worker`

### 5.4 Client Generation

#### REST Client Generation

- [ ] `p1` - **ID**: `cpt-cf-fr-rest-client-gen`

The framework MUST provide a mechanism to generate typed REST clients from an annotated Rust trait
(`#[toolkit::rest_contract]`). Generated clients MUST implement the gear's SDK trait, use `toolkit-http` for transport,
and propagate SecurityContext automatically. `openapi.json` is a published *output* for non-Rust consumers, not a
codegen input.

- **Rationale**: Hand-written HTTP clients are error-prone, tedious, and diverge from the spec over time.
  Trait-first codegen ensures type-safe, always-in-sync clients with a single Rust trait as the source of truth.
- **Actors**: `cpt-cf-actor-gear-dev`

### 5.5 Lifecycle

#### Eventual Readiness and Self-Registration

- [ ] `p1` - **ID**: `cpt-cf-fr-eventual-readiness`

The ToolKit OoP runtime MUST manage the full gear startup lifecycle without gear developer involvement:

1. Start the HTTP server immediately and serve liveness probe (`/healthz` → 200).
2. In the background, retry registration with Flight Control (DirectoryService) using exponential backoff until
   successful. Re-register on connection loss.
3. In the background, resolve all dependencies declared in the gear's `deps` via DirectoryService. Wire REST clients
   into ClientHub as dependencies become available.
4. Serve readiness probe (`/readyz`) returning 503 with unresolved dependency list until all critical `deps` are
   resolved; then 200.
5. Gear developers MUST NOT write retry loops, health-check polling, or registration code. The `deps` declaration in
   `#[toolkit::gear(deps = [...])]` is the only input required.

Gears with no `deps` (e.g., Flight Control, types-registry) become ready immediately after `start()`.

- **Rationale**: Kubernetes-native — no init containers, no deployment ordering, no single point of failure. Gears
  start independently and converge to ready state. Readiness probes gate traffic naturally. The same `deps` metadata
  already used for topo-sort in Profile 1 drives readiness gating in Profiles 2 and 3.
- **Actors**: `cpt-cf-actor-oop-worker`

#### Internal Gear Authentication

- [ ] `p1` - **ID**: `cpt-cf-fr-internal-auth`

System-level communication between gears (registration, heartbeats, background inter-gear calls without user
context) MUST be authenticated. The ToolKit runtime MUST provide this authentication transparently to gear
developers.

These calls use the **platform plane**: a dedicated **`PlatformSecurityContext`** — a type **distinct** from the tenant
`SecurityContext`, **not** a tenant context with a nil/sentinel tenant id, per
[ADR-0008](ADR/0008-cpt-cf-adr-two-plane-auth.md). It is first-class from the first phase and MUST NOT be passed to the
tenant `PolicyEnforcer`. Reusing the request-scoped tenant context for platform calls is the seam where confused-deputy
/ cross-tenant-leak regressions begin, so the type makes "no tenant" unrepresentable-as-a-tenant.

The plane is chosen by **tenant-scoped vs non-tenant-scoped**, not user-vs-system. System/root actors performing a
*tenant-scoped* operation (e.g. posting an event for a tenant) stay on the **tenant plane** with a real tenant
`SecurityContext` (from a user JWT or S2S client-credentials) and are authorized normally — the root tenant uses the
identical AuthZ flow as any tenant. Only genuinely **non-tenant** operations (e.g. registering a *global* GTS type) use
`PlatformSecurityContext`. Adding a new platform AuthN method (mTLS+SPIFFE, future methods) MUST NOT change the structure
of `PlatformSecurityContext` — it is only another way to populate it, dispatched entirely inside `InternalAuthMiddleware`.

- **First phase (K8s)**: projected K8s ServiceAccount tokens (audience: `toolkit-internal`), validated via the
  TokenReview API. Builds the `PlatformSecurityContext` from the validated SA identity and sets
  `PeerAuthenticated { gear }` for workload-policy decisions only.
- **Next phase**: mTLS + SPIFFE issued by cert-manager (embedded CA / boot-token on-prem), populating the same
  `PlatformSecurityContext` with the SPIFFE variant of the tenant-less, method-agnostic `PlatformIdentity` enum
  (`ServiceAccount` from phase 1, `Spiffe { trust_domain, gear, version }` added here from the X.509 SAN
  `spiffe://<trust_domain>/gear/<gear>/<version>`; `#[non_exhaustive]` for future methods).
- **Profile 2 (single-node)**: ephemeral bootstrap token, or plain UDS when all gears share one uid.

Gear developers MUST NOT implement internal auth logic. The `InternalCredential` abstraction (`attach_internal_auth`
/ `validate_internal_auth`) is managed entirely by the ToolKit runtime; the SA token and the mTLS identity are two
backends of it, so swapping does not change gear code.

- **Rationale**: Without system-level auth, any process that can reach DirectoryService can register as a gear or
  deregister others. Tenant-plane auth (re-validated JWT) does not cover system calls, which carry no user. SA tokens
  are K8s-native and ship without PKI; mTLS+SPIFFE hardens the same plane later without touching gear code.
- **Actors**: `cpt-cf-actor-oop-worker`, `cpt-cf-actor-platform-host`

#### OoP Worker Registration and Deregistration

- [ ] `p1` - **ID**: `cpt-cf-fr-oop-lifecycle`

OoP Workers MUST register their REST endpoint and OpenAPI spec with DirectoryService on startup and deregister on
graceful shutdown.

Active heartbeat-timeout reaping of unresponsive workers by the Platform Host is **Profile 2 (Host + Workers)** and is
**deferred to P2**; in **Profile 3 (K8s)** k8s readiness probes + the Endpoints controller handle dead-pod removal.

- **Rationale**: Stale service entries cause routing failures. Clean lifecycle management ensures the directory reflects
  the actual cluster state.
- **Actors**: `cpt-cf-actor-oop-worker`, `cpt-cf-actor-platform-host`

#### Public vs. Internal API Distinction

- [ ] `p1` - **ID**: `cpt-cf-fr-api-visibility`

Gears MUST be able to mark API endpoints as public (registered in gateway for external access) or internal (available
only for inter-gear communication via DirectoryService). The default MUST be internal. The existing
`OperationBuilder.public()` / `OperationBuilder.authenticated()` mechanism and the resulting `OperationSpec.is_public`
field MUST be used to drive this distinction.

- **Rationale**: Not all gear APIs should be exposed to external consumers. Separating public and internal APIs
  reduces attack surface and keeps the gateway configuration clean. The `OperationBuilder` already captures this intent
  via `is_public`.
- **Actors**: `cpt-cf-actor-gear-dev`, `cpt-cf-actor-platform-op`

### 5.6 Plugin Transparency (P2)

#### OoP Plugin Transport Abstraction

- [ ] `p2` - **ID**: `cpt-cf-fr-plugin-transparency`

Plugin host gears MUST resolve plugin implementations via `ClientHub` scoped (
`try_get_scoped::<dyn PluginTrait>(gts_id)`) without knowing whether the plugin is in-process or remote. When a plugin
gear runs OoP, the framework MUST provide a transport-generated client that implements the plugin trait and routes
calls over REST or gRPC to the remote plugin. The existing GTS-based discovery (types-registry + `BaseToolkitPluginV1` +
`GtsPluginSelector`) MUST continue to work unchanged.

- **Rationale**: Plugin host gears (authn-resolver, authz-resolver, mini-chat) should not contain transport-specific
  code. The same `choose_plugin_instance` + `try_get_scoped` pattern must work whether the plugin is an internal gear
  or a remote OoP process.
- **Actors**: `cpt-cf-actor-gear-dev`

#### Plugin Macro

- [ ] `p2` - **ID**: `cpt-cf-fr-plugin-macro`

The framework MUST provide a `#[toolkit::plugin]` proc-macro (or equivalent codegen) that, given a plugin trait
definition, generates: (a) server-side endpoints exposing the trait methods over the chosen transport (REST or gRPC), (
b) a client-side implementation of the trait that calls those endpoints via `toolkit-http` or tonic, and (c) the
`register_scoped` wiring for ClientHub. This eliminates hand-written boilerplate for OoP plugin authors.

- **Rationale**: Without codegen, every OoP plugin would require manually writing REST endpoints, a client
  implementation, and the ClientHub registration — tripling the effort compared to in-process plugins.
- **Actors**: `cpt-cf-actor-gear-dev`

### 5.7 Edge Architecture

#### Edge Modes

- [ ] `p1` (Mode A), `p2` (Mode B) - **ID**: `cpt-cf-fr-edge-modes`

The platform MUST support two named edge modes without requiring changes to internal gear code:

- **Mode A (Embedded Edge)**: self-contained deployment with the built-in api-gateway, authn-resolver, authz-resolver,
  and identity broker handling all external traffic. This is the on-prem baseline.
- **Mode B (External Edge)**: an external gateway (Tyk, Kong, Envoy, k8s Ingress) sits in front of the platform. Two
  sub-modes:
    - **B1 (pass-through)**: external gateway forwards platform-issued tokens as-is.
    - **B2 (token exchange)**: external gateway authenticates via its own IdP, then exchanges the external credential
      for a platform token via a dedicated exchange endpoint.

In both modes, downstream gears receive the same `SecurityContext`. The edge mode is transparent to application
gears.

- **Rationale**: On-prem customers need a self-contained stack. Enterprise customers have existing gateway and IdP
  infrastructure that the platform must integrate with rather than replace. Mandating a single edge topology would force
  one group to compromise.
- **Actors**: `cpt-cf-actor-platform-op`, `cpt-cf-actor-platform-user`

#### Gateway Minimal Constraint

- [ ] `p1` - **ID**: `cpt-cf-fr-gateway-minimal`

The built-in api-gateway MUST remain a minimal HTTP edge component. Its scope is limited to: HTTP routing (
OperationBuilder-registered routes), CORS, basic rate limiting, auth middleware delegation (Bearer extraction →
authn-resolver), and SecurityContext propagation. The api-gateway MUST NOT implement: login orchestration (OAuth2 flows,
social login callbacks), token issuance or signing, social provider integration, account linking or user provisioning,
API analytics or usage metering, or advanced traffic management (circuit breaker, canary, A/B routing).

- **Rationale**: The built-in gateway is maintained by a small team. Turning it into a Tyk/Kong competitor is neither
  feasible nor desirable. Login orchestration is fundamentally different from HTTP routing and must live in a dedicated
  subsystem.
- **Actors**: `cpt-cf-actor-gear-dev`, `cpt-cf-actor-platform-op`

#### Identity Broker / Login Service

- [ ] `p2` - **ID**: `cpt-cf-fr-identity-broker`

The platform MUST define a separate identity broker / login service subsystem (future gear) responsible for: login
orchestration (OAuth2/OIDC flows, social login via GitHub/Google/etc., local username/password), identity binding (
mapping external identity → platform `subject_id`), tenant resolution (`subject_id` → `tenant_id`, membership
management), and platform token issuance. The identity broker is the only component that creates platform tokens —
api-gateway and authn-resolver do not issue tokens. The identity broker MUST support multiple backends: built-in social
login, Keycloak-backed login, and external broker exchange (for Mode B integration).

- **Rationale**: Login orchestration (redirects, callbacks, state management, account linking) does not belong in a
  stateless token validation service (authn-resolver) or in a routing layer (api-gateway). A dedicated subsystem follows
  single-responsibility and enables independent iteration on login features.
- **Actors**: `cpt-cf-actor-platform-user`, `cpt-cf-actor-platform-op`

#### Token Exchange Endpoint

- [ ] `p2` - **ID**: `cpt-cf-fr-token-exchange`

The platform MUST expose a token exchange endpoint (via the identity broker) that allows external gateways and IdPs to
convert their externally-authenticated credential into a platform-internal token. This enables Mode B2 (External Edge
with token exchange): the external gateway authenticates the user through its own provider, then calls the exchange
endpoint to obtain a platform token that authn-resolver can validate. The exchange endpoint MUST validate the external
credential, perform identity binding, and return a platform token. The endpoint contract MUST be vendor-independent (not
Tyk-specific or Kong-specific).

- **Rationale**: Without a token exchange mechanism, Mode B2 would require authn-resolver to understand vendor-specific
  token formats, coupling the core auth pipeline to external providers. Token exchange isolates vendor specifics in the
  identity broker.
- **Actors**: `cpt-cf-actor-platform-op`

## 6. Non-Functional Requirements

### 6.1 Gear-Specific NFRs

#### OoP Call Latency

- [ ] `p1` - **ID**: `cpt-cf-nfr-oop-latency`

OoP REST call overhead (framework + network, excluding handler execution time) MUST be under 5 ms at p95 on localhost
and under 10 ms at p95 within a k8s cluster.

- **Threshold**: p95 < 5 ms (localhost), p95 < 10 ms (k8s intra-cluster)
- **Rationale**: OoP deployment must not introduce perceptible latency for synchronous request paths. Exceeding this
  threshold would make OoP impractical for interactive APIs.
- **Verification Method**: Automated benchmark with 1000 sequential requests, measuring framework overhead only (echo
  handler).
- **Architecture Allocation**: See DESIGN.md § NFR Allocation

#### Per-Hop JWT Re-validation

- [ ] `p1` - **ID**: `cpt-cf-nfr-per-hop-revalidation`

The JWT MUST be re-validated at every hop via `AuthNResolverClient::authenticate()` (~0.5 ms with cached JWKS). The
verification is a local crypto operation on cache hit, not a network call. There is a single trust root (the IdP
signature), re-checked at each boundary — tenant integrity does not depend on transport trust. There is no fast path
that skips signature verification; `PeerAuthenticated` (platform plane) never substitutes for tenant-plane validation.

- **Threshold**: ≤ ~0.5 ms per hop with warm JWKS cache; ≤ 1 ms for a typical 2-hop chain (budget: 5 ms p95).
- **Rationale**: The JWT is self-authenticating, so re-validating is cheap and removes the dependency on mTLS being
  correctly deployed everywhere before an unsigned envelope can be trusted. Defense in depth: a compromised peer cannot
  forward a forged token undetected.
- **Verification Method**: Latency benchmark of per-hop re-validation; security test asserting a forged-signature token
  is rejected even when `PeerAuthenticated` is set.
- **Architecture Allocation**: See DESIGN.md § NFR Allocation

#### Graceful Degradation

- [ ] `p1` - **ID**: `cpt-cf-nfr-graceful-degradation`

When an OoP dependency is unavailable, the caller MUST receive an HTTP 503 (Service Unavailable) response with a Problem
Details body. The caller process MUST NOT crash.

- **Threshold**: 100% of calls to an unavailable OoP gear return 503 within the configured timeout.
- **Rationale**: Cascading failures across OoP boundaries would negate the isolation benefits of out-of-process
  deployment.
- **Verification Method**: Integration test: stop a target OoP gear, verify caller returns 503.
- **Architecture Allocation**: See DESIGN.md § NFR Allocation

## 7. Public Library Interfaces

### 7.1 Public API Surface

#### GatewayProvider Trait

- [ ] `p1` - **ID**: `cpt-cf-interface-gateway-provider`

- **Type**: Rust trait
- **Stability**: unstable (will stabilize after Profile 2 and Profile 3 are validated)
- **Description**: Abstraction for registering/deregistering gear routes with a gateway (built-in or external).
- **Breaking Change Policy**: Minor version bump during unstable phase; major version bump after stabilization.

#### REST Client Codegen

- [ ] `p1` - **ID**: `cpt-cf-interface-rest-codegen`

- **Type**: Proc-macro code generation (`#[toolkit::rest_contract]`)
- **Stability**: unstable
- **Description**: Generates typed `RestXxxClient` structs implementing gear SDK traits from annotated Rust traits.
  `openapi.json` is a published output, not a codegen input.
- **Breaking Change Policy**: Generated code API may change during unstable phase.

#### SecurityContext HTTP Propagation

- [ ] `p1` - **ID**: `cpt-cf-interface-secctx-http`

- **Type**: HTTP header contract
- **Stability**: stable
- **Description**: Single-header propagation: `Authorization: Bearer <jwt>`, re-validated at each hop via AuthN
  Resolver (see [ADR-0008](ADR/0008-cpt-cf-adr-two-plane-auth.md)). `x-secctx-bin` is not used over HTTP.
- **Breaking Change Policy**: Header name is stable. Changes require a new ADR.

### 7.2 External Integration Contracts

#### Kubernetes Service Discovery

- [ ] `p2` - **ID**: `cpt-cf-contract-k8s-discovery`

- **Direction**: required from environment
- **Protocol/Format**: Kubernetes DNS (`{service}.{namespace}.svc.cluster.local`)
- **Compatibility**: Standard Kubernetes API; no custom CRDs required for basic discovery.

#### External Gateway Admin API

- [ ] `p2` - **ID**: `cpt-cf-contract-external-gw-admin`

- **Direction**: required from external gateway
- **Protocol/Format**: Gateway-specific admin API (Kong Admin API, Tyk Gateway API) or CRD-based configuration
- **Compatibility**: Adapter per gateway vendor behind GatewayProvider trait.

## 8. Use Cases

#### Deploy Gear as OoP Worker (On-Premise)

- [ ] `p2` - **ID**: `cpt-cf-usecase-deploy-oop-onprem`

**Actor**: `cpt-cf-actor-platform-op`

**Preconditions**:

- Platform Host is running with DirectoryService and api-gateway.
- Gear binary is available on the same host.

**Main Flow**:

1. Platform operator configures the gear in the deployment manifest as OoP.
2. Platform Host spawns the OoP Worker process.
3. OoP Worker starts its HTTP server and registers with DirectoryService (REST endpoint + OpenAPI spec).
4. Platform Host's api-gateway adds reverse-proxy routes for the gear's public API.
5. External clients access the gear through the gateway; other gears call it directly via ClientHub.

**Postconditions**:

- Gear is serving REST API on its own HTTP port.
- Gateway routes external traffic to the gear.
- DirectoryService holds the gear's endpoint for inter-gear resolution.

**Alternative Flows**:

- **Worker fails to start**: Platform Host logs the error and marks the gear as unhealthy. Gateway does not add
  routes.

#### Deploy Gear as K8s Pod

- [ ] `p1` - **ID**: `cpt-cf-usecase-deploy-k8s`

**Actor**: `cpt-cf-actor-platform-op`

**Preconditions**:

- Kubernetes cluster with ToolKit operator or Helm chart deployed.
- External gateway (Kong/Tyk) is configured.

**Main Flow**:

1. Platform operator applies the gear's Deployment + Service manifest.
2. K8s schedules the pod; the gear starts its HTTP server.
3. Gear registers with DirectoryService (if available) or relies on k8s DNS.
4. Gear's public routes are registered in the external gateway via GatewayProvider.
5. External clients access the gear through the gateway; other gears resolve via k8s DNS or DirectoryService.

**Postconditions**:

- Gear pod is running with a k8s Service exposing its HTTP port.
- External gateway routes public traffic to the gear's Service.

**Alternative Flows**:

- **Pod crashes**: K8s restarts the pod. GatewayProvider deregisters stale routes on health check failure.

#### Inter-Gear Call Across Process Boundary

- [ ] `p1` - **ID**: `cpt-cf-usecase-intergear-oop`

**Actor**: `cpt-cf-actor-gear-dev`

**Preconditions**:

- Gear A and Gear B are running (in any deployment profile).
- Gear A has Gear B's SDK crate with a generated REST client.

**Main Flow**:

1. Gear A calls `client_hub.get::<GearBClient>()`.
2. ClientHub returns a `RestGearBClient` (resolved via DirectoryService or k8s DNS).
3. Gear A invokes a method on the client. The generated client serializes the request, attaches SecurityContext
   headers, and sends an HTTP request via `toolkit-http`.
4. Gear B's HTTP server receives the request, reconstructs SecurityContext from headers, executes the handler, and
   returns the response.
5. The generated client deserializes the response and returns a typed result to Gear A.

**Postconditions**:

- Gear A received a typed response from Gear B.
- SecurityContext was propagated with the original bearer token intact.

**Alternative Flows**:

- **Gear B is unavailable**: `toolkit-http` returns a connection error. The generated client maps it to an HTTP 503
  Problem response.

## 9. Acceptance Criteria

- [ ] A gear compiled once runs both in-process (Profile 1) and as an OoP gear (Profile 3 / K8s) without source changes.
- [ ] ClientHub resolution returns a working REST client for an OoP gear and an in-process client for an embedded
  gear, using identical calling code.
- [ ] An external HTTP request passes through api-gateway to an OoP Worker with SecurityContext fully reconstructed (
  including bearer_token).
- [ ] A generated REST client successfully calls an OoP gear and returns a typed response.
- [ ] OoP REST call overhead is below 5 ms at p95 on localhost (benchmark with echo handler).
- [ ] Stopping an OoP Worker results in 503 responses to its callers, not a crash (`cpt-cf-nfr-graceful-degradation`).
- [ ] A gear deployed as a k8s pod is reachable via k8s DNS and has its public routes registered in the external
  gateway.

## 10. Dependencies

| Dependency                             | Description                                                              | Criticality |
|----------------------------------------|--------------------------------------------------------------------------|-------------|
| toolkit-http                            | HTTP client library used as transport for generated REST clients         | p1          |
| DirectoryService / gear-orchestrator | Service discovery for OoP Workers                                        | p1          |
| api-gateway                            | Built-in gateway for on-premise profiles; reverse-proxies to OoP Workers | p1          |
| authn-resolver                         | JWT validation at the gateway edge                                       | p1          |
| OperationBuilder / utoipa              | OpenAPI spec generation from gear routes                               | p1          |
| postcard                               | Binary serialization for SecurityContext in gRPC metadata (Profile 1 in-process only) | p1          |
| Kubernetes 1.26+                       | Required for Profile 3 (K8s Native)                                      | p1          |
| External gateway (Kong/Tyk)            | Required for Profile 3 public API ingress                                | p2          |

## 11. Assumptions

- All OoP communication within a single host or k8s cluster occurs over a trusted network. Multi-host on-premise (P2)
  will add mTLS.
- The existing `OperationBuilder` (type-state builder pattern with compile-time enforcement of handler + response +
  auth) already generates correct OpenAPI 3.x specs via `utoipa` and `OpenApiRegistryImpl.build_openapi()`. No major
  changes to OpenAPI generation are needed.
- Gear SDK crates already define client traits (e.g., `UsersClient`). Generated REST clients implement these existing
  traits.
- `postcard` serialization of SecurityContext is deterministic and backward-compatible across gear versions. The
  existing `toolkit_security::bin_codec` with `SECCTX_BIN_VERSION = 1` provides versioned encoding. This encoding is
  used only for in-process gRPC metadata (Profile 1); HTTP propagation uses JWT re-validation per ADR-0008.
- `SecurityContext` will gain a `set_bearer_token(&mut self, token: SecretString)` method for post-deserialization token
  injection (currently only settable via builder).
- External gateways (Kong, Tyk) expose a programmatic admin API sufficient for dynamic route registration.

## 12. Risks

| Risk                                                             | Impact                                                    | Mitigation                                                                      |
|------------------------------------------------------------------|-----------------------------------------------------------|---------------------------------------------------------------------------------|
| SDK trait and server implementation diverge                    | Type mismatches at runtime, deserialization failures      | Trait-first codegen: the SDK trait IS the source of truth; `#[rest_contract]` generates client and server from it |
| SecurityContext binary format changes break cross-version calls  | Auth failures during rolling upgrades                     | `x-secctx-bin` used only for in-process gRPC (Profile 1); HTTP uses JWT re-validation (ADR-0008)                 |
| OoP latency exceeds threshold under high load                    | Degraded user experience, profile becomes impractical     | Benchmark early; connection pooling in toolkit-http; consider gRPC for hot paths |
| External gateway admin APIs differ significantly between vendors | High implementation cost for each GatewayProvider adapter | Start with one adapter (Kong); keep the trait surface minimal                   |
| UDS/named pipe communication limits on Windows                   | Reduced functionality on Windows on-premise               | Test on Windows CI early; fall back to TCP localhost if needed                  |

## 13. Open Questions

- How do we handle API versioning across OoP gear upgrades (rolling deployments with breaking API changes)?

## 14. Traceability

- **Design**: [DESIGN.md](./DESIGN.md)
- **ADRs**: [ADR/](./ADR/)
