---
title: Data Flow Diagrams
description: How data moves through the ObjectStack system — from defineStack() configuration to database queries, realtime events, and metadata resolution
---

# Data Flow Diagrams

This guide provides detailed diagrams showing how data moves through every layer of the ObjectStack system — from developer configuration to database operations and back to the client.

<Callout type="info">
**Architecture Overview:** For a high-level view of how protocol layers connect, see the [Protocol Relationship Diagram](/docs/guides/cheatsheets/protocol-diagram).
</Callout>

---

## 1. Configuration Flow: defineStack() → Database

How your `objectstack.config.ts` becomes a running application.

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant Cfg as objectstack.config.ts
    participant CLI as CLI / Build
    participant Man as Manifest
    participant Ker as Kernel
    participant Plu as Plugins
    participant Drv as Driver
    participant DB as Database

    Dev->>Cfg: defineStack({ objects, views, flows })
    Cfg->>CLI: Export configuration
    CLI->>Man: Compile & validate (Zod)

    alt Validation Failed
        Man-->>CLI: Schema errors
        CLI-->>Dev: Error output with suggestions
    end

    CLI->>Ker: Boot Kernel with Manifest
    Ker->>Plu: Initialize plugins
    Plu-->>Ker: Contracts registered

    Ker->>Drv: Connect to datasource
    Drv->>DB: Run migrations (if needed)
    DB-->>Drv: Schema ready

    Ker-->>CLI: Server ready
    CLI-->>Dev: Listening on http://localhost:3000
```

| Stage | What Happens |
|:---|:---|
| `defineStack()` | Developer declares objects, fields, views, flows in TypeScript (array or map format) |
| Compile | CLI validates all definitions against Zod schemas |
| Manifest | Compiled metadata package containing all configuration |
| Kernel Boot | Kernel loads the manifest and initializes the plugin graph |
| Plugin Init | Each plugin registers its contracts (DataEngine, AuthService, etc.) |
| Driver Connect | Database driver connects and runs any pending migrations |

---

## 2. Read Flow: Client → API → Database → Response

How a GET request for a single record flows through the system.

```mermaid
sequenceDiagram
    participant C as Client
    participant API as REST / GraphQL
    participant Auth as Auth Middleware
    participant K as Kernel
    participant Sec as Security
    participant QE as Query Engine
    participant H as Hooks
    participant D as Driver
    participant DB as Database

    C->>API: GET /api/v1/data/task/tsk_123
    API->>Auth: Verify JWT token
    Auth-->>API: AuthUser resolved

    API->>K: read('task', 'tsk_123', context)

    K->>Sec: checkPermission('task', 'read', user)
    Sec-->>K: ✓ Allowed

    K->>QE: buildQuery('task', { id: 'tsk_123' })
    QE->>Sec: applyRLS(query, user)
    Sec-->>QE: Filtered query

    QE->>D: execute(filteredQuery)
    D->>DB: SELECT * FROM tasks WHERE id = 'tsk_123' AND ...
    DB-->>D: Row data
    D-->>QE: Typed record

    QE-->>K: DataRecord

    K->>H: afterRead hooks
    H-->>K: Transformed record

    K->>Sec: applyFieldSecurity(record, user)
    Sec-->>K: Fields filtered by profile

    K-->>API: Final record
    API-->>C: 200 OK { data: { ... } }
```

<Callout type="tip">
**Row-Level Security (RLS):** Security filters are applied at the query level, not after fetching. This means the database only returns rows the user is allowed to see — efficient even on large tables.
</Callout>

---

## 3. Write Flow: Client → Validation → Hooks → Database → Response

How a POST (create) request is processed with full validation and hook execution.

```mermaid
sequenceDiagram
    participant C as Client
    participant API as REST API
    participant K as Kernel
    participant V as Validator
    participant BH as Before Hooks
    participant D as Driver
    participant DB as Database
    participant AH as After Hooks
    participant EB as Event Bus
    participant WS as WebSocket

    C->>API: POST /api/v1/data/task { data: {...} }
    API->>K: create('task', data, context)

    K->>V: validate(schema, data)

    alt Validation Failed
        V-->>K: ZodError
        K-->>API: 400 VALIDATION_ERROR
        API-->>C: { error: { details: [...] } }
    end

    V-->>K: ✓ Valid data

    K->>BH: beforeCreate hooks
    Note over BH: Transform data, enforce rules,<br/>set defaults, check permissions

    alt Hook Throws Error
        BH-->>K: BusinessError
        K-->>API: 4xx/5xx Error
        API-->>C: Error Response
    end

    BH-->>K: Transformed data

    K->>D: insert('task', transformedData)
    D->>DB: INSERT INTO tasks (...) VALUES (...)
    DB-->>D: Inserted row with ID
    D-->>K: DataRecord { id: 'tsk_456', ... }

    K->>AH: afterCreate hooks
    Note over AH: Send notifications,<br/>update counters,<br/>trigger workflows

    K->>EB: emit('task.created', record)
    EB->>WS: Broadcast to subscribers

    K-->>API: Created record
    API-->>C: 201 Created { data: { id: 'tsk_456', ... } }
```

---

## 4. Metadata Flow: defineStack() → IDE Autocomplete

How object definitions flow from configuration to runtime API and IDE tooling.

```mermaid
flowchart LR
    subgraph "Development"
        A["defineStack()"] -->|"TypeScript"| B[objectstack.config.ts]
        B -->|"compile"| C[Zod Validation]
    end

    subgraph "Build Time"
        C -->|"generate"| D[JSON Schema]
        C -->|"generate"| E[TypeScript Types]
        D -->|"feed"| F[IDE Autocomplete]
        E -->|"feed"| F
    end

    subgraph "Runtime"
        C -->|"bundle"| G[Manifest]
        G -->|"load"| H[Schema Registry]
        H -->|"serve"| I["GET /api/v1/meta/objects"]
        I -->|"discover"| J[Client SDK]
        J -->|"generate"| K[Typed Client]
    end
```

| Output | Used By | Purpose |
|:---|:---|:---|
| JSON Schema | VS Code, IntelliJ | Autocomplete and validation in `*.object.ts` files |
| TypeScript Types | Plugin developers | Type-safe access to object definitions |
| Manifest | Kernel | Runtime metadata for query validation and execution |
| Metadata API | Client SDK | Dynamic object/field discovery |
| Typed Client | Frontend apps | `client.task.create({ title: '...' })` |

---

## 5. Realtime Flow: Database Change → Client

How data changes propagate to connected clients in real time.

```mermaid
sequenceDiagram
    participant W as Writer Client
    participant API as API Layer
    participant K as Kernel
    participant DB as Database
    participant EB as Event Bus
    participant Sub as Subscription Manager
    participant R1 as Reader Client 1
    participant R2 as Reader Client 2

    Note over R1,R2: Clients subscribe via WebSocket

    R1->>Sub: subscribe('task', { filter: { project: 'prj_1' } })
    R2->>Sub: subscribe('task', { filter: { assigned_to: 'usr_2' } })
    Sub-->>R1: Subscribed (channelId: ch_1)
    Sub-->>R2: Subscribed (channelId: ch_2)

    W->>API: PATCH /api/v1/data/task/tsk_123 { status: 'done' }
    API->>K: update(...)
    K->>DB: UPDATE tasks SET status = 'done' WHERE id = 'tsk_123'
    DB-->>K: Updated record

    K->>EB: emit('task.updated', { record, changes })

    EB->>Sub: Route event to matching subscriptions
    Sub->>Sub: Check filters

    Note over Sub: tsk_123.project = 'prj_1' ✓ → ch_1
    Note over Sub: tsk_123.assigned_to = 'usr_2' ✓ → ch_2

    Sub->>R1: { event: 'task.updated', data: { id: 'tsk_123', ... } }
    Sub->>R2: { event: 'task.updated', data: { id: 'tsk_123', ... } }

    K-->>API: Updated record
    API-->>W: 200 OK { data: { ... } }
```

<Callout type="info">
**Subscription Filtering:** The Subscription Manager applies each client's filter server-side. Clients only receive events for records that match their subscription criteria — no over-fetching.
</Callout>

---

## 6. Query Execution Flow

How a structured query is translated to SQL and executed.

```mermaid
flowchart TD
    A[QueryInput] -->|parse| B[Query Parser]
    B -->|validate| C[Schema Validator]
    C -->|optimize| D[Query Optimizer]
    D -->|secure| E[RLS Filter Injection]
    E -->|translate| F[SQL Generator]
    F -->|execute| G[Database]
    G -->|map| H[Result Mapper]
    H -->|paginate| I[QueryResult]
```

| Stage | Description |
|:---|:---|
| **Query Parser** | Parses the JSON `QueryInput` into an AST |
| **Schema Validator** | Verifies field names and operators against the object schema |
| **Query Optimizer** | Reorders filters, selects indexes, applies batch strategies |
| **RLS Filter Injection** | Appends row-level security conditions based on the user's profile |
| **SQL Generator** | Translates the AST to the target database dialect (PostgreSQL, MySQL, SQLite) |
| **Database** | Executes the query and returns raw rows |
| **Result Mapper** | Converts raw rows to typed `DataRecord` objects |
| **Pagination** | Calculates `total`, `hasMore`, `offset` for the response |

---

## 7. Hook Execution Order

The precise order in which hooks fire during each operation.

```mermaid
flowchart TD
    subgraph "Create Operation"
        C1[beforeValidate] --> C2[validate]
        C2 --> C3[afterValidate]
        C3 --> C4[beforeCreate]
        C4 --> C5["DB INSERT"]
        C5 --> C6[afterCreate]
        C6 --> C7[afterCommit]
    end

    subgraph "Update Operation"
        U1[beforeValidate] --> U2[validate]
        U2 --> U3[afterValidate]
        U3 --> U4[beforeUpdate]
        U4 --> U5["DB UPDATE"]
        U5 --> U6[afterUpdate]
        U6 --> U7[afterCommit]
    end

    subgraph "Delete Operation"
        D1[beforeDelete] --> D2["DB DELETE"]
        D2 --> D3[afterDelete]
        D3 --> D4[afterCommit]
    end
```

| Hook | Phase | Can Modify Data? | Can Abort? |
|:---|:---|:---|:---|
| `beforeValidate` | Pre-validation | ✅ Yes | ✅ Yes |
| `validate` | Validation | ❌ No | ✅ Yes |
| `afterValidate` | Post-validation | ✅ Yes | ✅ Yes |
| `beforeCreate/Update` | Pre-persist | ✅ Yes | ✅ Yes |
| `afterCreate/Update/Delete` | Post-persist | ❌ No (record is saved) | ❌ No |
| `afterCommit` | Post-transaction | ❌ No | ❌ No |

<Callout type="tip">
**Side Effects:** Place side effects (emails, notifications, external API calls) in `afterCommit` hooks. These run after the database transaction is committed, so you won't send a notification for a record that was rolled back.
</Callout>
