src — canvas

Module: src-canvas Cohesion: 0.80 Members: 0

src — canvas

The src/canvas module provides a comprehensive framework for AI agents to interact with and render visual interfaces. It encompasses two primary, albeit distinct, systems:

  1. A2UI (Agent-to-UI) Protocol Implementation: A structured, component-based UI system for building dynamic, interactive interfaces. This includes the A2UIManager for state management, A2UIServer for real-time communication, and A2UITool for agent interaction.
  2. Canvas Manager: A more free-form drawing surface for managing graphical elements, their positions, sizes, and history.

This documentation will cover both systems, highlighting their individual functionalities and how they integrate within the broader codebase.

Module Overview

The src/canvas module serves as the foundation for visual interaction within the system. It allows AI agents to:

Core Concepts

Architecture Diagram (A2UI)

The A2UI system follows a clear separation of concerns: 

graph TD
    A[AI Agent] -->|Calls A2UITool actions| B(A2UITool)
    B -->|Processes messages| C(A2UIManager)
    C -->|Emits UI state changes| D(A2UIServer)
    D -->|Broadcasts via WebSocket| E[UI Client (Browser/Terminal)]
    E -->|Sends user actions| D
    D -->|Forwards user actions| C
    C -->|Renders to HTML/Terminal| E

A2UI Components

a2ui-types.ts: The A2UI Protocol Definition

This file defines the entire A2UI protocol, acting as the contract between agents, the manager, and UI clients.

Key Interfaces & Types:

A2UIManager: The A2UI State Machine

The A2UIManager class is the core logic unit for the A2UI system. It manages the lifecycle and state of all A2UI surfaces and their components.

Key Responsibilities:

Execution Flow Example: Agent updates data, UI client reacts

  1. AI Agent calls a2uiTool.execute({ action: 'update_data', surfaceId: 'my-surface', data: { counter: 5 } }).
  2. A2UITool.updateData calls a2uiManager.processMessage({ dataModelUpdate: { surfaceId: 'my-surface', contents: { counter: 5 } } }).
  3. A2UIManager.processDataModelUpdate updates surfaces.get('my-surface').dataModel.
  4. A2UIManager.processDataModelUpdate emits 'data:updated' and calls notifyDataObservers.
  5. A2UIServer (which listens to A2UIManager events) receives the 'data:updated' event.
  6. A2UIServer.setupManagerListeners calls broadcastToSurface('my-surface', { dataModelUpdate: ... }).
  7. Subscribed UI clients receive the dataModelUpdate WebSocket message and update their UI.

A2UIServer: The A2UI Communication Hub

The A2UIServer class provides the network interface for the A2UI system, enabling real-time communication with UI clients and serving static HTML views.

Key Responsibilities:

A2UITool: Agent Interface for A2UI

The A2UITool class provides a high-level, simplified interface for AI agents to interact with the A2UI system without needing to understand the intricacies of the A2UIManager or A2UIServer directly.

Key Responsibilities:

Example Agent Interaction: 

import { getA2UITool } from './a2ui-tool.js';

const a2uiTool = getA2UITool();

async function agentWorkflow() {
  class="hl-cmt">// Start the A2UI server
  let result = await a2uiTool.execute({ action: 'start_server', port: 8080 });
  console.log(result.output); class="hl-cmt">// "A2UI server started at http://127.0.0.1:8080..."

  class="hl-cmt">// Create a new surface
  result = await a2uiTool.execute({ action: 'create_surface', surfaceId: 'dashboard-1' });
  console.log(result.output); class="hl-cmt">// "Surface 'dashboard-1' created"

  class="hl-cmt">// Add components
  result = await a2uiTool.execute({
    action: 'add_components',
    surfaceId: 'dashboard-1',
    components: [
      { id: 'header', type: 'heading', props: { level: 1, value: 'Welcome to Dashboard' } },
      { id: 'counter-text', type: 'text', props: { path: 'counterValue' } },
      { id: 'increment-btn', type: 'button', props: { label: 'Increment', action: { name: 'increment' } } },
    ],
  });
  console.log(result.output); class="hl-cmt">// "3 component(s) added to surface 'dashboard-1'"

  class="hl-cmt">// Update data model
  result = await a2uiTool.execute({ action: 'update_data', surfaceId: 'dashboard-1', data: { counterValue: 0 } });
  console.log(result.output); class="hl-cmt">// "Data model updated in surface 'dashboard-1'"

  class="hl-cmt">// Begin rendering
  result = await a2uiTool.execute({ action: 'begin_rendering', surfaceId: 'dashboard-1', root: 'header' });
  console.log(result.output); class="hl-cmt">// "Surface 'dashboard-1' is now rendering from root 'header'"

  class="hl-cmt">// Render to terminal
  result = await a2uiTool.execute({ action: 'render_terminal', surfaceId: 'dashboard-1' });
  console.log(result.output); class="hl-cmt">// ASCII art representation of the UI
}

agentWorkflow();

Canvas Manager

canvas-manager.ts: Free-form Canvas Management

The CanvasManager class provides functionality for managing a more traditional, free-form graphical canvas, distinct from the A2UI component-based surfaces. It allows for the creation and manipulation of various CanvasElement types.

Key Responsibilities:

Distinction from A2UI:

It's important to note that the CanvasManager operates on a different model than the A2UI system.

While both are "visual," they serve different purposes and have distinct internal representations.

Integration Points and Extensibility

How to Extend

  1. Define the new ComponentType and its Props interface in a2ui-types.ts.
  2. Add the new component to the ComponentProps union type.
  3. Implement rendering logic for the new component type in A2UIManager.renderNodeToTerminal and A2UIManager.renderNodeToHTML.
  4. If the component is interactive, ensure A2UIServer.getClientScript includes event listeners for it, converting client-side events into CanvasEventMessages or UserActionMessages.

  1. Add the new action name to the A2UIAction type in a2ui-tool.ts.
  2. Implement a new private method in A2UITool to handle the action, calling the appropriate A2UIManager or A2UIServer methods.
  3. Add a case for the new action in A2UITool.execute.

  1. Define new CanvasElementType and its properties in types.ts (the one imported by canvas-manager.ts).
  2. Implement rendering logic for the new element type in CanvasManager.elementToSVG (and potentially renderToPNG, renderToPDF).
  3. Add methods to CanvasManager for creating/manipulating this new element type.