Skip to main content

# Overview

Otoroshi provides a powerful workflow engine that lets you describe and execute sequences of logic using a JSON-based pseudo-language. Workflows allow you to orchestrate actions, transform data, trigger functions, and control flow based on conditional logic, all in a low-code fashion.

The Otoroshi LLM Extension adds a set of workflow functions and workflow nodes specifically designed for AI and LLM use cases. These additions allow you to build complex AI pipelines directly within Otoroshi workflows, including LLM calls, audio processing, image/video generation, embeddings, vector store operations, content moderation, guardrails validation, MCP tool calls, persistent memory management, and full agentic workflows.

## How workflows work​

A workflow is a JSON document with a `kind: "workflow"` root, a `steps` array containing nodes, and a `returned` field for the final output. Each node has a `kind` field that determines its behavior. The workflow engine maintains an isolated memory space where variables are stored and accessed across steps using expression language syntax like `${variable_name}`.

To call a function from a workflow, you use a `call` node:

    {
      "kind": "call",
      "function": "extensions.com.cloud-apim.llm-extension.llm_call",
      "args": {
        "provider": "provider_xxxxx",
        "payload": {
          "messages": [
            { "role": "user", "content": "Hello!" }
          ]
        }
      },
      "result": "llm_response"
    }

The result of the function call is stored in the workflow memory under the name specified by the `result` field, and can be referenced in subsequent steps.

## Available functions​

The LLM extension registers the following workflow functions:

Function | Description
--- | ---
`llm_call` | Call an LLM provider
`audio_tts` | Convert text to speech
`audio_stt` | Convert speech to text
`compute_embedding` | Compute text embeddings
`ocr_call` | Extract text from an image or pdf document
`generate_image` | Generate images from text
`generate_video` | Generate videos from text
`tool_function_call` | Call a tool function
`mcp_function_call` | Call an MCP connector function
`moderation_call` | Call a moderation model
`guardrail_call` | Validate input with a guardrail
`vector_store_add` | Add an entry to a vector store
`vector_store_remove` | Remove an entry from a vector store
`vector_store_search` | Search in a vector store
`memory_add_messages` | Add messages to persistent memory
`memory_get_messages` | Get messages from persistent memory
`memory_clear_messages` | Clear messages from persistent memory

## Available nodes​

The LLM extension also registers custom workflow nodes that can be used directly in workflow steps:

Node | Description
--- | ---
`AI Agent` | Execute an AI agent with tool calling, handoffs, and memory
`AI Agent Router` | Use an LLM to choose which path to follow
`MCP Tools` | Select an MCP connector for an agent

## Complete example​

Here is a complete workflow that uses the LLM extension to build a simple RAG (Retrieval Augmented Generation) pipeline:

    {
      "kind": "workflow",
      "steps": [
        {
          "kind": "call",
          "function": "extensions.com.cloud-apim.llm-extension.compute_embedding",
          "args": {
            "provider": "embedding-model_xxxxx",
            "payload": {
              "input": ["${input.question}"],
              "model": "text-embedding-ada-002"
            }
          },
          "result": "question_embedding"
        },
        {
          "kind": "call",
          "function": "extensions.com.cloud-apim.llm-extension.vector_store_search",
          "args": {
            "provider": "embedding-store_xxxxx",
            "payload": {
              "embedding": {
                "vector": "${question_embedding.data.0.embedding}"
              },
              "max_results": 5,
              "min_score": 0.7
            }
          },
          "result": "search_results"
        },
        {
          "kind": "call",
          "function": "extensions.com.cloud-apim.llm-extension.llm_call",
          "args": {
            "provider": "provider_xxxxx",
            "payload": {
              "messages": [
                {
                  "role": "system",
                  "content": "Answer the user question based on the following context: ${search_results}"
                },
                {
                  "role": "user",
                  "content": "${input.question}"
                }
              ]
            }
          },
          "result": "llm_response"
        }
      ],
      "returned": "${llm_response}"
    }

## Agent workflow with various AI/LLM functions usage​