---
title: "@outputai/llm"
description: "Generate and stream text, objects, and arrays from LLMs with prompt file management"
---

The `@outputai/llm` package is how you call LLMs from your steps and evaluators. It wraps the [AI SDK](https://sdk.vercel.ai/docs) and adds [prompt files](/prompts) — version-controlled `.prompt` files that live alongside your code and define the provider, model, temperature, and message template in one place.

## Generate Functions

`generateText` is the primary function for LLM calls. Use the `output` parameter with `Output.*` helpers to control the response shape. Use `streamText` for streaming responses:

| Output Shape | How | Use when you need |
|--------------|-----|-------------------|
| Unstructured text | `generateText({ prompt })` | Summaries, emails, explanations |
| Streamed text | `streamText({ prompt })` | Real-time output, long responses, UX responsiveness |
| Typed object | `generateText({ prompt, output: Output.object({ schema }) })` | Structured data, evaluator judgments |
| Array of objects | `generateText({ prompt, output: Output.array({ element }) })` | Lists, multiple items |
| One of N choices | `generateText({ prompt, output: Output.choice({ options }) })` | Classification, routing |

### Text Output

Generate unstructured text from a prompt file:

```typescript steps.ts
import { step } from '@outputai/core';
import { generateText } from '@outputai/llm';
import { GenerateSummaryInput, GenerateSummaryOutput } from './types.js';

export const generateSummary = step({
  name: 'generateSummary',
  description: 'Generate a company summary from research data',
  inputSchema: GenerateSummaryInput,
  outputSchema: GenerateSummaryOutput,
  fn: async (input) => {
    const { result } = await generateText({
      prompt: 'generate_summary@v1',
      variables: {
        companyName: input.name,
        industry: input.industry,
        size: input.size
      }
    });

    return result;
  }
});

// types.ts
// import { z } from '@outputai/core';
//
// export const GenerateSummaryInput = z.object({
//   name: z.string(),
//   industry: z.string(),
//   size: z.number()
// });
//
// export const GenerateSummaryOutput = z.string();
```

`result` is a convenience alias for `response.text`.

### Streaming

Stream text from a prompt file. Unlike `generateText`, `streamText` returns immediately with a stream result — properties like `text`, `usage`, and `finishReason` are promises that resolve when the stream completes.

```typescript steps.ts
import { step } from '@outputai/core';
import { streamText } from '@outputai/llm';
import { GenerateContentInput, GenerateContentOutput } from './types.js';

export const generateContent = step({
  name: 'generateContent',
  description: 'Streams text generation and collects chunks',
  inputSchema: GenerateContentInput,
  outputSchema: GenerateContentOutput,
  fn: async ({ topic }) => {
    const result = streamText({
      prompt: 'stream_content@v1',
      variables: { topic }
    });

    const chunks: string[] = [];
    for await (const chunk of result.textStream) {
      chunks.push(chunk);
    }

    const content = chunks.join('');
    return {
      content,
      chunkCount: chunks.length,
      avgChunkSize: Math.round(content.length / chunks.length)
    };
  }
});
```

Note that `streamText` is not `async` — it returns a stream result synchronously. Iterate `textStream` to process chunks as they arrive. You can also `await result.text` to get the full text in one shot, but that collapses the stream and is functionally identical to `generateText`.

To process chunks with side effects (e.g., writing to stdout):

```typescript
const result = streamText({
  prompt: 'generate@v1',
  variables: { topic: 'AI safety' }
});

for await (const chunk of result.textStream) {
  process.stdout.write(chunk);
}
```

You can apply stream transforms like `smoothStream` for more natural output pacing:

```typescript
import { streamText, smoothStream } from '@outputai/llm';

const result = streamText({
  prompt: 'generate@v1',
  variables: { topic },
  experimental_transform: smoothStream()
});
```

Use streaming callbacks for side effects without consuming the stream manually:

```typescript
const result = streamText({
  prompt: 'generate@v1',
  variables: { topic },
  onChunk({ chunk }) {
    // Called for each chunk
  },
  onFinish({ text, usage }) {
    // Called when generation completes
  },
  onError({ error }) {
    // Called on stream error
  }
});
```

### Object Output

Generate a structured object matching a Zod schema. This is what you'll use most in evaluators:

```typescript evaluators.ts
import { evaluator, EvaluationBooleanResult } from '@outputai/core';
import { generateText, Output } from '@outputai/llm';
import { z } from '@outputai/core';
import { JudgeSummaryInput } from './types.js';

export const judgeSummaryQuality = evaluator({
  name: 'judgeSummaryQuality',
  description: 'Judge whether a company summary is accurate and useful',
  inputSchema: JudgeSummaryInput,
  fn: async (input) => {
    const { output } = await generateText({
      prompt: 'judge_summary@v1',
      variables: {
        summary: input.summary,
        companyName: input.companyName
      },
      output: Output.object({
        schema: z.object({
          reasoning: z.string(),
          passes: z.boolean(),
          confidence: z.number()
        })
      })
    });

    return new EvaluationBooleanResult({
      value: output.passes,
      confidence: output.confidence,
      reasoning: output.reasoning
    });
  }
});

// types.ts
// import { z } from '@outputai/core';
//
// export const JudgeSummaryInput = z.object({
//   summary: z.string(),
//   companyName: z.string()
// });
```

`output` contains the typed object matching your schema.

### Array Output

Generate an array of structured items:

```typescript
import { generateText, Output } from '@outputai/llm';
import { z } from '@outputai/core';

const { output } = await generateText({
  prompt: 'extract_contacts@v1',
  variables: { companyData: JSON.stringify(company) },
  output: Output.array({
    element: z.object({
      name: z.string(),
      role: z.string(),
      email: z.string().optional()
    })
  })
});

// output is an array of { name, role, email } objects
```

### Choice Output

Select one value from a set of options:

```typescript
import { generateText, Output } from '@outputai/llm';

const { output } = await generateText({
  prompt: 'classify_lead@v1',
  variables: { activity: leadActivity },
  output: Output.choice({ options: ['hot', 'warm', 'cold', 'unknown'] })
});

// output is one of 'hot', 'warm', 'cold', 'unknown'
```

## Agents

The `Agent` class wraps AI SDK's `ToolLoopAgent` with Output [prompt files](/prompts) and the [skills](/prompts/skills) system. Use it when you need multi-step tool execution, conversation history, or a reusable agent instance with a fixed configuration. For single-shot LLM calls without tools, `generateText` is simpler.

### Construction

The prompt file is loaded and rendered at construction time. Variables, skills, and tools are fixed at construction. The agent is ready to call `generate()` or `stream()` immediately.

```typescript steps.ts
import { step } from '@outputai/core';
import { Agent, Output, skill } from '@outputai/llm';
import { z } from '@outputai/core';

const audienceSkill = skill({
  name: 'audience_adaptation',
  description: 'Tailor feedback for the specified expertise level',
  instructions: '# Audience Adaptation\n...'
});

export const reviewContent = step({
  name: 'reviewContent',
  description: 'Review content with structured feedback',
  inputSchema: ReviewContentInput,
  outputSchema: ReviewContentOutput,
  fn: async (input) => {
    const agent = new Agent({
      prompt: 'writing_assistant@v1',
      variables: {
        content_type: input.contentType,
        focus: input.focus,
        content: input.content
      },
      skills: [audienceSkill],
      output: Output.object({ schema: reviewSchema }),
      maxSteps: 5
    });
    const { output } = await agent.generate();
    return output;
  }
});
```

**Constructor options:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `prompt` | `string` | *(required)* | Prompt file name (e.g. `'writing_assistant@v1'`) |
| `variables` | `Record<string, unknown>` | `{}` | Template variables rendered at construction |
| `skills` | `Skill[]` | `[]` | [Skill](/prompts/skills) packages for the LLM |
| `tools` | `ToolSet` | `{}` | AI SDK tools available during the loop |
| `maxSteps` | `number` | `10` | Maximum tool-loop iterations |
| `stopWhen` | `StopCondition` | - | Custom stop condition (overrides `maxSteps`) |
| `output` | `Output` | - | Structured output spec (e.g. `Output.object({ schema })`) |
| `conversationStore` | `ConversationStore` | - | Pluggable store for multi-turn history |
| `temperature` | `number` | - | Override prompt file temperature |
| `onStepFinish` | `Function` | - | Callback after each tool-loop step |
| `prepareStep` | `Function` | - | Customize each step before execution |

### generate()

Run the agent and return when complete:

```typescript
const result = await agent.generate();
console.log(result.text);   // Generated text
console.log(result.output); // Structured output (when using Output.object)
console.log(result.usage);  // Token counts
```

The result has the same shape as `generateText`: `text`, `result` (alias for `text`), `output`, `usage`, `finishReason`, `toolCalls`, etc.

Pass additional messages to extend the conversation:

```typescript
const result = await agent.generate({
  messages: [{ role: 'user', content: 'Now focus on the introduction section.' }]
});
```

### stream()

Stream the agent's response:

```typescript
const stream = await agent.stream();

for await (const chunk of stream.textStream) {
  process.stdout.write(chunk);
}
```

Like `streamText`, the stream result provides `textStream` and `fullStream` iterables, plus promise-based properties (`text`, `usage`, `finishReason`) that resolve on completion.

### Structured Output

Use `Output.object()` with Agent to get typed responses:

```typescript steps.ts
import { Agent, Output } from '@outputai/llm';
import { z } from '@outputai/core';

const reviewSchema = z.object({
  issues: z.array(z.string()).describe('List of issues found'),
  suggestions: z.array(z.string()).describe('Actionable suggestions'),
  score: z.number().describe('Quality score 0-100'),
  summary: z.string().describe('Brief overall assessment')
});

const agent = new Agent({
  prompt: 'writing_assistant@v1',
  variables: { content_type: 'documentation', focus: 'clarity', content: markdownContent },
  output: Output.object({ schema: reviewSchema }),
  maxSteps: 5
});

const { output } = await agent.generate();
// output: { issues: string[], suggestions: string[], score: number, summary: string }
```

### Conversation Store

By default, Agent is stateless. Each `generate()` call starts fresh with only the initial prompt messages. Pass a `conversationStore` to maintain history across calls:

```typescript
import { Agent, createMemoryConversationStore } from '@outputai/llm';

const store = createMemoryConversationStore();
const chatbot = new Agent({
  prompt: 'chatbot@v1',
  conversationStore: store
});

const r1 = await chatbot.generate({
  messages: [{ role: 'user', content: 'Hello, tell me about Output.' }]
});
// r1.text: "Output is an AI framework for..."

const r2 = await chatbot.generate({
  messages: [{ role: 'user', content: 'How does it handle retries?' }]
});
// r2 sees the full conversation history from r1
```

For custom storage backends, implement the `ConversationStore` interface:

```typescript
interface ConversationStore {
  getMessages(): ModelMessage[] | Promise<ModelMessage[]>;
  addMessages(messages: ModelMessage[]): void | Promise<void>;
}
```

`createMemoryConversationStore()` is the built-in in-memory implementation. For production, implement the interface with your database.

<Note>
`stream()` does not automatically append messages to the conversation store. If you use streaming with a conversation store, persist messages manually in the `onFinish` callback.
</Note>

### When to Use Agent vs generateText

| | `generateText` | `Agent` |
|---|---|---|
| **Best for** | Single-shot LLM calls | Multi-step tool loops |
| **Tools** | Supported | Supported |
| **Skills** | Supported | Supported |
| **Conversation history** | Manual | Built-in with `conversationStore` |
| **Reusable instance** | No (function call) | Yes (construct once, call many) |
| **Structured output** | `Output.object()` | `Output.object()` |

Start with `generateText`. Move to `Agent` when you need conversation state or a reusable instance with a fixed configuration.

## Response Object

`generateText` returns the full [AI SDK response](https://sdk.vercel.ai/docs/reference/ai-sdk-core/generate-text#returns):

| Field | Description |
|-------|-------------|
| `result` | Convenience alias for `text` |
| `text` | The raw generated text |
| `output` | The structured output when using `Output.*` helpers |
| `usage` | Token counts: `inputTokens`, `outputTokens`, `totalTokens` |
| `finishReason` | Why generation stopped (`'stop'`, `'length'`, `'tool-calls'`, etc.) |
| `response` | Raw provider response metadata |
| `warnings` | Any warnings from the provider |
| `toolCalls` | Tool calls made by the model (when using tools) |
| `cost` | LLM usage attribute with `modelId`, `usage`, `total`, and `tokensUsed`; `null` when pricing could not be computed |

The `cost` property is an LLM usage attribute:

```json
{
  "type": "llm:usage",
  "modelId": "gpt-4o",
  "usage": [
    { "type": "input", "ppm": 5, "amount": 217, "total": 0.001085 },
    { "type": "output", "ppm": 15, "amount": 9, "total": 0.000135 }
  ],
  "total": 0.00122,
  "tokensUsed": 226
}
```

Only available, finite usage dimensions are included in `usage`. For example, `reasoning` is omitted when the model does not define separate reasoning pricing.

**Streaming response shape.** `streamText` returns a different result type. Stream iterables (`textStream`, `fullStream`) provide real-time chunks, while scalar properties (`text`, `usage`, `finishReason`, etc.) are promises that resolve when the stream completes:

| Field | Type | Description |
|-------|------|-------------|
| `textStream` | `AsyncIterable<string>` | Async iterable of text chunks |
| `fullStream` | `AsyncIterable<TextStreamPart>` | Async iterable of all stream events (text deltas, tool calls, etc.) |
| `text` | `Promise<string>` | Full text, resolved on completion |
| `usage` | `Promise<LanguageModelUsage>` | Token counts, resolved on completion |
| `finishReason` | `Promise<FinishReason>` | Why generation stopped, resolved on completion |
| `toolCalls` | `Promise` | Tool calls made during streaming, resolved on completion |
| `response` | `Promise` | Raw provider response metadata |
| `warnings` | `Promise` | Any warnings from the provider |

## Prompt Files

Instead of hardcoding model config and messages in your code, you write `.prompt` files that live in your workflow's `prompts/` folder. See the [Prompts Guide](/prompts) for the full documentation.

```yaml prompts/generate_summary@v1.prompt
---
provider: anthropic
model: claude-sonnet-4-20250514
temperature: 0.7
---

<system>
You write concise company summaries for sales teams.
</system>

<user>
Write a 2-3 paragraph summary of {{ companyName }}.

Industry: {{ industry }}
Company size: {{ size }} employees
</user>
```

### Configuration Options

| Option | Type | Description |
|--------|------|-------------|
| `provider` | `string` | `anthropic`, `openai`, `azure`, `vertex`, `bedrock`, `perplexity`, or a provider registered with `registerProvider` |
| `model` | `string` | Model identifier |
| `temperature` | `number` | Sampling temperature (0.0-2.0) |
| `maxTokens` | `number` | Maximum output tokens |
| `tools` | `object` | Provider-specific tools (web search, etc.) |
| `providerOptions` | `object` | Provider-specific options — see [ProviderOptions Guide](/prompts#provider-options) |

## Providers

### Anthropic

```yaml
---
provider: anthropic
model: claude-sonnet-4-20250514
---
```

Requires `ANTHROPIC_API_KEY` environment variable.

### OpenAI

```yaml
---
provider: openai
model: gpt-4o
---
```

Requires `OPENAI_API_KEY` environment variable.

### Azure OpenAI

```yaml
---
provider: azure
model: gpt-4o
---
```

Requires `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, and `AZURE_OPENAI_API_VERSION`.

### Vertex AI

```yaml
---
provider: vertex
model: gemini-1.5-pro
---
```

Requires Google Cloud authentication and configuration.

### Amazon Bedrock

```yaml
---
provider: bedrock
model: anthropic.claude-sonnet-4-20250514-v1:0
---
```

Requires AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`) or IAM role-based authentication. Set `AWS_SESSION_TOKEN` when using temporary credentials (e.g., from `aws sts assume-role`).

For cross-region inference, use the regional inference profile format: `us.anthropic.claude-sonnet-4-20250514-v1:0`.

Always set `maxTokens` in your Bedrock prompt files. Unlike the direct Anthropic provider (which auto-detects per-model limits), the Bedrock SDK has no client-side defaults and relies on server-side defaults that may be lower than the model's capacity.

When using `providerOptions`, use the `bedrock` namespace (not `anthropic`):

```yaml
providerOptions:
  bedrock:
    guardrailConfig:
      guardrailIdentifier: my-guardrail
      guardrailVersion: "1"
```

### Custom Providers

Use `registerProvider` when you want prompt files to reference an AI SDK provider that is not built in, or when you need a custom provider instance:

```typescript
import { createVertexAnthropic } from '@ai-sdk/google-vertex/anthropic';
import { registerProvider } from '@outputai/llm';

registerProvider('vertex-anthropic', createVertexAnthropic({
  project: process.env.GOOGLE_VERTEX_PROJECT,
  location: process.env.GOOGLE_VERTEX_LOCATION
}));
```

Then use the registered provider name in prompt front matter:

```yaml
---
provider: vertex-anthropic
model: claude-haiku-4-5
---
```

Built-in providers use Output's default fetch configuration, including longer Undici response timeouts for LLM calls that take time before returning headers or body chunks. Custom registered providers are used exactly as you register them; they do not automatically receive that custom fetch. If your custom provider also needs longer network timeouts, configure its provider instance directly.

## Provider Tools

Many providers offer built-in tools like web search. Configure them in YAML front matter:

```yaml prompts/research@v1.prompt
---
provider: vertex
model: gemini-2.0-flash
tools:
  googleSearch:
    mode: MODE_DYNAMIC
    dynamicThreshold: 0.8
---

<user>
Research {{ topic }} and provide sources
</user>
```

This is equivalent to calling `vertex.tools.googleSearch({ mode: 'MODE_DYNAMIC', dynamicThreshold: 0.8 })` at the code level, but keeps your prompt self-contained.

YAML tools are merged with code-level tools, so you can combine provider tools (from YAML) with custom tools (from code). Code-level tools take precedence if names conflict.

For provider-specific tool options, see:
- [Vertex AI Tools](https://sdk.vercel.ai/docs/ai-sdk-providers/google-vertex#provider-instance-methods)
- [OpenAI Tools](https://sdk.vercel.ai/docs/ai-sdk-providers/openai#provider-instance-methods)
- [Anthropic Tools](https://sdk.vercel.ai/docs/ai-sdk-providers/anthropic#provider-instance-methods)
- [Amazon Bedrock Tools](https://sdk.vercel.ai/docs/ai-sdk-providers/amazon-bedrock#provider-instance-methods)

## Tool Calling

Use tools with `generateText` to enable function calling:

```typescript
import { generateText, tool } from '@outputai/llm';
import { z } from '@outputai/core';

const { result, toolCalls } = await generateText({
  prompt: 'agent@v1',
  variables: { task: 'Research competitor pricing' },
  tools: {
    searchWeb: tool({
      description: 'Search the web for information',
      parameters: z.object({ query: z.string() }),
      execute: async ({ query }) => fetchSearchResults(query)
    })
  },
  toolChoice: 'auto'
});
```

## AI SDK Pass-Through Options

All generate functions accept additional [AI SDK options](https://sdk.vercel.ai/docs) passed through to the provider:

| Option | Type | Description |
|--------|------|-------------|
| `tools` | `ToolSet` | Tools the model can call (generateText and streamText) |
| `toolChoice` | `'auto' \| 'none' \| 'required'` | Tool selection strategy |
| `maxRetries` | `number` | Max retry attempts (default: 0) |
| `seed` | `number` | Seed for deterministic output |
| `abortSignal` | `AbortSignal` | Cancel the request |
| `topP` | `number` | Nucleus sampling (0-1) |
| `topK` | `number` | Top-K sampling |
| `onChunk` | `Function` | Callback for each stream chunk (streamText only) |
| `onFinish` | `Function` | Callback when stream completes (streamText only) |
| `onError` | `Function` | Callback on stream error (streamText only) |
| `experimental_transform` | `Function` | Stream transform, e.g. `smoothStream()` (streamText only) |

Options set in the prompt file (temperature, maxTokens) can be overridden at call time.

### Retries and Network Timeouts

`generateText` and `streamText` set `maxRetries: 0` by default. In Output workflows, LLM calls usually run inside steps, and steps are Temporal activities. When a provider error is allowed to fail the step, Temporal records the failed activity attempt and retries it according to the workflow's retry policy.

AI SDK retries are still available, but they happen inside one activity attempt. Use them when you want quick provider-level retries before the step fails. Keep the default when you want Temporal to be the single place that controls retries.

Pass `maxRetries` in the function call when you want the AI SDK to retry provider requests:

```typescript
const result = await generateText({
  prompt: 'summarize@v1',
  maxRetries: 2
});
```

Built-in providers are initialized with a custom fetch that extends Undici's `headersTimeout` and `bodyTimeout` to 15 minutes. This helps long-running LLM responses where the provider accepts the request but takes longer to return response headers or body chunks, for example reasoning-heavy calls. Active cancellation still works: if you pass `abortSignal`, or the AI SDK/provider aborts the request, that cancellation wins.

## LLM call cost event

Each `generateText` and `streamText` call emits a `cost:llm:request` event after the LLM responds and cost can be computed. You can observe it with the same [hooks mechanism](/packages/core#error-hooks) as error hooks: register a handler with `on('cost:llm:request', handler)` from `@outputai/core/hooks` in a hook file listed under `outputai.hookFiles`. The payload is the same LLM usage attribute exposed on `response.cost`. For payload details, see [Cost Events](/costs/cost-events).

## loadPrompt

Load and render a prompt file without generating — useful for debugging:

```typescript
import { loadPrompt } from '@outputai/llm';

const prompt = loadPrompt('generate_summary@v1', {
  companyName: 'Acme Corp',
  industry: 'SaaS',
  size: 250
});

console.log(prompt.config);   // { provider: 'anthropic', model: '...', temperature: 0.7 }
console.log(prompt.messages);  // Rendered message array
```

## API Reference

For complete TypeScript API documentation, see the [LLM Module API Reference](https://output-ai-reference-code-docs.onrender.com/modules/llm_src.html).
