---
id: chat-completion
title: Chat Completion
hide_title: false
hide_table_of_contents: false
description: How to use the SAP Cloud SDK for AI to perform chat completion tasks using orchestration service from SAP AI Core.
keywords:
  - sap
  - cloud
  - sdk
  - ai
  - orchestration
  - chat
  - completion
---

The `@sap-ai-sdk/orchestration` package provides a client for the orchestration service of SAP AI Core.
The orchestration service harmonizes the API of various generative AI models, enabling seamless integration and interaction with different models through a unified interface.
Additionally, it provides features like [templating](#prompt-templating), [content filtering](#content-filtering), [grounding](#grounding) and more to enhance the interaction with generative AI models.

Find more details about orchestration workflow [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/orchestration-workflow-v2).

## Installation

```bash
npm install @sap-ai-sdk/orchestration
```

## Quick Start

Initialize a client with proper orchestration configuration for each module.
Below is an example for configuring the mandatory `promptTemplating` module.
In addition, you can find more sample code [here](https://github.com/SAP/ai-sdk-js/blob/main/sample-code/src/orchestration.ts).

```ts
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
});
```

Here, the model name is specified along with a user message as the prompt.

To send a chat completion request, use the `chatCompletion()` method.
Use the following convenience methods for handling chat completion response:

- `getContent()` parses the response and returns the model's output as a string.
- `getFinishReason()` retrieves the `finish_reason` explaining why chat completion request stopped.
- `getTokenUsage()` provides token usage details, including `total_tokens`, `prompt_tokens`, and `completion_tokens`.
- `getAllMessages()` parses the response and returns a list of all messages.
- `getAssistantMessage()` parses the response and returns the assistant message.
- `getToolCalls()` parses the response and returns a list of tool calls generated by the model.
- `getRefusal()` parses the response and returns the refusal message from the model.
- `getCitations()` returns the list of citations from the response, useful when using models like [Perplexity Sonar](#using-citations-with-perplexity-sonar-models).
- `getIntermediateFailures()` returns information about non-fatal errors that occurred during the request, such as when using [module fallback](#module-fallback).

```ts
const response = await orchestrationClient.chatCompletion({
  messages: [
    { role: 'user', content: 'Hello World! Why is this phrase so famous?' }
  ]
});

console.log(response.getContent());
console.log(response.getFinishReason());
console.log(JSON.stringify(response.getTokenUsage()));
```

The following sections provide detailed configuration for each module.

## Prompt Templating

### LLM Configuration

Choose the LLM by setting the `name` property in the `promptTemplating.model` configuration.
Optionally, define `version` (default: `latest`) and `params` for custom settings.

```ts
const promptTemplating = {
  model: {
    name: 'gpt-5',
    version: '2024-08-06', // optional
    params: {
      // optional
      max_tokens: 50
    }
  }
};
```

### Module Fallback

The orchestration service supports module fallback, which allows you to configure multiple complete orchestration module configurations that are tried in sequence until one succeeds.
This feature is useful for implementing resilience and fallback strategies when working with AI models.

Each configuration in the fallback chain is a complete `OrchestrationModuleConfig`, allowing you to customize all aspects of orchestration for each fallback option, not just the model.
This includes [system prompts](#template-configuration), [content filtering](#content-filtering) rules, [data masking](#data-masking) settings, [grounding](#grounding) configurations, and other orchestration module settings.
This flexibility accommodates model-specific requirements, such as varying parameters, system prompts, or content filtering rules, enabling tailored fallbacks for different use cases or constraints.

To use module fallback, provide an array of `OrchestrationModuleConfig` objects when initializing the client.
The orchestration service will try each configuration in order until one succeeds.

```ts
import {
  OrchestrationClient,
  buildAzureContentSafetyFilter
} from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient([
  {
    // Primary configuration with strict filtering and detailed prompt
    promptTemplating: {
      model: {
        name: 'gpt-5',
        timeout: 10 // 10 seconds timeout for the LLM request on the orchestration server side
      },
      prompt: {
        template: [
          {
            role: 'system',
            content:
              'You are an expert AI analyst. Provide detailed, comprehensive responses.'
          },
          {
            role: 'user',
            content: '{{?question}}'
          }
        ]
      }
    },
    filtering: {
      input: {
        filters: [
          buildAzureContentSafetyFilter('input', {
            hate: 'ALLOW_SAFE',
            violence: 'ALLOW_SAFE'
          })
        ]
      }
    }
  },
  {
    // Fallback configuration: faster model with relaxed filtering and concise prompt
    promptTemplating: {
      model: {
        name: 'amazon--nova-micro'
      },
      prompt: {
        template: [
          {
            role: 'system',
            content: 'Provide a brief, concise response.'
          },
          {
            role: 'user',
            content: '{{?question}}'
          }
        ]
      }
    },
    filtering: {
      input: {
        filters: [
          buildAzureContentSafetyFilter('input', {
            hate: 'ALLOW_SAFE_LOW_MEDIUM',
            violence: 'ALLOW_SAFE_LOW_MEDIUM'
          })
        ]
      }
    }
  }
]);

const response = await orchestrationClient.chatCompletion({
  placeholderValues: {
    question: 'Write a comprehensive analysis of AI trends.'
  }
});

console.log(response.getContent());
```

:::info Timeout Parameter
The `timeout` parameter in the model configuration sets the maximum time (in seconds) for the LLM request.
If the request exceeds this timeout in non-streaming scenarios, the orchestration service triggers the fallback to the next configuration.
The timeout parameter applies to the individual LLM call, not the entire orchestration request.
For more information on timeouts in orchestration, refer to the [Orchestration Retry and Timeout](#orchestration-retry-and-timeout) section.
:::

The `getIntermediateFailures()` method returns information about any failed attempts before a successful response is obtained.

```ts
const failures = response.getIntermediateFailures();
if (failures) {
  failures.forEach(failure => {
    console.log(`${failure.location}: ${failure.message}`);
  });
}
```

:::tip
When using module fallback with different models, the SAP Cloud SDK for AI logs a warning if duplicate models are detected in the fallback chain.
Consider using different models for meaningful fallback behavior.
:::

#### Module Fallback Conditions

Module fallback is triggered under specific conditions, which may change in future versions of the orchestration service.

**For streaming requests:** Fallback happens before the stream starts.
The model chosen by the orchestration service will not switch mid-stream.

- The model is not available.

**For non-streaming requests:**
Fallback may occur even after the request starts being handled by the first provider.
The following additional conditions can trigger fallbacks:

- Request timeout
- Too many requests
- Internal Service errors

Module fallback is **not** triggered for content-filter hits or other errors, such as invalid input errors.

### Harmonized API

The [Harmonized API](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/harmonized-api?locale=en-US) lets you use different foundation models without the need to change the client code.
You can switch from one LLM to another easily like the example below.

```javascript
// Original config for using GPT 4o
/*
const promptTemplating = {
  model: {
    name: 'gpt-5'
  }
};
*/

// Switch to Claude 3.5 Sonnet
const promptTemplating = {
  model: {
    name: 'anthropic--claude-3.5-sonnet'
  }
};
```

:::tip Available LLMs on SAP Generative AI Hub
Thanks to the harmonized API, all available LLMs on the SAP Generative AI Hub can be accessed through orchestration, including:

- OpenAI GPT 4o
- OpenAI o1
- OpenAI o3 mini
- AWS Anthropic Claude
- AWS Amazon Nova
- GCP Gemini
- Mistral AI
- Perplexity Sonar
- SAP ABAP 1
  :::

Check the [SAP Notes](https://me.sap.com/notes/3437766) for all available LLMs on SAP Generative AI Hub.

#### Using Citations with Perplexity Sonar Models

Some models provide citations that reference the sources used to generate responses, making them ideal for use cases requiring up-to-date information and source transparency.
Citations can be accessed using the `getCitations()` method on the response object.
For details on the citation properties, refer to the [API documentation](pathname:///api/v2/types/orchestration_src.Citation.html).

Each citation contains the following properties:

- `ref_id`: Unique numeric identifier for the inline citation
- `title`: Title of the cited source
- `url`: URL of the cited source

:::warning
Citations are currently only supported by Perplexity Sonar models.
:::

Use the `getCitations()` method to retrieve citations from the response:

```ts
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'sonar'
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  messages: [
    {
      role: 'user',
      content: 'What are the latest developments in quantum computing?'
    }
  ]
});

console.log(response.getContent());

// Get citations from the response
const citations = response.getCitations();
if (citations) {
  citations.forEach(citation => {
    console.log(`[${citation.ref_id}] ${citation.title}: ${citation.url}`);
  });
}
```

#### Consuming SAP ABAP 1 Model

SAP-ABAP-1 is a foundation model built by SAP that helps with coding tasks and can be only be consumed through the orchestration service.
It is fine-tuned on a large amount of SAP ABAP code and is optimized for explaining ABAP code snippets.
When using this model, input and output filtering is configured by default.

The following example shows how to consume the `sap-abap-1` model:

```ts
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'sap-abap-1',
      version: 'latest' // optional
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  messages: [
    {
      role: 'user',
      content:
        'Please explain the following ABAP class definition and its purpose: {{?abapCode}}'
    }
  ],
  placeholderValues: {
    abapCode: `CLASS zcl_customer_manager DEFINITION
  PUBLIC
  FINAL
  CREATE PUBLIC.

  PUBLIC SECTION.
    METHODS: get_customer_data
      IMPORTING iv_customer_id TYPE kunnr
      RETURNING VALUE(rs_customer) TYPE zst_customer_data
      RAISING zcx_customer_not_found.

  PRIVATE SECTION.
    DATA: lo_customer_db TYPE REF TO zcl_customer_database.
ENDCLASS.`
  }
});

console.log(response.getContent());
console.log(response.getFinishReason());
console.log(JSON.stringify(response.getTokenUsage()));
```

For detailed example payloads and more details, refer to the [SAP AI Core documentation](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/sap-abap-1?q=sap+abap).

### Template Configuration

Use the orchestration client with the `promptTemplating.prompt.template` configuration to define a static prompt.
This prompt can include placeholders, which are replaced with values from `placeholderValues` during a `chatCompletion()` method call.
This setup is useful when the base structure of your prompt remains the same across requests.

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    },
    prompt: {
      template: [
        { role: 'user', content: 'What is the capital of {{?country}}?' }
      ]
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  placeholderValues: { country: 'France' }
});
```

:::info
The `promptTemplating.prompt.template` messages defined in the client configuration are static - they are always included in every `chatCompletion()` request.
For more dynamic prompts, i.e., you want to vary the full message list per request without re-initializing the client, use the `messages` property instead.
:::

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  messages: [{ role: 'user', content: 'What is the capital of {{?country}}?' }],
  placeholderValues: { country: 'France' }
});
```

### Prompt Registry

You can define template content within the templating configuration, or refer to a template defined in the [Prompt Registry](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/prompt-registry).

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    },
    prompt: {
      template_ref: {
        name: 'my-get-capital-template',
        scenario: 'my-scenario',
        version: '0.0.1'
      }
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  placeholderValues: { country: 'France' }
});
```

:::info Messages with Prompt Template References
When using the `template_ref` property, messages passed to the `chatCompletion()` or `stream()` methods are automatically appended to the `messages_history` array as they cannot be merged into the stored prompt template.
The `messages` property defined in the `template` array are appended after any existing `messagesHistory` entries.
:::

A prompt template can be referenced either by ID, or by a combination of name, scenario and version.
For details on storing a template in the Prompt Registry, refer to [this guide](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-prompt-template-imperative).

### Scoped Prompt Registry

:::info Tutorial: Creating Scoped Prompt Templates with Headers
For a step-by-step guide on creating orchestration deployment and configuring scoped prompt registry templates, see the [Using Scoped Prompt Registry Templates](../tutorials/using-scoped-prompt-registry-templates) tutorial.
:::

Prompt templates stored in the [Prompt Registry](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/prompt-registry) can be scoped to different levels.
By default, prompts are stored in the `tenant` scope, making them accessible across all resource groups within the tenant, ensuring broad availability.

Alternatively, you can use the `resource_group` scope to store prompts specific to a particular resource group.
This is useful when prompt templates need to be isolated and should not be shared with others in the same tenant.

To reference a scoped prompt template, specify the `scope` property in the `template_ref` alongside either the template `id` or template `name`, `scenario`, and `version`.
When using a non-default resource group, make sure to also provide the `resourceGroup` parameter in the client deployment config.

```ts
const orchestrationClient = new OrchestrationClient(
  {
    promptTemplating: {
      model: {
        name: 'gpt-5'
      },
      prompt: {
        template_ref: {
          name: 'my-scoped-template',
          scenario: 'my-scenario',
          version: '0.0.1',
          scope: 'resource_group' // or 'tenant'
        }
      }
    }
  },
  {
    resourceGroup: 'my-resource-group'
  }
);
```

### Local Prompt Template

You can define the template in a YAML file and pass its content as a string to the `promptTemplating.prompt` property.
This is useful for testing a prompt template locally before storing it in the [Prompt Registry](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/prompt-registry).

```ts
import { readFileSync } from 'fs';
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

// Read the YAML file containing the prompt template
const yamlTemplate = readFileSync('./path/to/prompt-template.yaml', 'utf-8');

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    },
    prompt: yamlTemplate
  }
});

const response = await orchestrationClient.chatCompletion({
  placeholderValues: { country: 'France' }
});
```

The YAML string is parsed and validated against the expected schema.
If the YAML is invalid or does not conform to the schema, an error will be thrown.

This [section](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-prompt-template-declarative#procedure) shows the format in which a prompt template can be defined.

### Function Calling

Define and pass [tool definitions](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/tool-calling) to enable the model to call specific functions.
Here's an example of temperature conversion using tool calls:

First, define the tool with `name`, `description` and `parameters` properties:

```ts
const convertTemperatureTool: ChatCompletionTool = {
  type: 'function',
  function: {
    name: 'convert_temperature_to_fahrenheit',
    description: 'Converts temperature from Celsius to Fahrenheit',
    parameters: {
      type: 'object',
      properties: {
        temperature: {
          type: 'number',
          description: 'The temperature value in Celsius to convert.'
        }
      },
      required: ['temperature']
    }
  }
};
```

Set `strict` to `true` to ensure function calls adhere to the function schema.
For more information refer to Orchestration [Tool Calling](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/tool-calling).

Initialize the client and send the initial request with the tool definition:

```ts
const client = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    },
    prompt: {
      tools: [convertTemperatureTool]
    }
  }
});

const response = await client.chatCompletion({
  messages: [
    { role: 'user', content: 'Convert 20 degrees Celsius to Fahrenheit.' }
  ]
});
```

When the model decides to use a tool, it returns the function name and input arguments in the response.
Use the model response to execute the function.

```ts
const initialResponse = response.getAssistantMessage();
let toolMessage: ToolChatMessage;

if (initialResponse && initialResponse.tool_calls) {
  const toolCall = initialResponse.tool_calls[0];
  const name = toolCall.function.name;
  const args = JSON.parse(toolCall.function.arguments);

  // Execute the function with the provided arguments
  const toolResult = callFunction(name, args);

  toolMessage: ToolChatMessage = {
    role: 'tool',
    content: toolResult,
    tool_call_id: toolCall.id
  };
}
```

The `callFunction()` function routes the calls to the actual function implementations.

```ts
function callFunction(name: string, args: any): string {
  switch (name) {
    case 'convert_temperature_to_fahrenheit':
      return convertTemperatureToFahrenheit(args.temperature);
    default:
      throw new Error(`Function: ${name} not found!`);
  }
}

function convertTemperatureToFahrenheit(temperature: number): string {
  return `The temperature in Fahrenheit is ${(temperature * 9) / 5 + 32}°F.`;
}
```

Send the function result back to the model to get its final response:

```ts
const finalResponse = await client.chatCompletion({
  messages: [toolMessage],
  messagesHistory: response.getAllMessages()
});

console.log(finalResponse.getContent());
```

### Response Format

For general response formatting, use the `response_format` parameter.
It is useful when the model is **not calling a tool** and should still return a structured response.

The example below returns a JSON Schema with `strict` set to `true` to let the response adhere to the schema definition.

```ts
const templating: TemplatingModuleConfig = {
  response_format: {
    type: 'json_schema',
    json_schema: {
      name: 'capital_response',
      strict: true,
      schema: {
        type: 'object',
        properties: {
          country_name: {
            type: 'string',
            description: 'The name of the country provided by the user.'
          },
          capital: {
            type: 'string',
            description: 'The capital city of the country.'
          }
        },
        required: ['country_name', 'capital']
      }
    }
  }
};
```

:::info

We recommend using Zod v4 for full compatibility with this package.
If you're upgrading from an earlier version, refer to the [Zod v4 migration guide](https://zod.dev/v4/changelog) and pay attention to breaking changes like the switch from `describe('...')` to `meta({ description: '...' })`

:::

You can also define JSON schema using [Zod](https://zod.dev/) schema as shown below:

```ts
import * as z from 'zod';
import { toJsonSchema } from '@langchain/core/utils/json_schema';
import { ResponseFormatJsonSchema } from '@sap-ai-sdk/orchestration';

const countryCapitalSchema = z.strictObject({
  country_name: z.string(),
  capital: z.string()
});

const response_format: ResponseFormatJsonSchema = {
  type: 'json_schema',
  json_schema: {
    name: 'capital_response',
    strict: true,
    schema: toJsonSchema(countryCapitalSchema)
  }
};
```

### Message History

The orchestration service supports chat completion with message history.
This allows the model to remember previous messages in the conversation, enabling a more coherent and context-aware interaction.

Provide the message history in the `messagesHistory` parameter when calling the `chatCompletion()` method.

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  messages: [{ role: 'user', content: 'What is my name?' }],
  messagesHistory: [
    {
      role: 'system',
      content:
        'You are a helpful assistant who remembers all details the user shares with you.'
    },
    {
      role: 'user',
      content: "Hi! I'm Bob"
    },
    {
      role: 'assistant',
      content:
        "Hi Bob, nice to meet you! I'm an AI assistant. I'll remember that your name is Bob as we continue our conversation."
    }
  ]
});
```

### Image Recognition

Many models in the orchestration service have image recognition capabilities, meaning the models can take images as input and answer questions about them.

:::note
The `image_url` content type can only be used in messages with `role: 'user'`.
Attempting to use `image_url` in non-user messages will result in an error.
:::

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  messages: [
    {
      role: 'user', // only user role supports image_url
      content: [
        {
          type: 'text',
          text: 'What is the content of the image?'
        },
        {
          type: 'image_url',
          image_url: {
            url: '{{?imageUrl}}'
          }
        }
      ]
    }
  ],
  placeholderValues: {
    imageUrl: 'IMAGE_URL'
  }
});
```

`IMAGE_URL` can either be a public URL or a base64 encoded image, e.g., `data:image/jpeg;base64,...`.
You can send a single chat completion request with multiple images by defining multiple content blocks with `image_url` type in the template.
The model will process each image and use the information from all of them to respond.

### Prompt Caching

:::note
This section covers explicit cache control for AWS Bedrock-hosted models (Anthropic Claude and Amazon Nova).
Other model providers, such as OpenAI and Gemini, use implicit context caching that is enabled by default and requires no additional configuration.
For a full overview, refer to the [Prompt Caching documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/prompt-caching?locale=en-US).
:::

Cache control improves performance by caching requests at the model provider level.
Add a `cache_control` breakpoint to a content block to mark a cache breakpoint in the prompt.
Availability of different time-to-live (TTL) options may depend on the model, refer to the [orchestration prompt caching documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/prompt-caching?locale=en-US) for details.

:::note
Prompt caching requires a minimum number of tokens in the cached content.
If the prompt is too short, the cache breakpoint may not take effect.
:::

Set the `cache_control` property as part of message contents.
Some models can also apply `cache_control` to tool calls.
Use the `getTokenUsage()` method to inspect cache activity.
For supported models, the `prompt_tokens_details` property on the tokens usage object will contain the amount of tokens read and written to the cache.

```ts
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'anthropic--claude-4.7-opus'
    },
    prompt: {
      template: [
        {
          role: 'system',
          content: [
            {
              type: 'text',
              text:
                'You are an expert news article classifier.' +
                ' Classify each article into exactly one category.'.repeat(512),
              cache_control: { type: 'ephemeral' }
            }
          ]
        },
        { role: 'user', content: 'input: Comcast launches prepaid plans' },
        { role: 'assistant', content: 'Business' },
        {
          role: 'user',
          content: [
            {
              type: 'text',
              text: 'input: {{?article}}',
              cache_control: { type: 'ephemeral' }
            }
          ]
        }
      ]
    }
  }
});

const response = await orchestrationClient.chatCompletion({
  placeholderValues: {
    article:
      'Scaling up neural models has yielded significant advancements in language generation'
  }
});

const usage = response.getTokenUsage();
console.log('Cache details', usage.prompt_tokens_details ?? usage);
```

### File Input

Some models in the orchestration service accept files as input, such as PDFs, CSV files, Word documents, and audio files.

:::note

The `file` content type can only be used in messages with `role: 'user'`.
Attempting to use `file` in non-user messages will result in an error.

:::

For file input, the [content block](pathname:///api/v2/types/orchestration_src.UserChatMessageContentItem.html) should have `type: 'file'` and include a [`file`](pathname:///api/v2/types/orchestration_src.FileContent.html) object with a `file_data` property containing a URL.
You can either provide a public HTTPS URL or a data URL with base64 encoded file content.

:::note

A model's support for file input via the orchestration service may not match the file input capabilities of each individual model when accessed directly.
Please always check if the specific file type you want to use is supported via the orchestration service for your chosen model.

:::

Data URLs must follow this format:

```
data:MIME_TYPE;base64,BASE64_ENCODED_FILE_CONTENT
```

It is recommended to include a `filename` in the payload; otherwise, the orchestration service may assign one automatically.

The following example reads a local file and sends it to the model.
Adjust the file path, MIME type and model name as needed, for PDF files use `application/pdf` as MIME type.

<!-- vale off -->

```ts
import { readFile } from 'node:fs/promises';
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: { name: 'MODEL_NAME' }
  }
});

const data = await readFile('PATH_TO_FILE', 'base64');
const fileUrl = `data:MIME_TYPE;base64,${data}`;

const response = await orchestrationClient.chatCompletion({
  messages: [
    {
      role: 'user', // only user role supports file
      content: [
        { type: 'text', text: 'Summarize this document.' },
        { type: 'file', file: { file_data: fileUrl, filename: 'FILENAME' } }
      ]
    }
  ]
});
```

<!-- vale on -->

## Content Filtering

Configure content filtering to restrict content that is passed to and received from a generative AI model.

This feature allows filtering both [input](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/input-filtering) and [output](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/output-filtering) of a model based on content safety criteria.

The following example demonstrates how to use content filtering with the orchestration client.
See the sections below for details on the available content filters and how to build them.

```ts
const inputFilter: InputFilterConfig = ... // Use a build function to create an input content filter
const outputFilter: OutputFilterConfig = ... // Use a build function to create an output content filter

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  },
  filtering: {
    input: {
      filters: [inputFilter] // Multiple filters can be applied
    },
    output: {
      filters: [outputFilter]
    }
  }
});

try {
  const response = await orchestrationClient.chatCompletion({
    messages: [{ role: 'user', content: 'I hate you!' }]
  });
  console.log(response.getContent());
} catch (error: any) {
  console.error(error.message);
  console.error(error.cause?.response?.data);
}
```

Multiple filters can be applied at the same time for both input and output filtering.

:::tip
The `chatCompletion()` method can throw an error with HTTP status code `400` if content filters hit.
In case of a `200` HTTP response, the `getContent()` method can throw an error if the output filters hit.
See the [Error Handling](../error-handling) page for more details.
:::

### Azure Content Filter

Use `buildAzureContentSafetyFilter()` function to build an Azure content filter.
Input and output filters are differentiated.
Set the `type` parameter to `input` to build input filter configuration and `output` to build output filter configuration.
Each category of the filter can be assigned a specific severity level, which corresponds to an Azure threshold value.

| Severity Level          | Azure Threshold Value |
| ----------------------- | --------------------- |
| `ALLOW_SAFE`            | 0                     |
| `ALLOW_SAFE_LOW`        | 2                     |
| `ALLOW_SAFE_LOW_MEDIUM` | 4                     |
| `ALLOW_ALL`             | 6                     |

#### Prompt Attack Detection

A prompt attack is a malicious input that is designed to bypass a model's safety mechanisms or override previous instructions.
Prompt attacks can lead to the generation of harmful content or the execution of malicious actions.

The Azure Content Safety service also supports prompt attack detection for input text via a prompt shield configuration.
Set the `prompt_shield` property to `true` to enable the detection.
This option is only available for input filters.
For more information, refer to [Prompt Shields in Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection).

```ts
import { buildAzureContentSafetyFilter } from '@sap-ai-sdk/orchestration';

const inputFilter = buildAzureContentSafetyFilter('input', {
  hate: 'ALLOW_SAFE_LOW',
  violence: 'ALLOW_SAFE_LOW_MEDIUM',
  prompt_shield: true
});

const outputFilter = buildAzureContentSafetyFilter('output', {
  hate: 'ALLOW_SAFE',
  violence: 'ALLOW_SAFE_LOW_MEDIUM'
});
```

#### Protected material detection for code

Protected material code detection identifies and safeguards against the inclusion of proprietary or copyrighted code snippets in AI-generated responses.
This scan proactively monitors for protected code snippets that match those in software libraries, source code, algorithms, and other intellectual property sourced from known GitHub repositories, preventing unauthorized use or potential legal issues.
To enable protected material code detection, set the `protected_material_code` property to `true` in the Azure content safety input filter configuration.
The property is only supported for output filter configurations.
For more information, refer to [Azure AI Content Safety Protected Material Detection](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/protected-material).

```ts
import { buildAzureContentSafetyFilter } from '@sap-ai-sdk/orchestration';

const filterConfig = buildAzureContentSafetyFilter('output', {
  hate: 'ALLOW_SAFE',
  protected_material_code: true
});
```

### Llama Guard Filter

Use `buildLlamaGuard38BFilter()` function to build a Llama Guard 3 8B content filter.
Input and output filters are differentiated.
Set the `type` parameter to `input` to build input filter configuration and `output` to build output filter configuration.

Available categories can be found with autocompletion.
Pass the categories as an array to the function to enable them.

```ts
import { buildLlamaGuard38BFilter } from '@sap-ai-sdk/orchestration';

const inputFilter = buildLlamaGuard38BFilter('input', ['self_harm']);
const outputFilter = buildLlamaGuard38BFilter('output', [
  'self_harm',
  'violence'
]);
```

## Data Masking

Use the orchestration client with the masking module to mask sensitive information in the prompt while preserving necessary context for the generative AI model.

The following example demonstrates how to use data masking with the orchestration client.
See the sections below for details on the available masking providers and how to build them.

```ts
const maskingProvider: MaskingProviderConfig = ... // Use a build function to create a masking provider

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  },
  masking: {
    masking_providers: [maskingProvider] // Multiple masking providers can be applied
  }
});

const response = await orchestrationClient.chatCompletion({
  messages: [
    {
      role: 'user',
      content:
        'Please write an email to {{?user}} ({{?email}}) about the amazing capabilities of SAP AI Core!'
    }
  ],
  placeholderValues: { user: 'Jane Doe', email: 'jane.doe@example.com' }
});
```

### SAP Data Privacy Integration

Orchestration service offers a masking provider "SAP Data Privacy Integration (DPI)" to anonymize or pseudonymize sensitive information.
Use `buildDpiMaskingProvider()` function to build a DPI masking provider with standard or custom entities.
Provide regular expression to match a custom entity.
Configure `replacement_strategy` to control how entities are masked.
Set method to `constant` to replace an entity with the specified value followed by an incrementing number.
Set method to `fabricated_data` to replace an entity with a randomly generated value appropriate to its type.

```ts
const maskingProvider = buildDpiMaskingProvider({
  method: 'annonymization',
  entities: [
    'profile-person',
    {
      type: 'profile-email',
      replacement_strategy: {
        method: 'fabricated_data'
      }
    },
    {
      type: 'custom',
      regex: '\\b[0-9]{4}-[0-9]{4}-[0-9]{3,5}\\b',
      replacement_strategy: {
        method: 'constant',
        value: 'REDACTED_ID'
      }
    }
  ],
  // mask_grounding_input: false, // optional
  allowlist: ['SAP'] // optional
});
```

The `allowlist` property specifies terms which will be kept unmasked.
Set `mask_grounding_input` to `true` to mask [Grounding](#grounding) input as well.

## Grounding

Grounding enables integrating external, contextually relevant, domain-specific, or real-time data into AI processes.
Use `buildDocumentGroundingConfig()` function to build configuration for document grounding service.

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  },
  grounding: buildDocumentGroundingConfig({
    placeholders: {
      input: ['groundingRequest'],
      output: 'groundingOutput'
    },
    // metadata_params: ['PARAM_NAME']
    filters: [
      {
        id: 'FILTER_ID',
        // data_repository_type: 'vector', // optional, default to 'vector'
        data_repositories: ['REPOSITORY_ID']
      }
    ]
  })
});

const response = await orchestrationClient.chatCompletion({
  messages: [
    {
      role: 'user',
      content:
        'UserQuestion: {{?groundingRequest}} Context: {{?groundingOutput}}'
    }
  ],
  placeholderValues: {
    groundingRequest: 'Give me a short introduction of SAP AI Core.'
  }
});
```

By default, the optional filter property `data_repository_type` is set to `vector`.
Set it to `help.sap.com` to retrieve context from the SAP Help Portal.

Set `metadata_params` property with an array of parameter names to include [Metadata](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/contextualized-retrieval-using-metadata-and-vector-search?locale=en-US) in the grounding result, which can be mentioned when writing the prompt.
If set the value to `'*'`, all metadata will be included.

Set `data_repositories` property with an array of repository IDs to search in specific data repositories.
Skip this property to search in all available data repositories.

## Translation

The translation module translates content sent to and received from a generative AI model into a chosen target language.

The module supports SAP's Document Translation service and allows translating both [input](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/input-translation) and [output](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/output-translation) of a model.
The target language is mandatory, while source language will be auto-detected if not provided.

For available translation options, consult the [list of supported languages](https://help.sap.com/docs/translation-hub/sap-translation-hub/supported-languages-6854bbb1bd824ffebc3a097a7c0fd45d) to identify compatible language pairs.
Input and output translation configurations are differentiated.
Set the `type` parameter to `input` to build the input configuration and `output` to build the output configuration.

```ts
const inputTranslation = buildTranslationConfig('input', {
  sourceLanguage: 'en-US',
  targetLanguage: 'de-DE'
});

const outputTranslation = buildTranslationConfig('output', {
  sourceLanguage: 'en-US',
  targetLanguage: 'de-DE'
});

const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  },
  translation: {
    input: inputTranslation,
    output: outputTranslation
  }
});

const response = await orchestrationClient.chatCompletion({
  messages: [
    {
      role: 'user',
      content: 'Write an abstract for a thriller playing at SAP headquarters.'
    }
  ]
});
```

### ApplyTo Selector

The `applyTo` property in the translation configuration allows you to selectively translate which messages or parts of messages in the conversation should be translated.
This provides granular control over translation in multi-turn conversations.

This selector is optional and allows specifying:

- **`category`**: can be either `placeholders` or `template_roles`.
- **`items`**: array of specific placeholder or role names to apply the translation to.
- **`sourceLanguage`** (optional): source language for the items. If not specified, the detected language will be applied.

When you don't define an `applyTo` selector, the system translates all applicable content without selective filtering.
For more information about this feature, refer to the [translation documentation](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/enhance-model-consumption-with-translation#loio152a5f4796f9493d98323786571356bf__section_rck_4gp_2hc).

##### Placeholders

The `placeholders` category allows translating specific placeholder values from the `placeholderValues` parameter of the `chatCompletion()` method.

##### Template Roles

The `template_roles` category allows translating template messages with specific message roles, such as `user`, `assistant`, `system` or `tool` messages in the prompt.

#### Automatic Target Language Inference

By detecting the source language from translation, the output translation can automatically respond in the user's original language.
There is no need to specify a fixed target language, instead reference the `applyTo` item from the input translation.

For more information check out [this](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/enhance-model-consumption-with-output-translation-48be058d01bd4df2a82e5430561e6f44#automatic-target-language-inference) link.

```ts
const inputTranslation = buildTranslationConfig('input', {
  sourceLanguage: 'en-US',
  targetLanguage: 'de-DE',
  applyTo: [
    {
      category: 'placeholders',
      items: ['country'],
      sourceLanguage: 'en-US'
    },
    {
      category: 'placeholders',
      items: ['groundingRequest'],
      sourceLanguage: 'de-DE'
    }
  ]
});

const outputTranslation = buildTranslationConfig('output', {
  sourceLanguage: 'en-US',
  targetLanguage: 'de-DE',
  applyTo: [
    {
      category: 'placeholders',
      items: ['country'],
      sourceLanguage: 'en-US'
    }
  ]
});
```

#### Chat History Control

By default, chat history messages are translated.
If chat history message translation is not necessary, set the `translateMessagesHistory` to `false`.

```ts
const translationConfig = buildTranslationConfig('input', {
  sourceLanguage: 'de-DE',
  targetLanguage: 'en-US',
  translateMessagesHistory: false
});
```

## Use JSON Configuration from AI Launchpad

If you already have an orchestration workflow created in your SAP AI Launchpad instance, you can either download the configuration as a JSON file or copy the JSON string in code to configure the orchestration client.

```ts
const jsonConfig = await fs.promises.readFile(
  'path/to/orchestration-config.json',
  'utf-8'
);
// Alternatively, you can provide the JSON string in code directly.
// const jsonConfig = 'YOUR_JSON_CONFIG'

const orchestrationClient = new OrchestrationClient(jsonConfig);
```

## Streaming

The `OrchestrationClient` supports streaming responses for chat completion requests based on the [Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events) standard.

Use the `stream()` method to receive a stream of chunk responses from the model.
After consuming the stream, call the helper methods to get the finish reason and token usage information.

```ts
const response = await orchestrationClient.stream({
  placeholderValues: { country: 'France' }
});

for await (const chunk of response.stream) {
  console.log(JSON.stringify(chunk));
}

const finishReason = response.getFinishReason();
const tokenUsage = response.getTokenUsage();

console.log(`Finish reason: ${finishReason}\n`);
console.log(`Token usage: ${JSON.stringify(tokenUsage)}\n`);
```

### Streaming the Delta Content

Use `toContentStream()` method to create a stream generating delta content string.

```ts
for await (const chunk of response.stream.toContentStream()) {
  console.log(chunk); // will log the delta content
}
```

### Streaming with Tool Calls

Use `getToolCalls()` method to get the tool calls at the end of a stream.
While `getDeltaToolCalls()` method can be called on individual chunks, partial tool calls are typically not useful.
Therefore, it is recommended to use the `getToolCalls()` method on the full response instead.

```ts
for await (const _ of response.stream) {
  console.log('Waiting for the stream to end ...');
}

const toolCalls = response.getToolCalls();
```

### Streaming with Abort Controller

Streaming request can be aborted using the `AbortController` API.
In case of an error, SAP Cloud SDK for AI will automatically close the stream.
It can also be manually aborted if an `AbortSignal` object was provided when calling the `stream()` method.

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
});

const controller = new AbortController();
const response = await orchestrationClient.stream(
  {
    messages: [
      { role: 'user', content: 'Give a long history of {{?country}}?' }
    ],
    placeholderValues: { country: 'France' }
  },
  controller.signal
);

// Abort the streaming request after one second
setTimeout(() => {
  controller.abort();
}, 1000);

for await (const chunk of response.stream) {
  console.log(JSON.stringify(chunk));
}
```

In this example, streaming request will be aborted after one second.
Abort controller can be useful, e.g., when end-user wants to stop the stream or refreshes the page.

### Streaming with Citations

When using models that support citations, citations are available after the stream completes and can be retrieved using the `getCitations()` method.
For more information on the citation properties, refer to the [API documentation](pathname:///api/v2/types/orchestration_src.Citation.html).

```ts
const response = await orchestrationClient.stream({
  messages: [
    {
      role: 'user',
      content: 'What are the latest developments in quantum computing?'
    }
  ]
});

for await (const chunk of response.stream) {
  console.log(chunk.getDeltaContent());
}

// Get citations after stream completes
const citations = response.getCitations();
```

:::note
Citations may not always be present in the response depending on the query and whether the model used web sources.
:::

### Streaming Options

The orchestration service offers multiple streaming options, which you can configure in addition to the LLM streaming options.
These include options like defining the maximum number of characters per chunk or modifying the output filter behavior.
There are two ways to add specific streaming options to your client, either at initialization of orchestration client, or when calling the `stream()` method.

You can check the list of available stream options [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/streaming).

Below is an example for setting the streaming options when calling the `stream()` method.

```ts
const response = orchestrationClient.stream(
  {
    messages: [
      { role: 'user', content: 'Give a long history of {{?country}}?' }
    ],
    placeholderValues: { country: 'France' }
  },
  controller,
  {
    llm: { include_usage: false },
    global: { chunk_size: 10 },
    outputFiltering: { overlap: 200 }
  }
);
```

Set the `include_usage` property to `false` if usage metrics should not be returned by default.
Set the `llm` property to `null` to not send any LLM streaming options.

:::note
When initializing a client with JSON module config, it is not possible to provide streaming options.
:::

#### Module Fallback with Streaming

[Module fallback](#module-fallback) is also supported when using the `stream()` method.
The service tries each configuration in sequence, and you can access intermediate failures after the stream completes.
For details on configuring module fallback, see the [Module Fallback section](#module-fallback).

```ts
const response = await orchestrationClient.stream({
  messages: [
    { role: 'user', content: 'Explain quantum computing in simple terms.' }
  ]
});

for await (const chunk of response.stream) {
  console.log(chunk.getDeltaContent());
}

// Check for fallback attempts after streaming completes
const failures = response.getIntermediateFailures();
if (failures) {
  console.log(`Used ${failures.length} fallback(s) before success`);
  failures.forEach((failure, index) => {
    console.log(
      `Failure ${index + 1} (Code: ${failure.code} at ${failure.location}): ${failure.message}`
    );
  });
}
```

:::note
Providing streaming option overrides is only possible when your orchestration configuration uses module fallback with more than one configuration.
Overrides cannot change global options like `chunk_size`.
:::

When using streaming with module fallback, you can provide streaming option overrides for each configuration in the fallback chain.

The streaming options structure consists of a `global` object containing options shared across all configurations, such as `chunk_size`.
It also includes module-specific options, like `promptTemplating` or `outputFiltering`, that apply to all configurations unless overridden, and an optional `overrides` object that maps configuration indices to option overrides.
When an override is provided for a specific configuration index, it replaces the base module-specific options entirely for that configuration.
If no override is provided for a configuration, it uses the base options from the top-level properties (everything except `global` and `overrides`).
Overrides cannot modify global options or contain additional overrides.

```ts
const response = await orchestrationClient.stream(
  {
    messages: [{ role: 'user', content: 'Tell me a story.' }]
  },
  undefined,
  {
    // Global Options
    global: { chunk_size: 100 },
    // Base Options
    promptTemplating: { include_usage: false }, // Config inherited by all modules if not overriden.
    // Module-specific overrides
    overrides: {
      // Config 0 replaces base options fully.
      0: { promptTemplating: { include_usage: true } },
      // Config 1 uses base options because no override is provided.
      // Config 2 replaces base options fully.
      2: { outputFiltering: { overlap: 50 } }
    }
  }
);
```

## Initialization with Orchestration Configuration

Instead of defining the orchestration configuration inline, you can reference a stored configuration from the prompt registry.
This approach enables configuration reuse across applications and centralized management of orchestration settings.

### By Configuration ID

Reference an orchestration configuration by its unique identifier:

```ts
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient({
  id: 'c1a4f2e2-BETA-4d2b-9f7a-1234567890ab'
});
```

### By Scenario, Name, and Version

Reference an orchestration configuration by its scenario, name, and version:

```ts
import { OrchestrationClient } from '@sap-ai-sdk/orchestration';

const orchestrationClient = new OrchestrationClient({
  scenario: 'customer-support',
  name: 'my-orchestration-config',
  version: '0.0.1'
});
```

Once initialized with an orchestration configuration reference, use the client as usual:

```ts
const response = await orchestrationClient.chatCompletion({
  placeholderValues: {
    customerQuestion: 'How do I reset my password?'
  }
});

console.log(response.getContent());
```

:::info Messages with Orchestration Configuration References
When initializing the client with an orchestration configuration reference, any messages passed to the `chatCompletion()` or `stream()` methods are automatically appended to the `messages_history` array as they cannot be merged into the stored configuration's prompt template.
The `messages` property defined in the `template` array are appended after any existing `messagesHistory` entries.
:::

:::note
When using an orchestration configuration reference, [streaming](#streaming) is only available if the stored configuration includes streaming settings.
:::

For details on creating and managing orchestration configurations, refer to the [Orchestration Configuration](../ai-core/prompt-registry#orchestration-configuration) section in the prompt registry guide.

## Custom Deployment Configuration

By default, there should be an orchestration deployment in the `default` [Resource Group](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/resource-groups).

If the orchestration service has been deployed in a different resource group, it is possible to specify the resource group name in the `resourceGroup` property when creating the client.

```ts
const orchestrationClient = new OrchestrationClient(orchestrationConfig, {
  resourceGroup: 'my-custom-resource-group'
});
```

Additionally, it is possible to manually specify a deployment ID using the `deploymentId` property instead of letting the SDK resolve it.
Make sure to set a correct resource group, in which the deployment was created.

```ts
const orchestrationClient = new OrchestrationClient(orchestrationConfig, {
  deploymentId: 'my-custom-deployment-id'
});
```

Refer to [Create a Deployment for Orchestration](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-orchestration) for more details on how to create and manage deployments for orchestration.

## Custom Request Configuration

Set custom request configuration in the `requestConfig` parameter when calling the `chatCompletion()` method.

```ts
const response = await orchestrationClient.chatCompletion(
  {
    ...
  },
  {
    headers: {
      'x-custom-header': 'custom-value'
      // Add more headers here
    },
    params: {
      // Add more parameters here
    }
    // Add more request configuration here
  }
);
```

Use a custom request configuration to attach SAP internal metering headers to enable internal cost allocation and context propagation.
Use `premium-billing` to indicate the billing plan or token (e.g., "premium-plan") and `user-context` to pass caller metadata.

```ts
const response = await orchestrationClient.chatCompletion(
  {
    messages: [{ role: 'user', content: 'Generate a report.' }]
  },
  {
    headers: {
      'premium-billing': 'premium-plan',
      'user-context': 'userId=12345;role=developer' // or a JSON/base64 string depending on your internal convention
    }
  }
);
```

## Access to Raw HTTP Response

The SAP Cloud SDK for AI provides access to the underlying HTTP response through the `rawResponse` property on response objects.
Use this for advanced scenarios such as accessing response headers, inspecting HTTP status codes, or debugging.

```ts
const response = await orchestrationClient.chatCompletion({
  messages: [{ role: 'user', content: 'Hello World!' }]
});

// Recommended: Use convenience methods for typical use cases
console.log(response.getContent());

// Advanced: Access HTTP response details when needed
console.log(response.rawResponse.status); // HTTP status code
console.log(response.rawResponse.headers); // Response headers
console.log(response.rawResponse.data); // Raw response data
```

:::tip
For most use cases, prefer the convenience methods like `getContent()`, `getMessages()`, and `getTokenUsage()`.
Use `rawResponse` only when you need direct access to HTTP-level details.
:::

The `rawResponse` property is available on chat completion and streaming response objects.
For embedding responses, access the raw HTTP response through the `response` property.

## Custom Destination

When initializing the client, it is possible to provide a custom destination.
For example, when targeting a destination with the name `my-destination`, the following code can be used:

```ts
const orchestrationClient = new OrchestrationClient(
  orchestrationConfig,
  deploymentConfig,
  {
    destinationName: 'my-destination'
  }
);
```

By default, the fetched destination is cached.
To disable caching, set the `useCache` parameter to `false` together with the `destinationName` parameter.

For more information about configuring a destination, refer to the [Using a Destination](../connecting-to-ai-core#using-a-destination) section.

## Resilience

The orchestration service provides built-in resilience features to handle transient failures and improve reliability.

:::note
The choice between Orchestration Service resilience and SAP Cloud SDK resilience should be made based on the specific requirements of the use case:

Orchestration Service Resilience should be used when resilience is required at the orchestration layer, such as retries or timeouts for orchestration API calls, with a preference for service-level configuration.

SAP Cloud SDK Resilience should be used when advanced resilience patterns, such as circuit breakers, are required, or when consistent fault tolerance is needed across multiple service integrations.
:::

### Orchestration Retry and Timeout

:::info

Models provided by GCP Gemini currently do not support the Orchestration Service `timeout` and `max_retries` options.

:::

The orchestration service supports built-in retry and timeout configurations on the model configuration.
These can be configured when initializing the client.

For example, to configure retry and timeout options:

<!-- vale off -->

```ts
const orchestrationClient = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5',
      timeout: 30, // 30 seconds
      max_retries: 3
    }
  }
});
```

<!-- vale on -->

### SAP Cloud SDK Resilience Package

:::info

Some orchestration service resilience features (timeout and retry) are also enabled by default.

:::

Use the `resilience()` function from `@sap-cloud-sdk/resilience` to add SDK-level resilience.
By default, it enables a circuit breaker and a 10-second timeout.

```ts
import { resilience } from '@sap-cloud-sdk/resilience';

const request = {
  messages: [
    { role: 'user', content: 'Hello World! Why is this phrase so famous?' }
  ]
};

const response = await orchestrationClient.chatCompletion(request, {
  middleware: resilience()
});
```

:::tip
`resilience()` returns an array of middleware.
You can pass it directly to `middleware` or combine it with other middleware: `[...resilience(), myMiddleware]`.
:::

Customize the behavior by passing options:

<!-- vale off -->

```ts
const response = await orchestrationClient.chatCompletion(request, {
  middleware: resilience({
    timeout: 5000, // 5 seconds; true for default 10s, false to disable
    circuitBreaker: true, // true by default, false to disable
    retry: 3 // false by default; true for 3 retries, or pass a number
  })
});
```

<!-- vale on -->

For advanced resilience patterns, refer to the [SAP Cloud SDK documentation on resilience](https://sap.github.io/cloud-sdk/docs/js/guides/resilience).

### Request Cancellation

Cancel HTTP requests using the `AbortSignal` API to stop in-flight requests when they are no longer needed.
This is useful when end-users navigate away from a page, close a dialog, or explicitly cancel an operation.
Pass an `AbortSignal` through the `signal` property of the request configuration parameter and call the `abort()` method on the controller when you want to cancel the request.

```ts
const controller = new AbortController();

const response = orchestrationClient.chatCompletion(
  {
    messages: [
      { role: 'user', content: 'What is the capital of {{?country}}?' }
    ],
    placeholderValues: { country: 'France' }
  },
  { signal: controller.signal }
);

// Cancel the request after one second
setTimeout(() => {
  controller.abort();
}, 1000);

try {
  await response;
} catch (error) {
  console.error('Request was cancelled:', error.message);
}
```

:::note
HTTP request cancellation may not cancel the request on the orchestration service.
The cancellation stops the HTTP connection from the client side, but the service may continue processing the request.
Metering information may not be available for cancelled requests.
:::
