---
id: orchestration
title: Orchestration Integration
hide_title: false
hide_table_of_contents: false
description: How to integrate orchestration client of SAP Cloud SDK for AI into LangChain.
keywords:
  - sap
  - cloud
  - sdk
  - ai
  - langchain
  - orchestration
---

The `@sap-ai-sdk/langchain` packages provides `OrchestrationClient` client for LangChain integration with orchestration service.

## Client Initialization

The client reuses the orchestration client from `@sap-ai-sdk/orchestration` and implements [LangChain's interface](https://docs.langchain.com/oss/javascript/langchain/models).
Therefore, the client initialization combines the configuration of the orchestration client and LangChain options.

Similar to the orchestration client, the `OrchestrationClient` LangChain client can be initialized with an orchestration configuration.
Refer to [Orchestration Chat Completion](../orchestration/chat-completion) for more information about the configuration.

```ts
import { OrchestrationClient } from '@sap-ai-sdk/langchain';

const config: OrchestrationModuleConfig = {
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
};
const client = new OrchestrationClient(config);
```

Optionally, you can also specify LangChain options, resource group in deployment config, and a custom destination.

### Custom Destination

When initializing the client, it is possible to provide a custom destination for your SAP AI Core instance.

```ts
const client = new OrchestrationClient(
  orchestrationConfig,
  langchainOptions,
  deploymentConfig,
  { destinationName: 'my-destination' }
);
```

By default, the fetched destination is cached.
To disable caching, set the `useCache` parameter to `false` together with the `destinationName` parameter.

For more information about configuring a destination, refer to the [Using a Destination](../connecting-to-ai-core#using-a-destination) section.

## Chat Completion

Pass a message history and, in most cases, input parameters for the orchestration templating module.

```ts
const systemMessage = new SystemMessage('Be a helpful assistant!');
const history = [systemMessage];
const response = await client.invoke(history, {
  placeholderValues: { subject: 'Paris' }
});
```

### Module Fallback

The [`OrchestrationClient`](pathname:///api/v2/classes/langchain_src.OrchestrationClient.html) supports module fallback, which allows you to provide multiple orchestration configurations that are tried in sequence until one succeeds.
Pass a non-empty array of [`LangChainOrchestrationModuleConfig`](pathname:///api/v2/types/langchain_src.LangChainOrchestrationModuleConfig.html) objects when initializing the client, using the [`LangChainOrchestrationModuleConfigList`](pathname:///api/v2/types/langchain_src.LangChainOrchestrationModuleConfigList.html) type.

For a full explanation of when fallback is triggered and how it works with the orchestration service, refer to [Module Fallback](../orchestration/chat-completion#module-fallback) in the orchestration documentation.

```ts
import { OrchestrationClient } from '@sap-ai-sdk/langchain';
import type { LangChainOrchestrationModuleConfigList } from '@sap-ai-sdk/langchain';

const orchestrationConfigs: LangChainOrchestrationModuleConfigList = [
  {
    // Primary configuration — fails if the model is unavailable
    promptTemplating: {
      model: {
        name: 'gpt-5',
        timeout: 10 // seconds; triggers fallback on timeout (non-streaming only)
      }
    }
  },
  {
    // Fallback configuration
    promptTemplating: {
      model: {
        name: 'anthropic--claude-4.5-haiku'
      }
    }
  }
];

const response = await new OrchestrationClient(orchestrationConfigs).invoke([
  { role: 'user', content: 'Tell me about SAP Cloud SDK.' }
]);
```

Module fallback also works with the `stream()` method.
Note that in streaming scenarios, only model unavailability triggers fallback.
Unlike non-streaming requests, timeouts will not trigger fallback during streaming.

```ts
import type { AIMessageChunk } from '@langchain/core/messages';

const client = new OrchestrationClient(orchestrationConfigs);
const stream = await client.stream([
  { role: 'user', content: 'Tell me about SAP Cloud SDK.' }
]);

let finalOutput: AIMessageChunk | undefined;
for await (const chunk of stream) {
  finalOutput = finalOutput ? finalOutput.concat(chunk) : chunk;
}
```

### Streaming

The client supports streaming responses for chat completion requests.
Use the `stream()` method to receive a stream of chunk responses from the model.

By default, the last chunk contains the finish reason and token usage information.

:::warning
The orchestration service currently doesn't support multiple choices during streaming.
:::

```ts
const orchestrationConfig: LangchainOrchestrationModuleConfig = {
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
};

const client = new OrchestrationClient(orchestrationConfig);
const response = await client.stream([
  {
    role: 'user',
    content:
      'Write a 100 word explanation about SAP Cloud SDK and its capabilities'
  }
]);

let finalResult: AIMessageChunk | undefined;

for await (const chunk of response) {
  console.log(chunk.content);
  finalResult = finalResult ? finalResult.concat(chunk) : chunk;
}

console.log(finalResult?.response_metadata?.finish_reason);

console.log(finalResult?.usage_metadata);
/*
  { input_tokens: 13, output_tokens: 30, total_tokens: 43 }
*/

// Token usage is also available in `response_metadata` property
console.log(finalResult?.response_metadata?.token_usage);
/*
  { completion_tokens: 30, prompt_tokens: 13, total_tokens: 43 }
*/
```

#### Streaming with Abort Controller

The client supports aborting streaming requests using the `AbortController` API.
In case of an error, SAP Cloud SDK for AI will automatically close the stream.
It can also be manually aborted if an `AbortSignal` object associated with an `AbortController` was provided when calling the `stream()` method.

```ts
const orchestrationConfig: LangchainOrchestrationModuleConfig = {
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
};

const client = new OrchestrationClient(orchestrationConfig);
const controller = new AbortController();
const { signal } = controller;
const response = await client.stream(
  [
    {
      role: 'user',
      content:
        'Write a 100 word explanation about SAP Cloud SDK and its capabilities'
    }
  ],
  { signal }
);

// Abort the streaming request after one second
setTimeout(() => {
  controller.abort();
}, 1000);

for await (const chunk of response) {
  console.log(chunk.content);
}
```

In this example, streaming request will be aborted after one second.
Abort controller can be useful, e.g., when end-user wants to stop the stream or refreshes the page.

#### Streaming Configuration

By default, the `stream()` method supports streaming, while the `invoke()` method uses non-streaming requests.

##### Auto-Streaming

Set the `streaming` option to `true` to enable automatic streaming for all requests made with the `invoke()` method:

```ts
const client = new OrchestrationClient(orchestrationConfig, {
  streaming: true
});

const response = await client.invoke(messageHistory);
```

With auto-streaming enabled, the `invoke()` method uses streaming to retrieve responses, returning the complete result upon completion.
This option is automatically enabled by LangGraph when a LangGraph-based agent is used with a [streaming mode](https://docs.langchain.com/oss/javascript/langgraph/streaming#supported-stream-modes) that can output partial responses, such as "updates".

##### Disabling Streaming

Set the `disableStreaming` option to turn off streaming entirely.

```ts
const client = new OrchestrationClient(orchestrationConfig, {
  disableStreaming: true
});

const response = await client.stream(messageHistory);
```

When streaming is disabled, both the `invoke()` and `stream()` methods use non-streaming requests internally.
The `stream()` method will still return an iterable, but it will yield a single chunk containing the complete response.

### Tool Calling

LangChain offers a unified way to connect tools to language models.
Use the `bindTools()` method to define the set of tools a model can access.
For more details, see the [official LangChain documentation on tool binding](https://docs.langchain.com/oss/javascript/langchain/models#tool-calling).
For a usage example, refer to the [getting started with agents tutorial](../tutorials/getting-started-with-agents#define-tools).

### Basic Agent Usage

The `OrchestrationClient` can be used with LangChain's agent framework through the `createAgent()` function.

:::note
In LangChain v1, the function `createReactAgent` was replaced by `createAgent`.
:::

Create an agent by providing the orchestration client as the model, along with any tools you want the agent to use:

<!-- vale off -->

```ts
import { createAgent } from 'langchain';
import { OrchestrationClient } from '@sap-ai-sdk/langchain';

const model = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
});

const agent = createAgent({
  model,
  tools: []
});

const agentInputs = {
  messages: [{ role: 'user', content: 'What is SAP?' }]
};

const result = await agent.invoke(agentInputs);
```

<!-- vale on -->

### Structured Output

It is often useful to have a model return output that matches a specific schema.
This schema can be defined using either a [Zod](https://zod.dev/) schema or a JSON schema.
We recommend using Zod v4 for full compatibility with this package.
If you're upgrading from an earlier version, refer to the [Zod v4 migration guide](https://zod.dev/v4/changelog) and pay attention to breaking changes like the switch from `describe('...')` to `meta({ description: '...' })`.
For more details on structured output, refer to the [official LangChain documentation on structured output](https://docs.langchain.com/oss/javascript/langchain/models#structured-outputs).
Below is an example using `json_schema` response type and passing in a Zod schema.

```ts
import * as z from 'zod';
import { OrchestrationClient } from '@sap-ai-sdk/langchain';

const llm = new OrchestrationClient({
  promptTemplating: {
    model: {
      name: 'gpt-5'
    }
  }
});

const joke = z.object({
  setup: z.string().meta({ description: 'The setup of the joke' }),
  punchline: z.string().meta({ description: 'The punchline to the joke' }),
  rating: z
    .number()
    .meta({ description: 'How funny the joke is, from 1 to 10' })
});

const structuredLlm = llm.withStructuredOutput(joke, {
  name: 'joke',
  strict: true
});

const finalResponse = await structuredLlm.invoke('Tell me a joke about cats');
```

## Resilience

<!-- vale off -->

:::warning
The Langchain, Orchestration layer and the SAP Cloud SDK resilience mechanisms should not be combined, or should be integrated only with careful consideration.
:::

<!-- vale on -->

Use LangChain options such as `maxRetries` and `timeout` to provide resilience.

### Retry

By default, LangChain client retries up to six times with exponential delay.
To modify this behavior, set the `maxRetries` option during the client initialization.

```ts
const client = new OrchestrationClient(orchestrationConfig, {
  maxRetries: 0
});
```

:::note
If the error is caused by input content filtering, the client will throw immediately without retrying.
:::

### Timeout

By default, no timeout is set in the client.
To limit the maximum duration for the entire request including retries, specify a timeout in milliseconds when calling the `invoke` method.
A request that times out will be [retried](#retry) by default.

```ts
const response = await client.invoke(messageHistory, { timeout: 10000 });
```

Timeout can also be set for streaming requests.

```ts
const response = await client.stream(
  [
    {
      role: 'user',
      content: 'Hello world! Why is this phrase so famous?'
    }
  ],
  { timeout: 1000 }
);
```
