---
id: orchestration
title: Orchestration Integration
hide_title: false
hide_table_of_contents: false
description: How to integrate orchestration client of SAP Cloud SDK for AI into LangChain.
keywords:
  - sap
  - cloud
  - sdk
  - ai
  - langchain
  - orchestration
---

The `@sap-ai-sdk/langchain` packages provides `OrchestrationClient` client for LangChain integration with orchestration service.

## Client Initialization

The client reuses the orchestration client from `@sap-ai-sdk/orchestration` and implements [LangChain's interface](https://js.langchain.com/docs/introduction).
Therefore, the client initialization combines the configuration of the orchestration client and LangChain options.

Similar to the orchestration client, the `OrchestrationClient` LangChain client can be initialized with an orchestration configuration.
Refer to [Orchestration Chat Completion](../orchestration/chat-completion) for more information about the configuration.

```ts
import { OrchestrationClient } from '@sap-ai-sdk/langchain';

const config: OrchestrationModuleConfig = {
  llm: {
    model_name: 'gpt-4o'
  }
};
const client = new OrchestrationClient(config);
```

Optionally, you can also specify LangChain options, resource group in deployment config, and a custom destination.

### Custom Destination

When initializing the client, it is possible to provide a custom destination for your SAP AI Core instance.

```ts
const client = new OrchestrationClient(
  orchestrationConfig,
  langchainOptions,
  deploymentConfig,
  { destinationName: 'my-destination' }
);
```

By default, the fetched destination is cached.
To disable caching, set the `useCache` parameter to `false` together with the `destinationName` parameter.

For more information about configuring a destination, refer to the [Using a Destination](../connecting-to-ai-core#using-a-destination) section.

## Chat Completion

Pass a message history and, in most cases, input parameters for the orchestration templating module.

```ts
const systemMessage = new SystemMessage('Be a helpful assistant!');
const history = [systemMessage];
const response = await client.invoke(history, {
  inputParams: { subject: 'Paris' }
});
```

### Streaming

The client supports streaming responses for chat completion requests.
Use the `stream()` method to receive a stream of chunk responses from the model.

By default, the last chunk contains the finish reason and token usage information.

:::warning
The orchestration service currently doesn't support multiple choices during streaming.
:::

```ts
const orchestrationConfig: LangchainOrchestrationModuleConfig = {
  llm: {
    model_name: 'gpt-4o'
  }
};

const client = new OrchestrationClient(orchestrationConfig);
const response = await client.stream([
  {
    role: 'user',
    content:
      'Write a 100 word explanation about SAP Cloud SDK and its capabilities'
  }
]);

let finalResult: AIMessageChunk | undefined;

for await (const chunk of response) {
  console.log(chunk.content);
  finalResult = finalResult ? finalResult.concat(chunk) : chunk;
}

console.log(finalResult?.response_metadata?.finish_reason);

console.log(finalResult?.usage_metadata);
/*
  { input_tokens: 13, output_tokens: 30, total_tokens: 43 }
*/

// Token usage is also available in `response_metadata` property
console.log(finalResult?.response_metadata?.token_usage);
/*
  { completion_tokens: 30, prompt_tokens: 13, total_tokens: 43 }
*/
```

#### Streaming with Abort Controller

The client supports aborting streaming requests using the `AbortController` API.
In case of an error, SAP Cloud SDK for AI will automatically close the stream.
It can also be manually aborted if an `AbortSignal` object associated with an `AbortController` was provided when calling the `stream()` method.

```ts
const orchestrationConfig: LangchainOrchestrationModuleConfig = {
  llm: {
    model_name: 'gpt-4o'
  }
};

const client = new OrchestrationClient(orchestrationConfig);
const controller = new AbortController();
const { signal } = controller;
const response = await client.stream([
  {
    role: 'user',
    content:
      'Write a 100 word explanation about SAP Cloud SDK and its capabilities'
  },
  { signal }
]);

// Abort the streaming request after one second
setTimeout(() => {
  controller.abort();
}, 1000);

for await (const chunk of response) {
  console.log(chunk.content);
}
```

In this example, streaming request will be aborted after one second.
Abort controller can be useful, e.g., when end-user wants to stop the stream or refreshes the page.

### Tool Calling

LangChain offers a unified way to connect tools to language models.
Use the `bindTools()` method to define the set of tools a model can access.
For more details, see the [official LangChain documentation on tool binding](https://js.langchain.com/docs/concepts/tool_calling/#tool-binding).
For a usage example, refer to the [getting started with agents tutorial](../tutorials/getting-started-with-agents#define-tools).

## Resilience

Use LangChain options such as `maxRetries` and `timeout` to provide resilience.

### Retry

By default, LangChain client retries up to six times with exponential delay.
To modify this behavior, set the `maxRetries` option during the client initialization.

```ts
const client = new OrchestrationClient(orchestrationConfig, {
  maxRetries: 0
});
```

:::note
If the error is caused by input content filtering, the client will throw immediately without retrying.
:::

### Timeout

By default, no timeout is set in the client.
To limit the maximum duration for the entire request including retries, specify a timeout in milliseconds when calling the `invoke` method.
A request that times out will be [retried](#retry) by default.

```ts
const response = await client.invoke(messageHistory, { timeout: 10000 });
```

Timeout can also be set for streaming requests.

```ts
const response = await client.stream(
  [
    {
      role: 'user',
      content: 'Hello world! Why is this phrase so famous?'
    }
  ],
  { timeout: 1000 }
);
```
