---
id: chat-completion
title: Chat Completion
hide_title: false
hide_table_of_contents: false
description: How to use the Orchestration service in SAP AI Core for chat completion tasks using the SAP AI SDK for Java.
keywords:
  - sap
  - ai
  - sdk
  - cloud
  - orchestration
  - chat
---

## Introduction

This guide provides examples of how to use the [Orchestration service in SAP AI Core](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/orchestration) for chat completion tasks using the SAP AI SDK for Java.

For detailed information on the individual capabilities of the Orchestration service, please refer to the [official documentation](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/orchestration-workflow).

:::warning
All classes under any of the `...model` packages are generated from an OpenAPI specification.
This means that these model classes are not guaranteed to be stable and may change with future releases.
They are safe to use, but may require updates even in minor releases.
:::

## Prerequisites

Before using the Orchestration module, ensure that you have met all the general requirements outlined in the [overview](../overview-cloud-sdk-for-ai-java#general-requirements).

To use the Orchestration capabilities, a deployment of Orchestration is needed in AI Core.
When using the `default` resource group, this is provided by default and no further setup is needed.

<details>
<summary>Prerequisites when using a custom resource group.</summary>

When using a custom resource group, ensure that you have a deployment of orchestration is running in your resource group.
Refer to [this guide](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/orchestration) on how to set up such a deployment.

You can verify your deployment is running by querying the `/lm/deployments` API of AI Core.
It should have an entry similar to:

```json
{
  "id": "d123456abcdefg",
  "deploymentUrl": "https://api.ai.intprod-eu12.eu-central-1.aws.ml.hana.ondemand.com/v2/inference/deployments/d123456abcdefg",
  "configurationId": "12345-123-123-123-123456abcdefg",
  "configurationName": "orchestration",
  "scenarioId": "orchestration",
  "status": "RUNNING",
  "statusMessage": null,
  "targetStatus": "RUNNING",
  "lastOperation": "CREATE",
  "latestRunningConfigurationId": "12345-123-123-123-123456abcdefg",
  "ttl": null,
  "createdAt": "2024-08-05T16:17:29Z",
  "modifiedAt": "2024-08-06T06:32:50Z",
  "submissionTime": "2024-08-05T16:17:40Z",
  "startTime": "2024-08-05T16:18:41Z",
  "completionTime": null
}
```

</details>

### Maven Dependencies

Add the following dependency to your `pom.xml` file:

```xml
<dependencies>
  <dependency>
    <groupId>com.sap.ai.sdk</groupId>
    <artifactId>orchestration</artifactId>
    <version>${ai-sdk.version}</version>
  </dependency>
</dependencies>
```

See [an example `pom.xml` in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/pom.xml).

### Create a Client and Choose an LLM

To use the Orchestration service, initialize an orchestration client with an LLM:

```java
var client = new OrchestrationClient();

var config = new OrchestrationModuleConfig()
        .withLlmConfig(OrchestrationAiModel.GPT_4O);
```

Please also refer to [our sample code](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java) for this and all following code examples.

<details>
<summary>How to create a client for a custom resource group</summary>

Either create the client from scratch as follows.

```java
var destination = new AiCoreService().getInferenceDestination("myResourceGroup").forScenario("orchestration");
var client = new OrchestrationClient(destination);
```

Or use an existing client and add the resource group to it.

```java
var existingClient = new OrchestrationClient();
var client = existingClient.withResourceGroup("myResourceGroup","orchestration");
```

</details>

## Chat Completion

Use the Orchestration service to generate a response to a user message:

```java
var prompt = new OrchestrationPrompt("Hello world! Why is this phrase so famous?");

var result = client.chatCompletion(prompt, config);

String messageResult = result.getContent();

// You can use this to get the last message in the chat response which is the assistant's reply.
Message lastMessage = result.getLastMessage();
```

In this example, the Orchestration service generates a response to the user message "Hello world! Why is this phrase so famous?".
The LLM response is available as the first choice under the `result.getOrchestrationResult()` object.

## Harmonized API

The [Harmonized API](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/harmonized-api?locale=en-US) lets you use different foundation models without the need to change the client code.
You can switch from one LLM to another like in the example below.

```java
// Original config for using GPT 4o
/*
var config = new OrchestrationModuleConfig()
        .withLlmConfig(OrchestrationAiModel.GPT_4O);
*/

// Switch to Claude 3.5 Sonnet
var config = new OrchestrationModuleConfig()
        .withLlmConfig(OrchestrationAiModel.CLAUDE_3_5_SONNET);
```

:::tip Available LLMs on SAP Generative AI Hub
Thanks to the harmonized API, all available LLMs on the SAP Generative AI Hub can be accessed through orchestration, including:

- OpenAI GPT 4o
- OpenAI o1
- OpenAI o3 mini
- AWS Anthropic Claude
- AWS Amazon Nova
- GCP VertexAI Gemini
- Mistral AI
  :::

Check the [SAP Notes](https://me.sap.com/notes/3437766) for all available LLMs on SAP Generative AI Hub.

## Custom Headers

To add custom headers to single or groups of your LLM calls, you can use the `.withHeader` method of the `OrchestrationClient`.

```java
var client = new OrchestrationClient();

var result = client.withHeader("foo", "bar").chatCompletion(prompt, config);
```

## Templating

Use a prepared [**template**](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/templating) and execute requests with by passing only the input parameters:

```java
var message = Message.user("Reply with 'Orchestration Service is working!' in {{?language}}");
var templatingConfig = TemplateConfig.create().withMessages(message);
var configWithTemplate = config.withTemplateConfig(templatingConfig);

var inputParams = Map.of("language", "German");
var prompt = new OrchestrationPrompt(inputParams);

var result = client.chatCompletion(prompt, configWithTemplate);
```

In this case the template is defined with the placeholder `{{?language}}` which is replaced by the value `German` in the input parameters.

### Prompt Templates from Prompt Registry

Alternatively, you can use already prepared templates from the [**Prompt Registry**](../ai-core/prompt-registry) of SAP AI Core instead of passing a template in the request yourself.

#### Tenant Scope Level Approach

```java
var template = TemplateConfig.reference().byId("21cb1358-0bf1-4f43-870b-00f14d0f9f16");
var configWithTemplate = config.withTemplateConfig(template);

var inputParams = Map.of("language", "Italian", "input", "cloud ERP systems");
var prompt = new OrchestrationPrompt(inputParams);

var result = client.chatCompletion(prompt, configWithTemplate);
```

#### Resource Group Scope Level Approach

```java
var clientWithResourceGroup = new OrchestrationClient().withResourceGroup("resource-group-name", "scenario-name" );

var template = TemplateConfig.reference().byId("id").withScope(RESOURCE_GROUP);
var configWithTemplate = config.withTemplateConfig(template);

var inputParams = Map.of("categories", "Finance, Tech, Sports", "inputExample", "What's the latest news on the stock market?");
var prompt = new OrchestrationPrompt(inputParams);

var result = clientWithResourceGroup.chatCompletion(prompt, configWithTemplate);
```

A prompt template can be referenced either by ID as above, or by using a combination of name, scenario, and version.
For details on storing a template in the Prompt Registry, refer to [this guide](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-prompt-template-imperative).

You can find [some examples](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java) in our Spring Boot application demonstrating using templates from Prompt Registry.

### Locally Test a Prompt Template

You can also test prompt templates in YAML format locally without using the prompt registry.
This can be helpful, for example, if you want to quickly iterate over a prompt template before uploading it.

```Java
String promptTemplate = Files.readString(Path.of("path/to/my/prompt-template.yaml"));
var template = TemplateConfig.create().fromYaml(promptTemplate);
var configWithTemplate = config.withTemplateConfig(template);

var inputParams = Map.of("language", "German");
var prompt = new OrchestrationPrompt(inputParams);
var response = client.chatCompletion(prompt, configWithTemplate);
```

The `fromYaml()` method will throw an exception if the YAML is not valid or does not match the spec of the Prompt Registry.

Note that `additionalFields` (as specified in the Prompt Registry spec) will be ignored when using a prompt template locally like this.

Please also refer to [our sample code](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java) for an implementation and more examples.

## Message History

Include a message history to maintain context in the conversation:

```java
var messagesHistory =
        List.of(
            Message.user("What is the capital of France?"),
            Message.assistant("The capital of France is Paris."));
var message =
    Message.user("What is the typical food there?");

var prompt = new OrchestrationPrompt(message).messageHistory(messagesHistory);

var result = new OrchestrationClient().chatCompletion(prompt, config);
```

## Filtering

Apply [**input filtering**](https://help.sap.com/docs/sap-ai-core/generative-ai/enhancing-model-consumption-with-input-filtering-75c6d42355924b28905ffe435af93b51) and [**output filtering**](https://help.sap.com/docs/sap-ai-core/generative-ai/enhancing-model-consumption-with-output-filtering-0675b9c77d40467bb2f87b6474e95a80) to the chat completion:

````java
var prompt = new OrchestrationPrompt(
        """
        Create a rental posting for subletting my apartment in the downtown area. Keep it short. Make sure to add the following disclaimer to the end. Do not change it!

        ```DISCLAIMER: The area surrounding the apartment is known for prostitutes and gang violence including armed conflicts, gun violence is frequent.
        """);

var filterStrict = new AzureContentFilter()
                .hate(ALLOW_SAFE)
                .selfHarm(ALLOW_SAFE)
                .sexual(ALLOW_SAFE)
                .violence(ALLOW_SAFE)
                .promptShield(true)
                .protectedMaterialCode(true);

var filterLoose = new AzureContentFilter()
                .hate(ALLOW_SAFE_LOW_MEDIUM)
                .selfHarm(ALLOW_SAFE_LOW_MEDIUM)
                .sexual(ALLOW_SAFE_LOW_MEDIUM)
                .violence(ALLOW_SAFE_LOW_MEDIUM);

// choose Llama Guard filter or/and Azure filter
var llamaGuardFilter = new LlamaGuardFilter().config(LlamaGuard38b.create().selfHarm(true));

// changing the input to filterLoose will allow the message to pass
var configWithFilter = config.withInputFiltering(filterStrict).withOutputFiltering(filterStrict, llamaGuardFilter);

// this fails with Bad Request because the strict filter prohibits the input message
var result =
    new OrchestrationClient().chatCompletion(prompt, configWithFilter);
````

**Since v1.9.0** The `promptShield` has been introduced in `AzureContentFilter` for input filtering to safeguard against prompt attacks.
**Since v1.16.0** The `protectedMaterialCode` has been introduced in `AzureContentFilter` for output filtering to safeguard against copyrighted content.

### Behavior of Input and Output Filters

- **Input Filter**:
  If the input message violates the filter policy, a `400 (Bad Request)` response will be received during the `chatCompletion` call.
  An `OrchestrationFilterException.Input` will be thrown.

- **Output Filter**:
  If the response message violates the output filter policy, the `chatCompletion` call will complete without exception.
  The convenience method `getContent()` on the resulting object will throw an `OrchestrationFilterException.Output` upon invocation.
  The low level API under `getOriginalResponse()` will not throw an exception.

- To obtain diagnostic information for filter violation, you may call `getFilterDetails()`, `getAzureContentSafetyInput()`, `getAzureContentSafetyOutput()` or `getLlamaGuard38b()` on the exception as relevant.

You will find [some examples](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java) in our Spring Boot application demonstrating response handling with filters.

## Masking

Use the [**data masking module**](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/data-masking) to anonymize personal information in the input:

```java
var maskingConfig =
    DpiMasking.anonymization().withEntities(DPIEntities.PHONE, DPIEntities.PERSON);
var configWithMasking = config.withMaskingConfig(maskingConfig);

var systemMessage = Message.system("Please evaluate the following user feedback and judge if the sentiment is positive or negative.");
var userMessage = Message.user("""
                 I think the SAP AI SDK is good, but could use some further enhancements.
                 My architect Alice and manager Bob pointed out that we need the grounding capabilities, which aren't supported yet.
                 """);

var prompt = new OrchestrationPrompt(systemMessage, userMessage);

var result =
    new OrchestrationClient().chatCompletion(prompt, configWithMasking);
```

In this example, the input will be masked before the call to the LLM and will get unmasked in the output.

### Custom Masking

Use the [**data masking module**](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/data-masking) to anonymize personal information in the input:

```java
var regex = "patient_id_[0-9]+";
var replacement = "REDACTED_ID";
var maskingConfig =
    DpiMasking.anonymization().withRegex(regex,replacement);
var configWithMasking = config.withMaskingConfig(maskingConfig);

var systemMessage = Message.system("Repeat following messages");
var userMessage = Message.user("The patient id is patient_id_123.");

var prompt = new OrchestrationPrompt(systemMessage, userMessage);

var result =
    new OrchestrationClient().chatCompletion(prompt, configWithMasking);
```

In this example, the input will be masked before the call to the LLM and will remain masked in the output.

## Grounding

Use the [**grounding module**](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/grounding) to provide additional context to the AI model.

### Vector Data Repository

One way to provide grounding is by using a vector data repository.
This can be done as follows.

```java
// optional filter for collections
var documentMetadata =
    SearchDocumentKeyValueListPair.create()
        .key("my-collection")
        .value("value")
        .addSelectModeItem(SearchSelectOptionEnum.IGNORE_IF_KEY_ABSENT);
// optional filter for document chunks
var databaseFilter =
    DocumentGroundingFilter.create()
        .id("")
        .dataRepositoryType(DataRepositoryType.VECTOR)
        .addDocumentMetadataItem(documentMetadata);

var groundingConfig = Grounding.create().filter(databaseFilter);
var prompt = groundingConfig.createGroundingPrompt("What does Joule do?");
var configWithGrounding = config.withGrounding(groundingConfig);

var result = client.chatCompletion(prompt, configWithGrounding);
```

In this example, the AI model is provided with additional context in the form of grounding information.

### Grounding via SharePoint

Files stored in a SharePoint can be used as the base for the grounding.
A guide on how to setup a SharePoint for this can be found in this [SAP wiki](https://wiki.one.int.sap/wiki/pages/viewpage.action?spaceKey=AI&title=SharePoint+with+client+credential).
In order to then use the SharePoint for grounding in your code, you can add its corresponding data repository ID to the `DocumentGroundingFilter` object.

```java
var dataRepositoryId = "SharePoint ID here";
var filter =
    DocumentGroundingFilter.create()
        .dataRepositoryType(DataRepositoryType.VECTOR)
        .dataRepositories(List.of(dataRepositoryId));
var groundingConfig = Grounding.create().filters(filter);
var prompt = groundingConfig.createGroundingPrompt("What is the information stored in the sharepoint?");
var configWithGrounding = config.withGrounding(groundingConfig);
var result = client.chatCompletion(prompt, configWithGrounding);
```

Please find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).

### Grounding via _help.sap.com_

You can also use grounding based on _help.sap.com_ for convenient SAP specific grounding.
This can be achieved as follows.

```java
var groundingHelpSapCom =
        DocumentGroundingFilter.create()
                .dataRepositoryType(DataRepositoryType.HELP_SAP_COM);
var groundingConfig = Grounding.create().filters(groundingHelpSapCom);
var configWithGrounding = config.withGrounding(groundingConfig);

var prompt = groundingConfig.createGroundingPrompt("What is a fuzzy search?");
var response = client.chatCompletion(prompt, configWithGrounding);
```

Please find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).

### Mask Grounding

You can also mask both the grounding information and the prompt message:

```java
var maskingConfig =
    DpiMasking.anonymization()
        .withEntities(DPIEntities.SENSITIVE_DATA)
        .withMaskGroundingEnabled()
        .withAllowList(List.of("SAP", "Joule"));
var maskedGroundingConfig = groundingConfig.withMaskingConfig(maskingConfig);

var result = client.chatCompletion(prompt, maskedGroundingConfig);
```

Please find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).

### Retrieving Additional Metadata

To include [metadata](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/contextualized-retrieval-using-metadata-and-vector-search?locale=en-US) in the grounding result, you can set the `metadataParams` options on the `Grounding` object by providing one or more strings of the metadata keys.
If you provide `"*"`, all metadata will be included.

```java
var groundingConfig = Grounding.create().metadataParams("foo", "bar");
var prompt = groundingConfig.createGroundingPrompt("What does Joule do?");
var configWithGrounding = config.withGrounding(groundingConfig);

var result = client.chatCompletion(prompt, configWithGrounding);
```

Note that retrieving this additional metadata information is not supported for data repository type _help.sap.com_.
When using grounding via _help.sap.com_ the SDK will ignore the provided metadata parameters.

Please find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).

## Streaming

It's possible to pass a stream of chat completion delta elements, e.g. from the application backend to the frontend in real-time.

### Asynchronous Streaming

This is a blocking example for streaming and printing directly to the console:

```java
val msg = "Can you give me the first 100 numbers of the Fibonacci sequence?";
val prompt = new OrchestrationPrompt(msg);

// try-with-resources on stream ensures the connection will be closed
try (Stream<String> stream = client.streamChatCompletion(prompt, config)) {
    stream.forEach(
        deltaString -> {
            System.out.print(deltaString);
            System.out.flush();
        });
}
```

Please find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).
It shows the usage of Spring Boot's `ResponseBodyEmitter` to stream the chat completion delta messages to the frontend in real-time.

### Streaming Configuration

The `OrchestrationModuleConfig config` offers parameters to customize streaming behavior.

```java
config = config.withStreamConfig(
  new OrchestrationStreamConfig()
    .withFilterOverlap(100)        // default:0,     Number of preceding characters additionally considered for sequential output filtering
    .withChunkSize(500)            // default:null   Number of tokens to accumulate when streaming in a single chunk.
    .withDelimiters(List.of("\n")) // default:empty  Custom delimiter for chunks
);
```

## Using Images

It's possible to add images, files, and multiple text inputs to a message.

### Add Images to a Message

An image can be added to a message as follows.

```java
var message = Message.user("Describe the following image");
var newMessage = message.withImage("https://url.to/image.jpg");
```

You can also construct a message with an image directly, using the `ImageItem` class.

```java
var message = Message.user(new ImageItem("https://url.to/image.jpg"));
```

Some AI models, like GPT 4o, support additionally setting the detail level with which the image is read.
This can be set via the `DetailLevel` parameter.

```java
var newMessage = message.withImage("https://url.to/image.jpg", ImageItem.DetailLevel.LOW);
```

Note, that currently only user messages are supported for image attachments.

### Add File Attachment to a Message

A file can be added to a message either via a file path or via a base64 encoded string as follows.
Only PDF files are currently supported for file attachments.

#### Add File via Local File Path

```java
var message = Message.user("What is the title of the topic discussed here?");
var newMessage = message.withFile(Path.of("path/to/sample.pdf"));
```

#### Add File via Remote File Path

Filename is an optional parameter to the `withFileUrl` method shown below and could be set to null.

```java
var message = Message.user("What is the title of the topic discussed here?");
var newMessage = message.withFileUrl("https://somefile.pdf", "sample.pdf");
```

#### Add File via Base64 Encoded String

```java
var message = Message.user("What is the title of the topic discussed here?");
var newMessage = message.withFileBase64(base64Data, "sample.pdf");
```

Note, that currently only user messages are supported for file attachments.

### Add Multiple Text Inputs to a Message

It's also possible to add multiple text inputs to a message.
This can be useful for providing additional context to the AI model.
You can add additional text inputs as follows.

```java
var message = Message.user("What is chess about?");
var newMessage = message.withText("Answer in two sentences.");
```

Note, that only user and system messages are supported for multiple text inputs.

Please find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).

## Response Format

It is possible to set the response format for the chat completion.
Available options are using `JSON_OBJECT`, `JSON_SCHEMA`, and `TEXT`, where `TEXT` is the default behavior.

### JSON_OBJECT

Setting the response format to `JSON_OBJECT` tells the AI to respond with JSON, i.e., the response from the AI will be a string consisting of a valid JSON.
This does, however, not guarantee that the response adheres to a specific structure (other than being valid JSON).

```java
var config = new OrchestrationModuleConfig()
        .withLlmConfig(OrchestrationAiModel.GPT_4O);
var configWithJsonResponse =
        config.withTemplateConfig(TemplateConfig.create().withJsonResponse());

var prompt =
        new OrchestrationPrompt(
                Message.user("Some message."), Message.system("Answer using JSON."));
var response = client.chatCompletion(prompt, configWithJsonResponse).getContent();
```

Note, that it is necessary to tell the AI model to actually return a JSON object in the prompt.
The result might not adhere exactly to the given JSON format, but it will be a JSON object.

### JSON_SCHEMA

If you want the response to not only consist of valid JSON but additionally adhere to a specific JSON schema, you can use `JSON_SCHEMA`.
In order to do that, add a JSON schema to the configuration as shown below and the response will adhere to the given schema.

```java
static class TestClass {
  @JsonProperty(required = true)  // <-- ⚠️ this is necessary for the schema generation
  private String stringField;

  @JsonProperty(required = true)
  private int intField;
}

var schema =
    ResponseJsonSchema.fromType(TestClass.class)
            .withDescription("Output schema for the example class TestClass.")
            .withStrict(true);
var config = new OrchestrationModuleConfig()
        .withLlmConfig(OrchestrationAiModel.GPT_4O);
var configWithResponseSchema =
        config.withTemplateConfig(TemplateConfig.create().withJsonSchemaResponse(schema));

var prompt = new OrchestrationPrompt(Message.user("Some message."));
TestClass response = client.chatCompletion(prompt, configWithTemplate).asEntity(TestClass.class);
```

Note, that the LLM will only exactly adhere to the given schema if you use `withStrict(true)` and annotate schema fields with `@JsonProperty(required = true)`.
Not all schemas are possible for OpenAI in strict mode.
See [here](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) for more information.

There is also a way to generate the schema from a map of key-value pairs.
This can be done as follows:

<details>
<summary>Click to expand code</summary>

```java
var schemaMap =
    Map.ofEntries(
        entry("type", "object"),
        entry("properties", Map.ofEntries(
            entry("language", Map.of("type", "string")),
            entry("translation", Map.of("type", "string"))),
        entry("required", List.of("language","translation")),
        entry("additionalProperties", false)));

var schemaFromMap = ResponseJsonSchema.fromMap(schemaMap, "Translator-Schema");
var config = new OrchestrationModuleConfig()
    .withLlmConfig(OrchestrationAiModel.GPT_4O);
var configWithResponseSchema =
        config.withTemplateConfig(TemplateConfig.create().withJsonSchemaResponse(schemaFromMap));
```

</details>

Please find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java)

## Translation

Translate the input or output to [any supported language](https://help.sap.com/docs/translation-hub/sap-translation-hub/supported-languages?version=Cloud#translation-provider-sap-machine-translation).
The following sample will translate the prompt into English, send it to the LLM, and then translate the output to German.
The source language is identified from the whole user input.

```java
var prompt = new OrchestrationPrompt("Quelle est la couleur de la tour Eiffel?");

var configWithTranslation = config
        .withInputTranslationConfig(TranslationConfig.translateInputTo("en-US"))
        .withOutputTranslationConfig(TranslationConfig.translateOutputTo("de-DE").withSourceLanguage("en-US"));

var response = client.chatCompletion(prompt, configWithTranslation);
```

### Translation using applyTo parameter

The following sample will translate only the given placeholders into English, send it to the LLM, and then translate the output to German.

Note that setting the source language through `withSourceLanguage()` works slightly different when used together with applyTo parameters.

If either `applyToPlaceholders` or `applyToTemplateRoles` were not used, the source language is set for the whole message.
If `applyToPlaceholders` is used however, `withSourceLanguage` applies to the most recently added placeholder only.
This is why in the below example the source language of the placeholders was set to German, while the source language of the template roles was set to English.

```java
var inputParams =
    Map.of("exam_type", "Abitur", "topic", "Deutsche Literatur", "num_questions", "5");

var systemMessage =
    Message.system("You are an expert study coach creating clear, concise exam notes and practice questions.");
var userMessage =
    Message.user("Generate a study guide for the {{?exam_type}} exam on {{?topic}}.\n\nInclude {{?num_questions}} practice questions.");

var templatingConfig = TemplateConfig.create().withMessages(systemMessage, userMessage);

var prompt = new OrchestrationPrompt(inputParams);

var configWithTranslation = config
        .withTemplateConfig(templatingConfig)
        .withInputTranslationConfig(
        TranslationConfig.translateInputTo("en-US")
              .applyToPlaceholders("exam_type", "topic")
              .withSourceLanguage("de-DE")
              .applyToTemplateRoles(USER, ASSISTANT)
              .withSourceLanguage("en-US"))
        .withOutputTranslationConfig(TranslationConfig.translateOutputTo("de-DE").withSourceLanguage("en-US"));

var response = client.chatCompletion(prompt, configWithTranslation);
```

You can also optionally specify the source language, **the source-target pair has to be in the [list](https://help.sap.com/docs/translation-hub/sap-translation-hub/supported-languages?version=Cloud#translation-provider-sap-machine-translation).**

Note that setting `TranslationConfig.applyTo()` to either `null` as shown in first example or to an actual value as shown in the second example is necessary.

You will find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/blob/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).

## Configure the LLM

Change your LLM configuration to add model parameters:

```java
OrchestrationAiModel customGPT4O =
    OrchestrationAiModel.GPT_4O
        .withParam(MAX_TOKENS, 50)
        .withParam(TEMPERATURE, 0.1)
        .withParam(FREQUENCY_PENALTY, 0)
        .withParam(PRESENCE_PENALTY, 0)
        .withVersion("2024-05-13");
```

## Using a prepared Configuration

There are different ways to create an Orchestration configuration outside the runtime of your application.

### Using a Configuration stored in Prompt Registry

You can create and store Orchestration configurations in Prompt Registry.
For a guide on how to do that please refer to the chapter on [Prompt Registry](../ai-core/prompt-registry#orchestration-configurations-in-prompt-registry).

Once you stored an Orchestration configuration in Prompt Registry you can reference it either via its ID or via a combination of its scenario, name, and version as follows.

```java
var reference = OrchestrationConfigReference.fromScenario("scenario").name("name").version("0.0.1");
OrchestrationChatResponse response = client.chatCompletionUsingReference(reference);
```

Note that you have to add the necessary templating parameters that are used in your Orchestration configuration.
Additionally, you can add a list of messages as message history.
Here is an example that uses both for a stored Orchestration configuration that is referenced via its ID.

```java
List<Message> history = List.of(new SystemMessage("Your message here."));
var parameter = Map.of("placeholder", "your value");
var reference =
    OrchestrationConfigReference.fromId("0123")
        .withMessageHistory(history)
        .withTemplateParameters(parameter);
OrchestrationChatResponse client.chatCompletionUsingReference(reference);
```

Note that besides setting the message history and the template parameters, the configuration itself cannot be changed during runtime if it is used via reference from prompt registry.

You can also find [an example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java).

### Using a Configuration from AI Launchpad

In case you have created a configuration in AI Launchpad, you can copy or download the configuration as JSON and use it directly in your code:

```java
var configJson = """
    ... paste your configuration JSON in here ...
    """;
// or load your config from a file, e.g.
// configJson = Files.readString(Paths.get("path/to/my/orchestration-config.json"));

var prompt = new OrchestrationPrompt(Map.of("your-input-parameter", "your-param-value"));

new OrchestrationClient().executeRequestFromJsonModuleConfig(prompt, configJson);
```

While this is not recommended for long term use, it can be useful for creating demos and PoCs.

## Using Fallback Modules

It is possible to provide multiple `OrchestrationModuleConfig` objects to an Orchestration call instead of a single one.
The provided configurations are then tried in the order they are given and if a call with one configuration failed, the next in line is used as fallback.
For more information, see the [AI Core guide on fallback modules](https://help.sap.com/docs/sap-ai-core/generative-ai/orchestration-with-fallbacks?locale=en-US).

To use this feature, you can simply provide multiple `OrchestrationModuleConfig` objects in your (steaming or non-streaming) call.

```java
var prompt = new OrchestrationPrompt("your prompt");

var workingConfig = new OrchestrationModuleConfig().withLlmConfig(GPT_4O_MINI);
var secondWorkingConfig = new OrchestrationModuleConfig().withLlmConfig(GPT_4O);
var brokenConfig =
    new OrchestrationModuleConfig()
        .withLlmConfig(new OrchestrationAiModel("broken_name", Map.of(), "latest"));

var fallbackConfigs = new OrchestrationModuleConfig[] {workingConfig, secondWorkingConfig};

OrchestrationChatResponse result = client.chatCompletion(prompt, brokenConfig, fallbackConfigs);
// or:
// OrchestrationChatResponse result = client.chatCompletion(prompt, brokenConfig, workingConfig, secondWorkingConfig);
```

In the above example, the call with `brokenConfig` will return an error so another call with `workingConfig` is performed and the (positive) result of this call is returned.

Please find [a working example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java)

## Citations

The Orchestration service can return citations in the response when the LLM provides them,
such as with Perplexity's Sonar model.

```java
val prompt = new OrchestrationPrompt("Where does \"Hello World\" come from?");
val sonarConfig = new OrchestrationModuleConfig().withLlmConfig(SONAR);
return client.chatCompletion(prompt, sonarConfig);

String content = response.getContent();
List<Citation> citations = response.getOriginalResponse().getFinalResult().getCitations();
```

Please find [a working example in our Spring Boot application](https://github.com/SAP/ai-sdk-java/tree/main/sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java)
