---
title: "E-Commerce Customer Service Agent"
description: "End-to-end example of using ArkSim to simulate and evaluate an e-commerce shopping assistant agent with product recommendation and order handling scenarios."
---

## Overview

This example walks through running ArkSim against a shopping assistant agent built for an e-commerce use case. The agent is designed to help customers navigate product discovery, orders, returns, and general shopping queries.

The example includes two ready-to-run agent setups you can test against out of the box, and a guide for plugging in your own agent once you're familiar with the setup.

---

## Example Agents

You can run the example in two ways: **Option 1** uses the OpenAI API directly (no server, minimal setup). **Option 2** runs a local agent built with the OpenAI Agents SDK (e-commerce knowledge base) via A2A, Chat Completions, or a custom agent connector.

<CardGroup cols={2}>
  <Card icon="bolt" title="Option 1: OpenAI API">
    A lightweight agent that calls the OpenAI API directly. Quick start with minimal setup.
  </Card>
  <Card icon="diagram-project" title="Option 2: OpenAI Agents SDK">
    Agent built with the OpenAI Agents SDK, backed by an e-commerce knowledge base; supports A2A, Chat Completions, or custom agent connector.
  </Card>
</CardGroup>

---

## Scenarios

The example ships with a set of pre-built scenarios in `scenarios.json` representing realistic e-commerce customer interactions. Each scenario defines a simulated user with a distinct persona, goal, and background knowledge drawn from product and policy documentation.

Sample goals include:

- Asking about product availability, specifications, and delivery timelines
- Checking order status and tracking a shipment
- Initiating or following up on a return or refund request
- Comparing products and asking for recommendations

Scenarios are defined in `scenarios.json` in the example directory and can be edited or extended to reflect your own use case.

---

<Note>
  Before following either path, ensure [ArkSim is installed](./installation) (`pip install arksim`).
</Note>

## Option 1: OpenAI Agent

This agent calls the OpenAI API directly, with no server setup required.

<Steps>
  <Step title="Set your API key">
    ```bash
    export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
    ```
  </Step>
  <Step title="Run simulation and evaluation">
    Run from the `examples/e-commerce` directory:

    ```bash
    cd examples/e-commerce
    arksim simulate-evaluate config.yaml
    ```
  </Step>
</Steps>

---

## Option 2: OpenAI Agents SDK

This agent is a shopping assistant built with the OpenAI Agents SDK, backed by an e-commerce knowledge base. It can be exposed via the A2A Protocol, a Chat Completions-compatible endpoint, or loaded directly as a Python class.

<Steps>
  <Step title="Select agent config">
    In the example directory, use the config file for your chosen interface:

    - **A2A:** `config_a2a.yaml` (inline agent config; uses `${A2A_API_KEY}`)
    - **Chat Completions:** `config_chat_completions.yaml` (inline agent config; uses `${AGENT_API_KEY}`)
    - **Custom agent connector:** `config_custom.yaml` (loads agent directly as a Python class — no server needed)

    Set the matching environment variable before running.
  </Step>
  <Step title="Install agent dependencies">
    <Tabs>
      <Tab title="uv">
        ```bash
        uv venv --python 3.11
        source .venv/bin/activate
        uv pip install -r examples/e-commerce/agent_server/requirements.txt
        ```
      </Tab>
      <Tab title="conda">
        ```bash
        conda create -n ecommerce_venv python=3.11 -y
        conda activate ecommerce_venv
        pip install -r examples/e-commerce/agent_server/requirements.txt
        ```
      </Tab>
    </Tabs>
  </Step>
  <Step title="Start the agent server">
    <Tabs>
      <Tab title="A2A">
        Exposes an A2A-compatible agent on port `9999`. Run from the repository root:

        ```bash
        export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
        export A2A_API_KEY=1234-4567-8910
        python -m examples.e-commerce.agent_server.a2a.server
        ```

        Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/a2a/server.py` under the e-commerce example.
      </Tab>
      <Tab title="Chat Completions">
        Exposes a Chat Completions–compatible endpoint on port `8888` at `/chat/completions`. Run from the repository root:

        ```bash
        export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
        export AGENT_API_KEY=123456
        python -m examples.e-commerce.agent_server.chat_completions.server
        ```

        Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/chat_completions/server.py` under the e-commerce example.
      </Tab>
      <Tab title="Custom">
        No server needed. The custom agent connector loads `custom_agent.py` directly as a Python class at runtime. Skip this step and proceed to run simulation.
      </Tab>
    </Tabs>
  </Step>
  <Step title="Run simulation and evaluation">
    In a **new terminal**, from `examples/e-commerce`, run the commands for the same interface you used in step 3. Use the same API key you set there.

    <Tabs>
      <Tab title="A2A">
        ```bash
        cd examples/e-commerce
        export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
        export A2A_API_KEY=1234-4567-8910
        arksim simulate-evaluate config_a2a.yaml
        ```
      </Tab>
      <Tab title="Chat Completions">
        ```bash
        cd examples/e-commerce
        export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
        export AGENT_API_KEY=123456
        arksim simulate-evaluate config_chat_completions.yaml
        ```
      </Tab>
      <Tab title="Custom">
        No separate server terminal is needed. From `examples/e-commerce`:

        ```bash
        cd examples/e-commerce
        export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
        arksim simulate-evaluate config_custom.yaml
        ```

        Or run the full pipeline as a Python script:

        ```bash
        export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
        python run_pipeline.py
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

---

## Running with Your Own Agent

To test your own backend agent against these scenarios:

- **Chat Completions:** Follow the comments in `agent_server/chat_completions/server.py` to swap in your own backend logic or point to your endpoint.
- **A2A:** Implement your own A2A executor in `agent_server/a2a/agent_executor.py`.
- **Custom:** Subclass `BaseAgent` and point `config_custom.yaml` at your module. See [Custom agent configuration](./simulate-conversation#connection-types) for details.

Make sure the `agent_config` field in the config YAML is updated. Then run simulation and evaluation as above.

<Tip>
  See [Agent configuration](./simulate-conversation#agent-configuration) for supported protocols and how to configure agent connections.
</Tip>

---

## Configuration

The example uses a single config file for both simulation and evaluation.

```yaml
# AGENT CONFIGURATION

agent_config:
  agent_type: chat_completions
  agent_name: e-commerce
  api_config:
    endpoint: https://api.openai.com/v1/chat/completions
    headers:
      Content-Type: application/json
      Authorization: "Bearer ${OPENAI_API_KEY}"
    body:
      model: gpt-5.1
      messages:
        - role: system
          content: >-
            You are a shopping assistant. Please provide a complete answer
            to the user's question based on your knowledge. Response within
            50 words.

# SIMULATION SETTINGS

# Path to the scenarios file
scenario_file_path: ./scenarios.json

# Number of conversations per scenario to generate
num_conversations_per_scenario: 1

# Maximum turns per conversation
max_turns: 5

# Output file path for simulation results
output_file_path: ./results/simulation/simulation.json

# Jinja template for the simulated user's system prompt
simulated_user_prompt_template: null

# EVALUATION SETTINGS

# Output directory for evaluation results
output_dir: ./results/evaluation

# Paths to Python files defining custom
# QuantitativeMetric or QualitativeMetric subclasses
custom_metrics_file_paths:
  - ./custom_metrics.py

# Built-in metrics to run; if empty, all built-in metrics run
metrics_to_run:
  - faithfulness
  - helpfulness
  - coherence
  - verbosity
  - relevance
  - goal_completion
  - agent_behavior_failure

# Generate HTML report
generate_html_report: true

# Numeric thresholds: per-metric minimum scores on native scale
# numeric_thresholds:
#   overall_score: 0.7
#   goal_completion: 0.6

# SHARED SETTINGS

# LLM model
model: gpt-5.1

# LLM provider
provider: openai

# Workers for parallel processing
num_workers: 50
```

---

## Output

Results are written under the example directory:

| Location                                 | Contents                                                                 |
| ---------------------------------------- | ------------------------------------------------------------------------ |
| `./results/simulation/simulation.json`   | Simulated conversations from the simulation step                         |
| `./results/evaluation/evaluation.json`   | Evaluation results (per-turn and per-conversation scores, unique errors) |
| `./results/evaluation/final_report.html` | Interactive HTML report for browsing and sharing results                 |
