LLM-D基准测试

Prerequisites

Please refer to the official llm-d prerequisites for the most up-to-date requirements. For the client setup, the provided install.sh will install the necessary tools.

Administrative Requirements

Deploying the llm-d stack requires cluster-level admin privileges, as you will be configuring cluster-level resources. However, the scripts can be executed by namespace-level admin users, as long as the Kubernetes infrastructure components are configured and the target namespace already exists.

Prerequisites

Please refer to the official llm-d prerequisites for the most up-to-date requirements.

System Requirements

- Python 3.11+ - kubectl -- Kubernetes CLI - helm (>= 4.x) -- Helm package manager - curl, git -- Standard system tools - helmfile (>= 1.5) -- Required for modelservice deployments. Older helmfile is incompatible with Helm 4 (it probes helm with the removed helm version --client flag and panics). ./install.sh installs the pinned Helm 4 / helmfile combination for you. - kustomize, jq, yq -- Required for template rendering - skopeo, crane -- Required for container image management - oc (optional) -- Required for OpenShift clusters (either kubectl or oc must be present)

Administrative Requirements

[!IMPORTANT] Deploying the llm-d stack requires cluster-level admin privileges for configuring cluster-level resources. Namespace-level admin users can run the tool if Kubernetes infrastructure components are configured and the target namespace already exists. Use --non-admin to skip admin-only steps.

Dependencies

• llm-d-infra
• llm-d-modelservice
• inference-perf
• guidellm
• vllm
• inferencemax

Getting Started

Install

The install script supports both uv and the standard python3 -m venv for virtual environment creation. When run interactively, it will prompt you to choose; in non-interactive mode (e.g. curl pipe), it auto-selects uv if your system Python is missing or older than 3.11. You can also pass --uv or --no-uv to skip the prompt.

Quick install (one-liner):

curl -sSL https://raw.githubusercontent.com/llm-d/llm-d-benchmark/main/install.sh | bash
cd llm-d-benchmark
source .venv/bin/activate
llmdbenchmark --version

Or clone manually:

git clone https://github.com/llm-d/llm-d-benchmark.git
cd llm-d-benchmark
./install.sh              # or: --uv / --no-uv
source .venv/bin/activate
llmdbenchmark --version

Install a specific branch:

LLMDBENCH_BRANCH=main \
  curl -sSL https://raw.githubusercontent.com/llm-d/llm-d-benchmark/main/install.sh | bash

The install script auto-detects if the repo is present -- if not, it clones it first. It creates a virtualenv, validates system tools (kubectl, helm, Python 3.11+), and installs the llmdbenchmark package. See Installation for manual install and flags.

[!TIP] The last line of output from llmdbenchmark standup shows the workspace path where all rendered configs, manifests, and results are stored.

Deploy and benchmark (full pipeline)

Stand up the llm-d stack, run a quick sanity benchmark, and tear down:

```bash

Preview what would be deployed (no cluster changes)

llmdbenchmark --spec gpu --dry-run standup

Deploy for real

llmdbenchmark --spec gpu standup

Run a sanity benchmark against the deployed endpoint

llmdbenchmark --spec gpu run -l inference-perf -w sanity_random.yaml

Deploy multiple models behind one gateway

The multi-model-wva scenario deploys N models under a single gateway, each with its own EPP + InferencePool + VariantAutoscaling + HPA, sharing one WVA controller and one HTTPRoute with N backendRefs:

```bash

Standup - renders two stacks (qwen3-06b, llama-31-8b), installs shared

infra once, deploys a per-model Helm release + VA + HPA for each.

llmdbenchmark --spec guides/multi-model-wva standup -p my-namespace

See what's deployed: list detected endpoints + copy-paste run commands.

llmdbenchmark --spec guides/multi-model-wva run -p my-namespace --list-endpoints

infra (istio, Gateway, WVA controller, model PVC) installs normally,

Installation

Quick Install (recommended)

```bash

Manual Install w/o Install Script

git clone https://github.com/llm-d/llm-d-benchmark.git
cd llm-d-benchmark
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
pip install "git+https://github.com/llm-d-incubation/llm-d-planner.git@92b14fe09fea0ec9ff36539326b7a8df00f1022c"

Verify Installation

llmdbenchmark --version

[Deployment Methods](llmdbenchmark/standup/README.md#deployment-methods)

The standup phase supports two deployment paths:

• standalone -- Direct Kubernetes Deployments and Services for each model (step 06)
• modelservice -- Helm-based deployment with gateway infrastructure, GAIE, and LWS support (steps 07-09)

Both paths share steps 00-05 (infrastructure, namespaces, secrets) and step 10 (smoketest).

Benchmark qwen3-06b only with guidellm, two parallel harness pods

llmdbenchmark --spec guides/multi-model-wva run -p my-namespace \ --stack qwen3-06b \ -l guidellm \ -w sanity_random.yaml \ -j 2

Breakdown of the Example:

- `--stack qwen3-06b` filters per-stack steps to that pool. Endpoint
  detection (step 03) runs only for that stack and auto-resolves to
  `http://<gateway>:80/qwen3-06b` - including the routing prefix - so every
  downstream step targets the qwen3-06b InferencePool.
- `-l guidellm` selects the guidellm harness
  ([workload/harnesses/guidellm-llm-d-benchmark.sh](workload/harnesses/guidellm-llm-d-benchmark.sh)).
- `-j 2` launches two guidellm pods hitting the same endpoint. Both pods
  run the same treatment (`-w`) but write to distinct result
  subdirectories (`{experiment_id}_1`, `{experiment_id}_2`) on the
  workload PVC.

Want to compare pools side-by-side? Launch two invocations in parallel
shells (different `--workspace` each):

Example: set common defaults via env vars, override per-run via CLI

export LLMDBENCH_SPEC=inference-scheduling export LLMDBENCH_NAMESPACE=my-team-ns export LLMDBENCH_KUBECONFIG=~/.kube/my-cluster

Well-Lit Path Guides

llm-d-benchmark supports all available Well-Lit Path Guides. Each guide has a corresponding specification:

llmdbenchmark --spec inference-scheduling standup       # Inference scheduling
llmdbenchmark --spec pd-disaggregation standup          # Prefill-decode disaggregation
llmdbenchmark --spec tiered-prefix-cache standup        # Tiered prefix cache
llmdbenchmark --spec precise-prefix-cache-aware standup # Precise prefix cache-aware routing
llmdbenchmark --spec wide-ep-lws standup                # Wide expert-parallel with LWS

[!WARNING] wide-ep-lws requires RDMA/RoCE networking and LeaderWorkerSet (LWS) controller. Verify your cluster has working RDMA HCAs before deploying.

Terminal 1 - --workspace is a global option, placed before the subcommand

llmdbenchmark --spec guides/multi-model-wva --workspace /tmp/run-qwen run -p my-namespace \ --stack qwen3-06b \ -l guidellm -w sanity_random.yaml -j 2 &

Global Options

Plan Options

Standup Options

Teardown Options

Experiment Options

Run Options

Smoketest Options

Run post-deployment validation independently against an already-deployed stack.

llmdbenchmark --spec gpu smoketest -p my-namespace
llmdbenchmark --spec gpu smoketest -p my-namespace -s 2   # config validation only

Smoketests also run automatically after standup unless --skip-smoketest is passed. See llmdbenchmark/smoketests/README.md for details on what each step validates.

Environment Variables

Every CLI flag can be set via a LLMDBENCH_* environment variable (see tables above). The priority chain is:

1. CLI flag (highest) -- explicitly passed on the command line
2. Environment variable -- exported in the user's shell
3. Rendered config (lowest) -- defaults.yaml + scenario YAML

This is useful for CI/CD pipelines, .bashrc configuration, or migrating from the original bash-based workflow.

```bash

These use the env vars above; --dry-run overrides nothing, just adds a flag

llmdbenchmark standup --dry-run llmdbenchmark standup # live deploy to my-team-ns llmdbenchmark standup -p override-ns # CLI wins over env var ```

Boolean env vars accept 1, true, or yes (case-insensitive). Active LLMDBENCH_* overrides are logged at startup for debugging.

[Config Override Chain](config/README.md#config-override-chain)

Values flow through a merge pipeline during the plan phase:

Steps read from the rendered config.yaml and never define their own fallback defaults. If a required key is missing from the rendered config, the step raises a clear error. This ensures defaults.yaml is the single source of truth for all default values. Environment variables (LLMDBENCH_*) sit between scenario overrides and CLI flags in the priority chain.

See config/README.md for the full configuration reference, including how to override values.

Run - iterates every stack, each harness pod targets its own pool's endpoint.

llmdbenchmark --spec guides/multi-model-wva run -p my-namespace

Benchmark just one pool (no --endpoint-url needed - auto-resolves):

llmdbenchmark --spec guides/multi-model-wva run -p my-namespace \ --stack qwen3-06b \ -l inference-perf -w sanity_random.yaml

Benchmark an existing endpoint (run-only mode)

Already have a model-serving endpoint running? Skip deployment entirely:

llmdbenchmark --spec gpu run \
  --endpoint-url http://10.131.0.42:80 \
  --model meta-llama/Llama-3.1-8B \
  --namespace my-namespace \
  --harness inference-perf \
  --workload sanity_random.yaml

This uses the same harness, profile rendering, and result collection pipeline -- just without the standup and teardown phases.

[!TIP] run can also be used in debug mode (-d / --debug) which starts the harness pod with sleep infinity so you can exec into it and run commands interactively. See this example.

See workload/README.md for the full experiment file format and all pre-built experiments, as well as advanced functionality.

CLI Reference