Prerequisites

• Node 18+
• Go 1.25 (required: protoc-gen-go, protoc-gen-go-grpc)
• Conda
• Python 3.10+
• FFmpeg
• libopus-dev、libopusfile-dev、libsoxr-dev，pkg-config

For pure voice sessions, no local avatar GPU is required. Runtime cost depends on the realtime voice/omni/LLM/TTS/ASR providers you configure.

To verify, use:

node --version
go version
protoc --version
ffmpeg -version
conda --version

Step 5: Install project dependencies

make setup

This installs the base editable package ([dev,inference]), generates gRPC stubs, and installs frontend dependencies.

Install the voice-agent extras used by the default config:

```bash

Additional Requirements

• GPU with CUDA 12.8+
• PyTorch 2.8 (CUDA 12.8)
• FFmpeg with libvpx for video encoding
• Avatar model weights

Install PyTorch (CUDA 12.8):

pip3 install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128

Install vllm if you use LiveAct:

pip install vllm==0.11.0

SageAttention (source build)

git clone https://github.com/thu-ml/SageAttention.git cd SageAttention export EXT_PARALLEL=4 NVCC_APPEND_FLAGS="--threads 8" MAX_JOBS=32 # Optional python setup.py install

Quick Start

Demo

<p align="center"><em>The following characters are demo examples only. They are not bundled with CyberVerse and are not provided for commercial use.</em></p>

<p align="center"> <a href="docs/assets/character1.png"><img src="docs/assets/character1.png" alt="CyberVerse character selection gallery" width="100%"/></a> </p>

<p align="center"> <a href="docs/assets/character2.png"><img src="docs/assets/character2.png" alt="CyberVerse character gallery examples" width="100%"/></a> </p>

Step 2: Create Python environment

conda create -n cyberverse python=3.10
conda activate cyberverse

Step 3: Configure environment variables

cp -r infra/config config

Edit config/env and fill in the supported API keys:

Alibaba Cloud Qwen-series models:

DASHSCOPE_API_KEY=your_dashscope_api_key

Or Volcengine Doubao-series models:

DOUBAO_ACCESS_TOKEN=your_doubao_access_token
DOUBAO_APP_ID=your_doubao_app_id

Doubao Voice: follow the Volcengine quick start to get App ID / API Key, then fill in DOUBAO_APP_ID / DOUBAO_ACCESS_TOKEN.

After the stack is running, you can change API keys and service endpoints from the web UI at /settings instead of editing config/env only.

Omni, LLM, embedding, TTS, and ASR model definitions are discovered automatically from infra/config/*_models/. Create matching files under config/*_models/ only when you want local overrides.

Step 4: Create local config and enable voice-only mode

Edit config/cyberverse.yaml:

inference:
  avatar:
    enabled: false

With enabled: false, CyberVerse runs as a pure voice agent assistant.

all optional groups at once

pip install -e ".[all]" ```

Optional: Full Digital-Human Video

If you want to drive realtime Avatar video with FlashHead or LiveAct, follow the steps below.

Configure Avatar Inference

Set enabled: true in config/cyberverse.yaml. Model-specific settings live in one file per model under config/avatar_models/; update those paths to match your local checkpoints.

inference:
  avatar:
    enabled: true
    default: "flash_head"               # use "flash_head" or "live_act"
    idle_strategy: "silent_inference"
    runtime:
      cuda_visible_devices: 0      # shared GPU ID(s), e.g. 0,1 for multi-GPU
      world_size: 1                # shared GPU count, set to 2 for dual-GPU
    model_config_dir: "avatar_models"

Then edit the active model file, for example config/avatar_models/flash_head.yaml or config/avatar_models/live_act.yaml. The Web UI also edits model parameters in those per-model files.

Optional when the figure needs a fixed camera.

BAIDU_XILING_CAMERA_ID="0" ```

Baidu Xiling is selected per character in the Web UI. It is not an avatar inference model and should not be configured as inference.avatar.default. CyberVerse still runs ASR/LLM/TTS/history through the orchestrator, then sends 16 kHz 16-bit mono PCM chunks to the browser. The frontend embeds the Baidu H5 iframe and drives it with the official sendAudioData / AUDIO_STREAM_RENDER message format.

LiveAct FP4 GEMM (Optional)

FP4 acceleration requires building and installing lightx2v_kernel from LightX2V. Use PyTorch 2.7+ and a CUTLASS checkout on the build machine.

Preparation

pip install scikit_build_core uv

Build wheel

```bash git clone https://github.com/NVIDIA/cutlass.git git clone https://github.com/ModelTC/LightX2V.git cd LightX2V/lightx2v_kernel

SageAttention & FlashAttention (Optional)

```bash

FlashAttention (optional)

wget -O flash_attn-2.8.1+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl \ "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1%2Bcu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl"

pip install flash_attn-2.8.1+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl ```

Check API health

curl -s http://localhost:8080/api/v1/health ```

Open http://localhost:5173 in your browser.

export HF_ENDPOINT=https://hf-mirror.com

hf download Soul-AILab/SoulX-FlashHead-1_3B \ --local-dir ./checkpoints/SoulX-FlashHead-1_3B

hf download facebook/wav2vec2-base-960h \ --local-dir ./checkpoints/wav2vec2-base-960h

#### LiveAct (SoulX-LiveAct)

| ModelName | Download |
|-----------|----------|
| SoulX-LiveAct | [Hugging Face](https://huggingface.co/Soul-AILab/LiveAct), [ModelScope](https://modelscope.cn/models/Soul-AILab/LiveAct) |
| chinese-wav2vec2-base | [Hugging Face](https://huggingface.co/TencentGameMate/chinese-wav2vec2-base), [ModelScope](https://modelscope.cn/models/TencentGameMate/chinese-wav2vec2-base) |

hf download TencentGameMate/chinese-wav2vec2-base \ --local-dir ./checkpoints/chinese-wav2vec2-base ```

Plugin-Based Stack

Brain, voice, hearing, tools, memory, and face are all replaceable modules. Runtime behavior stays in config/cyberverse.yaml, while omni, LLM, TTS, ASR, and embedding provider definitions are loaded from the built-in infra/config/*_models/ directories and optional local overrides under config/*_models/. You can configure different vendors' API keys and service endpoints in the web UI at /settings to switch providers and model combinations by scenario. The LiteLLM plugin adds access to 100+ LLM providers (AWS Bedrock, Azure, Vertex AI, Mistral, Cohere, etc.) through a single unified interface.

QA — Self-Check

Use this section when avatar video stutters, freezes, or falls behind audio. The first step is to confirm whether inference can keep up with playback.