Prerequisites

• Node 18+
• Go 1.25 (required: protoc-gen-go, protoc-gen-go-grpc)
• Conda
• Python 3.10+
• FFmpeg

For pure voice sessions, no local avatar GPU is required. Runtime cost depends on the realtime voice/omni/LLM/TTS/ASR providers you configure.

To verify, use:

node --version
go version
protoc --version
ffmpeg -version
conda --version

Step 5: Install project dependencies

make setup

This installs the base editable package ([dev,inference]), generates gRPC stubs, and installs frontend dependencies.

Install the voice-agent extras used by the default config:

```bash

Additional Requirements

• GPU with CUDA 12.8+
• PyTorch 2.8 (CUDA 12.8)
• FFmpeg with libvpx for video encoding
• Avatar model weights

Install PyTorch (CUDA 12.8):

pip3 install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128

Install vllm if you use LiveAct:

pip install vllm==0.11.0

SageAttention (source build)

git clone https://github.com/thu-ml/SageAttention.git cd SageAttention export EXT_PARALLEL=4 NVCC_APPEND_FLAGS="--threads 8" MAX_JOBS=32 # Optional python setup.py install

Quick Start

Demo

<p align="center"><em>The following characters are demo examples only. They are not bundled with CyberVerse and are not provided for commercial use.</em></p>

<p align="center"> <a href="docs/assets/character1.png"><img src="docs/assets/character1.png" alt="CyberVerse character selection gallery" width="100%"/></a> </p>

<p align="center"> <a href="docs/assets/character2.png"><img src="docs/assets/character2.png" alt="CyberVerse character gallery examples" width="100%"/></a> </p>

Optional Digital Human Video

When you have GPU resources and want the Agent to be visible, enable avatar inference: a single character reference image can drive realtime facial animation, lip-sync, and cached idle video playback through configurable backends such as FlashHead and LiveAct. If you do not have a GPU or do not need video yet, disable it to return to a pure voice Agent; the same character and persona configuration continues to work.

Step 2: Create Python environment

conda create -n cyberverse python=3.10
conda activate cyberverse

Step 3: Configure environment variables

cp infra/.env.example .env

Edit .env and fill in the supported API keys:

Alibaba Cloud Qwen-series models:

DASHSCOPE_API_KEY=your_dashscope_api_key

Or Volcengine Doubao-series models:

DOUBAO_ACCESS_TOKEN=your_doubao_access_token
DOUBAO_APP_ID=your_doubao_app_id

Doubao Voice: follow the Volcengine quick start to get App ID / API Key, then fill in DOUBAO_APP_ID / DOUBAO_ACCESS_TOKEN.

After the stack is running, you can change API keys and service endpoints from the web UI at /settings instead of editing .env only.

Step 4: Create local config and enable voice-only mode

cp infra/cyberverse_config.example.yaml cyberverse_config.yaml

Edit cyberverse_config.yaml:

inference:
  avatar:
    enabled: false

With enabled: false, CyberVerse runs as a pure voice agent assistant.

all optional groups at once

pip install -e ".[all]" ```

Optional: Full Digital-Human Video

If you want to drive realtime Avatar video with FlashHead or LiveAct, follow the steps below.

Configure Avatar Inference

Set enabled: true, then update the model paths to match your local checkpoints:

inference:
  avatar:
    enabled: true
    default: "flash_head"               # selects which avatar model to start; if set to live_act, fill the live_act section below
    runtime:
      cuda_visible_devices: 0      # shared GPU ID(s), e.g. 0,1 for multi-GPU
      world_size: 1                # shared GPU count, set to 2 for dual-GPU
    flash_head:
      checkpoint_dir: "./checkpoints/SoulX-FlashHead-1_3B"  # ← your path
      wav2vec_dir: "./checkpoints/wav2vec2-base-960h"        # ← your path
      model_type: "lite"           # "pro" for higher quality (needs more GPU)
      compile_model: true
      compile_vae: true
      dist_worker_main_thread: true
      infer_params:
        frame_num: 33
        motion_frames_latent_num: 2
        tgt_fps: 20
        sample_rate: 16000
        sample_shift: 5
        color_correction_strength: 1.0
        cached_audio_duration: 8
        num_heads: 12
        height: 512
        width: 512
    live_act:
      ckpt_dir: "./checkpoints/LiveAct"                     # ← your path
      wav2vec_dir: "./checkpoints/chinese-wav2vec2-base"   # ← your path
      seed: 42
      fp8_gemm: true
      fp4_gemm: false
      compile_wan_model: false
      compile_vae_decode: false
      dist_worker_main_thread: true
      default_prompt: "一个人在说话"
      infer_params:
        size: "320*480"
        fps: 20
        audio_cfg: 1.0

You can also adjust these options later in the web UI.

SageAttention & FlashAttention (Optional)

```bash

FlashAttention (optional)

wget -O flash_attn-2.8.1+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl \ "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1%2Bcu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl"

pip install flash_attn-2.8.1+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl ```

Check API health

curl -s http://localhost:8080/api/v1/health ```

Open http://localhost:5173 in your browser.

export HF_ENDPOINT=https://hf-mirror.com

hf download Soul-AILab/SoulX-FlashHead-1_3B \ --local-dir ./checkpoints/SoulX-FlashHead-1_3B

hf download facebook/wav2vec2-base-960h \ --local-dir ./checkpoints/wav2vec2-base-960h

#### LiveAct (SoulX-LiveAct)

| ModelName | Download |
|-----------|----------|
| SoulX-LiveAct | [Hugging Face](https://huggingface.co/Soul-AILab/LiveAct), [ModelScope](https://modelscope.cn/models/Soul-AILab/LiveAct) |
| chinese-wav2vec2-base | [Hugging Face](https://huggingface.co/TencentGameMate/chinese-wav2vec2-base), [ModelScope](https://modelscope.cn/models/TencentGameMate/chinese-wav2vec2-base) |

hf download TencentGameMate/chinese-wav2vec2-base \ --local-dir ./checkpoints/chinese-wav2vec2-base ```

Plugin-Based Stack

Brain, voice, hearing, tools, memory, and face are all replaceable modules. You can combine omni models, LLMs, TTS, ASR, embeddings, RAG, tool calls, and Avatar backends in cyberverse_config.yaml, then configure different vendors' API keys and service endpoints in the web UI at /settings to switch providers and model combinations by scenario.