Installation

Environment Setup

git clone https://github.com/Soul-AILab/SoulX-Transcriber.git
cd SoulX-Transcriber

conda create -n soulx_transcriber python=3.12 -y
conda activate soulx_transcriber

Install MS-Swift and dependencies:

pip install ms-swift

install uv：

curl -LsSf https://astral.sh/uv/install.sh | sh

install vllm：

uv pip install vllm --torch-backend=auto --index-url https://mirrors.aliyun.com/pypi/simple/

install vllm-omni:

uv pip install vllm-omni --index-url https://mirrors.aliyun.com/pypi/simple/

install gradio (Optional)：

uv pip install 'vllm-omni[demo]' --index-url https://mirrors.aliyun.com/pypi/simple/

If you meet an "Undefined symbol" error while using VLLM_USE_PRECOMPILED=1, please use "pip install -e . -v" to build from source.

git clone https://github.com/vllm-project/vllm-omni.git cd vllm-omni uv pip install -e . --index-url https://mirrors.aliyun.com/pypi/simple/ ``` > For more details on compiling vLLM from source, refer to the vLLM official documentation.

🎬 Demo Video

create new uv environment（using aliyun mirror）

uv venv vllm_omni --python 3.12 --seed --index-url https://mirrors.aliyun.com/pypi/simple/

activate uv environment

source vllm_omni/bin/activate

🧪 Multi-speaker Dialogue Simulation Pipeline

To improve out-of-domain generalization, we build an agent-based multi-speaker dialogue simulation pipeline with a speaker-aware prompt audio matching mechanism. Given a target dialogue text, the system analyzes speaker tags, selects the most suitable reference audio for each speaker using multi-dimensional speaker representations, and synthesizes context-consistent multi-turn dialogue audio.

Workflow: building dialogue text database → building reference audio database → target text analysis → reference audio matching → dialogue audio generation. Detailed information is shown on the figure below. <div align="center"> <img src="figs/simulation_pipeline.png" width="100%" alt="simulation_pipeline"> </div>

• Dialogue text database. We collect multi-speaker dialogue texts from Chinese/English podcasts and novels. An LLM annotates speaker tags and controls the number of speakers; we keep segments with 3–8 speakers to ensure natural, coherent dialogue context.

• Dialogue context analysis: We use Qwen3-8B as the LLM brain for speaker-tag and context analysis, and SoulX-Podcast & MOSS-TTSD for long-form, multi-speaker multi-turn TTS synthesis.

• Reference audio database. We run VAD on long-form drama audio, cut it into 3–10s clips, and filter by UTMOS and SNR to ensure quality. Each clip is annotated by Gemini-3.1-pro-preview with multi-dimensional speaker attributes (e.g., gender/age/emotion/speech rate/pitch/timbre/style/tone/role state). We embed each attribute using bge-m3 and stack them into a per-clip feature matrix, forming an embedding index for retrieval.

• Best reference–audio matching. Given a target dialogue with speaker tags, an LLM analyzes each speaker’s attributes and builds the same multi-dimensional embedding representation. We compute similarity against all reference clips, apply a weighted score across attribute dimensions, and retrieve top-k (k=3) candidates per speaker. A final selection enforces diversity (different source speakers) and UTMOS consistency (|Δ| ≤ 0.5) to produce the best reference audio set for synthesis.