AI Skill Hub 强烈推荐：ComfyUI_Simple_Qwen3 是一款优质的AI工具。AI 综合评分 8.0 分，在同类工具中表现稳健。如果你正在寻找可靠的AI工具解决方案，这是一个值得深入了解的选择。
Simple gguf LLM Qwen3-VL等模型加载器
ComfyUI_Simple_Qwen3 是一款基于 Python 开发的开源工具，专注于 AI、LLM、Qwen3-VL 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
Simple gguf LLM Qwen3-VL等模型加载器
ComfyUI_Simple_Qwen3 是一款基于 Python 开发的开源工具，专注于 AI、LLM、Qwen3-VL 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
# 方式一：pip 安装（推荐）
pip install comfyui_simple_qwen3-vl-gguf
# 方式二：虚拟环境安装（推荐生产环境）
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install comfyui_simple_qwen3-vl-gguf
# 方式三：从源码安装（获取最新功能）
git clone https://github.com/KLL535/ComfyUI_Simple_Qwen3-VL-gguf
cd ComfyUI_Simple_Qwen3-VL-gguf
pip install -e .
# 验证安装
python -c "import comfyui_simple_qwen3_vl_gguf; print('安装成功')"
# 命令行使用
comfyui_simple_qwen3-vl-gguf --help
# 基本用法
comfyui_simple_qwen3-vl-gguf input_file -o output_file
# Python 代码中调用
import comfyui_simple_qwen3_vl_gguf
# 示例
result = comfyui_simple_qwen3_vl_gguf.process("input")
print(result)
# comfyui_simple_qwen3-vl-gguf 配置文件示例（config.yml） app: name: "comfyui_simple_qwen3-vl-gguf" debug: false log_level: "INFO" # 运行时指定配置文件 comfyui_simple_qwen3-vl-gguf --config config.yml # 或通过环境变量配置 export COMFYUI_SIMPLE_QWEN3_VL_GGUF_API_KEY="your-key" export COMFYUI_SIMPLE_QWEN3_VL_GGUF_OUTPUT_DIR="./output"
<img width="2048" height="448" alt="15184-43452264163153+" src="https://github.com/user-attachments/assets/92b22216-aa55-4411-8718-8ec82e1b88b6" />
Simple gguf LLM Qwen3-VL, Qwen3.5, Qwen3.6, Gemma4 and others model loader for Comfy-UI.
The node is split into two parts. All work is isolated in a subprocess. Why? To ensure everything is cleaned up and nothing unnecessary remains in memory after this node runs and llama.cpp. I've often encountered other nodes leaving something behind, and that's unacceptable to me. > 💡 Update: The llama_python_cpp code has been improved and no longer leaks memory, so it is now possible to call llama_cpp directly.
| Mode | Characteristics | Benefits |
|---|---|---|
| subprocess | Inference runs in a separate Python process. The model is loaded and unloaded for each execution. | ✅ Complete isolation – no VRAM leaks. ✅ Safe main script - no crash. 💡 Frees VRAM after each use. |
| direct_clean | Inference runs in the main ComfyUI process. The model is cached between calls, but unloaded immediately after each inference (VRAM freed). Images are transmitted directly (no temporary files). | ✅ Faster than subprocess (no process spawn overhead). 💡 Still frees VRAM after each use. |
direct_clean or subprocess, this cache will be unloaded. |
| save1-save3 | Allows you to keep the model in VRAM for a long time, if it makes sense. The difference with the keep_vram mode is that the model is not cleared after switching modes, and the only way to unload the model is to use the UnloadQwenModel node and unload a specific cache or all caches. | ✅ Suitable for storing small models in memory, such as local translators or embedders. 💡 The main thing is to remember unload model manually from VRAM when no longer needed. |
Qwen3 support hasn't been added to the standard library, llama-cpp-python, which is downloaded via pip install llama-cpp-python - this didn't work. The standard version llama-cpp-python hasn't been updated for a long time. llama-cpp-python 0.3.16 last commit on Aug 15, 2025 and it doesn't support qwen3.
Check the version number of llama-cpp-python from JamePeng you're using: - Version 0.3.17 or latest supports qwen3-VL. - Version 0.3.30 or latest supports qwen3.5. - Version 0.3.35 or latest supports gemma4.
<details>
<summary>Installing software before compilation</summary>
1. Check that you have CUDA Toolkit installed. For example: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0 - Try installing: https://developer.nvidia.com/cuda-downloads - Check that the PATH in Environment Variables includes the CUDA Toolkit bin folder (For example: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0\bin). - After installing CUDA Toolkit, restart your computer.
2. Check that the NVIDIA Driver and CUDA Toolkit versions match (the driver can and most often should be newer than the CUDA Toolkit version): Run command in CMD nvidia-smi.
3. Check that you have Visual C++ Redistributable installed. - Try installing: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 - Install both versions (x86 and x64).
4. Check that you have Visual Studio 2022 installed. - Install Visual Studio 2022. - Install the following packages (they will not be installed by default): ☑ Desktop development with C++ (in Workloads tab).
☑ MSVC v143 - VS 2022 C++ x64/x86 build tools (in Individual components tab).
☑ Windows 10/11 SDK (in Individual components tab).
☑ CMake tools for Visual Studio (in Individual components tab).
- The environment variable for MSVC is not added to the PATH by default. Run this command every time in your terminal before compiling: call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"
5. If you use python_embeded for Comfy-UI, may need to add missing libs folders: python_embeded\include, python_embeded\libs (Not Lib\site-packages), python_embeded\DLLs: - From here https://github.com/astral-sh/python-build-standalone/releases download Python appropriate version (for example cpython-3.13.11+20251217-x86_64-pc-windows-msvc-install_only.tar.gz) - unzip and copy the necessary folders to python_embeded. </details>
<details>
<summary>Build llama-cpp-python from source code</summary>
1. Clone the repositories using Git: - https://github.com/JamePeng/llama-cpp-python - https://github.com/ggml-org/llama.cpp
git clone https://github.com/JamePeng/llama-cpp-python.git
git clone https://github.com/ggml-org/llama.cpp.gitllama.cpp\ in the llama-cpp-python\vendor\ folder
3. Automatically set the paths to MSVC (Windows only):
call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"<details>
<summary>4. Optional: For fast build with Ninja</summary>
Using Ninja enables parallel compilation across CPU cores, significantly reducing build time (but may increase CPU temperature). Verify Ninja is installed with Visual Studio 2022:
ninja --version
1.12.1set CMAKE_GENERATOR=Ninja
set MAX_JOBS=16</details>
5. Go to the llama-cpp-python folder
cd *path_to_src*\llama-cpp-python*path_to_comfyui*\python -m pip install json_repair,colorama
set CMAKE_ARGS=-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=120 -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Release
*path_to_comfyui*\python_embeded\python -m pip install . ✅ The command above is for embedded Python (typical for ComfyUI). Adjust the Python path if you're using a system or virtual environment.
Replace 120 with your сompute сapability number:
RTX 50-series (Blackwell) → 120
RTX 40-series → 89
RTX 30-series → 86
RTX 20-series → 75⏱️ Build time: Without Ninja, compilation may take 30–60 minutes depending on your hardware.
⏱️ Build time: With Ninja, compilation may take 1–2 minutes depending on your hardware.
💡 Tip: In subprocess mode, you can launch it immediately. In other modes, you need to restart Comfu-ui.
</details>
<details>
<summary>Simple bat file for fast rebuild</summary>
cd llama-cpp-python\vendor\llama.cpp\
git pull
cd ..\..\
git pull --rebase
call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"
set CMAKE_GENERATOR=Ninja
set MAX_JOBS=16
set CMAKE_ARGS=-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=120 -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Release
H:\ComfyUI128\python_embeded\python.exe -m pip install . --no-cache-dir --no-build-isolation
pause✅ The command above is for embedded Python (typical for ComfyUI). Adjust the Python path if you're using a system or virtual environment.
Replace 120 with your сompute сapability number:
RTX 50-series (Blackwell) → 120
RTX 40-series → 89
RTX 30-series → 86
RTX 20-series → 75💡 Tip: In subprocess mode, you can launch it immediately. In other modes, you need to restart Comfu-ui.
</details>
1.Installation to custom_nodes - Use ComfyUI Manager and find ComfyUI_Simple_Qwen3-VL-gguf - OR copy this project to the folder path_to_comfyui\ComfyUI\custom_nodes
cd path_to_comfyui\ComfyUI\custom_nodes
git clone https://github.com/KLL535/ComfyUI_Simple_Qwen3-VL-gguf<img width="836" height="460" alt="image" src="https://github.com/user-attachments/assets/db89cd37-e974-4a3a-8987-886b10393513" />
Configurations can be stacked. Each additional configuration, via the "config_override" input, overwrites the specified fields and leaves the others unchanged.
<img width="1331" height="696" alt="Image" src="https://github.com/user-attachments/assets/320192ed-d0c2-46bb-bc44-7f24d8348f3a" />
You don't have to follow the JSON format exactly. If json_repair is installed - it will fix it.
You can save your favorite configs to a JSON file and they will be available for selection in the drop-down list model preset.
Configurations loads from file system_prompts_user.json.
You must fill out this file yourself. An example of how to fill it out can be found in the file system_prompts_user.example.json.
Config values can be taken from the configurator or from the examples below.
<img width="1366" height="578" alt="image" src="https://github.com/user-attachments/assets/80a211c8-713b-4e6e-9ee5-0b731853ec31" />
<img width="2048" height="448" alt="03522-929995336568847" src="https://github.com/user-attachments/assets/0dc6c148-c049-4fc4-9363-eedb04db2785" />
<details>
<summary>Ernie Image Prompt Enhancer</summary>
Highly specialized LLM for Ernie Image.
For example: Ernie-Image-Prompt-Enhancer-Ministral-3.8B-Q4_K_M.gguf + mmproj-BF16.gguf
💡 TIP: A specialsystem promptis required; in the templates it is calledErnie Prompt Enhancer.
💡 TIP: Theuser promptshould look like this:{"prompt": "{prompt}", "width": {width}, "height": {height}}
💡 TIP: mmproj should be left empty "" if image input are not needed.
💡 TIP: The result will only be in Chinese.
"Ernie-Image-Prompt-Enhancer": {
"model_path": "H:\\LLM3\\ernie\\Ernie-Image-Prompt-Enhancer-Ministral-3.8B-Q4_K_M.gguf",
"mmproj_path": "H:\\LLM3\\ernie\\mmproj-BF16.gguf",
"max_tokens": 2048,
"image_min_tokens": 1024,
"image_max_tokens": 1024,
"ctx": 4096,
"n_batch": 2048,
"n_ubatch": 512,
"gpu_layers": -1,
"temperature": 0.8,
"top_p": 0.8,
"min_p": 0.05,
"top_k": 64,
"repeat_penalty": 1.05,
"chat_handler": "llava15",
"script": "qwen3vl_run.py",
"raw_mode": true,
"prompt_template": "[SYSTEM_PROMPT]{system}[/SYSTEM_PROMPT][INST]{user}{images}[/INST]",
"stop": ["</s>", "[INST]", "[/INST]"],
"debug": true,
"verbose": false
},</details>
<details>
<summary>gemma-4-12B</summary>
"Gemma-4-12B": {
"model_path": "H:\\LLM2\\lmstudio-community\\gemma-4-12B-it-QAT-GGUF\\gemma-4-12B-it-QAT-Q4_0.gguf",
"mmproj_path": "H:\\LLM2\\lmstudio-community\\gemma-4-12B-it-QAT-GGUF\\mmproj-gemma-4-12B-it-QAT-BF16.gguf",
"max_tokens": 10240,
"n_ctx": 12288,
"n_batch": 2048,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"split_mode": 0,
"temperature": 0.5,
"top_p": 0.9,
"min_p": 0.05,
"repeat_penalty": 1.05,
"presence_penalty": 0.0,
"top_k": 20,
"chat_handler": "gemma4",
"script": "qwen3vl_run.py",
"debug": true,
"enable_thinking": true,
"force_mmproj": false,
"verbose": false
},💡 TIP: It works well for ideogram4 json.
<img width="719" height="813" alt="image" src="https://github.com/user-attachments/assets/275815ba-3b26-4082-8cd5-4ea2a71bcd3e" />
</details>
<details>
<summary>HY-MT2 (translate)</summary>
For example: Hy-MT2-1.8B-Q4_K_M.gguf
💡 TIP: Here I made a prompt template in which the target_language is set through thesystem_prompt_overrideinput. Just supply the text with the target language there, for exampleRussian. And the text that needs to be translated should be submitted to theuser_promptinput.
💡 WARNING: The model is highly specialized and understands only strictly defined tasks.
"HY-1.8B (translate)": {
"model_path": "H:\\LLM3\\Hy-MT2-1.8B-Q4_K_M.gguf",
"raw_mode": true,
"prompt_template": "<｜hy_begin▁of▁sentence｜>Translate the following segment into {system}, without additional explanation.<｜hy_place▁holder▁no▁3｜><｜hy_User｜>{user}<｜hy_Assistant｜>",
"stop": [ "<｜hy_place▁holder▁no▁2｜>" ],
"system_prompt_default": "Russian",
"max_tokens": 2048,
"top_k": 20,
"top_p": 0.6,
"temperature": 0.7,
"repeat_penalty": 1.05,
"min_p": 0.05,
"script": "qwen3vl_run.py",
"n_ctx": 4096,
"n_batch": 4096,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"use_mmap": true,
"use_mlock": false,
"offload_kqv": true,
"verbose": false,
"debug": true
},For example: Hy-MT2-7B-Q4_K_M.gguf
"HY-7B (translate)": {
"model_path": "H:\\LLM3\\Hy-MT2-7B-Q4_K_M.gguf",
"raw_mode": true,
"prompt_template": "<|startoftext|>Translate the following segment into {system}, without additional explanation.<|extra_4|>{user}<|extra_0|>",
"stop": [ "<|eos|>" ],
"system_prompt_default": "Russian",
"max_tokens": 2048,
"top_k": 20,
"top_p": 0.6,
"temperature": 0.7,
"repeat_penalty": 1.05,
"min_p": 0.05,
"script": "qwen3vl_run.py",
"n_ctx": 4096,
"n_batch": 4096,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"use_mmap": true,
"use_mlock": false,
"offload_kqv": true,
"verbose": false,
"debug": true
},For example: Hy-MT2-30B-A3B-Q4_K_M.gguf
💡 WARNING: Not yet supported by the library. A patch for llama.cpp is required, or wait until support is added to llama.cpp.
💡 TIP: "n_cpu_moe": 12 to 16G VRAM
"HY-30B-A3B (translate)": {
"model_path": "H:\\LLM3\\Hy-MT2-30B-A3B-Q4_K_M.gguf",
"raw_mode": true,
"prompt_template": "<|start_header_id|>user<|end_header_id|>\\n\\nTranslate the following segment into {system}, without additional explanation.\\n\\n{user}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n",
"stop": [ "<|eot_id|>", "<|start_header_id|>" ],
"system_prompt_default": "Russian",
"max_tokens": 2048,
"top_k": 20,
"top_p": 0.6,
"temperature": 0.3,
"repeat_penalty": 1.05,
"min_p": 0.05,
"script": "qwen3vl_run.py",
"n_ctx": 4096,
"n_batch": 4096,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_cpu_moe": 12,
"n_threads": 8,
"use_mmap": false,
"use_mlock": false,
"offload_kqv": true,
"verbose": false,
"debug": true
},</details>
<details>
<summary>Qwen3.6-35B-A3B</summary>
For example: Qwen3.6-35B-A3B-Q4_K_M.gguf + mmproj-Qwen3.6-35B-A3B-BF16.gguf
💡 Tip: Q4_K_M is already quite an old quantization. Search for models on huggingface and choose models with better quantization, such as UD_IQ from unsloth. They will be smarter and lighter.
For example: Qwen3.6-35B-A3B-APEX-I-Quality.gguf + mmproj.gguf
For example: Qwen3.6-35B-A3B-UD-IQ4_XS.gguf + mmproj-BF16.gguf
💡 Tip: If there is a BF16 version for mmproj, choose it, it is better than F16.
Examples:
This model not fit in 16 Gb VRAM. Settings for n_cpu_moe offloading:
💡 Tip: use_mmap = false - Provides better speed, but the model may take longer to load, it needs to be tested.
💡 Tip: split_mode = 0 - Provides better speed on a single GPU, eliminating performance drops after launch.
"Qwen3.6-35B-A3B-Q4_K_M": {
"model_path": "H:\\LLM\\lmstudio-community\\Qwen3.6-35B-A3B-GGUF\\Qwen3.6-35B-A3B-Q4_K_M.gguf",
"mmproj_path": "H:\\LLM\\lmstudio-community\\Qwen3.6-35B-A3B-GGUF\\mmproj-Qwen3.6-35B-A3B-BF16.gguf",
"max_tokens": 4096,
"image_min_tokens": 1024,
"image_max_tokens": 2048,
"n_ctx": 8192,
"n_batch": 2048,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"n_cpu_moe": 20,
"use_mmap": true,
"split_mode": 0,
"temperature": 0.8,
"top_p": 0.95,
"min_p": 0.05,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"top_k": 40,
"chat_handler": "qwen35",
"enable_thinking": true,
"script": "qwen3vl_run.py",
"debug": true,
"verbose": false
}, "Qwen3.6-35B-A3B-UD-IQ4_XS": {
"model_path": "H:\\LLM2\\qwen\\Qwen3.6-35B-A3B-UD\\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf",
"mmproj_path": "H:\\LLM2\\qwen\\Qwen3.6-35B-A3B-UD\\mmproj-BF16.gguf",
"max_tokens": 4096,
"image_min_tokens": 1024,
"image_max_tokens": 2048,
"n_ctx": 8192,
"n_batch": 2048,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"n_cpu_moe": 16,
"use_mmap": true,
"split_mode": 0,
"temperature": 0.8,
"top_p": 0.95,
"min_p": 0.05,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"top_k": 40,
"chat_handler": "qwen35",
"enable_thinking": true,
"script": "qwen3vl_run.py",
"debug": true,
"verbose": false
}, "Qwen3.6-35B-A3B-APEX-I-Quality": {
"model_path": "H:\\LLM2\\qwen\\Qwen3.6-35B-A3B-APEX\\Qwen3.6-35B-A3B-APEX-I-Quality.gguf",
"mmproj_path": "H:\\LLM2\\qwen\\Qwen3.6-35B-A3B-APEX\\mmproj.gguf",
"max_tokens": 4096,
"image_min_tokens": 1024,
"image_max_tokens": 2048,
"n_ctx": 8192,
"n_batch": 2048,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"n_cpu_moe": 20,
"use_mmap": true,
"split_mode": 0,
"temperature": 0.8,
"top_p": 0.95,
"min_p": 0.05,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"top_k": 40,
"chat_handler": "qwen35",
"enable_thinking": true,
"script": "qwen3vl_run.py",
"debug": true,
"verbose": false
},</details>
<details>
<summary>Qwen3.6-27B</summary>
For example: Qwen3.6-27B-UD-IQ3_XXS.gguf + mmproj-BF16.gguf
Fit in 16 Gb VRAM:
"Qwen3.6-27B-UD-IQ3_XXS": {
"model_path": "H:\\LLM\\lmstudio-community\\Qwen3.6-27B-GGUF\\Qwen3.6-27B-UD-IQ3_XXS.gguf",
"mmproj_path": "H:\\LLM\\lmstudio-community\\Qwen3.6-27B-GGUF\\mmproj-BF16.gguf",
"max_tokens": 2048,
"image_min_tokens": 1024,
"image_max_tokens": 2048,
"n_ctx": 8192,
"n_batch": 2048,
"n_ubatch": 512,
"n_gpu_layers": -1,
"split_mode": 0,
"temperature": 0.7,
"top_p": 0.95,
"min_p": 0.05,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"top_k": 40,
"chat_handler": "qwen35",
"enable_thinking": true,
"script": "qwen3vl_run.py",
"debug": true,
"verbose": false
},</details>
<details>
<summary>Nemotron-3-Nano-Omni-30B</summary>
For example: NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-UD-IQ4_NL.gguf + mmproj-BF16.gguf
Not fit in 16 Gb VRAM -> Use n_cpu_moe = 24:
"NVIDIA-Nemotron-3-Nano-Omni-30B": {
"model_path": "H:\\LLM2\\nemotron\\NVIDIA-Nemotron-3-Nano-Omni-30B\\NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-UD-IQ4_NL.gguf",
"mmproj_path": "H:\\LLM2\\nemotron\\NVIDIA-Nemotron-3-Nano-Omni-30B\\mmproj-BF16.gguf",
"max_tokens": 4096,
"n_ctx": 8192,
"n_batch": 8192,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"n_cpu_moe": 24,
"split_mode": 0,
"temperature": 0.6,
"top_p": 0.95,
"min_p": 0.05,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"top_k": 40,
"enable_thinking": true,
"script": "qwen3vl_run.py",
"debug": true,
"verbose": false,
"chat_handler": "qwen35"
},💡 WARNING: Chat handler qwen35 is not compatible with this model. The model will work, but the quality may deteriorate.
There is an alternative solution: override the chat template. If you only need to process text and/or images, you can use this template overrides: Thinking version (add these lines):
"chat_handler": "llava15",
"raw_mode": true,
"prompt_template": "<|im_start|>system\\n{system}<|im_end|>\\n<|im_start|>user\\n{images}{user}<|im_end|>\\n<|im_start|>assistant\\n<think>\\n",
"stop": ["<|endoftext|>"]Non-thinking version (add these lines):
"chat_handler": "llava15",
"raw_mode": true,
"prompt_template": "<|im_start|>system\\n{system}<|im_end|>\\n<|im_start|>user\\n{images}{user}<|im_end|>\\n<|im_start|>assistant\\n",
"stop": ["<|endoftext|>"]</details>
<details>
<summary>Gemma4-26B-A4B</summary>
For example: Huihui-gemma-4-26B-A4B-it-abliterated-MXFP4_MOE.gguf + mmproj-BF16.gguf
Not fit in 16 Gb VRAM -> set n_cpu_moe.
"gemma-4-26B-A4B": {
"model_path": "H:\\LLM2\\gemma\\Huihui-gemma-4-26B-A4B-it-abliterated-MXFP4_MOE\\Huihui-gemma-4-26B-A4B-it-abliterated-MXFP4_MOE.gguf",
"mmproj_path": "H:\\LLM2\\gemma\\Huihui-gemma-4-26B-A4B-it-abliterated-MXFP4_MOE\\mmproj-BF16.gguf",
"max_tokens": 4096,
"n_ctx": 4096,
"n_batch": 512,
"n_ubatch": 512,
"n_gpu_layers": -1,
"n_threads": 8,
"split_mode": 0,
"n_cpu_moe": 10,
"temperature": 0.7,
"top_p": 0.95,
"min_p": 0.05,
"repeat_penalty": 1.1,
"presence_penalty": 0.0,
"top_k": 40,
"chat_handler": "gemma4",
"script": "qwen3vl_run.py",
"debug": true,
"enable_thinking": false,
"force_mmproj": true,
"verbose": false,
},</details>
<details>
<summary>Gemma4-E4B</summary>
Possible model configurations that can be passed to the config_override input.
<details>
<summary>Configs</summary>
| Field | Type | Default | Description | |
|---|---|---|---|---|
| model_path | string | Path to the GGUF model file. Relative paths are supported. The path is specified relative to ComfyUI\custom_nodes\ComfyUI_Simple_Qwen3-VL-gguf | ||
| mmproj_path | string | Path to the multimodal projector file (required for vision models) | ||
| force_mmproj | bool | false | 💡 If there are no images or audio or viseo, mmproj still loads into memory (and takes up space for no reason), but the correct template is not lost (enable_thinking will work). | |
| n_ctx or ctx | int | 8192 | Context size, maximum tokens the model can process. 💡 Increasing this parameter increases memory consumption, but if there are many pictures and the answer is big, then the answer can be truncated or error if the input data does not fit into the context. Rule: image_max_tokens + input_text_max_tokens + max_tokens <= n_ctx | |
| n_batch | int | 2048 | Batch size for prompt processing. A smaller number saves memory. Setting n_batch = n_ctx can speed up processing | |
| n_ubatch | int | 512 | Micro-batch size for advanced memory management | |
| image_min_tokens | int | Minimum number of tokens to allocate for image embeddings. | ||
| image_max_tokens | int | Maximum number of tokens to allocate for image embeddings. | ||
| user_prompt_after_content | bool | True | Inserts user_prompt after the image, otherwise before the image. | |
| max_tokens or output_max_tokens | int | 2048 | Maximum number of tokens to generate. A smaller number saves time, but may result in a truncated response. Thinking models require many output tokens | |
| temperature | float | 0.7 | Sampling temperature; Lower values (e.g., 0.1) make output more deterministic and focused; higher values (e.g., 1.5) increase randomness and creativity | |
| top_p | float | 0.92 | Nucleus sampling probability (0.0–1.0). The model considers only the tokens whose cumulative probability reaches top_p. Lower values make output more focused | |
| min_p | float | 0.05 | Minimum probability for a token to be considered in sampling. Tokens with probability below min_p are ignored | |
| top_k | int | 0 | Top-k sampling. limits to the k most likely tokens. 0 disables top-k | |
| repeat_penalty | float | 1.1 | Penalty for repeating tokens (≥1.0). Values >1 discourage repetition | |
| frequency_penalty | float | 0.0 | Penalty based on token frequency. Positive values reduce the likelihood of frequently used tokens | |
| presence_penalty or present_penalty | float | 0.0 | Penalty based on token presence. Positive values reduce the likelihood of tokens that have already appeared | |
| swa_full | bool | False | Enable full Stochastic Weight Averaging (SWA). 💡 Enabling this setting may cause higher memory consumption. | |
| use_mmap | bool | Enable mmap. 💡 Observation. For Windows, it's better to turn it off. | ||
| use_mlock | bool | Enable mlock. | ||
| offload_kqv | bool | True | Offload KV Cache to GPU. Turn OFF (slow) for safe VRAM. | |
| n_cpu_moe | int | For MoE models that don't fit in VRAM. The number of expert layers that will be in RAM and processed by the CPU. This is a more advanced replacement for n_gpu_layers, which is twice as fast. | ||
| cpu_moe | bool | For MoE models unloads all experts into RAM. Allows to save VRAM memory | ||
| pool_size | int | 4194304 | Memory pool size for the model (llama.cpp). | |
| n_threads or cpu_threads | int | os.cpu_count() or 8 | Number of CPU threads to use for inference. | |
| image_quality | int | 95 | JPEG quality (1–100) when encoding images to data URIs. Higher values give better quality but larger size. | |
| frame_quality | int | 75 | JPEG quality (1–100) when encoding video frames to data URIs. Higher values give better quality but larger size. | |
| ctx_checkpoints | int | 0 | Max number of context checkpoints to create per slot | |
| merge_system_and_user | bool | False | If True, combines system and user prompts into a single user message.Used for some llava-type models. | |
| n_gpu_layers or gpu_layers | int | -1 | Number of layers to offload to GPU; -1 means all layers in GPU. 0 means all layers in CPU. Setting a lower number (40 -> 35 -> 30) can help, sometimes even speeding up by avoiding out-of-memory errors. | |
| script | string | Name of the Python script to execute ("qwen3vl_run.py"). This field must be specified in the config. | ||
| verbose | bool | False | Enables verbose logging from llama.cpp | |
| silent | bool | False | 💡 Unstable function. Disable. | |
| debug | bool | False | Enables output of the time count for each stage to the console. (e.g., [DEBUG] total time: 7.818s | 397 word (50.8 word/sec)) |
| force_gc_start | bool | False | Enables garbage collection after memory clearing when the unload_all_models flag is active. 💡 If you have a lot of garbage accumulating in your memory, enable this option, but it will increase the time. | |
| force_gc_unload | bool | False | Enables garbage collection after deleting the LLM model. 💡 If you have a lot of garbage accumulating in your memory, enable this option, but it will increase the time. | |
| chat_handler | string | Type of chat handler: "gemma4", "qwen35", "qwen3", "qwen25", "gemma3", "llava15", "llava16", "bakllava", "moondream", "minicpmv26", "minicpmv45", "glm41v", "glm46v", "granite", "lfm2vl", "paddleocr", "obsidian", "nanollava", "llama3visionalpha". 💡 Specify for multimodal models. | ||
| chat_format | string | Type of chat format for text model: "llama-2", "llama-3", "alpaca", "vicuna", "oasst_llama", "baichuan-2", "baichuan", "openbuddy", "redpajama-incite", "snoozy", "phind", "intel", "open-orca", "mistrallite", "zephyr", "pygmalion", "chatml", "mistral-instruct", "chatglm3", "openchat", "saiga", "gemma", "qwen" 💡 Required to be specified for text models only (or multimodal model in text mode). | ||
| chat_format_from_gguf | bool | false | Forces the chat template to be loaded from the gguf model. 💡 If there are pictures, audio, video, it doesn't work. | |
| enable_thinking | bool | False | For "Gemma4, "Qwen35, "minicpmv45", "glm46v" enables the thinking process in the response. | |
| add_vision_id | bool | auto | For "Qwen35", "Qwen3" adds a vision ID token to the prompt. If not set, it will be calculated automatically (True if number of images != 1) | |
| force_reasoning | bool | False | For "Qwen3" forces reasoning mode. | |
| stop | list of strings | Stop sequences that halt generation. When any of these strings is generated, the process stops. (e.g., ["tag1", "tag2"]). 💡 Important: Llama automatically adds stop tokens based on chat_handler or chat_format. Pass stop only if you want to override the default behavior. |
master_preset list from the system prompt to the user prompt, if the model understands the task better this way. |
| system_prompt_default | string | | 💡 Allows you to set the default system prompt for the model. |
| raw_output | bool | False | If True disables output.strip() |
| max_images | int | 10 | You can set a limit on the number of incoming images (in batch mode, you can transfer many images) |
| max_audios | int | 3 | You can set a limit on the number of incoming audio (in batch mode, you can transfer many audio) |
| max_frames | int | 24 | Allows you to limit the frame size for video, which will result in frame scaling. Transferring many frames will require significantly increasing the context window, which may run out of memory. On the other hand, scaling frames may result in the loss of important motion information. The player may see a slideshow instead of a video, which will be helpfully reported. |
| audio_sample_rate | int | | You can set a new sampling frequency and then the audio will be resampled. |
| print_config | bool | false | Prints the full configuration to the console for debugging. |
| type_k | int | | Define the data format (degree of compression/quantization) that is used to store the KV-cache (Context Cache) 💡 Some variant may not work. Default: F16 |
| type_v | int | | Define the data format (degree of compression/quantization) that is used to store the KV-cache (Context Cache) 💡 Some variant may not work. Default: F16 |
| enable_variables | bool | False | Enables substitution of placeholders { } in system and user prompts. Example: "enable_variables": true |
| add_image_id | str | None | Allows you to additionally mark images with the specified text before inserting them. Example: "add_image_id": "\\n[Image {num}]:" |
| add_frame_id | str | None | Allows you to additionally mark frames with the specified text before inserting them. Example: "add_frame_id": "\\n[Frame {num}]:" |
| add_audio_id | str | None | Allows you to additionally mark audios with the specified text before inserting them. |
Multi-GPU settings https://github.com/KLL535/ComfyUI_Simple_Qwen3-VL-gguf/issues/24: <div class="rdm-tbl-wrap"><table class="rdm-tbl"><thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td>cuda_device</td><td>int/str</td><td>None</td><td>System level. Sets CUDA_VISIBLE_DEVICES environment variable before initialization. Restricts GPU visibility for the entire Python process. Accepts single index (0) or comma-separated list ("0,1"). Remaps logical GPU indices for llama-cpp (e.g., cuda_device=2 makes physical GPU2 appear as logical 0, so main_gpu must be 0). Must be set before any CUDA library loads; runtime changes are ignored. Use for strict GPU isolation in multi-GPU or shared environments. to the specified device(s). 💡 It may not work correctly in direct_clear and keep_vram modes, since comfi-ui is already running llama.cpp with its own settings.</td></tr><tr><td>main_gpu</td><td>int</td><td>0</td><td>Library level. Index of the primary GPU to use when split_mode=0 (NONE). Ignored in LAYER/ROW modes except for KV-cache placement. Works with CUDA_VISIBLE_DEVICES filtering: if CUDA_VISIBLE_DEVICES=1, then main_gpu=0 refers to physical GPU1.</td></tr><tr><td>split_mode</td><td>int</td><td>1</td><td>GPU splitting mode: 0=NONE (No splitting. The model is loaded onto a single main_gpu), 1=LAYER (distribute layers across GPUs. This is the most common mode. Different layers of the neural network are assigned to different GPUs. For example, layers 1-16 go to GPU0, and 17-32 go to GPU1.), 2=ROW (tensor parallelism, this splits the actual weight matrices across GPUs. It can be faster for certain operations but usually requires higher bandwidth between GPUs). Use 0 for single-GPU setups to avoid distribution overhead.</td></tr><tr><td>tensor_split</td><td>list</td><td>None</td><td>List of floats specifying the fraction of the model to offload to each GPU (e.g., [0.7, 0.3] for 70%/30% split). Only effective when split_mode=1 (LAYER). Length must match number of visible GPUs. If not set, llama-cpp auto-balances based on VRAM.</td></tr></tbody></table></div>
Encoder options: <div class="rdm-tbl-wrap"><table class="rdm-tbl"><thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td>extract_embedding</td><td>bool</td><td>false</td><td>true - allows you to get embeddings.</td></tr><tr><td>tokenizer_path</td><td>str</td><td>""</td><td>Allows you to override the tokenizer, in case the built-in gguf does not work correctly. You need to specify the path to the folder. May slow down performance as it requires calling transformers.</td></tr><tr><td>prompt_template</td><td>str</td><td>{user}</td><td>Some models require a prompt template to work correctly.</td></tr><tr><td>convert_emb_to_cond</td><td>bool</td><td>false</td><td>true - The output will be conditioning, understandable comfy, false - embeddings.</td></tr><tr><td>embedding_scale</td><td>float</td><td>None</td><td>Allows you to multiply all weights by a given constant.</td></tr><tr><td>pooling_type</td><td>bool</td><td>0</td><td>Determines the format of the output vectors: -1 (LLAMA_POOLING_TYPE_UNSPECIFIED) — The type is not specified. The system will attempt to determine it automatically (if the metadata is embedded in the GGUF file). 0 (LLAMA_POOLING_TYPE_NONE) — Pooling is disabled. The model returns an array of vectors for each token (the same two-dimensional list [N × Hidden_Dim]). 1 (LLAMA_POOLING_TYPE_MEAN) — The arithmetic mean. The library will automatically add the token vectors and divide by their number. The output will be a single combined vector. 2 (LLAMA_POOLING_TYPE_CLS) — Only the CLS token. Will take the vector of the very first token in the sequence. 3 (LLAMA_POOLING_TYPE_LAST) — Only the last token. Takes the token vector where the sentence ends. 4 (LLAMA_POOLING_TYPE_RANK) — Specific pooling for reranking models (used to attach the classification head to the graph).</td></tr></tbody></table></div>
Custom prompt templates: <div class="rdm-tbl-wrap"><table class="rdm-tbl"><thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td>raw_mode</td><td>bool</td><td>False</td><td>Allows you to enable custom templates mode.</td></tr><tr><td>prompt_template</td><td>string</td><td>default to joycaption</td><td>Prompt format. See the model recommendations. The template must include placeholders {system}, {images}, {user}</td></tr><tr><td>stop</td><td>list of strings</td><td>default to joycaption</td><td>Stop sequences that halt generation. In this mode it is necessary to set it. See the model recommendations.</td></tr></tbody></table></div>
<details> <summary>default prompt_template example</summary>
"raw_mode": true,
"prompt_template": "<|start_header_id|>system<|end_header_id|>\n\n{system}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{images}{user}<|eot_id|><|start_header_id|>assistant<|end_header_id|>",
"stop": ["<|eot_id|>", "<|end_of_text|>"],💡 Note: There is no need to write the first token <|begin_of_text|>, it is inserted by llama automatically.
#### Ministral:
"raw_mode": true,
"prompt_template": "[INST]{system}\n\n{images}{user}[/INST]",
"stop": ["</s>", "[INST]", "[/INST]"],💡 Note: There is no need to write the first token <s>, it is inserted by llama automatically.
</details>
Extra options. <div class="rdm-tbl-wrap"><table class="rdm-tbl"><thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td>extra_chat_handler_*</td><td></td><td></td><td>Allows you to pass any arguments to the function *ChatHandler</td></tr><tr><td>extra_llama_*</td><td></td><td></td><td>Allows you to pass any arguments to the function Llama</td></tr><tr><td>extra_chat_completion_*</td><td></td><td></td><td>Allows you to pass any arguments to the function create_chat_completion</td></tr></tbody></table></div>
The following settings are generated automatically. They DO NOT need to be write in the config. <div class="rdm-tbl-wrap"><table class="rdm-tbl"><thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td>system_prompt</td><td>string</td><td>System prompt that sets the behavior and context for the model. - add automatically in node</td></tr><tr><td>user_prompt</td><td>string</td><td>User input query or instruction. - add automatically in node</td></tr><tr><td>seed</td><td>int</td><td>Random seed for reproducible generation. - add automatically in node</td></tr><tr><td>images or images_path</td><td>list</td><td>List of images (PIL images or file paths) – add automatically in node</td></tr><tr><td>audios or audios_path</td><td>list</td><td>List of images (WAV bytes or file paths) – add automatically in node</td></tr><tr><td>videos or videos_path</td><td>list</td><td>List of images (only file paths) – add automatically in node</td></tr><tr><td>config_hash</td><td>string</td><td>Hash of the configuration for model caching – generated automatically in node</td></tr></tbody></table></div>
</details>
<details> <summary>config_override input</summary>
You can pass config_override as a JSON dictionary or without formatting.
config_override example:
"model_path": "H:\LLM2\Qwen3.5-9B-Q4_K_M\Qwen3.5-9B-Q4_K_M.gguf",
"mmproj_path": "H:\LLM2\Qwen3.5-9B-Q4_K_M\mmproj-BF16.gguf",
"max_tokens": 2048,
"image_min_tokens": 1024,
"image_max_tokens": 2048,
"n_ctx": 8192,
"n_batch": 2048,
"n_ubatch": 512,
"n_gpu_layers": -1,
"temperature": 0.7,
"top_p": 0.8,
"min_p": 0.05,
"top_k": 20,
"repeat_penalty": 1.0,
"presence_penalty": 1.5,
"pool_size": 4194304,
"chat_handler": "qwen35",
"enable_thinking": true,
"script": "qwen3vl_run.py",
"silent": false,
"debug": true,</details>
高质量的AI工具，支持多种LLM模型
AI Skill Hub 为第三方内容聚合平台，本页面信息基于公开数据整理，不对工具功能和质量作任何法律背书。
建议在沙箱或测试环境中充分验证后，再部署至生产环境，并做好必要的安全评估。
✅ MIT 协议 — 最宽松的开源协议之一，可自由商用、修改、分发，仅需保留版权声明。
总体来看，ComfyUI_Simple_Qwen3 是一款质量优秀的AI工具，在同类工具中具备一定竞争力。AI Skill Hub 将持续追踪其更新动态，建议收藏备用，结合自身场景选择合适时机引入使用。
| 原始名称 | ComfyUI_Simple_Qwen3-VL-gguf |
| Topics | AILLMQwen3-VLPython |
| GitHub | https://github.com/KLL535/ComfyUI_Simple_Qwen3-VL-gguf |
| License | MIT |
| 语言 | Python |
收录时间：2026-07-05 · 更新时间：2026-07-05 · License：MIT · AI Skill Hub 不对第三方内容的准确性作法律背书。