TensorSharp

TensorSharp

<p align="center"> <img src="imgs/banner_1.png" alt="TensorSharp logo" width="320"> </p>

English | 中文

Native .NET LLM inference engine for GGUF models — autoregressive LLMs and DiffusionGemma-style text-diffusion, plus Qwen-Image-Edit image editing and MiniMax-H3 video with native 32 kHz stereo audio (and Wan 2.1/2.2 for video alone). Ships a console app, a browser chat UI, and Ollama/OpenAI-compatible HTTP APIs. A pure-.NET engine that trades wins with the hand-tuned C++ llama.cpp on identical GGUF files and the same GPU. The optional TensorSharp.AgentHost layer adds Agent Skills and a bounded, in-process model-to-tool loop for sandboxed file and shell work.

Highlights

• ⚡ Trades wins with llama.cpp — from pure .NET. On identical GGUF files and the same GPU, TensorSharp matches or beats llama.cpp on the workloads that matter: Gemma 4 E4B and 2-bit Qwen 3.6 35B-A3B MoE prefill 1.28× faster on CUDA with first tokens 1.27× sooner (multi-turn up to 1.49×); Gemma 4 12B decodes 1.21× faster on Vulkan (up to 1.32× on long context). → Benchmarks
• 🚀 Continuous batching & paged KV cache. vLLM-style paged KV pool with block-hash prefix sharing and an iteration-level scheduler, on by default in the server. The pool's bytes are host-resident, so it buys memory efficiency and prefix reuse rather than throughput that scales with concurrency. → deep dive
• 🧬 DeepSeek V4 Flash (284B MoE) with three whole-model executors. The compressed-sparse-attention 1M-context architecture runs on a direct-CUDA engine (--backend cuda), the native ggml executor (--backend ggml_cuda / ggml_vulkan), and a 100% pure-C# CPU executor (--backend cpu, no native dependencies). Weights layer-split automatically across every visible GPU, so a model far larger than one card still runs; the server hosts it with per-sequence slots and continuous batching. → DeepSeek V4 card
• 🧠 GLM-5.2 (744B-A40B MoE) with tensor parallelism and CPU MoE offload. Multi-head Latent Attention plus a DeepSeek Sparse Attention "lightning indexer" that picks which 2048 cached tokens each query may attend to. --tp N runs every layer on every GPU (heads column/row-parallel, every expert split row-wise; this native GLM path is local/single-process) and --cpu-moe keeps the routed experts — 92% of the checkpoint — in system RAM. The default layer split and --cpu-moe reproduce llama.cpp token-for-token on the same backend; --tp sums per-rank partials, so on a 2-bit MoE its last-bit difference reaches the top-8 router and the near-tied tokens can differ. Head-to-head on 3x RTX PRO 6000: pp2048 918.9 vs llama.cpp's 763.1 tok/s, tg64 43.7 vs 42.2. The advertised 1M context (~93 GiB of KV) is a ceiling rather than a promise — once the weights land the loader sizes the context to the VRAM actually free and logs its pick (342,272 tokens on the layer split, 646,400 with --n-cpu-moe 30); MAX_CONTEXT makes a specific length a hard requirement instead. → GLM card
• 🧠 GLM-5.3-Flash (320B MoE) decodes 2.0× llama.cpp. The hybrid successor — 288 routed experts, KDA linear attention on 34 of 45 trunk layers, NoPE MLA + DSA on the other 11 with a pooled indexer (top-k 2048 over 4-cell pools, then expanded to their members), Sinkhorn hyper-connections ×4 — loads through the same native executor and the same GlmDsaModel as GLM-5.2. On 2× RTX PRO 6000 Blackwell (96 GB), GLM-5.3-Flash-UD-Q2_K_XL (101 GiB), layer split, both engines at n_ubatch 2048 and back to back: tg64 73.5 vs llama.cpp's 36.6 tok/s, with prefill within a few percent either way (pp2048 2014 vs 2070, pp16384 1692 vs 1690, pp32768 1446 vs 1483). Vision through mmproj-BF16.gguf (the GLM-OCR ViT): --image, multi-image and multi-turn image sessions. The layer split across every visible GPU remains the default when --tp is omitted; on GGML GPU backends, --tp N instead runs native local tensor parallelism with head-sharded KDA/MLA, per-rank KDA state and row-sharded routed-expert hidden dimensions. Rank partials reduce before the nonlinear Sinkhorn hyper-connections. --cpu-moe / --n-cpu-moe and per-sequence native slots also work; NextN/MTP speculation is not implemented yet. It now also runs on the 100% pure-C# --backend cpu path (text), though well behind the GGML backends there — ~5.7× off ggml_cpu on prefill and ~2.4× on decode, and its prefill logits sit at cosine 0.9567 against ggml_cpu, which is close but is not proven to be only 2-bit quantization sensitivity, so treat that path as a reference implementation to A/B against rather than bit-parity. → GLM card
• ⚡ Qwen 3.8 Flash Next — the whole token as one graph. A hybrid MoE: GatedDeltaNet recurrent layers on 36 of 48 layers interleaved with full-attention layers (some behind Qwen Sparse Attention's indexer), a PLE n-gram embedding block, ×4 hyper-connection streams and 512 experts with 10 used. Embedding, in-graph PLE, all 48 layers, the final mixer and the LM head run as (almost) one captured graph per token, from a shape-keyed cache; vision rides the Qwen3.5-VL tower with (T,H,W) IMRoPE, including multi-image and multi-turn image sessions with KV reuse across turns (extend-only — the GDN recurrence cannot rewind). --tp N runs a layer split here, not tensor parallelism: on 2× A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB) lands 24.2 GB + 26.2 GB across the two cards with prefill ~1520–1550 t/s and decode ~56 tok/s either way, and greedy output byte-identical to the 1-GPU run — capacity, not speed. → Qwen 3.8 Flash Next card
• 🔮 Speculative decoding — four algorithms over one draft-verify runtime. Multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (NextN block embedded in the trunk — use an MTP-retaining GGUF such as unsloth/Qwen3.6-35B-A3B-MTP-GGUF; the base repo ships the same file names with the block stripped), GLM 5.2 (its NextN block ships in the stock checkpoint — ~1.3× decode, 94% draft acceptance, on 2× RTX PRO 6000 with --n-cpu-moe 20) and Gemma 4 (separate gemma4-assistant draft GGUF, --draft-model); DeepSeek V4 adds DSpark block drafting (--draft-model), which proposes a whole block of tokens per step for 1.3–1.4× decode (up to 2.0× on multi-turn chat). A fourth algorithm needs no trained weights at all: --spec-type ngram matches the sequence's own suffix against the tokens it has already seen, works on every checkpoint, and measured 45.2 tok/s against 31.4 plain (1.44×) on Qwen3.5-9B (Q8_0, ggml_metal, M5 Pro) — a model that ships no draft head — with byte-identical output. In every case the draft proposes, the trunk verifies in one batched forward, and the output matches standard decode. Off by default; opt in with --spec on either host for the trunk-embedded heads, while naming a --draft-model enables speculation by itself. → Speculative decoding
• 🔗 Tensor parallelism & distributed clustering. Split a model across multiple GPUs with --tp N — on the direct cuda backend and on GGML CUDA / Vulkan — and extend across machines with peer-to-peer TCP clustering (--tp-node-id / --tp-peers). Megatron-LM column/row-parallel pattern with hierarchical AllReduce; MoE expert parallelism and per-rank GatedDeltaNet kernels on GGML. Fused per-rank execution makes --tp 2 decode 1.39× a single GPU on Gemma 4 E4B and 1.57× on Muse-Glimmer 30B (which also gains 1.34× prefill — the one model that beats a single GPU on both phases), and runs models that do not fit one card at all (Qwen 3.5-35B-A3B; Muse-Glimmer 30B Q8_0 at 28.2 GB on 24 GB cards). Architectures that shard no weights take the same --tp N as a layer split instead — each GPU holds a contiguous run of whole layers (Qwen 3.8 Flash Next and DeepSeek V4) — which buys capacity rather than speed. GLM 5.x also layer-splits by default, while --tp N selects its native local/single-process TP path on GGML GPU backends for both GLM-5.2 and GLM-5.3-Flash; and an architecture that supports neither now says so on stderr and runs on one GPU instead of silently leaving the others idle. Optional Redis-backed KV cache and Responses API store. → Tensor Parallelism
• 🎨 Qwen-Image-Edit image editing. Prompt + input image → edited image, driving a 60-block MMDiT with a Qwen-Image VAE and Qwen2.5-VL-7B text encoder. CUDA-graph-captured DiT, FlowMatch-Euler true-CFG denoise, live Web UI previews, and a Lightning distillation LoRA fast path (--qwen-image-lora, applied as a runtime side-path over the untouched quantized weights) that takes the default 30 steps × CFG — 60 DiT forwards — down to 4. Beat stable-diffusion.cpp 1.19× on a warm 4-step edit. → Qwen-Image-Edit card
• 🎬🔊 MiniMax-H3 joint audio-video generation. Prompt → video with a native 32 kHz stereo soundtrack, generated together rather than dubbed on: one 19.3B diffusion transformer denoises a packed video+audio latent in a single token sequence, up to 15 s at 24 fps. Text-to-video, image-to-video (the photo becomes the first frame and the prompt drives the motion), first/last-frame morphing, and reference-to-video on the separate Ref2VA checkpoint — up to nine references in any mix of stills (--ref-image), clips (--ref-video, with --ref-video-audio for a clip's own soundtrack) and standalone audio (--ref-audio), each taking its own stretch of the shared timeline before the generated clip, so the person or product carries over while camera, background and composition come entirely from the prompt. All of it CFG-free at 4–8 steps against a 20-step default. Seven native ggml graphs — a 50-layer Qwen3-VL-32B text encoder with its 27-block vision tower and DeepStack taps, the packed-latent DiT with its learned AdaLN curve table and 3-axis float RoPE, a pure-transformer video VAE (36 blocks, no deconvolutions), and an alias-free BigVGAN audio VAE. Frame counts snap to a 17k+5 grid (5, 22, 39, 56, 73, 90 …) and any grid length decodes correctly — the video VAE runs 5 latent frames at a time with a 2-frame look-ahead and cross-fades the seams, while h3_attend pre-scales V by a power of two derived from the key count so that a long clip's unmasked bidirectional attention (8646 packed tokens at 107 frames, against 2364 at 22) stays finite in ggml's FP16 flash-attention accumulator; before that fix a 107-frame clip came back with every pixel black and the audio clamped. Runs 2.4× faster end-to-end than stable-diffusion.cpp at 256×256 and 1.7× at 640×384 on an M5 Pro (ggml_metal); on a 16 GB RTX 3080 Laptop (ggml_cuda) stable-diffusion.cpp takes the end-to-end win instead — 1.15× at 256×256, 1.07× at 640×384 — while TensorSharp stays ahead per denoise step (3.325 s vs 3.338 s), the gap being fixed setup cost of which ~3 s is H.264 encoding and .NET startup rather than inference. Every network verified against the reference: text encoder cos 0.999999, DiT cos 0.998, both VAEs cos 1.000000/0.99999. CLI (--image, --end-image, --ref-image, --video-mode, --no-audio; the soundtrack is written as a sidecar .wav next to the MP4), /api/video-generate, /v1/videos/generations, and two auto-downloading configs — config/minimax-h3-fl2va.json and config/minimax-h3-ref2va.json — that fetch all four networks (~33.5 GB; only the denoiser differs between the two) and load them one at a time, so peak VRAM is the largest of them rather than their sum. All of it also runs on the 100% pure-C# --backend cpu path — text→video, image→video, first/last frame and reference conditioning — which agrees closely with the native ggml route without being bit-identical to it. → MiniMax-H3 card
• 🎬 Wan 2.1 / 2.2 video generation — video only (text → video and image → video). The video-only alternative to MiniMax-H3, and the home of the repository's single biggest speed lever. Prompt → H.264 MP4; on the Wan 2.2 models (TI2V-5B, I2V-A14B) an uploaded image becomes the video's first frame while the prompt drives motion, camera and scene changes. One resident-weight ggml graph per denoise step (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V i2v), causal 3D video VAE encode+decode each as a single graph, A14B's two 14B experts hot-swapped at the timestep boundary, stagewise VRAM handoff — TI2V-5B generates 81-frame 480p image-to-video on a 16 GB GPU in under 8 min, and Wan 2.1 runs 6.0× faster end-to-end than stable-diffusion.cpp on the identical workload. Step-distilled checkpoints are auto-detected from the DiT file name (Turbo / distill / Lightning / lightx2v / FastWan / …-4steps-…) and switch to that step count with guidance off — 4 DiT passes instead of the official recipe's 100, which took the same 1088×832×121-frame image-to-video from 3 h 30 m to 17 m 30 s on an M5 Pro. It is the single biggest speed lever in the repository and needs no flag, only a different --model file. Numerics verified against diffusers (DiT cos > 0.995, VAE encoders cos > 0.999, decode 59.9 dB / >35 dB PSNR). CLI (--image), /v1/videos/generations, and Web UI chat with image upload. → Wan card
• 🌫️ DiffusionGemma text diffusion. Block-wise EntropyBound denoising over a Gemma-4-derived MoE backbone, with CLI flags and a Web UI denoising preview stream. → DiffusionGemma card
• 🖼️ Multimodal. Image / video / audio (Gemma 4); image input for Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer; PDF documents via CLI and Web UI. → Multimodal
• 🛠️ Tool calling & thinking mode. Multi-turn tool calls and structured chain-of-thought across Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer (ATEM markup), and DeepSeek V4 (DSML markup). → Features
• 🧩 Agent Skills. Point the CLI or the server at a folder of skills — a SKILL.md of model-facing instructions plus the scripts, references and assets it needs — and select them per request with "skills": ["pdf"] on any chat API or --skill on the CLI. Selection scopes reach and preference; on tool-capable models only the one-line metadata costs context up front, and the model activates a skill by pulling SKILL.md and references through built-in skills_list / skills_read tools that TensorSharp executes itself, in process. An ordinary OpenAI client therefore never sees a built-in tool call it cannot service. The injected block is a pure function of the sorted selection, so the KV prefix cache keeps matching turn to turn; every model-named path is confined to that skill's own directory. Script execution stays off unless --skills-allow-exec is passed, and the default --skills-sandbox required then runs it under OS confinement or refuses it. Loads the published open-source skills unmodified. → Agent Skills
• 🖥️ Sandboxed code execution. With --code-exec the model gets a patch-first coding surface: bounded read_file, exact-match edit_file, deliberate whole-file write_file, atomic multi-file apply_patch, and shell for tests and commands. A source-locatable failure returns a bounded excerpt and asks for the smallest repair before rerunning. Web/CLI chats retain one workspace; each OpenAI/Ollama HTTP request gets a private workspace across its internal repair rounds and deletes it afterward. Skill scripts share that workspace when code execution is enabled, otherwise they use per-call scratch. Commands start offline; --code-exec-allow-network grants unrestricted host IP networking and is separate from --skills-allow-network and host-performed package installation. Code execution is off by default and the default policy is sandbox-or-refuse: macOS uses Seatbelt and Linux needs bwrap 0.12.0+, with the documented shared-temp/local-IPC and detached-child caveats. Windows requires the explicit --code-exec-unconfined escape hatch, which intentionally leaves filesystem and network access unconfined. → Usage
• 🔌 Ollama- & OpenAI-compatible APIs plus a browser chat UI — drop-in for existing tooling. → HTTP APIs
• 📄 Config files with auto-download. Put CLI/Server options in a reusable JSON file with ${variables} and { "path", "urls" } entries that fetch the model on first run. → config/README.md
• 🧮 Native quantized compute. Q4_K_M / Q8_0 / MXFP4 / IQ2_XXS / IQ2_S / IQ3_S / IQ3_XXS / IQ4_XS and more run in matmul without dequantizing to FP32. Runs on GGML Metal / CUDA / Vulkan, a direct CUDA/cuBLAS backend, MLX (Apple Silicon), and a pure-C# CPU path — all with CPU fallbacks. On MLX the IQ decode kernels amortize the codebook and scale loads across a whole sub-block instead of re-reading them per weight, which took a mixed-IQ 30B from 3.6 to 14.3 tok/s on an M5 Pro (67% of the fused ggml-metal graph, output byte-identical). → Backends
• 🧵 The 100% pure-C# CPU backend now uses many cores. --backend cpu still needs no native library, but its matmuls run on a persistent worker pool instead of a fresh Parallel.For per matmul: ~15% prefill and ~2.8× decode on gemma-4-E4B-it-Q8_0, and the managed video/image paths share the same pool. It deliberately does not take every core — its workers spin, and the rest of the CPU path still uses the ThreadPool — with TS_CPU_THREADS / TS_CPU_POOL / TS_CPU_SPIN to tune it. It also binds quantized weights zero-copy from the GGUF mapping now, exactly as the GGML backends already did, instead of expanding them to F32 at load: GLM-5.3-Flash-UD-Q2_K_XL went from a load that never finished to ~48 s, and any model whose weights used to be copied benefits (TS_DIRECT_QUANT_WEIGHTS=0 restores the old behaviour for an A/B). → Backends · Env-var matrix
• 🧩 Plug-in architecture. A model family, a modality and a chat format are each one table entry — an architecture descriptor, a capability interface, a ChatProtocol — so adding one touches its own directory plus one registration line, and no architecture-name switch is left in the loader, planner, CLI or server. → Adding a model, a modality, or a chat format