TensorSharp

简介

# TensorSharp

<p align="center"> <img src="imgs/banner_1.png" alt="TensorSharp logo" width="320"> </p>

English | 中文

A C# inference engine for running GGUF language models locally, including autoregressive LLMs and DiffusionGemma-style text-diffusion models. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access.

Highlights

• Continuous batching & paged KV cache — vLLM-style paged KV pool with block-hash prefix sharing and an iteration-level scheduler, on by default in the server. → deep dive
• MTP / NextN speculative decoding — multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (NextN block embedded in the trunk GGUF) and Gemma 4 (separate gemma4-assistant draft GGUF). The draft proposes several tokens per step and the trunk verifies them in one batched forward, with the request's own sampler driving both. Opt in with --mtp-spec (+ --mtp-draft-model for Gemma 4). → Speculative decoding
• DiffusionGemma text diffusion — block-wise EntropyBound denoising over a Gemma-4-derived MoE backbone, with CLI generation flags and a Web UI denoising preview stream. → DiffusionGemma card
• Multimodal — image / video / audio inputs (Gemma 4); image inputs for Gemma 3, Qwen 3.5-family, Mistral 3, and Nemotron-H Omni. → Multimodal Support
• Tool calling / function calling — multi-turn tool calls across all three API styles, with architecture-agnostic output parsing. → Tool Calling
• Thinking / reasoning mode — structured chain-of-thought for Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, and Nemotron-H. → Thinking Mode
• Ollama- & OpenAI-compatible APIs — drop-in endpoints for existing tooling, plus a browser chat UI. → HTTP APIs
• Native quantized compute — Q4_K_M / Q8_0 / MXFP4 / IQ2_XXS and more run in matmul without dequantizing to FP32.

---

Everything below is detailed reference. New here? The five sections above are all you need to get running.

Features

• Multi-architecture support -- Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Mistral 3
• Multimodal inference -- image, video, and audio inputs (Gemma 4); images for Gemma 3 / Qwen 3.5-family / Mistral 3 / Nemotron-H Omni
• Thinking / reasoning mode -- structured chain-of-thought output with <think> / <|channel>thought / <|channel>analysis tags (Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H)
• Tool calling / function calling -- models can invoke user-defined tools; multi-turn tool-call conversations supported across all three API styles
• Quantized model support -- loads GGUF files with Q4_K_M, Q8_0, F16, MXFP4, and other quantization formats; performs native quantized matmul without dequantizing to FP32, including memory-efficient pure C# CPU loading for large GGUFs
• GPU-accelerated -- GGML Metal on macOS, GGML CUDA on Windows/Linux with NVIDIA GPUs, a direct CUDA/cuBLAS backend with PTX kernels, and an MLX backend for Apple Silicon (mlx-c / Metal), all with CPU fallbacks for unsupported ops
• Optimized pure C# CPU backend -- managed GEMM fast paths plus fused SIMD kernels for RMSNorm, RoPE, softmax, fused activations, and other inference hot paths
• Continuous batching & paged KV cache -- vLLM-style block-paged KV pool with block-hash prefix sharing across requests, iteration-level scheduler that admits / preempts sequences mid-batch, optional SSD-backed tier for very large KV working sets, and a native fused paged-attention kernel (TSGgml_PagedAttentionForward) that drives ggml_flash_attn_ext on Metal/CUDA. Enabled by default in TensorSharp.Server; opt-out with --no-continuous-batching. See docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md.
• MTP / NextN speculative decoding -- multi-token-prediction draft heads accelerate solo (non-concurrent) decode. Qwen 3.6 ships its NextN block fused into the trunk GGUF; Gemma 4 loads a separate EAGLE-style gemma4-assistant draft GGUF via --mtp-draft-model whose draft layers attend the target's own KV cache. The draft proposes up to --mtp-draft tokens per step (kept while draft confidence ≥ --mtp-pmin) and the trunk verifies them in a single batched forward; the request's own sampler — penalties included — drives both drafting and verification, so output is identical to standard decode. Opt in with --mtp-spec (off by default). On ggml backends fused multi-token-verify / draft-step kernels make it a clear win; the pure-C# cuda backend runs a fully GPU-resident per-op verify/draft and is also a win. CPU / MLX stay on standard decode. Env: TS_MTP_* (shared) and TS_GMTP_* (Gemma 4 tuning).
• Batched / parallel inference -- IBatchedPagedModel.ForwardBatch implementations for Mistral 3, Gemma 4, GPT OSS, Qwen 3, Qwen 3.5/3.6-family, and Nemotron-H all run by default and pack N sequences into a single forward pass with paged K/V scatter and per-sequence attention via the native kernel. Each model exposes a TS_<FAMILY>_BATCHED=0 escape hatch (e.g. TS_GEMMA4_BATCHED=0, TS_QWEN35_BATCHED=0, TS_GPTOSS_BATCHED=0, TS_NEMOTRON_BATCHED=0) to fall back to the per-sequence KV-swap path for A/B comparison or regression isolation.
• Ollama & OpenAI API compatibility -- drop-in replacement endpoints for existing tooling
• Configurable sampling -- temperature, top-k, top-p, min-p, repetition/presence/frequency penalties, seed, stop sequences
• Chat templates -- auto-loaded from GGUF metadata (Jinja2), with hardcoded fallbacks per architecture
• Inference engine -- the new InferenceEngine (worker-thread scheduler + paged block pool) replaces the legacy single-request FIFO queue inside TensorSharp.Server. The old queue object is now a compatibility shim for status/event shapes; the engine itself handles concurrency.
• Batch processing -- JSONL input support in the console application, plus a built-in inference benchmark for prefill/decode throughput
• Streaming -- token-by-token output via SSE (web) or stdout (console), with abort/stop support for in-flight generations
• Text-diffusion generation -- DiffusionGemma uses an iterative EntropyBound denoising sampler instead of autoregressive Forward(). The CLI exposes --diffusion-steps, --diffusion-seed, and --diffusion-blocks; the Web UI streams whole-message replace events for live denoising previews and batches concurrent diffusion requests through DiffusionBatchScheduler.
• Hybrid SSM-Transformer -- Nemotron-H mixes Mamba2 SSM layers, attention-only layers, and MoE FFN layers in a single model. The Mamba2 step has both a per-sequence native kernel and a batched native kernel (TSGgml_NemotronMamba2BatchedStepF32, NEON SIMD + GCD parallelism) used by the batched path.
• Hybrid Attention-Recurrent -- Qwen 3.5/3.6-family models mix full-attention layers with GatedDeltaNet recurrent layers; the batched path keeps recurrent running state in a per-slot recurrent-state pool
• Mixture of Experts -- Gemma 4 MoE variants (e.g. gemma-4-26B-A4B), GPT OSS MoE (e.g. gpt-oss-20b), Qwen 3.5/3.6-family MoE (qwen35moe / qwen3next variants such as Qwen3.5-35B-A3B), and Nemotron-H MoE FFN layers
• Batched GPU MoE -- a single fused GGML graph dispatch handles all selected experts (plus the optional shared expert and residual add) for Qwen 3.5/3.6-family and Nemotron-H decode, eliminating per-expert round-trips
• KV cache codecs -- pluggable codec interface (IKvBlockCodec) with a built-in TurboQuant (Q4 / Q8) compressed codec for paged blocks, configurable via --paged-kv-quant-bits
• Message editing -- edit or delete previous messages in the web chat UI and regenerate from that point
• Text/Image/Audio/Video uploads -- the web UI accepts file uploads up to 500 MB, with automatic token-budget-aware truncation for large text files
• Per-turn observability -- structured logs capture the full user input and the full raw assistant output (both <think> reasoning and the final result) plus the KV cache hit ratio. The same cache-hit stats are surfaced through every API: prompt_cache_hit_tokens / prompt_cache_hit_ratio (Ollama), usage.prompt_tokens_details.cached_tokens (OpenAI), and promptTokens / kvReusedTokens / kvReusePercent in the Web UI SSE done event