Krasis

Tradeoffs And Requirements

- Krasis currently targets NVIDIA GPUs with CUDA, including Ampere and newer architectures. The production HQQ attention and compact KV cache modes do not require FP8 support. - Input models should be BF16 safetensors from Hugging Face or another local safetensors source. - First run is slower because Krasis builds optimized local caches. Later runs reuse those caches. - Disk usage must cover the source model plus Krasis cache artifacts under ~/.krasis. - System RAM should be sized for the selected quantized cache and HCS backing store. Larger models need substantial RAM even when GPU VRAM is limited. - Production runs should use quantized INT4/INT8 expert caches and HQQ attention. BF16-heavy modes are validation/debug modes, not normal deployment targets.

Requirements

- Native x86-64 Windows, Linux (including Ubuntu 24.04+), or WSL2 - Python 3.10+ on Linux/WSL; native Windows uses the release-pinned private Python included by the installer - NVIDIA GPU with CUDA drivers installed - Rust is only needed for source builds, not normal wheel installs - Enough disk/RAM for the source model and generated Krasis caches

2. Install CUDA Dependencies

krasis-setup

This installs runtime CUDA/PyTorch dependencies when needed. It is usually only required once per machine.

Install

The current release is v1.0.16.

Native Windows: Download the Krasis Windows installer. It installs Krasis for the current user, includes its own Python runtime, and adds Krasis to the Start Menu.

Linux or WSL2:

curl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash -s -- prerelease

See all releases for older versions and individual wheel/source assets.

You can contact me here, but for bugs, setup problems, model requests, or feature requests please open a GitHub issue.

If you want to monitor Krasis during runs, check out ktop.

1. Install Krasis

Linux/WSL:

curl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash -s -- prerelease

This creates a managed environment at ~/.krasis/venv, installs Krasis, symlinks commands into ~/.local/bin, and updates PATH for the current shell. No sudo is required for the Krasis install itself. Omit prerelease when installing the latest stable release.

Native Windows:

Download KrasisSetup-1.0.16-win64.exe. The installer creates a per-user install under %LOCALAPPDATA%\Programs\Krasis, installs and validates a release-pinned private Python/Krasis/PyTorch runtime, and adds Krasis and Krasis Manager shortcuts to the Start Menu folder. It never uses or modifies a system Python. Krasis opens the native interactive launcher in a maximized, resizable console; Krasis Manager starts the localhost management dashboard. Models and caches still live under %USERPROFILE%\.krasis. The first install downloads the pinned CUDA PyTorch wheel and can take several minutes.

Native Windows packages Marlin, FlashAttention, and FLA sidecars for supported Ampere and newer NVIDIA architectures.

Uninstall Krasis, keeping model files

curl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash -s -- --uninstall ```

On native Windows, download and run the desired stable or prerelease installer from the Krasis releases page.

Source Build

For development builds:

git clone https://github.com/brontoguana/krasis.git
cd krasis
./dev build
./dev run qcn

DeepSeek-V4-Flash-0731 is available as ./dev run dsv4 (also deepseek-v4). Fresh launcher selections use INT4 experts, HQQ6 attention, and the architecture-owned Native cache. HQQ6 measured 4.838535 WikiText-2 perplexity, +0.466% versus HQQ8/Native, and passed the four-prompt witness. The measured HQQ8/expanded-BF16 comparison profile matched the accepted BF16 quality gate and measured 30.08/29.86/29.29 tok/s internal decode for 50/100/250-token outputs. An explicit Native cache stores the checkpoint's existing E4M3/E2M1 QAT state without a second quantizer and increases the measured 1,000 MiB context capacity from 149,808 to 294,432 tokens. Native currently regresses 1K/5K prefill by 8.45%/6.22% against expanded BF16 while remaining effectively flat at 10K and above; the launcher nevertheless defaults to Native to prioritize context capacity, with expanded BF16 retained as an explicit faster-short-prefill mode.

The ./dev entry point handles environment setup and is preferred for local development commands. Its general config shortcuts are qcn, dsv4, and gemma; pass an explicit validated .conf path for other models. Shortcuts that resolved to deprecated KV configurations were removed rather than kept as commands that fail at startup.

Quick Start

Usage

Use saved config

krasis --non-interactive

Use a config file

krasis --config tests/qcn-k4v4-hqq8-int4-benchmark.conf

API

Krasis exposes an OpenAI-compatible chat endpoint:

http://localhost:8012/v1/chat/completions

Useful endpoints:

• GET /health
• GET /v1/models
• POST /v1/timing