FlashRT

Getting Started

• Install FlashRT
• Quick Start
• API snippets — Pi0 / Pi0.5 / GROOT / Pi0-FAST / Qwen3.6
• Qwen3.6-27B NVFP4 LLM path — quickstart, K selection, measured throughput · parameter reference · OpenAI-compatible server example
• Adding a new model
• Contributing
• Architecture

Pi0-FAST max-performance mode (for fixed-prompt 24h deployment):

model = flash_rt.load_model( checkpoint="/path/to/pi0_fast_base", config="pi0fast", decode_cuda_graph=True, # capture decode loop as CUDA Graph decode_graph_steps=46, # action tokens per inference (50 total with text prefix) )

#### Qwen3.6-27B NVFP4 (LLM, RTX 5090)

The LLM path uses a dedicated frontend — same kernel binary, separate
generation API since chat completion has a different surface from VLA
control. See [`docs/qwen36_usage.md`](docs/qwen36_usage.md) for the
full parameter reference and [`docs/qwen36_nvfp4.md`](docs/qwen36_nvfp4.md)
for the K-curve / measured throughput / model-dependency notes.

Build & install

This is the hands-on "go from a fresh machine to a green benchmark" section. For a single-page install reference (prerequisites, troubleshooting table, JAX/transformers pin rationale) see docs/INSTALL.md.

Docker and native Linux paths both produce the same two extension modules:

Crucially — no pip install flash-attn required. The FA2 kernel is vendored at source level and built into flash_rt_fa2.so during cmake/make; at runtime import flash_rt loads both .so files directly, so you never hit the flash-attn wheel's torch × CUDA × driver × glibc compatibility matrix. Setting FVK_RTX_FA2=0 is still supported as a fall-back to pip flash-attn for debugging, but the default path has zero pip-wheel dependency.

Option A — Prebuilt Docker image (fastest, recommended)

The published image already has CUDA 13.0, PyTorch 2.9, the FlashRT kernels prebuilt, and CUTLASS vendored — pull and run, no local compile, no flash-attn wheel hunting:

```bash docker pull ghcr.io/liangsu8899/flashrt:latest docker run --rm --gpus all -it ghcr.io/liangsu8899/flashrt:latest

Option B — Build the Docker image yourself

If you need a different GPU arch, want to pin a specific commit, or prefer to vet the image source:

git clone https://github.com/LiangSu8899/FlashRT.git
cd FlashRT
docker build -t flashrt:dev -f docker/Dockerfile .
docker run --rm --gpus all -it flashrt:dev

Build args (GPU_ARCH, FA2_HDIMS, BASE_IMAGE, CUTLASS_REF) documented in docker/README.md. Cold build on a fresh host is ~25 min (NGC pull + FA2 codegen); warm rebuild ~12 min.

Option C — Native Linux (no Docker)

System requirements:

Create an isolated Python environment first. The build step calls python3 -m pybind11 --cmakedir to locate pybind11 headers, so the Python that runs cmake .. MUST be the same interpreter the .so files will be imported from. System-Python + conda-Python mix-ups are the #1 native-install failure mode.

python3.12 -m venv .venv         # 3.10 / 3.11 / 3.12 all supported
source .venv/bin/activate

Minimum pip list (for the torch frontend; everything must be installed before cmake ..):

```bash

2. Build helpers

pip install pybind11 cmake "numpy>=1.24" safetensors

tracked upstream — see docs/INSTALL.md §JAX for rationale.

pip install jax==0.5.3 jax-cuda12-pjrt==0.5.3 jax-cuda12-plugin==0.5.3 ml_dtypes==0.5.3

Then build:

pip install -e ".[torch]" # or "[jax]" / "[all]"

NOTE: editable mode (-e) is required. The cmake build below drops

compiled .so files into flash_rt/ in the source tree; editable

install makes that directory importable directly. A non-editable

`pip install .` would install a copy BEFORE the .so files exist and

`make install` / `ninja install` step needed.

```

Build timing (one-time)

On a 5090 with CUDA 13 in a warm container, make -j$(nproc):

Subsequent rebuilds of only the hand-written kernels take ~2 min — FA2 is a separate CMake target and is only re-linked, not recompiled, unless the vendored source itself changes.

Slim-build flags (developer iteration speed)

FA2's CUTLASS 3.x templates dominate cold-build cost. The default matrix covers every RTX family card × fp16+bf16 × all 3 hdim buckets, which is right for distribution but overkill when you're iterating on a single 5090/4090 and a single model family. Three opt-in CMake flags trade binary coverage for iteration speed:

Shipped flash_rt_fa2.so size also shrinks — the all-three-slim build produces 17.8 MB (vs 135 MB default), a 87% reduction in binary size on the FA2 module.

Dropped entries still resolve at the Python layer — calling a stubbed entry (e.g. fa2.fwd_bf16 on a build with FA2_DTYPES="fp16") aborts the process with a clear "rebuild with -DFA2_DTYPES=…" message instead of linker errors or silent wrong output.

ccache (iterative C++ rebuild speedup)

If ccache is on PATH at CMake-config time, it is enabled automatically for both C++ and CUDA compiles. First build is unchanged. Hit rate on the .cpp side (pybind bindings) is high, so repeat edits to csrc/bindings.cpp / csrc/fa2_bindings.cpp get fast rebuilds. CUDA .cu files — nvcc's invocation style makes ccache hit rate unreliable, so treat CUDA speedup as a bonus rather than a guarantee. Tip: set CCACHE_DIR to a host-mounted path so the cache survives container rebuilds.

Install via apt-get install ccache (Ubuntu) or equivalent.

Quick Start

Already built? Run the snippet below. Not yet built? See Build & install first — cmake .. && make -j produces the kernel .so files this snippet imports. About 6 minutes from git clone to first inference.

```python import flash_rt # Python module name; project is FlashRT (see About)

model = flash_rt.load_model( checkpoint="/path/to/pi05_checkpoint", config="pi05", # or "pi0", "groot", "groot_n17", "pi0fast" framework="torch", # or "jax" )

actions = model.predict( images=[base_img, wrist_img], prompt="pick up the red block", )

The NVFP4 ckpt has no MTP head; point this env var at a paired

4. JAX-side (optional — only if you will load Orbax checkpoints).

API snippets

Already built? Jump to API examples below. Not yet built? See Build & install for the full Docker / native Linux flow, then come back.

tokenizer API.

pip install "transformers<4.56" pandas pillow pyarrow

to RtxTorchGroot; on Jetson Thor it resolves to ThorPipelineTorchGroot.

model = flash_rt.load_model( "/path/to/groot_checkpoint", config="groot", embodiment_tag="gr1", # see GROOT embodiment slots below ) ```

Versions are pinned because the Orbax/jaxlib/PJRT plugin ABI is

`import flash_rt` would fail at runtime with a missing-module error.

cmake -B build -S . # auto-detects GPU arch cmake --build build -j$(nproc)

Supported Models

Latency columns below are 2-view, pure CUDA Graph replay (p50, see Measurement protocol). All per-view breakdowns live in the Latency sections further down.

---

Comparison

On the same Jetson AGX Thor hardware, FlashRT goes from the original openpi JAX baseline (1.4 Hz) to 23 Hz (FP8) / 25 Hz (NVFP4) — a ~16-18× speedup at zero accuracy loss (cosine ≥ 0.9996 vs the production reference).

FlashRT Pi0.5 Thor numbers above are the NVFP4 production preset (use_fp4=True); the FP8 baseline is 44.0 ms 2v / 54.8 ms 3v at the same task success (491/500). See Latency (Thor) for the full sweep.

<a name="community-benchmarks"></a>