GPU Llama3

Prerequisites

• Java 21 or 22+ — required for the Vector API & TornadoVM. Each line has its own artifact (-jdk21 / -jdk22plus), matching TornadoVM's. jitllm runs on either line; jitllm4j needs a Java 25 java on the PATH to start, and runs the model on the JDK in JAVA_HOME.
• TornadoVM with an OpenCL, CUDA, or Metal backend. jitllm/jitllm4j auto-detect whichever backend your installed SDK was built with.
• GCC/G++ 13+ — to build TornadoVM's native components.

`jitllm4j` — zero-dependency Java 25 script

Same backend auto-detection as jitllm. A single-file Java 25 launcher for basic runs (needs java 25+ on your PATH). It has no subcommands and only the core options; use jitllm for the server, multi-GPU (--devices) and the prefill, native-library and cache options.

./jitllm4j --gpu --verbose-init --metal \
  --model Mistral-7B-Instruct-v0.3.Q8_0.gguf --prompt "what is java"

1. Install a TornadoVM SDK (bundles the GPU runtime)

curl -s "https://get.sdkman.io" | bash && source "$HOME/.sdkman/bin/sdkman-init.sh" sdk install tornadovm tornado --devices # confirm your GPU is listed

2. Run a model on the GPU — no build required, via JBang

jbang jitllm@beehive-lab -m beehive-llama-3.2-1b-instruct-fp16.gguf -p "Explain GPU acceleration in one sentence." ```

Ready-to-run GGUF models are in the beehive-lab Hugging Face collections.

-----------

🛠️ Install & build

Build from source

git clone https://github.com/beehive-lab/jitllm.git && cd jitllm

export TORNADOVM_HOME=/path/to/tornadovm-sdk     # e.g. ~/.sdkman/candidates/tornadovm/current
export PATH="$TORNADOVM_HOME/bin:$PATH"
mvn clean package -DskipTests                    # or ./mvnw; drop -DskipTests to run the unit tests

./jitllm --gpu --model model.gguf --prompt "..."

Nothing is resolved from a Maven repository for TornadoVM, so there is no version to pass: switch SDKs by changing TORNADOVM_HOME, and clean when you do.

scripts/tornadovm-dev.sh is what CI uses to build TornadoVM develop. Installations live under ~/.jitllm/tornadovm/<backend>-jdk<N>/<commit>-r<recipe>/ and are immutable; current is only a convenience pointer to the last one prepared, and nothing is deleted unless you run scripts/tornadovm-dev.sh prune --yes (which removes every installation on that line except current; do not run it while a build or an inference process is using an older one).

-----------

🚀 JBang — run without building

Script-like startup à la Jlama, powered by JBang:

```bash curl -Ls https://sh.jbang.dev | bash -s - app setup

⏱️ Quickstart (60 seconds)

```bash

Run configuration at startup

With -v / --verbose, the CLI prints an aligned summary to stderr, after preparing the session and before printing generated text. It shows the model filename, parameter count and file size, GGUF quantization alongside loaded weight types, device, TornadoVM version on GPU runs, execution mode, context capacity, batched prefill chunk width, KV-cache precision, MMA/tensor-core selection, native libraries, CUDA graphs, staged transfers, GPU allocation budget, and sampling settings. GPU memory estimates split weights, KV cache, and workspace (including staging and control buffers), with a total. These predict allocation-budget charges, not physical VRAM use; conservative predictions are labeled. CPU runs instead show common-pool worker parallelism plus the caller and the configured tensor/Q4 Vector API settings. Tensor-core and native-library details describe the selected prefill path; native libraries choose their own kernel algorithms.

Startup timings separate model loading, plan construction, TornadoVM JIT precompilation, and initial device setup. Device setup includes uploads and execution (including CUDA graph capture when enabled), so it is not a pure transfer measurement. “Ready to generate” measures elapsed startup time through session preparation. Prefill and decode performance still appear after generation.

Without --verbose, startup diagnostics are hidden and session preparation stays lazy; errors, warnings, generated text, and final performance metrics remain visible. Verbose output includes SDK location, dimensions/head counts, training context, execution path, and memory-estimate assumptions. Startup timings are printed once; the ending performance block contains only request metrics.

--verbose-init remains a hidden deprecated alias for --verbose. For direct Java launches, use -Djitllm.verbose=true. The legacy -Djitllm.EnableTimingForTornadoVMInit=true setting still enables the report and, in addition, the older per-stage initialization log lines; --verbose no longer sets it. The full Java command is printed by --show-command, not by --verbose. The summary is CLI-only; library callers can explicitly use GenerationSession.prepare() to prepare a session and obtain its execution settings without advancing its position.

-----------

Force a specific backend (only needed for multi-backend SDKs)

./jitllm --gpu --cuda \ --model beehive-llama-3.2-1b-instruct-fp16.gguf \ --prompt "Explain the benefits of GPU acceleration." ```

Swap in any supported GGUF model — e.g. beehive-llama-3.2-3b-instruct-fp16.gguf or ...-8b-....

GPU runs prefill the prompt in batches and use cuBLAS/cuDNN (CUDA) or MLX (Metal) where they are installed; --no-batch-prefill and --no-native-libraries turn these off. See Fast prompt processing on CUDA.