AMD Strix Halo LLM 微调指南

The stack we'll build

---

1. Install latest stable mainline kernel from kernel.ubuntu.com/mainline/ — 6.19.14 was tested,

Apply scripts/fix-kernel-run-parts.py to the .debs before installing

7. Build flash-linear-attention from patched source (see Step 4 below)

8. Build bitsandbytes from source for ROCm gfx1151 (see Step 5 below)

Install

Download the four .deb files from https://kernel.ubuntu.com/mainline/v6.19.14/amd64/:

linux-headers-6.19.14-061914_*_all.deb
linux-headers-6.19.14-061914-generic_*_amd64.deb
linux-image-unsigned-6.19.14-061914-generic_*_amd64.deb
linux-modules-6.19.14-061914-generic_*_amd64.deb

Install:

sudo dpkg -i linux-headers--all.deb -fixed.deb sudo update-grub && sudo reboot ```

Install the script — explicit root:root + 0755 so the NOPASSWD sudoers

for reproducible dates. The quick-start's torch-only install gets them too.

(it's a verbatim copy of the patched cumsum.py from a working install).

GUIDE=/path/to/strix-halo-llm-finetune-guide python3 $GUIDE/scripts/fla_repatch.py \ --fla-root /path/to/fla-patched \ --cumsum-backup $GUIDE/scripts/cumsum-pytorch.py

Install editable

pip install -e . ```

Re-run fla_repatch.py after every git pull of FLA. It's idempotent — running it on already-patched code is a no-op.

---

Build

PATH=/opt/rocm-7.1.0/bin:$PATH cmake --build build --config Release

on PyTorch 2.10/2.11 + HIP 7.13, but the build produced rocm83.so.

Install editable (replaces PyPI bnb)

pip uninstall -y bitsandbytes pip install -e . ```

Step 6 — llama.cpp HIP build (for inference)

If you want to run the resulting fine-tune via llama-server, build llama.cpp with the --gcc-install-dir flag (without it, ROCm 7.1.0's clang-20 can't find <cmath>):

git clone https://github.com/ggml-org/llama.cpp /path/to/llama.cpp
cd /path/to/llama.cpp
PATH=/opt/rocm-7.1.0/bin:$PATH \
cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON \
  -DGGML_HIP_ROCWMMA_FATTN=OFF \
  -DGGML_HIP_GRAPHS=ON \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DGGML_HIP_NO_VMM=ON \
  -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_HIP_FLAGS="--gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/13"
PATH=/opt/rocm-7.1.0/bin:$PATH cmake --build build --parallel $(nproc)

Then symlink the binaries to where §6b and the eval harness expect them — RUNPATH is baked to build/bin (see the move warning below), so a symlink is safe; the binary still resolves its .sos from the real build dir:

sudo ln -sf "$PWD/build/bin/llama-server"     /usr/local/bin/llama-server
sudo ln -sf "$PWD/build/bin/llama-perplexity" /usr/local/bin/llama-perplexity

GGML_HIP_GRAPHS=ON is now upstream default (b867+) but explicitly enabling doesn't hurt.

GGML_HIP_ROCWMMA_FATTN=OFF is intentional despite being the AMD-recommended setting for RDNA 3.5. On gfx1151 specifically, the rocwmma flash-attention path is dramatically slower than llama.cpp's runtime FA at any non-trivial context depth — about 2.4× slower on prefill at 8k context on both dense Qwen3.5-27B and MoE Qwen3.6-A3B. TG is unaffected (memory-bandwidth-bound). Hardware-verified A/B with numbers + reproduction scripts in rocwmma-fattn-sweep/. Earlier versions of this guide recommended ON; that was wrong and is now corrected.

Minimum build for --spec-type draft-mtp GPU dispatch: b9180+ (community-reported — see the u/kant12 credit in §6b; our own b1270 attempt also used llama-cli, so it doesn't independently bisect the floor). Older builds (we tried b1270 via lemonade's prebuilt) will accept the --spec-type draft-mtp flag without complaint but never dispatch the draft model to the GPU — the process pegs a CPU core at 0% GPU and never makes progress. Symptom is silent. And use llama-server, not llama-cli for the speculation path; we burned hours on this and the llama-cli path doesn't wire the draft dispatcher the same way. Settings shown in §6b below.

Build in the directory you intend to keep it in. cmake bakes the absolute build/bin path into the binary's RUNPATH, so if you build in /tmp/llama.cpp-test/ and then move the tree to /srv/aurora-ai/llama.cpp/, the resulting binary will fail to find its shared libraries (libllama-server-impl.so etc.) on launch. Reconfigure + rebuild in the final location, or use patchelf --set-rpath. We hit this swapping our own production build from a staging dir.

---

Fine-Tuning 27B+ LLMs on AMD Strix Halo — A Home Enthusiast's Guide

A reproducible recipe for fine-tuning Qwen3.5-27B (or larger) hybrid LLMs on a single AMD Strix Halo APU (Ryzen AI MAX+ 395, Radeon 8060S, gfx1151) with 128 GB of unified memory — including the patches, system tuning, and out-of-process evaluation orchestrator that make multi-day training runs survivable on consumer hardware.

Status: Tested on a Corsair AI Workstation 300 (Sixunited AXB35-02 board) running Ubuntu 24.04 LTS, mainline kernel 6.19.14 (as tested; 6.19 now EOL — use 7.0.x, see Upgrade-path gotchas), ROCm 7.13 nightly. The same recipe should work on Framework Desktop, GMKtec EVO-X2, FEVM FA-EX9, Bosgame M5 — any AXB35-02 / Strix Halo system.

---

Quick start (for the impatient)

```bash

(see configs/grub-cmdline.example), then sudo update-grub && reboot

6. Set up venv + nightly PyTorch

python3 -m venv /path/to/venv source /path/to/venv/bin/activate pip install --pre \ "torch==2.11.0+rocm7.13.0a20260506" \ "torchvision==0.26.0+rocm7.13.0a20260506" \ "torchaudio==2.11.0+rocm7.13.0a20260506" \ "triton==3.6.0+rocm7.13.0a20260506" \ --index-url https://rocm.nightlies.amd.com/v2-staging/gfx1151/ \ --extra-index-url https://pypi.org/simple/

9. Set up Telegram alerts (optional — see Step 9 below)

Configure with ROCm 7.1.0 toolchain + gcc-13 for clang's libstdc++ lookup

PATH=/opt/rocm-7.1.0/bin:$PATH \ cmake -G Ninja \ -DCOMPUTE_BACKEND=hip \ -DBNB_ROCM_ARCH="gfx1151" \ -DCMAKE_BUILD_TYPE=Release \ -DROCM_VERSION=83 \ -DCMAKE_HIP_FLAGS="--gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/13" \ -S . -B build

Step 6b — Inference settings for Qwen3.5 / Qwen3.6

If you're serving the fine-tune (or any Qwen3.5/3.6 base model) via llama-server for chat or tool-call use, a few runtime settings beyond the build flags matter on this hardware. These are what we run in production:

```ini

libhsa-runtime64.so has a null-ptr bug on gfx1151. Point at your venv's

harness use). Adjust python3.X to your venv.

Environment=LD_LIBRARY_PATH=/path/to/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib:/opt/rocm/lib ExecStart=/usr/local/bin/llama-server \ -m /path/to/your-qwen35.gguf \ -ngl 999 \ -c 32768 \ --fit off \ --no-mmap \ --reasoning-budget 0 \ --temp 1.0 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.00 \ --host 0.0.0.0 \ --port 8080 ```

Per-flag rationale:

• --no-mmap is the gfx1151 gotcha — mmap-only loading triggers a ~30 min GPU page-table setup wall on the unified-memory path. Either --no-mmap or --mmap --direct-io together work; mmap alone hangs. Documented across multiple Strix Halo issues; not specific to llama.cpp.
• --fit off disables llama-server's auto-fit; we keep it off across the board (with explicit -ngl/-c, the sizing heuristic is unnecessary).
• LD_LIBRARY_PATH overlay (the Environment= line above) — stock ROCm 7.1.0's libhsa-runtime64.so has a null-pointer bug on gfx1151 that surfaces as crashes/hangs at model load. Prepend the nightly runtime from PyTorch's _rocm_sdk_core wheel so it wins resolution. Same overlay the repo's benchmark (rocwmma-fattn-sweep/bench.sh) and eval harness (scripts/eval_via_llama_perplexity.py) rely on; the §6b numbers below were measured with it.
• --reasoning-budget 0 disables the thinking block. Strongly recommended for tool-call workflows — Qwen3.5/3.6's native chat template emits tool calls inside the <thinking> block, and if the reasoning budget runs out mid-call the response stream looks empty to the client. Leave thinking on only for pure-chat-no-tools workloads where reasoning visibly helps.
• Sampling: --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 is the unsloth-recommended set for Qwen3.5/3.6 with reasoning off. Their per-model sampling guidance is worth following — meaningfully better than llama.cpp's defaults for coherence on this family. See unsloth's Qwen3.6 docs for the per-mode (reasoning vs non-reasoning) recommendations.
• KV cache quantization (--cache-type-k q4_0 --cache-type-v q4_0) is reported to give measurable memory-bandwidth gains at long context with minimal quality loss on Qwen3.5/3.6. We haven't benched it ourselves yet on this hardware (production is at the F16 cache default, 8k context where the bandwidth pressure is lower) — adding when we do. If you're running long-context (32k+) chat workloads, it's worth trying.

For tool-call agents specifically (Continue, Codex CLI, Roo, OpenClaw, aichat, etc.), also note:

• Custom Jinja template required for Qwen3-Coder-Next. The native template emits XML <tool_call><function=...>...</function></tool_call> which trips clients expecting Hermes-style JSON {"name": ..., "arguments": ...}. Swap via --chat-template-file <your-hermes.jinja>. Templates for Qwen3-Coder-Next + Nemotron-3-Super in Hermes format are floating around HuggingFace and the ggml-org/llama.cpp issue tracker.
• Disable thinking for tool workflows specifically. Even on models where you want thinking for chat, route tool-call/agent workflows to a separate llama-server instance (or a separate role binding in your client config) with --reasoning-budget 0.

Step 9 — Telegram alerts (optional but nice)

scripts/tg_alert.sh is a 50-line bash helper that sends HTML messages to a Telegram bot. Set up:

1. Talk to @BotFather on Telegram, create a bot, save the token.
2. Message @userinfobot /start and it returns your numeric chat ID immediately.
3. Store the credentials. Quote the values — Telegram tokens contain : and _ and other characters that can confuse a source if the value isn't quoted:

sudo mkdir -p /etc/strix-halo
sudo tee /etc/strix-halo/telegram.env > /dev/null <<EOF
TELEGRAM_BOT_TOKEN="<your-token>"
TELEGRAM_CHAT_ID="<your-chat-id>"
EOF
sudo chown "root:$(whoami)" /etc/strix-halo/telegram.env
sudo chmod 0640 /etc/strix-halo/telegram.env

1. Test:

```bash ./scripts/tg_alert.sh "<b>Test</b> — Strix Halo guide setup OK"

rocm / rocm-sdk-* are pulled in transitively by torch; they're pinned above only

_rocm_sdk_core/lib (same overlay rocwmma-fattn-sweep/bench.sh + the eval

CRITICAL gotcha — the namespace package shadow

If a previous setup left /path/to/venv/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_rocm82.so lying around (a symlink to a non-existent file from an older bnb install), Python treats that directory as a namespace package — and silently shadows your editable install. Symptom: import bitsandbytes; print(bitsandbytes.__file__) returns None, no .optim attribute. Cure:

```bash rm -rf /path/to/venv/lib/python3.12/site-packages/bitsandbytes

ROCm vs Vulkan — backend selection depends on precision

Inference on Strix Halo can run through either of two llama.cpp backends, and the right choice is not the same for every workload:

• ROCm/HIP — the production backend this guide builds in Step 6. Used by all the numbers in the table above. Required for training (PyTorch + ROCm 7.13 nightly).
• Vulkan (RADV STRIX_HALO) — Mesa's Vulkan driver, with cooperative-matrix path. Built with -DGGML_VULKAN=ON (no HIP). Recipe in vulkan-vs-rocm-sweep/build-vulkan.sh.

Tested on Qwen3.6-35B-A3B at the same source commit (b9296), same hardware, same bench shape:

Q4_K_M (quantized) — Vulkan wins decode by ~22%:

BF16 (full precision) — ROCm wins decode by ~117%:

The reason is visible right in Vulkan's own capability report on launch:

ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | ...
                                                                    ^^^^^^^
                                                                no native BF16

bf16: 0 — RADV STRIX_HALO supports FP16 cooperative matrix natively but not BF16; the Vulkan backend falls back to slower kernels for BF16 ops. ROCm/HIP has BF16 wired through native HIP matmul kernels and dominates anything BF16-bound.

Practical recommendation:

Full sweep + per-shape numbers + capability extract + the build recipe for the Vulkan binary in vulkan-vs-rocm-sweep/. Long-form writeup with the methodology, all depths, and the bf16: 0 deep-dive: ROCm vs Vulkan on AMD Strix Halo: when each wins, and why it inverts at the precision boundary. The Vulkan canonical dashboard for Strix Halo (with deeper per-model Vulkan numbers) is bench.ciru.ai; this guide is the canonical ROCm + training reference.

---

Troubleshooting

The failure modes that cost us the most time, indexed. Each links to the step with the full fix.