NanoCamelid 推理引擎

Requirements

• Raspberry Pi 5 or another ARM64 Linux machine
• Rust toolchain
• A local GGUF model file

Raspberry Pi Deployment

Prepare a Pi workspace:

./scripts/pi/bootstrap.sh

Build and test remotely:

./scripts/remote_build.sh <pi-host> [ssh-key] [pi-user]

Preview the resolved deploy/build/readiness plan without SSH:

./scripts/remote_build.sh <pi-host> [ssh-key] [pi-user] git-ff --dry-run

On a prepared Pi workspace, remote_build.sh now runs the same 1B readiness gate as scripts/pi/ready-1b.sh: it prefers the Pi-local Llama-3.2-1B-Instruct-Q4_0.gguf, falls back to ...Q8_0.gguf, then runs inspect, scalar-vs-selected smoke validation, and one direct chat turn by default. Disable that model-backed gate explicitly with:

NANOCAMELID_REMOTE_SMOKE=0 ./scripts/remote_build.sh <pi-host> [ssh-key] [pi-user]

false and no are also accepted falsey values for NANOCAMELID_REMOTE_SMOKE.

To force a specific GGUF path that already exists on the Pi:

NANOCAMELID_REMOTE_SMOKE_GGUF=/path/on/pi/model.gguf \
./scripts/remote_build.sh <pi-host> [ssh-key] [pi-user]

To override the default smoke kind, prompt, or token budget:

NANOCAMELID_REMOTE_SMOKE_KIND=model \
NANOCAMELID_SMOKE_PROMPT="Hello" \
NANOCAMELID_SMOKE_TOKENS=1 \
./scripts/remote_build.sh <pi-host> [ssh-key] [pi-user]

To keep the remote gate at inspect+smoke only:

NANOCAMELID_REMOTE_READY_CHAT=0 ./scripts/remote_build.sh <pi-host> [ssh-key] [pi-user]

Remote builds default to clean git fast-forward deployment so Pi-local edits are not overwritten by validation runs. For explicit snapshot deployment, pass rsync as the fourth argument or set NANOCAMELID_DEPLOY_MODE=rsync.

Quick Start

Install the latest release build from GitHub:

curl -fsSL https://raw.githubusercontent.com/timtoole02/NanoCamelid/main/scripts/install.sh | bash

The installer clones NanoCamelid, builds the release binary with Cargo, and links nanocamelid into ~/.local/bin. Override paths when needed:

curl -fsSL https://raw.githubusercontent.com/timtoole02/NanoCamelid/main/scripts/install.sh | \
  env NANOCAMELID_INSTALL_DIR=/mnt/nanocamelid/src/NanoCamelid \
    CARGO_TARGET_DIR=/mnt/nanocamelid/target \
    bash

On Pi workspaces mounted at /mnt/nanocamelid, the installer uses /mnt/nanocamelid/target by default unless CARGO_TARGET_DIR or NANOCAMELID_TARGET_DIR is set.

Manual checkout still works:

git clone https://github.com/timtoole02/NanoCamelid.git
cd NanoCamelid

cargo run -- probe
cargo run -- inspect /path/to/model.gguf
cargo run --release -- smoke q8-model /path/to/model.gguf "Hello" 1
cargo run --release -- smoke q8-chat /path/to/model.gguf "Say hello in one sentence." 8
NANOCAMELID_MODEL_GGUF=/path/to/model.gguf cargo run --release -- tui 0.0 64

probe prints CPU and SIMD feature information. inspect reads GGUF metadata and tensor layout. smoke q8-model loads a Q8_0 model, checks scalar/runtime logit parity, and runs a short greedy generation path from directly tokenized prompt text. smoke q8-chat runs the same parity/generation validation through the tokenizer chat template so Llama 3.2 1B Instruct rows can be smoke-tested through the real instruct prompt path. Set NANOCAMELID_MODEL_GGUF to reuse the same GGUF path across repeated inspect, generate, chat, and tui runs, or NANOCAMELID_SMOKE_GGUF to override that shared default just for smoke validation.

For the standard local validation gate, use:

NANOCAMELID_TARGET_DIR="/Volumes/SSK Drive/nanocamelid-target" ./scripts/validate.sh
./scripts/validate.sh --dry-run

On macOS, validate.sh refuses to guess a default target directory or use a non-/Volumes target. Set CARGO_TARGET_DIR or NANOCAMELID_TARGET_DIR to an external drive path first so the repo does not create build artifacts on the internal disk. On prepared Pi workspaces, the same script defaults to /mnt/nanocamelid/target. The gate also runs cargo run -- ready 1b --dry-run so the default Llama 3.2 1B readiness command path stays covered without requiring the GGUF during local validation.

Single-turn generation is available through either raw prompt text or a rendered chat prompt:

NANOCAMELID_MODEL_GGUF=/path/to/model.gguf \
  cargo run --release -- generate "Hello" 0.0 32

NANOCAMELID_MODEL_GGUF=/path/to/model.gguf \
  cargo run --release -- chat "Say hello in one sentence." 0.0 32

tui opens an interactive terminal chat that keeps the model loaded, shows the connected model path/name, selected Q8 kernel, chat renderer, and per-turn plus session token-in/token-out counters, TTFT, and throughput. The prompt surface is closer to a modern assistant CLI: slash commands expose model switching, nearby model discovery, live decoding settings, system prompts, transcript saving, and session history without restarting the process.

NANOCAMELID_PREFILL_BATCH controls how many prompt tokens are ingested at once before decode begins. The default is 16. Set it to 1 for the old single-token reference behavior, or use bench q4-prefill to compare candidate batch sizes on the current host without loading a GGUF model. On a prepared Pi workspace, ./scripts/pi/bench-1b-prefill.sh sweeps the real Llama 3.2 1B chat path across prefill batch sizes and prints the model-backed prompt ingestion timing for each run.

Set NANOCAMELID_TRACE=1 on generate, chat, or tui runs to print an aggregate stage-level timing summary. It is intended for focused tuning: the summary groups decode and batched prefill work by layer stage so slow paths can be identified before changing kernels.

For very long-context GGUFs, NANOCAMELID_CONTEXT_LIMIT can cap the runtime KV cache during local smoke tests:

NANOCAMELID_CONTEXT_LIMIT=128 \
  NANOCAMELID_MODEL_GGUF=/path/to/model.gguf \
  cargo run --release -- generate "Hello" 0.0 8

This does not change the model metadata or make broad context-length support claims; it only bounds memory for short validation runs.

Inside the TUI:

- /model <path> loads a different GGUF without restarting the process. A successful switch resets the conversation and token counters. If the new model fails to load, the current model stays active. - /models lists GGUFs next to the current model. - /temp [value] and /tokens [count] show or change per-turn decoding settings. - /system [prompt] sets a system prompt and resets chat state; /system clear removes it. - /status, /history, /trim <turns>, and /save <path> manage the active session. - /clear, /exit, and /quit reset or leave the chat.

On a prepared Pi workspace with the Llama 3.2 1B Instruct Q4_0 or Q8_0 GGUF at the default model path, start the interactive 1B chat directly:

./scripts/pi/chat-1b.sh

For the matching one-command 1B validation path on that same Pi workspace:

./scripts/pi/smoke-1b.sh

For the fuller 1B readiness gate, including inspect, smoke, and one direct chat turn:

./scripts/pi/ready-1b.sh
./scripts/pi/ready-1b.sh --no-chat
./scripts/pi/ready-1b.sh --dry-run

The same gate is available through the CLI when you are already using the release binary or Cargo directly:

nanocamelid ready 1b
nanocamelid ready 1b --no-chat
nanocamelid ready 1b --dry-run
nanocamelid ready 1b /path/to/Llama-3.2-1B-Instruct-Q4_0.gguf chat "Say hello in one sentence." 8

Use --no-chat or --smoke-only for the inspect plus smoke-only form when you want the gate to validate the model path without launching the final direct chat turn. Use --dry-run to print the resolved 1B model path, smoke settings, and direct-chat gate without loading the GGUF. For non-interactive automation, the CLI and ready-1b.sh also honor NANOCAMELID_READY_SMOKE_KIND, NANOCAMELID_READY_SMOKE_PROMPT, and NANOCAMELID_READY_SMOKE_TOKENS as smoke defaults before the final direct chat turn.

For the supported Llama 3.2 3B Instruct Q4_0 row, place Llama-3.2-3B-Instruct-Q4_0.gguf under the same models/ directory and use the matching launchers:

./scripts/pi/smoke-3b.sh
./scripts/pi/chat-3b.sh

The launcher prefers a leading .gguf argument, then NANOCAMELID_MODEL_GGUF, then the Pi-local Q4_0 model when present, and finally Q8_0. It defaults the block dot path to SDOT on Pi-class ARM64 hardware. It runs a smoke 1b chat preflight before opening the TUI, so the 1B instruct path keeps the scalar-vs-selected-kernel parity gate in front of interactive chat. It still honors NANOCAMELID_Q8_DOT_KERNEL if you want to force a different kernel for comparison. When the helper needs to build through Cargo, it uses /mnt/nanocamelid/target by default, or an explicit CARGO_TARGET_DIR or NANOCAMELID_TARGET_DIR override.

Optional arguments set the model path, temperature, and maximum assistant output tokens:

./scripts/pi/chat-1b.sh /path/to/Llama-3.2-1B-Instruct-Q4_0.gguf 0.0 64
./scripts/pi/chat-1b.sh 0.0 64

smoke-1b.sh uses the same kernel defaults, but runs only the smoke gate and exits. Its model-selection precedence is a leading .gguf argument, NANOCAMELID_SMOKE_GGUF, NANOCAMELID_MODEL_GGUF, Pi-local Q4_0, then Pi-local Q8_0. By default it runs the real instruct prompt path with chat, the prompt Say hello in one sentence., and an 8-token response budget. Optional arguments let you override the model path, smoke kind, prompt, and token budget directly. Add --dry-run to print the resolved smoke plan without loading the model:

./scripts/pi/smoke-1b.sh /path/to/Llama-3.2-1B-Instruct-Q4_0.gguf chat "Say hello in one sentence." 8
./scripts/pi/smoke-1b.sh chat "Say hello in one sentence." 8
./scripts/pi/smoke-1b.sh --dry-run
./scripts/pi/smoke-1b.sh model "Hello" 1
./scripts/pi/smoke-3b.sh chat "Say hello in one sentence." 4

ready-1b.sh uses the same Pi target directory and model defaults, then runs inspect, smoke 1b, and chat against the resolved GGUF. A leading .gguf argument overrides the model path. An optional leading chat, model, q8-chat, or q8-model argument selects the smoke gate kind. The remaining optional arguments override the final direct-chat prompt and token budget; when omitted, direct chat reuses the selected smoke prompt and token budget. Set --no-chat, --smoke-only, or NANOCAMELID_READY_CHAT=0 to stop after inspect and smoke when you only need the readiness gate:

./scripts/pi/ready-1b.sh /path/to/Llama-3.2-1B-Instruct-Q4_0.gguf "Say hello in one sentence." 8
./scripts/pi/ready-1b.sh /path/to/Llama-3.2-1B-Instruct-Q4_0.gguf chat "Say hello in one sentence." 8
./scripts/pi/ready-1b.sh "Say hello in one sentence." 8
./scripts/pi/ready-1b.sh --smoke-only

For faster local iteration, disable the preflight smoke gate explicitly:

NANOCAMELID_CHAT_SMOKE=0 ./scripts/pi/chat-1b.sh
NANOCAMELID_CHAT_SMOKE=0 ./scripts/pi/chat-3b.sh

The preflight smoke defaults to chat with a one-token response budget, and you can override the gate with:

• NANOCAMELID_CHAT_SMOKE_KIND=model|chat
• NANOCAMELID_CHAT_SMOKE_PROMPT="..."
• NANOCAMELID_CHAT_SMOKE_TOKENS=1