LlamaCpp Elixir绑定

Prerequisites

• C++17 compiler (GCC, Clang, or MSVC)
• CMake 3.14+
• Git (for the llama.cpp submodule)

Hardware requirements

Installation

Add llama_cpp_ex to your list of dependencies in mix.exs:

def deps do
  [
    {:llama_cpp_ex, "~> 0.7.5"}
  ]
end

Build the speculative session once — it owns a target context and a

Install the HuggingFace CLI if needed: pip install huggingface-hub

huggingface-cli download Qwen/Qwen3.5-35B-A3B-GGUF Qwen3.5-35B-A3B-Q4_K_M.gguf --local-dir models/ ```

Quick Start

```elixir

Usage

Minimal: stream a single response

```elixir :ok = LlamaCppEx.init()

{:ok, model} = LlamaCppEx.load_model( Path.expand("~/Downloads/Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf"), n_gpu_layers: 999 )

Examples

The examples/ directory contains runnable scripts demonstrating key features:

```bash

Create context and sampler separately

{:ok, ctx} = LlamaCppEx.Context.create(model, n_ctx: 4096) {:ok, sampler} = LlamaCppEx.Sampler.create(model, temp: 0.7, top_p: 0.9)

Sample every 200 ms while the generation runs.

Stream.repeatedly(fn -> Process.sleep(200) s = LlamaCppEx.MTP.stats(mtp) IO.puts( "iters=#{s.iters} emitted=#{s.tokens_emitted} " <> "accept=#{Float.round(s.acceptance_rate * 100, 1)}% " <> "tok/s=#{Float.round(s.tokens_per_sec, 1)}" ) end) |> Stream.take_while(fn _ -> not Task.yield(gen_task, 0) |> match?({:ok, _}) end) |> Stream.run()

Task.await(gen_task, :infinity)

For in-band progress events (no separate process), use `stream_events/3` with `emit_stats_every`:

Options

LlamaCppEx.MTP.init/2:

• :n_draft — draft tokens proposed per iteration (default 3). On NVIDIA, 2–4 is the sweet spot. On Apple Silicon, set this to 1 — see the Apple Silicon performance note above.
• :n_ctx, :n_threads, :flash_attn, :type_k/:type_v, :offload_kqv, … — any LlamaCppEx.Context option; applied to both target and draft contexts.

LlamaCppEx.MTP.stream/3:

• :max_tokens (default 256), plus all sampling options (:temp, :top_k, :top_p, :min_p, :seed, :penalty_*, :grammar).
• :emit_stats_every — when set, periodic {:stats, _} events become available via stream_events/3.

Disable thinking via enable_thinking option (uses Jinja chat template kwargs)

{:ok, reply} = LlamaCppEx.chat(model, [ %{role: "user", content: "What is the capital of France?"} ], max_tokens: 256, enable_thinking: false, temp: 0.7, top_p: 0.8, top_k: 20, min_p: 0.0, penalty_present: 1.5) ```

Lower-level API

For fine-grained control over the inference pipeline:

```elixir

Pre-Tokenized API

Tokenize outside the GenServer to reduce contention under concurrent load:

model = LlamaCppEx.Server.get_model(server)
{:ok, tokens} = LlamaCppEx.Tokenizer.encode(model, prompt)
{:ok, text} = LlamaCppEx.Server.generate_tokens(server, tokens, max_tokens: 100)

Ecto Schema Integration

Convert Ecto schema modules to JSON Schema automatically (requires {:ecto, "~> 3.0"} — optional dependency):

```elixir defmodule MyApp.Person do use Ecto.Schema

embedded_schema do field :name, :string field :age, :integer field :active, :boolean field :tags, {:array, :string} end end

JSON Schema constrained generation + Ecto integration

LLAMA_MODEL_PATH=/path/to/model.gguf mix run examples/structured_output.exs