Lattice 苹果芯片LLM工具箱

CPU build (Linux/macOS). f16 is required to load the BF16/F16 safetensors

CLI binary (chat + serve). The f16 feature is required to load the

Metal GPU decode (macOS only, requires model weights)

cargo bench -p lattice-inference --features metal-gpu,f16 -- metal_decode

Install

Three ways to get lattice, in order of convenience:

1. cargo install (from crates.io):

```bash

Build and package

```bash

Skip Swift rebuild (use existing build output)

./apps/macos/scripts/package-app.sh --skip-build

Skip Cargo rebuild

./apps/macos/scripts/package-app.sh --skip-cargo ```

Install

Drag LatticeStudio.app from the .dmg to /Applications.

The app is ad-hoc signed. On first launch, right-click and choose "Open" to bypass Gatekeeper, then click "Open" in the dialog. macOS remembers the exception for subsequent launches. Alternatively:

xattr -dr com.apple.quarantine /Applications/LatticeStudio.app

Quick Start: Embeddings

[dependencies]
lattice-embed = "0.4"
tokio = { version = "1", features = ["full"] }

use lattice_embed::{EmbeddingService, EmbeddingModel, NativeEmbeddingService};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let service = NativeEmbeddingService::default();

    // Single embedding (BGE-small-en-v1.5, 384 dimensions)
    let embedding = service
        .embed_one("The quick brown fox jumps over the lazy dog", EmbeddingModel::default())
        .await?;

    println!("Dimensions: {}", embedding.len()); // 384

    // Batch
    let texts = vec![
        "First document".to_string(),
        "Second document".to_string(),
    ];
    let embeddings = service.embed(&texts, EmbeddingModel::BgeSmallEnV15).await?;

    // SIMD-accelerated similarity
    let similarity = lattice_embed::utils::cosine_similarity(&embeddings[0], &embeddings[1]);
    println!("Similarity: {:.4}", similarity);

    Ok(())
}

Model weights are downloaded from HuggingFace on first use and cached at ~/.lattice/models (or $LATTICE_MODEL_CACHE).

Quick Start: CLI

HTTP API

lattice serve exposes an OpenAI-compatible endpoint:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-0.8b",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

---

Supported Models

Context scaling (Qwen3.5-0.8B vs Ollama vs MLX)

./scripts/bench_context_scaling.sh

Performance depends on hardware, model size, batch size, and sequence length. Run benchmarks on your target hardware for representative numbers.

---