llama-rn AI技能包

Installation

npm install llama.rn

llama.rn downloads the pre-built ios/rnllama.xcframework and android/src/main/jniLibs from the matching GitHub release during postinstall. Existing downloads are reused, and each archive is verified with SHA-256 before extraction.

Bun

Bun does not run dependency lifecycle scripts unless the package is trusted. If you install llama.rn with Bun and want the native downloads to run automatically, add it to trustedDependencies and re-run bun install:

{
  "trustedDependencies": ["llama.rn"]
}

If you prefer not to trust dependency lifecycle scripts, run the downloader manually before npx pod-install or your Android build:

node ./node_modules/llama.rn/install/download-native-artifacts.js

iOS

Please re-run npx pod-install again.

By default, llama.rn will use pre-built rnllama.xcframework for iOS. If you want to build from source, please set RNLLAMA_BUILD_FROM_SOURCE to 1 in your Podfile.

Android

Add proguard rule if it's enabled in project (android/app/proguard-rules.pro):

```proguard

Setup

First, you need a multimodal model and its corresponding multimodal projector (mmproj) file, see how to obtain mmproj for more details.

Usage

💡 You can find complete examples in the example project.

Load model info only:

import { loadLlamaModelInfo } from 'llama.rn'

const modelPath = 'file://<path to gguf model>'
console.log('Model Info:', await loadLlamaModelInfo(modelPath))

Initialize a Llama context & do completion:

import { initLlama } from 'llama.rn'

// Initial a Llama context with the model (may take a while)
const context = await initLlama({
  model: modelPath,
  use_mlock: true,
  n_ctx: 2048,
  n_gpu_layers: 99, // number of layers to store in GPU memory (Metal/OpenCL)
  // embedding: true, // use embedding
})

const stopWords = ['</s>', '<|end|>', '<|eot_id|>', '<|end_of_text|>', '<|im_end|>', '<|EOT|>', '<|END_OF_TURN_TOKEN|>', '<|end_of_turn|>', '<|endoftext|>']

// Do chat completion
const msgResult = await context.completion(
  {
    messages: [
      {
        role: 'system',
        content: 'This is a conversation between user and assistant, a friendly chatbot.',
      },
      {
        role: 'user',
        content: 'Hello!',
      },
    ],
    n_predict: 100,
    stop: stopWords,
    // ...other params
  },
  (data) => {
    // This is a partial completion callback
    const { token } = data
  },
)
console.log('Result:', msgResult.text)
console.log('Timings:', msgResult.timings)

// Or do text completion
const textResult = await context.completion(
  {
    prompt: 'This is a conversation between user and llama, a friendly chatbot. respond in simple markdown.\n\nUser: Hello!\nLlama:',
    n_predict: 100,
    stop: [...stopWords, 'Llama:', 'User:'],
    // ...other params
  },
  (data) => {
    // This is a partial completion callback
    const { token } = data
  },
)
console.log('Result:', textResult.text)
console.log('Timings:', textResult.timings)

The binding's deisgn inspired by server.cpp example in llama.cpp:

• /completion and /chat/completions: context.completion(params, partialCompletionCallback)
• /tokenize: context.tokenize(content)
• /detokenize: context.detokenize(tokens)
• /embedding: context.embedding(content)
• /rerank: context.rerank(query, documents, params)
• ... Other methods

Please visit the Documentation for more details.

You can also visit the example to see how to use it.

Usage Examples

Vision (Image Processing)

const result = await context.completion({
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'What do you see in this image?',
        },
        {
          type: 'image_url',
          image_url: {
            url: 'file:///path/to/image.jpg',
            // or base64: 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD...'
          },
        },
      ],
    },
  ],
  n_predict: 100,
  temperature: 0.1,
})

console.log('AI Response:', result.text)

Audio Processing

// Method 1: Using structured message content (Recommended)
const result = await context.completion({
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'Transcribe or describe this audio:',
        },
        {
          type: 'input_audio',
          input_audio: {
            data: 'data:audio/wav;base64,UklGRiQAAABXQVZFZm10...',
            // or url: 'file:///path/to/audio.wav',
            format: 'wav', // or 'mp3'
          },
        },
      ],
    },
  ],
  n_predict: 200,
})

console.log('Transcription:', result.text)

Usage

import { initLlama } from 'llama.rn'

const context = await initLlama({
  model: modelPath,
  n_ctx: 8192,
  n_gpu_layers: 99,
  n_parallel: 4, // Max number of parallel slots supported
})

// Enable parallel mode with 4 slots
await context.parallel.enable({
  n_parallel: 4, // new_n_ctx (2048) = n_ctx / n_parallel
  n_batch: 512,
})

// Queue multiple completion requests
const request1 = await context.parallel.completion(
  {
    messages: [{ role: 'user', content: 'What is AI?' }],
    n_predict: 100,
  },
  (requestId, data) => {
    console.log(`Request ${requestId}:`, data.token)
  }
)

const request2 = await context.parallel.completion(
  {
    messages: [{ role: 'user', content: 'Explain quantum computing' }],
    n_predict: 100,
  },
  (requestId, data) => {
    console.log(`Request ${requestId}:`, data.token)
  }
)

// Cancel a request if needed
await request1.stop()

// Wait for completion
const result = await request2.promise
console.log('Result:', result.text)

// Disable parallel mode when done
await context.parallel.disable()

Optional space: by convention, applied in this grammar after literal chars when allowed

ws ::= | " " | "\n" [ \t]{0,20}

const gbnf = '...'

const context = await initLlama({ // ...params grammar: gbnf, })

const { text } = await context.completion({ // ...params messages: [ { role: 'system', content: 'You are a helpful assistant that can answer questions and help with tasks.', }, { role: 'user', content: 'Test', }, ], }) console.log('Result:', text) ```

Also, this is how json_schema works in response_format during completion, it converts the json_schema to gbnf grammar.

API

context.parallel.enable(config?): - config.n_parallel (number): Number of concurrent slots (default: 2) - config.n_batch (number): Batch size for processing (default: 512) - Returns: Promise<boolean>

context.parallel.disable(): - Disables parallel mode - Returns: Promise<boolean>

context.parallel.configure(config): - Reconfigures parallel mode (enables if not already enabled) - config.n_parallel (number): Number of concurrent slots - config.n_batch (number): Batch size for processing - Returns: Promise<boolean>

context.parallel.completion(params, onToken?): - params: Same completion parameters as completion() - onToken: Optional callback (requestId, data) => void for token streaming - requestId: Unique request identifier - data: Token data with token, content, reasoning_content, tool_calls, accumulated_text - Returns: Promise<{ requestId, promise, stop }> - requestId: Unique request identifier - promise: Resolves to NativeCompletionResult when complete - stop: Function to cancel this request

context.parallel.embedding(text, params?): - text: Text content to get embedding for - params: Optional embedding parameters - Returns: Promise<{ requestId, promise }> - requestId: Unique request identifier - promise: Resolves to embedding result when complete

context.parallel.rerank(query, documents, params?): - query: Query string for ranking - documents: Array of document strings to rank - params: Optional rerank parameters (e.g., normalize) - Returns: Promise<{ requestId, promise }> - requestId: Unique request identifier - promise: Resolves to rerank results when complete