浏览器LLM推理

/!\ IMPORTANT: Require having docker compose installed

Install the required modules

npm i

How to compile the binary yourself

This repository already come with pre-built binary from llama.cpp source code. However, in some cases you may want to compile it yourself: - You don't trust the pre-built one. - You want to try out latest - bleeding-edge changes from upstream llama.cpp source code.

You can use the commands below to compile it yourself:

```shell

Firstly, build llama.cpp into wasm

npm run build:wasm

Then, build ES module

npm run build ```

How to use

Simple usage with ES6 module

For complete code, see examples/basic/index.html

import { Wllama } from './esm/index.js';

(async () => {
  const CONFIG_PATHS = {
    default: './esm/wasm/wllama.wasm',
  };
  // Automatically switch between single-thread and multi-thread version based on browser support
  // If you want to enforce single-thread, add { "n_threads": 1 } to LoadModelConfig
  const wllama = new Wllama(CONFIG_PATHS);
  // Define a function for tracking the model download progress
  const progressCallback =  ({ loaded, total }) => {
    // Calculate the progress as a percentage
    const progressPercentage = Math.round((loaded / total) * 100);
    // Log the progress in a user-friendly format
    console.log(`Downloading... ${progressPercentage}%`);
  };
  // Load GGUF from Hugging Face hub
  // (alternatively, you can use loadModelFromUrl if the model is not from HF hub)
  await wllama.loadModelFromHF(
    { repo: 'ggml-org/models', file: 'tinyllamas/stories260K.gguf' },
    { progressCallback }
  );
  const response = await wllama.createChatCompletion({
    messages: [{ role: 'user', content: elemInput.value }],
    max_tokens: 50,
    temperature: 0.5,
    top_k: 40,
    top_p: 0.9,
  });
  console.log(response.choices[0].message.content);
})();

Alternatively, you can use the *.wasm files from CDN:

import WasmFromCDN from '@wllama/wllama/esm/wasm-from-cdn.js';
const wllama = new Wllama(WasmFromCDN);
// NOTE: this is not recommended, only use when you can't embed wasm files in your project

Code demo and documentation

Demo: - Basic usages with completions and embeddings: https://github.ngxson.com/wllama/examples/basic/ (source code) - Embedding and cosine distance: https://github.ngxson.com/wllama/examples/embeddings/ (source code) - Multimodal (vision) completion: https://github.ngxson.com/wllama/examples/multimodal/ (source code) - Tool calling: https://github.ngxson.com/wllama/examples/tools/ (source code)

Optionally, you can run this command to update llama.cpp to latest upstream version (bleeding-edge, use with your own risk!)

Clone the repository with submodule

git clone --recurse-submodules https://github.com/ngxson/wllama.git cd wllama

git submodule update --remote --merge