红烛AI工具

⚠️ Model Format Requirements

English NER (requires separate tokenizer)

ner = Candle::NER.from_pretrained("dslim/bert-base-NER", tokenizer: "bert-base-cased")

Load a LLaVA model (requires ~13GB download on first use)

vlm = Candle::VLM.from_pretrained("llava-hf/llava-v1.6-vicuna-7b-hf")

Dependencies

• Candle
• Magnus
• Outlines-core

Install & Chat in 30 Seconds

```bash

Install the gem

gem install red-candle

Usage

```ruby require "candle"

x = Candle::Tensor.new([1, 2, 3, 4, 5, 6], :i64) x = x.reshape([3, 2])

A note on memory usage

The default model (jinaai/jina-embeddings-v2-base-en with the sentence-transformers/all-MiniLM-L6-v2 tokenizer, both from HuggingFace) takes a little more than 3GB of memory running on a Mac. The memory stays with the instantiated Candle::EmbeddingModel class, if you instantiate more than one, you'll use more memory. Likewise, if you let it go out of scope and call the garbage collector, you'll free the memory. For example:

```ruby > require 'candle'

Basic Usage

```ruby require 'candle'

Basic Usage

```ruby require 'candle'

Use Cases

• Token Analysis: Understand how your text is being processed by models
• Debugging: See why certain inputs might cause unexpected model behavior
• Custom Preprocessing: Build your own text processing pipelines
• Educational: Teach how modern NLP models handle text
• NER Preparation: Get aligned tokens for named entity recognition tasks

Basic Usage

```ruby require 'candle'

Configuration

```ruby

Configure padding with more options

padded_tokenizer = tokenizer.with_padding( length: 128, # Fixed length padding direction: "right", # Pad on the right (default) pad_token: "[PAD]" # Padding token ) ```

Chat with it - no API calls, running locally in your Ruby process!

messages = [ { role: "user", content: "Explain Ruby in one sentence" } ]

puts llm.chat(messages)

Chat interface

messages = [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "Explain Ruby in one sentence." } ] response = llm.chat(messages) ```

Score: 0.0438 - Doc #3: London is the capital of England

Score: -3.0829 - Doc #3: London is the capital of England

Supported Models

• Gemma: Google's Gemma models (e.g., google/gemma-2b, google/gemma-7b, google/gemma-2b-it)
• Llama: Llama 2 and Llama 3 models (e.g., TinyLlama/TinyLlama-1.1B-Chat-v1.0, meta-llama/Llama-2-7b-hf, NousResearch/Llama-2-7b-hf)
• Mistral: All Mistral models (e.g., mistralai/Mistral-7B-Instruct-v0.1)
• Qwen: Qwen 2 and 2.5 models (e.g., Qwen/Qwen2-1.5B, Qwen/Qwen2.5-7B-Instruct)
• Phi: Microsoft's Phi-2, Phi-3, Phi-3.5, and Phi-4 models (e.g., microsoft/phi-2, microsoft/Phi-3-mini-4k-instruct, microsoft/phi-4)
• ⚠️ ⚠️ ⚠️ Note: Phi-3 and Phi-4 GGUF models have a known issue with KV cache persistence between generations. The reset_cache parameter doesn't work for GGUF models. Recreate the model instance for each generation.
• candle pull request about phi-3 gguf models: https://github.com/huggingface/candle/pull/2937

Supported Models

Red-Candle supports both BERT and XLM-RoBERTa reranker architectures. The model type is auto-detected from config.json.

Model Integration

All models expose their tokenizers:

```ruby

Supported Models

Run Rust unit and integration tests

cd ext/candle && cargo test

Generate LCOV format for CI integration

bundle exec rake rust:coverage:lcov

Use alternative pooling methods

Ask a question about an image

answer = vlm.ask("photo.jpg", "What animal is in this image?")