RK AI工具

Installation

Standard Installation (recommended create a virtual environment like: conda, uv, venv)

1. Clone the repository:

git clone https://github.com/notpunchnox/rkllama
cd rkllama

1. Install RKLLama:

python -m pip install .

Output:

Docker Installation

Pull the RKLLama Docker image:

docker pull ghcr.io/notpunchnox/rkllama:main

docker run -it --privileged -p 8080:8080 -v <local_models_dir>:/opt/rkllama/models ghcr.io/notpunchnox/rkllama:main 

Set up by: ichlaffterlalu

Docker Compose

Docker Compose facilities much of the extra flags declaration such as volumes:

docker compose up --detach --remove-orphans

**Manual Installation**

1. Download the Model - Download .rkllm models directly from Hugging Face. - Alternatively, convert your GGUF models into .rkllm format (conversion tool coming soon on my GitHub).

2. Place the Model - Create the models directory on your system. - Make a new subdirectory with model name. - Place the .rkllm files in this directory. - Create Modelfile and add this :

    FROM="file.rkllm"
    HUGGINGFACE_PATH="huggingface_repository"
    SYSTEM="Your system prompt"
    TEMPERATURE=1.0
    

Example directory structure:

   ~/RKLLAMA/models/
       └── TinyLlama-1.1B-Chat-v1.0
           |── Modelfile
           └── TinyLlama-1.1B-Chat-v1.0.rkllm
   

You must provide a link to a HuggingFace repository to retrieve the tokenizer and chattemplate. An internet connection is required for the tokenizer initialization (only once), and you can use a repository different from that of the model as long as the tokenizer is compatible and the chattemplate meets your needs. Tokenizer gets downloaded for the first time in the models directory

**For Multimodal Encoder Model (.rknn) Installation**

1. Download the encoder model .rknn - Download .rknn models directly from Hugging Face. - Alternatively, convert your ONNX models into .rknn format. - Place the .rknn model inside the models directory. RKLLama detected the encoder model present in the directory. - Include manually the following properties in the Modelfile according to the conversion properties used for the conversion of the vision encoder .rknn:

    IMAGE_WIDTH=448
    IMAGE_HEIGHT=
    N_IMAGE_TOKENS=
    IMG_START=
    IMG_END=
    IMG_CONTENT=

    # For example, for Qwen2VL/Qwen2.5VL:

    IMAGE_WIDTH=392
    IMAGE_HEIGHT=392
    N_IMAGE_TOKENS=196
    IMG_START=<|vision_start|>
    IMG_END=<|vision_end|>
    IMG_CONTENT=<|image_pad|>

    # For example, for MiniCPMV4:

    IMAGE_WIDTH=448
    IMAGE_HEIGHT=448
    N_IMAGE_TOKENS=64
    IMG_START=<image>
    IMG_END=</image>
    IMG_CONTENT=<unk>
   

Example directory structure for multimodal:

   ~/RKLLAMA/models/
       └── qwen2-vision\:2b
           |── Modelfile
           └── Qwen2-VL-2B-Instruct.rkllm
           └── Qwen2-VL-2B-Instruct.rknn
   

**For Image Generation Installation**

1. In a temporary folder, clone the repository https://huggingface.co/danielferr85/lcm-sd-1.5-rknn-2.3.2-rk3588 or https://huggingface.co/danielferr85/lcm-ssd-1b-rknn-2.3.2-rk3588 from Hugging Face for the desired SD model 2. Execute the ONNX to RKNN convertion of the models for your needs WITH RKNN TOOLKIT LIBRARY VERSION 2.3.2. For example: For LCM SD 1.5

    python convert-onnx-to-rknn.py --model-dir <directory_download_model> --resolutions 512x512 --components "text_encoder,unet,vae_decoder" --target_platform rk3588
   

For LCM SSD1B

    python convert-onnx-to-rknn.py --model-dir <directory_download_model> --resolutions 1024x1024 --components "text_encoder,text_encoder_2,unet,vae_decoder" --target_platform rk3588
   

For LCM SD 1.5

   ~/RKLLAMA/models/
       └── lcm-stable-diffusion
           |── scheduler
              |── scheduler_config.json
           └── text_encoder
              |── config.json
              |── model.rknn
           └── unet
              |── config.json
              |── model.rknn
           └── vae_decoder
              |── config.json
              |── model.rknn
           
   

or

For LCM SSD1B

   ~/RKLLAMA/models/
       └── lcm-segmind-stable-diffusion
           |── scheduler
              |── scheduler_config.json
           └── text_encoder
              |── config.json
              |── model.rknn
           └── text_encoder_2
              |── config.json
              |── model.rknn
           └── unet
              |── config.json
              |── model.rknn
           └── vae_decoder
              |── config.json
              |── model.rknn
           
   

1. Done! You are ready to test the OpenAI endpoint /v1/images/generations to generate images. You can add it to OpenWebUI in the Image Generation section.
2. Available converted models for RK3588/RK3576 and RKNN 2.3.2 at: https://huggingface.co/danielferr85/lcm-sd-1.5-rknn-2.3.2-rk3588 (only 512x512 resolutions) and https://huggingface.co/danielferr85/lcm-ssd-1b-rknn-2.3.2-rk3588 (only 1024x1024 resolutions)

**For Speech Generation (TTS) Installation**

For Piper:

1. Download a voice from https://huggingface.co/danielferr85/piper-checkpoints-rknn from Hugging Face. (You can convert new ones, see below)
2. Create a folder inside the models directory in RKLLAMA for the piper Audio model, For example: es_AR-daniela-high
3. Copy the encoder (.onnx), decoder (.rknn) and config (piper.json) file from the choosed voice to the new directory model created in RKLLMA.
4. The structure of the model MUST be like this:

   ~/RKLLAMA/models/
       └── es_AR-daniela-high
           |── encoder.onnx
           └── decoder.rknn
           └── piper.json
          
   

For MMS-TTS:

1. Download a voice from https://huggingface.co/danielferr85/mms-tts-rknn from Hugging Face. (You can convert new ones, see below)
2. Create a folder inside the models directory in RKLLAMA for the piper Audio model, For example: mms_tts_spa
3. Copy the encoder (.rknn), decoder (.rknn) and vocab (mms_tts.json) file from the choosed voice to the new directory model created in RKLLMA.
4. The structure of the model MUST be like this:

   ~/RKLLAMA/models/
       └── mms_tts_spa
           |── encoder/
               |── encoder.rknn
           └── decoder/
               └── decoder.rknn
           └── mms_tts.json   
          
   

1. Done! You are ready to test the OpenAI endpoint /v1/audio/speech to generate audio. You can add it to OpenWebUI in the Audio section for TTS.

IMPORTANT For Piper: - You must convert only your decoder (.rknn) for your specific platform (for example rk3588). - The encoder can have any name but must ended with extension .onnx - The decoder can have any name but must ended with extension .rknn - The config of the model must have the name piper.json - You must use rknn-toolkit 2.3.2 for RKNN conversion because is the one used by RKLLAMA - Always CHECK THE LICENSE of the voice that you are going to use. - In OpenAI request, the argument model is the name of the model folder that you create and the argument voice is the speaker of the voice if the voice is multispeaker (For example 'F' (Female) or 'M' (Male). Check the config of the model). If the model is monospeaker, then voice can be skipped. - You can convert any Piper TTS model (or create one, or finetuning one) creating first the encoder and decoder in ONNX format and then converting the decoder in RKNN: 1. Clone this repo and branch streaming: https://github.com/mush42/piper/tree/streaming 2. Create a virtual environemnt and install that repo 3. Clone the repository https://huggingface.co/danielferr85/piper-checkpoints-rknn from Hugging Face 4. Download the entire folder of the voice to convert from dataset: https://huggingface.co/datasets/rhasspy/piper-checkpoints and put it inside the repo piper-checkpoints-rknn (the structure must be for example: es/es_MX/ald/medium/). You can also use the script download_models.py from download automatically the model you want. 5. Execute the script export_encoder_decoder.py to export the encoder and decoder IN ONNX format. 6. Execute the script export_rknn.py to export the decoder in RKNN format (you must uhave installed the rknn-toolkit version 2.3.2).

For MMS-TTS: - You must convert your encoder and decoder (.rknn) for your specific platform (for example rk3588). - The encoder can have any name but must ended with extension .rknn and must be placed inside a folder called encoder - The decoder can have any name but must ended with extension .rknn and must be placed inside a folder called decoder - The vocab of the model must have the name mms_tts.json - You must use rknn-toolkit 2.3.2 for RKNN conversion because is the one used by RKLLAMA - Always CHECK THE LICENSE of the voice that you are going to use. - In OpenAI request, the argument model is the name of the model folder that you create and the argument voice is the speaker of the voice if the voice is multispeaker but in MMS-TTS always is monospeaker, then voice is skipped. - You can convert more mms_tts models. Check: https://github.com/airockchip/rknn_model_zoo/tree/main/examples/mms_tts

**For Transcriptions Generation (STT) Installation**

• For OmniASR CTC models:

1. Download a model from https://huggingface.co/danielferr85/omniASR-ctc-rknn from Hugging Face.
2. Create a folder inside the models directory in RKLLAMA for the model, For example: omniasr-ctc:300m
3. Copy the model (.rknn) and vocabulary (omniasr.txt) file from the choosed model to the new directory model created in RKLLMA.
4. The structure of the model MUST be like this:

   ~/RKLLAMA/models/
       └── omniasr-ctc:300m
           └── model.rknn
           └── omniasr.txt
          
   

IMPORTANT - The model can have any name but must ended with extension .rknn - The vocabulary of the model must be the name omniasr.txt - You must use rknn-toolkit 2.3.2 for RKNN conversion because is the one used by RKLLAMA

• For Whisper models:

1. Download a model from https://huggingface.co/danielferr85/whisper-with_past-models-rknn from Hugging Face.
2. Create a folder inside the models directory in RKLLAMA for the model, For example: whisper-large-v3-turbo
3. Copy the encoder model (.rknn), decoder model (.rknn), decoder with past model (.rknn), whisper.ini and tokenizer folder from the choosed model to the new directory model created in RKLLMA.
4. The structure of the model MUST be like this:

   ~/RKLLAMA/models/
       └── whisper-large-v3-turbo
           └── encoder
               └── model_encoder.rknn
           └── decoder
               └── model_decoder.rknn 
           └── decoder_with_past
               └── model_decoder_with_past.rknn
           └── tokenizer
              └── tokenizer_config.json
              └── tokenizer.json
           └── whisper.ini
          
   

Done! You are ready to test the OpenAI endpoint /v1/audio/transcriptions to generate transcriptions. You can add it to OpenWebUI in the Audio section for STT.

Uninstall

1. Remove the pyhton package rkllama

    pip uninstall rkllama
    

---

Usage

Tool Calling Quick Start

RKLLama supports advanced tool/function calling for enhanced AI interactions:

```bash

Example: Weather tool call

curl -X POST http://localhost:8080/api/chat \ -H "Content-Type: application/json" \ -d '{ "model": "qwen2.5:3b", "messages": [{"role": "user", "content": "What is the weather in Paris?"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get current weather", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"} }, "required": ["location"] } } }] }' ```

Features: - 🔧 Multiple model support (Qwen, Llama 3.2+, others) - 🌊 Streaming & non-streaming modes - 🎯 Robust JSON parsing with fallback methods - 🔄 Auto format normalization - 📋 Multiple tools in single request

For complete documentation: Tool Calling Guide

Tested Hardware and Environment

• Hardware: Orange Pi 5 Pro: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
• Hardware: Orange Pi 5 Plus: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
• Hardware: Orange Pi 5 Max: (Rockchip RK3588S, NPU 6 TOPS), 16GB RAM.
• Hardware: Radxa Rock 4d: (Rockchip RK3576, NPU 6 TOPS), 16GB RAM.
• OS: Ubuntu 24.04 arm64.
• OS: Armbian Linux 6.1.99-vendor-rk35xx (Debian stable bookworm), v25.2.2.

Configuration

RKLLAMA uses a flexible configuration system that loads settings from multiple sources in a priority order:

See the Configuration Documentation for complete details.