Type to search documentation...

# API 参考

完整的 CLI 参数、REST API 端点和环境变量参考。

## 模型参数

参数 | 说明 | 默认
--- | --- | ---
`--m, --model` | HuggingFace 模型 ID 或本地路径 | —
`--f, --model-file` | GGUF 文件名 | —
`--isq` | ISQ 量化类型 | —
`--kvcache-dtype` | KV 缓存类型 | `auto`
`--dtype` | 模型精度 | `auto`
`--chat-template` | 自定义 chat template | —
`--mtp` | MTP 数量 | `0`

## 服务器参数

参数 | 说明 | 默认
--- | --- | ---
`--port` | API 服务器端口 | `8000`
`--host` | 绑定地址 | `0.0.0.0`
`--ui-server` | 启动 Web UI | `false`
`--ui-port` | Web UI 端口 | `8001`

## 性能参数

参数 | 说明 | 默认
--- | --- | ---
`--max-batch` | 最大批量 | `32`
`--max-seq-len` | 最大序列长度 | 模型默认
`--graph-batch` | CUDA Graph 批量 | `1,2,4,8,16,32`

## 高级参数

参数 | 说明 | 默认
--- | --- | ---
`--num-ranks` | TP GPU 数量 | `1`
`--local-num-ranks` | 本节点 GPU 数 | —
`--rank-offset` | rank 偏移 | `0`
`--master-addr` | 主节点地址 | —
`--pd-config` | PD 配置 | —
`--mcp-config` | MCP 配置 | —

## Chat Completions

兼容 OpenAI Chat Completions API：

    POST /v1/chat/completions

    {
      "model": "default",
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is Rust?"}
      ],
      "max_tokens": 256,
      "temperature": 0.7,
      "top_p": 0.9,
      "stream": true,
      "response_format": {
        "type": "json_schema",
        "json_schema": { ... }
      }
    }

📘 采样参数

支持 temperature、top_p、top_k 等常用采样参数。

## Text Completions

    POST /v1/completions

    {
      "model": "default",
      "prompt": "Once upon a time",
      "max_tokens": 128
    }

## Embeddings

    POST /v1/embeddings

    {
      "model": "default",
      "input": "Hello world"
    }

    // Response
    {
      "data": [{
        "embedding": [0.0123, -0.0456, ...],
        "index": 0
      }]
    }

## Tokenize / Detokenize

直接分词器访问：

    curl -XPOST http://localhost:8000/tokenize \
      -H "Content-Type: application/json" \
      -d '{"prompt":"Hello, world!"}'

    // Response
    { "tokens": [9906, 11, 1917, 0], "count": 4, "max_model_len": 4096 }

    curl -XPOST http://localhost:8000/tokenize \
      -H "Content-Type: application/json" \
      -d '{
      "messages": [
        {"role":"system","content":"You are helpful."},
        {"role":"user","content":"Hello!"}
      ]
    }'

    // 应用 chat template 后返回 Token ID

    curl -XPOST http://localhost:8000/detokenize \
      -H "Content-Type: application/json" \
      -d '{"tokens":[9906,11,1917,0]}'

    // Response
    { "prompt": "Hello, world!" }

参数 | 端点 | 说明
--- | --- | ---
`prompt` | /tokenize | 纯文本
`messages` | /tokenize | Chat 消息
`add_special_tokens` | /tokenize | 特殊 Token
`tokens` | /detokenize | Token ID
`skip_special_tokens` | /detokenize | 跳过特殊

## Anthropic 兼容 API

Claude 兼容端点：

    curl http://localhost:8000/v1/messages \
      -H "Content-Type: application/json" \
      -d '{
      "model": "default",
      "max_tokens": 256,
      "messages": [{"role":"user","content":"Hello from Claude Code"}]
    }'

支持流式和 Token 计数。

## Models

    GET /v1/models

    // Response
    {
      "data": [{
        "id": "default",
        "object": "model",
        "owned_by": "xinfer"
      }]
    }

## Health Check

    GET /health

    // Response: 200 OK
    { "status": "ok" }

## 结构化输出

约束解码方式：

    curl -sXPOST localhost:8000/v1/chat/completions \
      -H 'Content-Type: application/json' \
      -d '{
        "messages": [{"role":"user","content":"Classify this sentiment: I love it!"}],
        "structured_outputs": {
          "choice": ["positive", "negative", "neutral"]
        },
        "max_tokens": 50
      }'

    curl -sXPOST localhost:8000/v1/chat/completions \
      -H 'Content-Type: application/json' \
      -d '{
        "messages": [{"role":"user","content":"Return a 3-digit code"}],
        "structured_outputs": {
          "regex": "^[0-9]{3}$"
        },
        "max_tokens": 10
      }'

    curl -sXPOST localhost:8000/v1/chat/completions \
      -H 'Content-Type: application/json' \
      -d '{
        "messages": [{"role":"user","content":"Generate a user profile"}],
        "structured_outputs": {
          "json": {
            "type": "object",
            "properties": {
              "name": {"type": "string"},
              "age": {"type": "integer", "minimum": 0, "maximum": 150},
              "email": {"type": "string", "pattern": "^[a-z]+@[a-z]+\\.[a-z]+$"}
            },
            "required": ["name", "age", "email"],
            "additionalProperties": false
          }
        },
        "max_tokens": 500
      }'

    curl -sXPOST localhost:8000/v1/chat/completions \
      -H 'Content-Type: application/json' \
      -d '{
        "messages": [{"role":"user","content":"Return yes or no"}],
        "structured_outputs": {
          "grammar": "start: \"yes\" | \"no\""
        },
        "max_tokens": 10
      }'

📘 约束源

三种约束源，只能同时使用一种。

## 采样参数

参数 | 类型 | 说明
--- | --- | ---
`temperature` | float | 采样温度
`top_p` | float | 核采样
`top_k` | int | Top-K 采样
`min_p` | float | 最小概率
`repetition_penalty` | float | 重复惩罚
`frequency_penalty` | float | 频率惩罚
`presence_penalty` | float | 存在惩罚
`seed` | int | 随机种子
`max_tokens` | int | 最大 Token 数
`reasoning_effort` | string | 推理努力级别

## 环境变量

变量 | 说明
--- | ---
`HF_TOKEN` | HuggingFace 访问令牌
`CUDA_VISIBLE_DEVICES` | 指定 GPU 设备
`XINFER_CHAT_LOGGER=1` | 启用聊天日志
`XINFER_SSE_BUFFER_SIZE` | SSE 缓冲区大小
`XINFER_NVFP4_FORCE_LUT=1` | 强制 LUT 解量化
`SM90_LOWER_PRECISION_GDN_PREFILL=1` | GDN 预填充加速
`NCCL_SOCKET_IFNAME` | NCCL 网络接口
`NCCL_IB_DISABLE=1` | 禁用 InfiniBand
`NCCL_DEBUG=INFO` | NCCL 调试日志

    # Blackwell 高精度 NVFP4 解码
    XINFER_NVFP4_FORCE_LUT=1 xinfer --m nvidia/Qwen3-30B-A3B-FP4 --ui-server

    # Hopper 快速 GDN 预填充
    SM90_LOWER_PRECISION_GDN_PREFILL=1 xinfer --m Qwen/Qwen3.5-35B-A3B-FP8 --ui-server