模型部署

Quick Start

One command, one model. Pick your hardware:

CPU — Linux, Windows, or macOS via Docker (images are multi-arch: amd64 + arm64):

docker run --rm --shm-size=8g -p 8000:8000 -v modelship-cache:/.cache \
  ghcr.io/modelship-ai/modelship:latest-cpu deploy \
  --model "Qwen/Qwen3-8B-GGUF:*Q4_K_M.gguf" --loader llama_server \
  --usecase generate --num-cpus 4

NVIDIA GPU — needs the NVIDIA Container Toolkit:

docker run --rm --shm-size=8g --gpus all -p 8000:8000 -v modelship-cache:/.cache \
  ghcr.io/modelship-ai/modelship:latest-cuda deploy \
  --model "Qwen/Qwen3-8B-GGUF:*Q4_K_M.gguf" --loader llama_server \
  --usecase generate --num-gpus 1

Apple Silicon — native, with Metal offload:

uv tool install mship && mship bootstrap --metal
mship deploy --model "Qwen/Qwen3-8B-GGUF:*Q4_K_M.gguf" --loader llama_server \
  --usecase generate --num-gpus 1

Wait for Deployed app 'modelship' successfully, then talk to it — this hits the Responses API and streams the model's reasoning as it thinks:

uvx --with httpx llm openai endpoint http://localhost:8000/modelship/v1 \
  -m qwen3-8b --responses "Which is larger, 9.11 or 9.9?"

Add --chat for an interactive session, -T to hand it a tool, or --models to list what's deployed.

The model serves as qwen3-8b, inferred from the reference. It pulls ~5 GB and wants ~8 GB of free RAM; --num-cpus 4 reserves four cores for it, so lower it if the container has fewer (the deploy waits for resources it can't get). On a small box, swap in lmstudio-community/Qwen3-0.6B-GGUF:*Q4_K_M.gguf.

Deploying several models at once uses a models.yaml instead; one model's tuning blocks have flags too (--llama-server-config.n-ctx 8192) — see Model Configuration. Hitting an error? Check Troubleshooting.

<details> <summary>Prefer <code>curl</code>?</summary>

curl http://localhost:8000/modelship/v1/responses \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-8b", "input": "Which is larger, 9.11 or 9.9?"}'

The response carries both output_text and a first-class reasoning output item. /modelship/v1/chat/completions is there too, if that's what your client speaks.

</details>

Supported OpenAI endpoints