LLMKube

Install the CLI

brew install defilantech/tap/llmkube

Install the operator on any K8s cluster

helm repo add llmkube https://defilantech.github.io/LLMKube helm install llmkube llmkube/llmkube --namespace llmkube-system --create-namespace

Deploy a model (one command, uses catalog-tested defaults)

llmkube deploy phi-4-mini

Installation

Quick Start

```bash

Use the API

Every deployment exposes an OpenAI-compatible API. Use any OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://llama-3b-service:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="llama-3b",
    messages=[{"role": "user", "content": "Explain Kubernetes in one sentence"}]
)

Works with LangChain, LlamaIndex, and any OpenAI-compatible client library.

---

Troubleshooting

<details> <summary><b>Model won't download</b></summary>

kubectl describe model <model-name>
kubectl logs <pod-name> -c model-downloader

<details> <summary><b>Pod OOM crash</b></summary>

llmkube deploy <model> --memory 8Gi  # Rule of thumb: file size x 1.2

<details> <summary><b>GPU not detected</b></summary>

kubectl get pods -n gpu-operator-resources
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds

<details> <summary><b>OpenShift / MicroShift / OKD: ship the bundled Helm preset</b></summary>

LLMKube is tested in CI against MicroShift to verify the OpenShift SCC admission path end-to-end on every PR. The repo ships a Helm values preset at charts/llmkube/values-openshift.yaml that disables the operator's default fsGroup so the restricted-v2 SCC can inject an appropriate value from the namespace's allocated supplemental-groups range.

Recommended install:

helm install llmkube ./charts/llmkube \
  -f charts/llmkube/values-openshift.yaml \
  -n llmkube-system --create-namespace

That single command produces an LLMKube deployment whose InferenceService pods are admitted cleanly under restricted-v2. The same values-openshift.yaml works on MicroShift, OKD, OpenShift Container Platform, and any other distribution that runs the SCC admission controller with the standard MustRunAs fsGroup strategy.

Per-InferenceService override (fallback for single-tenant cases).

If you would rather pin fsGroup per workload instead of disabling the default operator-wide:

```bash