---
id: 12302
title: Document llama.cpp cloud-deployment profile
state: CLOSED
labels:
  - documentation
  - enhancement
  - ai
  - architecture
assignees:
  - neo-gpt
createdAt: '2026-06-01T10:17:09Z'
updatedAt: '2026-06-01T10:41:45Z'
githubUrl: 'https://github.com/neomjs/neo/issues/12302'
author: neo-gpt
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-06-01T10:41:45Z'
---
# Document llama.cpp cloud-deployment profile

## Context

This ticket captures the residual docs scope surfaced after the #12090 closeout. #12090 is closed and was resolved by PR #12118, which shipped the generic provider-capacity warning substrate plus docs in `learn/agentos/DeploymentCookbook.md` and `learn/agentos/SharedDeployment.md`.

Operator correction on 2026-06-01: the missing work is not to reopen #12090. The missing work is a dedicated llama.cpp cloud-deployment guide or focused guide enhancement inside `learn/agentos/cloud-deployment/`.

## V-B-A Evidence

- Live GitHub state: #12090 is `CLOSED`, closed on 2026-05-27T23:44:44Z.
- PR #12118 is `MERGED`, merged on 2026-05-27T23:44:43Z, and closed #12090.
- PR #12118 docs-side files were `learn/agentos/DeploymentCookbook.md` and `learn/agentos/SharedDeployment.md`; it did not touch `learn/agentos/cloud-deployment/`.
- Current repo search shows llama.cpp only appears in `learn/agentos/cloud-deployment/Day0Tutorial.md` as part of the broad OpenAI-compatible fallback row. There is no dedicated `llama-server`, `/slots`, `/health`, `/metrics`, or llama.cpp residency-smoke profile in the cloud-deployment guide tree.
- Official llama.cpp server docs describe `llama-server` as an OpenAI-compatible HTTP server with chat/embeddings routes, context-size/server-slot flags, slots monitoring, health/metrics surfaces, and embeddings mode: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

## The Problem

Neo now has generic local-provider coexistence substrate (`requireParallelModels`, `/v1/models` probing, warning-only readiness), but cloud operators do not have a concrete llama.cpp profile that explains how to use that substrate safely.

The risky failure mode is operator-visible and severe: a llama.cpp deployment can appear to work for a single chat call or a single embedding call while still being invalid for Agent OS workloads if chat and embedding roles cannot remain resident together. REM/session-summary/embedding paths alternate between roles; model swapping or context rebuilds defeats the local-provider economics.

## The Architectural Reality

- Neo's provider key remains `openAiCompatible` for llama.cpp unless implementation evidence proves a first-class provider is necessary.
- `ai/config.template.mjs` already defines `openAiCompatible.model`, `openAiCompatible.embeddingModel`, `openAiCompatible.keep_alive`, and `openAiCompatible.requireParallelModels`.
- `ai/provider/OpenAiCompatible.mjs` owns chat-completions transport.
- `ai/services/memory-core/TextEmbeddingService.mjs` owns OpenAI-compatible `/v1/embeddings` calls.
- `ai/services/graph/ProviderReadinessHelper.mjs` uses `/v1/models` for OpenAI-compatible provider readiness/capacity visibility.
- `learn/agentos/cloud-deployment/Day0Tutorial.md` distinguishes native Ollama from OpenAI-compatible fallback, but does not yet document a llama.cpp deployment profile.

## The Fix

Add a new guide or focused guide enhancement under `learn/agentos/cloud-deployment/` for a llama.cpp OpenAI-compatible cloud profile.

The guide should cover:

- llama.cpp as an `openAiCompatible` backend, not a new Neo provider key by default.
- Supported topology for chat and embedding roles, including whether the safest shape is one `llama-server` endpoint that exposes both configured model ids or role-separated llama.cpp services.
- If role-separated services are required and current Neo config cannot express separate chat/embedding hosts, state that limitation explicitly and file/follow an implementation ticket rather than hiding it in docs.
- Env examples for `NEO_MODEL_PROVIDER=openAiCompatible`, `NEO_EMBEDDING_PROVIDER=openAiCompatible`, `NEO_GRAPH_PROVIDER=openAiCompatible`, `NEO_OPENAI_COMPATIBLE_HOST`, `NEO_OPENAI_COMPATIBLE_MODEL`, `NEO_OPENAI_COMPATIBLE_EMBEDDING_MODEL`, `NEO_OPENAI_COMPATIBLE_KEEP_ALIVE=-1`, `NEO_OPENAI_COMPATIBLE_REQUIRE_PARALLEL_MODELS=2`, and role-keyed `NEO_LOCAL_MODELS_*` context caps.
- llama.cpp-specific operator smoke: `/health`, `/v1/models`, `/v1/chat/completions`, `/v1/embeddings`, `/slots`, and optional metrics when enabled.
- Cross-links from `Day0Tutorial.md`, `Configuration.md`, or `Overview.md` so cloud-deployment readers can find the profile.

## Contract Ledger Matrix

| Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
|---|---|---|---|---|---|
| llama.cpp cloud profile | This ticket + PR #12118 residual scope | Document llama.cpp behind `openAiCompatible` with role-residency smoke | If first-class provider is required, defer to separate implementation ticket | `learn/agentos/cloud-deployment/` | Docs diff + source anchors |
| Provider selector guidance | `ai/config.template.mjs` + PR #12118 | Use `openAiCompatible`; do not invent `llamaCpp` selector in docs | Explicitly document limitation if endpoint separation is unsupported | Same guide | Config anchors |
| Residency invariant | #12090 / PR #12118 | Chat + embedding roles must stay resident together | Reject model-swapping topology as invalid for Agent OS local Brain path | Same guide | `/v1/models` + role smoke |
| Operator smoke | llama.cpp server docs + Neo health/readiness surfaces | List concrete checks before handing deployment to agents | Backend docs may drift; implementation must verify current flags/endpoints | Same guide | Official llama.cpp docs + Neo health output |

## Decision Record Impact

Aligned with ADR 0014 cloud-deployment topology and scheduler-task taxonomy. No ADR superseded.

## Acceptance Criteria

- [ ] A llama.cpp cloud-deployment guide or focused guide enhancement exists under `learn/agentos/cloud-deployment/`.
- [ ] The guide states that llama.cpp uses Neo's existing `openAiCompatible` provider key initially; adding `llamaCpp` is out of scope unless separately proven.
- [ ] The guide documents the chat+embedding dual-residency invariant and explicitly rejects swapping/rebuild topologies for Agent OS workloads.
- [ ] The guide includes an env example for provider selectors, model names, keep-alive, parallel-model requirement, and local role context caps.
- [ ] The guide includes llama.cpp-specific smoke checks for `/health`, `/v1/models`, `/v1/chat/completions`, `/v1/embeddings`, `/slots`, and metrics when enabled.
- [ ] The guide cross-links from the existing cloud-deployment navigation surface (`Overview.md`, `Day0Tutorial.md`, or `Configuration.md`) so operators can find it.
- [ ] If the only correct topology requires separate chat and embedding hosts that current Neo config cannot express, the PR documents that as a current limitation and links/files the implementation follow-up.

## Out of Scope

- Adding a first-class `llamaCpp` provider selector.
- Changing provider defaults.
- Retuning embedding context defaults.
- Reworking #12090's already-shipped warning substrate.
- Live benchmarking a production llama.cpp deployment.

## Avoided Traps

- Do not comment on or reopen closed #12090 as the active handoff.
- Do not treat generic OpenAI-compatible docs as sufficient for a backend with llama.cpp-specific slot/cache/runtime behavior.
- Do not fossilize unverified llama.cpp CLI flags. Verify against the current official server docs while implementing.
- Do not document a topology that works only by serially swapping chat and embedding models.

## Related

- Closed: #12090
- Resolving PR: #12118
- Generic cloud docs: `learn/agentos/DeploymentCookbook.md`, `learn/agentos/SharedDeployment.md`
- Target guide tree: `learn/agentos/cloud-deployment/`
- Official llama.cpp server docs: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Handoff Retrieval Hint: "llama.cpp openAiCompatible cloud deployment profile dual resident chat embedding guide #12118 residual"

## Timeline

- 2026-06-01T10:17:10Z @neo-gpt assigned to @neo-gpt
- 2026-06-01T10:17:11Z @neo-gpt added the `documentation` label
- 2026-06-01T10:17:11Z @neo-gpt added the `enhancement` label
- 2026-06-01T10:17:12Z @neo-gpt added the `ai` label
- 2026-06-01T10:17:12Z @neo-gpt added the `architecture` label
- 2026-06-01T10:17:42Z @neo-gpt cross-referenced by #12090
- 2026-06-01T10:26:10Z @neo-gpt cross-referenced by PR #12303
- 2026-06-01T10:32:40Z @neo-gpt referenced in commit `8f2c81e` - "docs(agentos): keep llama.cpp guide intent driven (#12302)"
- 2026-06-01T10:41:45Z @tobiu referenced in commit `4843531` - "docs(agentos): add llama.cpp cloud profile (#12302) (#12303)

* docs(agentos): add llama.cpp cloud profile (#12302)

* docs(agentos): keep llama.cpp guide intent driven (#12302)

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-06-01T10:41:45Z @tobiu closed this issue

