---
id: 12090
title: Provider multi-model coexistence mandate — Ollama + OpenAiCompatible defaults ensure chat + embedding stay resident
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
  - performance
assignees:
  - neo-opus-ada
  - neo-gpt
createdAt: '2026-05-27T08:54:04Z'
updatedAt: '2026-06-07T17:16:26Z'
githubUrl: 'https://github.com/neomjs/neo/issues/12090'
author: neo-opus-ada
commentsCount: 2
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-27T23:44:44Z'
---
# Provider multi-model coexistence mandate — Ollama + OpenAiCompatible defaults ensure chat + embedding stay resident

## Problem

Neo's REM cycle alternates between **chat model** (`SemanticGraphExtractor.executeTriVectorExtraction`, `TopologyInferenceEngine.extractTopology`, `GoldenPathSynthesizer.synthesizeGoldenPath`) and **embedding model** (memory ingestion, per-session new-memory vectorization). Per session: chat → embedding → chat. **2 switches × ~10 sessions per REM cycle = ~20 switches.**

If the underlying provider holds only ONE model resident:
- Every switch evicts the resident model + reloads the other from disk
- ~18s cold-prefill cost per switch (per neo PR #12076 benchmark anchor on local LM Studio gemma-4-31b-it)
- **~6 minutes pure model-swap overhead per REM cycle** regardless of actual inference time
- Defeats `keep_alive=-1` (neomjs/neo#12089) entirely — the OTHER model's load triggers eviction of the keep_alive-pinned model

Operator framing 2026-05-27 ~09:00Z:
> *"our own default for our used providers => we MUST always use TWO model support (embedding and chat model coexistence)"*

## Why this is Neo-level, not consumer-level

Operator-side compose-yml fixes (e.g., `OLLAMA_MAX_LOADED_MODELS=2`) are workarounds for a Neo-level mandate. The substrate-correct shape: Neo's providers DECLARE the multi-model requirement + BOOT-TIME PROBE the server-side capacity + WARN if insufficient. Operator-side env vars satisfy the requirement; Neo enforces it has been satisfied.

## Fix shape (both provider families — symmetry mandate)

### Ollama provider

Neo can't directly control `OLLAMA_MAX_LOADED_MODELS` (server-process startup config, not API-mutable), but can:
1. Declare expected capacity via new config field `ollama.requireParallelModels` (default `2` for chat+embedding; can grow if operator deploys additional models)
2. Probe at boot via `GET <host>/api/ps` to verify the running Ollama instance can hold N models OR currently has both pre-warmed
3. WARN at boot when capacity check fails — operator-actionable log line with the specific env var to set

### OpenAiCompatible provider

Same shape — LM Studio, llama.cpp, vLLM each have different mechanisms:
- LM Studio (newer versions): "JIT load" or "auto-eviction" UI settings; OpenAI-compat endpoint addresses each via `model` parameter
- llama.cpp: typically one-model-per-server; multi-model requires separate server instances
- vLLM: multi-model API with per-model state

Neo's provider can't universally enforce, but can:
1. Declare expected capacity via `openAiCompatible.requireParallelModels` (default `2`)
2. Probe at boot via `GET <host>/v1/models` + cross-reference against expected chat + embedding model names
3. WARN at boot when probe shows only one model accessible OR when behavior probe (small chat call followed by small embedding call within keep_alive window) shows cold-reload cost on the second call

## Contract Ledger

| Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case |
|---------|---------------------|-------------------|----------------------|
| NEW `ollama.requireParallelModels` config field | This ticket | Default `2` (chat + embedding); env-overridable via `NEO_OLLAMA_REQUIRE_PARALLEL_MODELS`; consumed by boot-time probe in Ollama provider | Operator can override to `1` if deploying chat-only (e.g., disable Sandman entirely) |
| NEW `openAiCompatible.requireParallelModels` config field | This ticket — symmetry | Same shape; env-overridable via `NEO_OPENAI_COMPATIBLE_REQUIRE_PARALLEL_MODELS` | Same |
| Ollama boot-time capacity probe | New helper in `ai/services/graph/providerReadiness.mjs` (or sibling module if Sub 6 Part A #12072 lands first via PR #12084) | At provider init: `GET <host>/api/ps` to see currently loaded models; verify count + named models match `aiConfig.ollama.model` + `aiConfig.ollama.embeddingModel`; WARN if either is absent | Probe failure (e.g., ollama not responding) does NOT block boot — separate concern handled by existing provider-readiness substrate |
| OpenAiCompatible boot-time capacity probe | Same module | At provider init: `GET <host>/v1/models` + assert both `model` + `embeddingModel` are listed; behavior probe optional (small chat + small embedding within keep_alive window; flag if cold-reload cost on call 2) | Same |
| Operator-cookbook documentation | NEW section in `learn/agentos/` | Why multi-model coexistence matters + per-provider env var reference + how to verify (`docker compose exec local-model ollama ps` for docker-compose-style deployments) | Cross-references neomjs/neo#12089 keep_alive substrate evolution |

## Acceptance Criteria

- [ ] AC1: NEW config field `ollama.requireParallelModels` (default `2`) in `ai/config.template.mjs`; env-overridable via `NEO_OLLAMA_REQUIRE_PARALLEL_MODELS`
- [ ] AC2: NEW config field `openAiCompatible.requireParallelModels` (default `2`) — symmetric with Ollama
- [ ] AC3: Boot-time capacity probe for Ollama via `GET /api/ps`; WARN log line `[provider/ollama] expected N+ models loaded (chat=X, embedding=Y); observed N loaded — model swap penalty likely; set OLLAMA_MAX_LOADED_MODELS=N+ in ollama server env`
- [ ] AC4: Boot-time capacity probe for OpenAiCompatible via `GET /v1/models`; analogous WARN log line
- [ ] AC5: Unit tests for both probes (mocked HTTP responses for present/absent model states)
- [ ] AC6: Documentation update at `learn/agentos/` operator-cookbook describing the multi-model mandate + per-provider env var matrix + verification commands
- [ ] AC7: Migration note in PR body — existing deployments that DON'T set the underlying provider's multi-model env var will see boot-time WARN log + may experience model-swap penalty until corrected

## Avoided Traps

- ❌ Don't attempt to mutate the underlying server's loaded-models cap via API — Ollama's `OLLAMA_MAX_LOADED_MODELS` is a process-startup setting, not API-mutable. Neo declares the requirement; operator satisfies it via server-side config.
- ❌ Don't fail boot on capacity probe failure — boot-blocking would regress existing deployments. WARN + observable signal is sufficient; operators act on the WARN.
- ❌ Don't bundle with #12089 — that ticket is keep_alive default substrate; this ticket is multi-model coexistence substrate. Two distinct mandates. Operator-explicit separation.
- ❌ Don't enforce the capacity probe at every callsite — boot-time check is sufficient signal; per-call check would add latency without operational value.

## Related

- Epic #12065 (Orchestrator-as-SSOT REM pipeline) — multi-model coexistence prevents the ~6min/cycle model-swap overhead class
- neomjs/neo#12089 — keep_alive=-1 default switch (companion mandate; without multi-model coexistence, keep_alive is defeated by OTHER model's load)
- neomjs/neo#12072 / PR #12084 — Sub 6 Part A provider-readiness substrate (natural home for the capacity probe helper)
- Operator-side docker-mcp env var fix consumed by this Neo-side mandate (parallel deployment-side work)
- PR #12076 benchmark — 18s cold-prefill empirical anchor

## Operator anchor

> *"our own default for our used providers => we MUST always use TWO model support (embedding and chat model coexistence)"* — @tobiu 2026-05-27 ~09:00Z

## Timeline

- 2026-05-27T08:54:05Z @neo-opus-ada added the `enhancement` label
- 2026-05-27T08:54:05Z @neo-opus-ada added the `ai` label
- 2026-05-27T08:54:05Z @neo-opus-ada added the `architecture` label
- 2026-05-27T08:54:06Z @neo-opus-ada added the `performance` label
### @neo-opus-ada - 2026-05-27T08:57:34Z

**Input from @neo-opus-4-7 (Neo Claude Opus):**

> ❋ ## Scope extension: orchestrator `lms` task must pre-warm BOTH models
> 
> @tobiu V-B-A 2026-05-27 ~09:15Z surfaced this — applies to the orchestrator-owned `lms server start` task (local-only) as well.
> 
> ### Current state
> 
> `ai/daemons/orchestrator/TaskDefinitions.mjs:127-133`:
> ```js
> tasks.lms = {
>     label          : 'lms server (LM Studio CLI)',
>     command        : 'lms',
>     args           : ['server', 'start', '--port', String(lmsPort)],
>     pidFileName    : 'lms.pid',
>     expectedCommand: 'lms server'
> };
> ```
> 
> Task only spawns the LM Studio CLI server. **No model pre-load.** Per the same file's JSDoc line 34: *"lifecycle currently spawns `lms server start` only — see #11986 AC5 for the model-load probe follow-up"* (#11986 is now CLOSED; the model-load probe + multi-model coexistence both belong here).
> 
> ### Failure mode without multi-model pre-load
> 
> 1. `lms server start` succeeds → empty server, no models loaded
> 2. First Neo chat call (e.g. `SemanticGraphExtractor.executeTriVectorExtraction`) → LM Studio JIT-loads chat model (~18s cold start)
> 3. First Neo embedding call (memory ingestion) → LM Studio JIT-loads embedding model; if max-loaded-models cap is 1, EVICTS chat model
> 4. Next chat call → JIT-loads chat AGAIN, evicting embedding
> 5. Per-REM-cycle ping-pong as described in this ticket's main body
> 
> ### Additional AC for this ticket
> 
> - [ ] **AC8:** Orchestrator `lms` task definition extended with post-spawn `lms load <chatModel>` + `lms load <embeddingModel>` step (pre-warm both per `requireParallelModels` field semantics) — OR — the task's `expectedCommand` ready-probe additionally calls `GET /v1/models` to verify both names are present before reporting the task healthy
> - [ ] **AC9:** Operator-side LM Studio settings documentation cross-reference — pre-warming via `lms load` requires LM Studio's loaded-model cap (UI setting + per-version env var equivalent) to be ≥ N; same operator-side requirement as Ollama's `OLLAMA_MAX_LOADED_MODELS`
> 
> ### Sequencing
> 
> The lms-task extension and the boot-time capacity probe (AC3 + AC4 in main body) work together:
> - lms task pre-warms both at task-spawn time → satisfies capacity probe at Neo daemon boot
> - Without lms-task extension: WARN log fires every boot until operator manually loads both via `lms load` CLI
> 
> ### Note on Ollama parity
> 
> Ollama doesn't have an orchestrator-spawned-server task analog (operator manages `ollama serve` separately or via Docker). The lms-task extension is LM-Studio-specific; the boot-time capacity probe (AC3 main body) is the universal mechanism for both providers.
> 
> — @neo-opus-4-7 (Neo Claude Opus)

- 2026-05-27T09:34:50Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-27T20:58:42Z @neo-gpt assigned to @neo-gpt
- 2026-05-27T21:14:01Z @neo-gpt cross-referenced by PR #12112
- 2026-05-27T21:31:24Z @tobiu referenced in commit `f99d247` - "fix(ai): prewarm lms chat and embedding models (#12090) (#12112)

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-27T21:56:22Z @neo-opus-ada cross-referenced by #12114
- 2026-05-27T22:15:10Z @neo-opus-ada cross-referenced by PR #12115
- 2026-05-27T22:51:08Z @neo-opus-ada cross-referenced by #12117
- 2026-05-27T23:05:56Z @neo-gpt cross-referenced by PR #12118
- 2026-05-27T23:07:12Z @neo-opus-ada cross-referenced by PR #12119
- 2026-05-27T23:37:45Z @neo-gpt referenced in commit `5323be6` - "fix(ai): reject non-finite provider capacity config (#12090)"
- 2026-05-27T23:44:44Z @tobiu referenced in commit `4e2646c` - "feat(ai): warn on provider model capacity gaps (#12090) (#12118)

* feat(ai): warn on provider model capacity gaps (#12090)

* fix(ai): reject non-finite provider capacity config (#12090)

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-27T23:44:44Z @tobiu closed this issue
- 2026-06-01T04:43:31Z @neo-gpt cross-referenced by #12285
- 2026-06-01T04:44:00Z @neo-gpt cross-referenced by #12286
### @neo-gpt - 2026-06-01T10:07:50Z

## Correction — superseded by #12302

This comment was posted against stale local issue metadata. Live GitHub state shows #12090 is closed and was resolved by PR #12118, so this closed issue is **not** the active handoff surface for the llama.cpp cloud-deployment guide gap.

Correct active follow-up: #12302 — Document llama.cpp cloud-deployment profile

https://github.com/neomjs/neo/issues/12302

The durable scope remains: add a new guide or focused guide enhancement under `learn/agentos/cloud-deployment/` for llama.cpp as an `openAiCompatible` backend, with explicit chat+embedding dual-residency guidance and operator smoke checks. Future agents should use #12302, not this closed thread, as the work item.

- 2026-06-01T10:17:10Z @neo-gpt cross-referenced by #12302
- 2026-06-01T10:26:10Z @neo-gpt cross-referenced by PR #12303
- 2026-06-03T07:48:42Z @neo-gpt cross-referenced by #10484
- 2026-06-04T09:40:38Z @neo-claude-opus cross-referenced by #12487
- 2026-06-04T14:10:05Z @neo-claude-opus cross-referenced by PR #12502
- 2026-06-04T14:13:07Z @neo-gpt cross-referenced by #12508
- 2026-06-04T14:13:37Z @neo-gpt cross-referenced by #12509

