% sections/memory_rag.tex

\section{Memory and Retrieval-Augmented Generation}
\label{sec:memory_rag}

Production routing systems must support multi-turn conversations with persistent context and knowledge-augmented responses.
We describe the memory and RAG subsystems that operate as plugins within the routing pipeline.

\subsection{Persistent Memory}

The memory system maintains user-scoped knowledge across conversation sessions, enabling personalized routing and context-aware responses.
\Cref{fig:memory_lifecycle} illustrates the full memory lifecycle.

\noindent\textbf{Memory storage.}
Each conversation turn is stored directly as an \emph{episodic chunk}---no external LLM is required.
The user message and assistant response are concatenated into a \texttt{Q:/A:} block, sanitized (UTF-8 validation, 16\,KB cap per entry), and embedded.
A lightweight \emph{entropy gate} discards turns that carry no retrievable signal---greetings, acknowledgments, and short one-word replies---before embedding, reducing index pollution.
Every $s$ turns a session-level sliding-window chunk spanning the last $w$ turns is additionally stored (defaults: $s{=}3$, $w{=}5$), creating overlapping windows so facts near turn boundaries co-occur in at least one chunk---improving multi-hop retrieval coverage.

\noindent\textbf{Retrieval gating.}
Not every query benefits from memory retrieval.
A lightweight heuristic determines whether memory search is warranted by filtering out general fact-check queries, tool-augmented requests, and simple greetings, avoiding unnecessary embedding lookups and reducing latency for queries where personal context is irrelevant.

\noindent\textbf{Retrieval.}
At query time, relevant memories are retrieved via the hybrid search pipeline (vector similarity, BM25, and n-gram matching) over the user's memory store.
An optional query-rewriting step reformulates the user's query for improved retrieval recall.
Adaptive thresholding adjusts the similarity cutoff based on retrieval mode, preventing cosine-calibrated thresholds from silently discarding all RRF-scored results.

\noindent\textbf{Post-retrieval filtering.}
Retrieved memories pass through a composable \textsc{ReflectionGate} before context injection:
\begin{enumerate}[nosep,leftmargin=*]
  \item \emph{Safety}: a regex block-list prevents prompt-injection payloads from being surfaced.
  \item \emph{Recency decay}: memories are weighted by recency, prioritizing recent context.
  \item \emph{Deduplication}: near-duplicate entries (high Jaccard similarity) are collapsed to a single representative.
  \item \emph{Budget}: the final set is capped at a configurable count to bound injected context length.
\end{enumerate}
Filtered context is injected as a separate conversation message positioned after system instructions but before user turns, following the context-injection pattern of the OpenAI Agents SDK.

\noindent\textbf{Background consolidation.}
A scheduled background job merges semantically related memories using greedy single-linkage clustering over word-level Jaccard similarity.
Memories within each cluster are replaced by a single deduplicated summary entry, reducing store redundancy and improving retrieval precision over time.

\begin{figure}[ht]
\centering
\resizebox{\linewidth}{!}{%
\begin{tikzpicture}[
    node distance=0.55cm and 0.7cm,
    box/.style={draw, rounded corners, fill=blue!6, minimum width=2.0cm,
                minimum height=0.8cm, align=center, font=\small},
    storebox/.style={draw, rounded corners, fill=green!6, minimum width=2.0cm,
                minimum height=0.8cm, align=center, font=\small},
    gatebox/.style={draw, rounded corners, fill=orange!8, minimum width=2.0cm,
                minimum height=0.8cm, align=center, font=\small},
    arr/.style={->, >=stealth, thick},
    darr/.style={->, >=stealth, thick, dashed},
    lbl/.style={font=\scriptsize, text=gray}
]

% --- Write path (top row) ---
\node[box] (conv) {Conv\\Turn};
\node[gatebox, right=of conv] (entropy) {Entropy\\Gate};
\node[box, right=of entropy] (sanitize) {Sanitize\\+ Embed};
\node[storebox, right=of sanitize] (store) {Memory\\Store};

\draw[arr] (conv) -- (entropy);
\draw[arr] (entropy) -- node[above, lbl] {pass} (sanitize);
\draw[arr] (sanitize) -- (store);

% Entropy gate discards low-signal turns
\draw[darr] (entropy.south) -- ++(0,-0.45) node[below, lbl] {discard};

% Session window annotation above store
\draw[darr] (sanitize.north) -- ++(0,0.45) -| node[above, lbl, pos=0.25] {session window (every $s$ turns)} (store.north);

% --- Read path (bottom row) ---
\node[box, below=2.6cm of conv] (query) {User\\Query};
\node[gatebox, right=of query] (rgate) {Retrieval\\Gating};
\node[box, right=of rgate] (hybrid) {Hybrid\\Search};
\node[gatebox, right=of hybrid] (reflect) {Reflection\\Gate};

\draw[arr] (query) -- (rgate);
\draw[arr] (rgate) -- node[above, lbl] {pass} (hybrid);
\draw[arr] (hybrid) -- (reflect);
\draw[arr] (reflect.east) -- ++(0.5,0) node[right, lbl, align=left] {Inject\\Context Msg};

% Store feeds into hybrid search
\draw[arr] (store.south) -- ++(0,-1.05) -| (hybrid.north);

% Skip arrow from gate
\draw[darr] (rgate.south) -- ++(0,-0.45) node[below, lbl] {skip};

% --- Path labels ---
\node[lbl, above=0.08cm of conv, xshift=1.0cm] {\textit{Write path}};
\node[lbl, above=0.08cm of query, xshift=1.0cm] {\textit{Read path}};

\end{tikzpicture}
}
\caption{Memory lifecycle. \textit{Write path}: each conversation turn passes an entropy gate (discarding low-signal turns such as greetings), is sanitized and embedded as an episodic \texttt{Q:/A:} chunk, and stored directly---no LLM inference required. Every $s$ turns an additional sliding-window chunk is stored. \textit{Read path}: a heuristic gate skips retrieval for irrelevant queries; qualifying queries run hybrid search (vector + BM25 + n-gram) over the user's store; retrieved memories pass through a \textsc{ReflectionGate} (safety blocklist, recency decay, deduplication, budget cap) before injection as a separate conversation message.}
\label{fig:memory_lifecycle}
\end{figure}

\subsection{Retrieval-Augmented Generation}

The RAG plugin retrieves relevant documents from vector stores and injects them as context before model invocation.

\noindent\textbf{Indexing pipeline.}
Documents are chunked (configurable size and overlap), embedded using the shared embedding model (\Cref{sec:ml_inference}), and indexed in a vector store.

\noindent\textbf{Hybrid retrieval.}
Pure vector search can miss lexically relevant results when the embedding model underweights rare terms.
We implement a three-signal hybrid retrieval pipeline (\Cref{fig:hybrid_search}) that scores each candidate chunk along three axes:
\begin{enumerate}[nosep,leftmargin=*]
  \item \emph{Vector similarity}: cosine similarity from the embedding index (already in $[0,1]$).
  \item \emph{BM25}: an Okapi BM25 inverted index built over chunk contents provides keyword-level relevance, with configurable $k_1$ (term-frequency saturation, default~1.2) and $b$ (length normalization, default~0.75) parameters.
  \item \emph{N-gram Jaccard}: character $n$-gram sets (default $n{=}3$) capture fuzzy lexical overlap, producing Jaccard similarity in $[0,1]$.
\end{enumerate}

\noindent\textbf{Score fusion.}
Two fusion modes combine the three retriever scores into a single ranking:
(1)~\emph{Weighted}: BM25 scores are min-max normalized to $[0,1]$, then the final score is $w_v \cdot s_{\text{vec}} + w_b \cdot s_{\text{bm25}} + w_n \cdot s_{\text{ngram}}$ with configurable weights (defaults: 0.7, 0.2, 0.1).
(2)~\emph{Reciprocal Rank Fusion (RRF)}: $\text{score}(d) = \sum_r 1/(k + \text{rank}_r(d))$, which is parameter-free beyond the constant $k$ (default~60).

\begin{figure}[t]
\centering
\resizebox{\linewidth}{!}{%
\begin{tikzpicture}[
    node distance=0.5cm and 0.8cm,
    box/.style={draw, rounded corners=2pt, fill=blue!6,
                minimum height=0.7cm, minimum width=1.6cm,
                align=center, font=\scriptsize},
    idx/.style={draw, rounded corners=2pt, fill=green!8,
                minimum height=0.7cm, minimum width=1.6cm,
                align=center, font=\scriptsize},
    fuse/.style={draw, rounded corners=3pt, fill=orange!10,
                 minimum height=0.9cm, minimum width=1.8cm,
                 align=center, font=\scriptsize\bfseries},
    score/.style={font=\tiny, text=gray!70},
    arr/.style={->, >=stealth, thick},
    arrs/.style={->, >=stealth, semithick, gray!60},
    lbl/.style={font=\tiny, text=gray},
  ]

  % === Query ===
  \node[box, minimum width=1.4cm] (query) {Query};

  % === Three retrieval paths ===
  % Path 1: Vector
  \node[box] (embed) at (2.8, 1.2) {Embed};
  \node[idx] (vecidx) at (5.0, 1.2) {Vector\\Index};

  % Path 2: BM25
  \node[box] (tok) at (2.8, 0.0) {Tokenize};
  \node[idx] (bm25idx) at (5.0, 0.0) {BM25\\Index};

  % Path 3: N-gram
  \node[box] (ngram) at (2.8, -1.2) {Char\\$n$-grams};
  \node[idx] (ngramidx) at (5.0, -1.2) {N-gram\\Index};

  % Query fan-out
  \draw[arr] (query.east) -- ++(0.35,0) |- (embed.west);
  \draw[arr] (query.east) -- ++(0.35,0) |- (tok.west);
  \draw[arr] (query.east) -- ++(0.35,0) |- (ngram.west);

  % Processing arrows
  \draw[arr] (embed) -- (vecidx);
  \draw[arr] (tok) -- (bm25idx);
  \draw[arr] (ngram) -- (ngramidx);

  % Score labels
  \node[score, above=0.01cm of vecidx.east, anchor=south west] {cosine $\in [0,1]$};
  \node[score, above=0.01cm of bm25idx.east, anchor=south west] {min-max $\to [0,1]$};
  \node[score, above=0.01cm of ngramidx.east, anchor=south west] {Jaccard $\in [0,1]$};

  % === Score Fusion ===
  \node[fuse] (fusion) at (8.0, 0.0) {Score\\Fusion};

  % Index → Fusion
  \draw[arrs] (vecidx.east) -- ++(0.3,0) |- (fusion.west);
  \draw[arrs] (bm25idx.east) -- (fusion.west);
  \draw[arrs] (ngramidx.east) -- ++(0.3,0) |- (fusion.west);

  % Fusion modes annotation
  \node[lbl, below=0.15cm of fusion] {Weighted / RRF};

  % === Output ===
  \node[box, minimum width=1.5cm, fill=green!8, font=\scriptsize\bfseries] (out) at (10.5, 0.0) {Ranked\\Results};
  \draw[arr] (fusion.east) -- (out.west);

  % === Fallback annotation ===
  \draw[densely dotted, gray!40] (5.0, -2.0) -- (vecidx.south);
  \node[lbl, anchor=north] at (5.0, -2.05)
    {non-native backends: $4{\times}$ top-$k$ expansion};

  % === Weight defaults ===
  \node[lbl, anchor=north] at (8.0, -0.85)
    {$w_v{=}0.7,\; w_b{=}0.2,\; w_n{=}0.1$};

\end{tikzpicture}
}
\caption{Hybrid search pipeline. Each candidate chunk is scored along three axes---vector cosine similarity, BM25 keyword relevance, and character $n$-gram Jaccard overlap---then combined via weighted fusion or Reciprocal Rank Fusion (RRF). Backends without native hybrid support fall back to a generic rerank path that fetches $4{\times}$ top-$k$ candidates from vector search before applying BM25 and $n$-gram scoring.}
\label{fig:hybrid_search}
\end{figure}

\noindent\textbf{Backend abstraction.}
The RAG plugin accesses vector stores through a common \texttt{VectorStoreBackend} interface, decoupling retrieval logic from storage implementation.
Six backend types are supported (\Cref{fig:rag_backends}):
\begin{itemize}[nosep,leftmargin=*]
  \item \emph{In-memory}: development and testing; no external dependencies.
  \item \emph{Milvus}~\cite{wang2021milvus}: production-grade distributed vector database with native hybrid search support.
  \item \emph{Llama Stack}: delegates vector storage and search to a Llama Stack deployment via its OpenAI-compatible \texttt{/v1/vector\_stores} API, enabling unified management of LLM serving and retrieval through a single platform. When the Llama Stack instance is configured with the Milvus \texttt{vector\_io} provider, the backend supports hybrid search by passing \texttt{ranking\_options: \{ranker: ``rrf''\}} in search requests, combining vector similarity with BM25 keyword matching at the provider level.
  \item \emph{External API}: any OpenAI-compatible vector store endpoint.
  \item \emph{MCP}: retrieval via Model Context Protocol tool servers.
  \item \emph{OpenAI file search}: delegated retrieval through OpenAI's hosted file search API.
\end{itemize}
Backends that implement native hybrid search (Milvus, Llama Stack with Milvus provider) use their own BM25 and keyword indexes; all other backends fall back to a generic rerank path that fetches an expanded candidate set ($4{\times}$ top-$k$) from vector search and applies BM25 and n-gram scoring as a post-retrieval reranking step.

\noindent\textbf{Score-range awareness.}
Different retrieval modes produce scores on fundamentally different scales: cosine similarity yields values in $[0, 1]$ where a threshold of $\sim$0.7 is typical, while RRF scores follow $\sum 1/(k + \text{rank})$ and typically range from 0.001 to 0.05.
Applying a cosine-calibrated threshold to RRF scores would silently discard all results.
The backend interface handles this transparently: when hybrid search is active, score-based filtering is bypassed and result volume is controlled solely by the top-$k$ parameter.

\begin{figure}[t]
\centering
\resizebox{\linewidth}{!}{%
\begin{tikzpicture}[
    node distance=0.5cm and 0.7cm,
    box/.style={draw, rounded corners, fill=blue!6, minimum width=2cm,
                minimum height=0.7cm, align=center, font=\small},
    backend/.style={draw, rounded corners, fill=green!6, minimum width=1.8cm,
                minimum height=0.6cm, align=center, font=\scriptsize},
    arr/.style={->, >=stealth, thick},
    lbl/.style={font=\scriptsize, text=gray}
]

\node[box] (query) {User Query};
\node[box, right=1.2cm of query] (rag) {RAG Plugin};
\node[box, right=1.2cm of rag] (iface) {VectorStore\\Interface};

\draw[arr] (query) -- (rag);
\draw[arr] (rag) -- (iface);

% Backends fan out
\node[backend, above right=0.4cm and 1.2cm of iface] (milvus) {Milvus};
\node[backend, right=1.2cm of iface] (llama) {Llama Stack};
\node[backend, below right=0.4cm and 1.2cm of iface] (others) {MCP / API /\\OpenAI};

\draw[arr] (iface) -- (milvus);
\draw[arr] (iface) -- (llama);
\draw[arr] (iface) -- (others);

% Hybrid label
\node[lbl, right=0.1cm of milvus] {hybrid};
\node[lbl, right=0.1cm of llama] {hybrid (via Milvus)};
\node[lbl, right=0.1cm of others] {vector only};

% Inject
\node[box, below=1.5cm of rag] (inject) {Inject into Prompt};
\draw[arr] (rag) -- (inject);

\end{tikzpicture}
}
\caption{RAG backend architecture. The RAG plugin accesses vector stores through a common interface. Milvus and Llama Stack (with Milvus provider) support native hybrid search combining vector similarity with BM25 keyword matching via Reciprocal Rank Fusion. Other backends use vector-only retrieval with optional post-retrieval reranking.}
\label{fig:rag_backends}
\end{figure}

\subsection{Stateful Conversations (Response API)}

The system supports the OpenAI Responses API for stateful multi-turn conversations:

\noindent\textbf{Conversation chaining.}
Each response is stored with a unique ID.
Subsequent requests reference \texttt{previous\_response\_id} to reconstruct the full conversation history without retransmitting the full message sequence.
A bidirectional translator converts between the Response API format and Chat Completions format for backend model invocation, enabling all routing, safety, and caching features to operate identically on both API surfaces.

\noindent\textbf{Routing continuity.}
Stored responses include routing metadata (decision, model selection, signal results), enabling consistent routing across conversation turns and providing context for feedback-driven model selection.

\noindent\textbf{State backends.}
Conversation state is persisted via three backends:
(1)~\emph{in-memory} for development;
(2)~\emph{Redis} for production deployments requiring distributed state with high availability---supporting both standalone and cluster modes;
(3)~\emph{Milvus} for deployments that benefit from semantic retrieval over conversation history.
The Redis backend enables horizontal scaling: multiple router replicas share conversation state, and Redis persistence ensures that conversation chains survive pod restarts.

\subsection{Integration with Signal-Decision Architecture}

Both memory and RAG operate as per-decision plugins.
Different decisions can activate different RAG configurations (different vector stores, different $k$ values, different chunk strategies, different search modes) or disable retrieval entirely.
This enables, for example, a ``research assistant'' decision that activates RAG with hybrid search over a technical knowledge base while a ``casual chat'' decision disables retrieval.
