There are many amazing LLMs and embedding models that can be run locally. R2R fully supports using these models, giving you full control over your data and infrastructure.

Running models locally can be ideal for sensitive data handling, reducing API costs, or situations where internet connectivity is limited. While cloud-based LLMs often provide cutting-edge performance,
local models offer a compelling balance of capability, privacy, and cost-effectiveness for many use cases.

<Steps>
### Serving Local Models

<Note>
For this cookbook, we'll serve our local models via Ollama. [You may follow the instructions on their official website to install.](https://ollama.com/)

You can also follow along using LM Studio. To get started with LM Studio, see our [Local LLM documentation](/self-hosting/local-rag).

R2R supports [LiteLLM](https://github.com/BerriAI/litellm) for routing embedding and completion requests. This allows for OpenAI-compatible endpoints to be called and seamlessly routed to, if you are serving local models another way.
</Note>


We must first download the models that we wish to run and start our ollama server. The following command will 'pull' the models and begin the Ollama server via `http://localhost:11434`.

```zsh
ollama pull llama3.1
ollama pull mxbai-embed-large
```

<Error>
Ollama has a default context window size of 2048 tokens. Many of the prompts and processes that R2R uses requires larger window sizes.

It is recommended to set the context size to a minimum of 16k tokens. The following guideline is generally useful to determine what your system can handle:
- 8GB RAM/VRAM: ~4K-8K context
- 16GB RAM/VRAM: ~16K-32K context
- 24GB+ RAM/VRAM: 32K+ context

To change the default context window you must first create a Modelfile for Ollama, where you can set `num_ctx`:
```Zsh
echo 'FROM llama3.1
PARAMETER num_ctx 16000' > Modelfile
```

Then you must create a manifest for that model:
```Zsh
ollama create llama3.1 -f Modelfile
```
</Error>

Then, we can start the Ollama server:
```Zsh
ollama serve
```

### Configuring R2R

Now that our models have been loaded and our Ollama server is ready, we can launch our R2R server.

The standard distribution of R2R includes a configuration file for running `llama3.1` and `mxbai-embed-large`. If you wish to utilize other models, you must create a custom config file and pass this to your server.

<AccordionGroup>
  <Accordion title="ollama.toml">
    ```Toml
    [app]
    # LLM used for internal operations, like deriving conversation names
    fast_llm = "ollama/llama3.1"

    # LLM used for user-facing output, like RAG replies
    quality_llm = "ollama/llama3.1"

    # LLM used for ingesting visual inputs
    vlm = "ollama/llama3.2-vision" # TODO - Replace with viable candidate

    # LLM used for transcription
    audio_lm = "ollama/llama3.1" # TODO - Replace with viable candidate

    [embedding]
    provider = "ollama"
    base_model = "mxbai-embed-large"
    base_dimension = 1_024
    batch_size = 128
    add_title_as_prefix = true
    concurrent_request_limit = 2

    [completion_embedding]
    provider = "ollama"
    base_model = "mxbai-embed-large"
    base_dimension = 1_024
    batch_size = 128
    add_title_as_prefix = true
    concurrent_request_limit = 2

    [agent]
    tools = ["local_search"]

    [agent.generation_config]
    model = "ollama/llama3.1"

    [completion]
    provider = "litellm"
    concurrent_request_limit = 1

    [completion.generation_config]
    temperature = 0.1
    top_p = 1
    max_tokens_to_sample = 1_024
    stream = false
    ```
  </Accordion>
</AccordionGroup>

We launch R2R by specifying this configuration file:
```Zsh
export R2R_CONFIG_NAME=ollama
python -m r2r.serve
```

Since we're serving with Docker, once R2R successfully launches the R2R dashboard opens for us. We can upload a document and see requests hit our Ollama server.

<Frame
    caption="The R2R Dashboard and Ollama server showing successful ingestion"
>
    <img src="../images/cookbooks/local/local_ingestion.png" alt="The processed document and the Ollama server logs." />
</Frame>

### Retrieval and Search

Now that we have ingested our file, we can perform RAG and chunk search over it. Here, we see that we are able to get relevant results and correct answers—all without needing to make a request out to an external provider!

<Frame
caption="A RAG search done using a local LLM"
>
    <img src="../images/cookbooks/local/local_rag.png" alt="A RAG search done with local LLMs." />
</Frame>

<Frame
caption="A chunk search done using a local LLM"
>
    <img src="../images/cookbooks/local/local_search.png" alt="A semantic serach done with LLMs." />
</Frame>

### Extracting Entities and Relationships

If we'd like to build a graph for our document, we must first extract the entities and relationships that it contains. Through the dashboard
we can select the 'Document Extraction' action in the documents table. This will start the extraction process in the background, which uses named entity
recognition to find entities and relationships.

Note that this process can take quite a bit of time, depending on the size of your document and the hardware running your model. Once the process is complete,
we will see that the `extraction` status has turned green.

<Frame
caption="A successful extraction shown on the documents table "
>
    <img src="../images/cookbooks/local/successful_extraction.png" alt="Successful extraction on the documents table." />
</Frame>

<Frame
caption="The entities extracted from our document"
>
    <img src="../images/cookbooks/local/extracted_entities.png" alt="A semantic serach done with LLMs." />
</Frame>

<Frame
caption="The relationships extracted from our document"
>
    <img src="../images/cookbooks/local/extracted_relationships.png" alt="A semantic serach done with LLMs." />
</Frame>

### Graph RAG

Now we must `pull` the document extractions into the graph. This is done at the collection level, and creates a copy of our extractions for searching over and creating communities with.

Then, we can conduct search, RAG, or agent queries that utilize the graph.

<Frame
caption="A RAG search that includes entities and relationships from the graph"
>
    <img src="../images/cookbooks/local/graph_search.png" alt="A search that utilizes the entities and relationships from the graph." />
</Frame>

<Frame
caption="Pulling extractions into the graph"
>
    <img src="../images/cookbooks/local/pulling_extractions.png" alt="A semantic serach done with LLMs." />
</Frame>

### Building communities

We can go one step further and create communities over the entities and relationships in the graph. By clustering over the closely related extractions, we can
further develop the understanding of how these entities and relationships interact. This can be particularly helpful in sets of documents where we see overarching
or recuring themes.

We trigger the extraction procedure, which produces a number of communities. Now, when we run queries over our graph we can utilize the communities to provide context that
better encompasses overall concepts and ideas throughout our documents.

<Frame
caption="A RAG query that utilizes communities"
>
    <img src="../images/cookbooks/local/graph_search_communities.png" alt="A RAG search that utilizes communities." />
</Frame>

<Frame
caption="The communities that were built from our document"
>
    <img src="../images/cookbooks/local/generated_communities.png" alt="A semantic serach done with LLMs." />
</Frame>


</Steps>
