3. Install dependencies (takes 5-10 min)

pip install -r requirements.txt

3. Install dependencies (takes 5-10 min)

pip install -r requirements.txt

3. Dependencies

pip install -r requirements.txt

Reinstall dependencies

pip install -r requirements.txt

Installation

Important — Keep dependencies up to date: If you are using a pre-built / downloaded package release (zip/tar), always install dependencies from the latest requirements.txt in this repository rather than any bundled copy. Package versions are updated regularly to address security vulnerabilities and compatibility issues.

> pip install -r requirements.txt
> 

Always fetch the current file from the repository before installing to ensure you have the latest, secure package versions.

Windows: 1. Download Crawllama 2. Extract to any folder (e.g., C:\Crawllama) 3. Install Ollama from ollama.ai/download 4. Start Ollama and load model:

 ollama serve
 ollama pull qwen3:4b
 

 setup.bat
 run.bat
 

Linux/macOS: 1. Download and extract:

 wget https://github.com/arn-c0de/Crawllama/archive/refs/tags/v1.4.7-preview.zip
 unzip v1.4.7-preview.zip
 cd Crawllama-v1.4.7-preview
 

 curl -fsSL https://ollama.ai/install.sh | sh
 ollama serve &
 ollama pull qwen3:4b
 

 chmod +x setup.sh run.sh
 ./setup.sh
 ./run.sh
 

---

Option 1: Setup Scripts (Recommended for Git Installation)

Windows:

setup.bat

Linux/macOS:

chmod +x setup.sh
./setup.sh

Note: After the initial setup, you must select at least one LLM model during setup. If a model is already installed, you can skip this step—otherwise, selection is required to avoid errors in the test program.

The setup script: - Checks Python version (3.10+) - Creates virtual environment - Lets you select features and LLM models to install (core is always installed) - Installs all selected dependencies - Creates necessary directories - Copies .env.example to .env - Checks Ollama status

Note for initial installation:

When running pip install -r requirements.txt for the first time within the newly created virtual environment, installing all dependencies—especially packages like torch, sentence-transformers, and scientific libraries—may take 5–10 minutes (or longer, depending on connection and hardware). Please wait until the process completes; afterward, the virtual environment is ready for use.

Note on disk space: After installation (including venv), the project typically requires about 1.2–1.5 GB of free disk space (v1.4: ~1.23 GB). This value may vary significantly depending on the operating system, Python packages (e.g., larger PyTorch/CUDA wheels), and additional models. Plan for ample additional space if storage is limited.

Model download sizes (approximate):

• qwen3:4b — ~2–4 GB (depending on format/quantization)
• qwen3:8b — ~8–12 GB
• deepseek-r1:8b — ~6–10 GB
• llama3:7b — ~6–9 GB
• mistral:7b — ~4–8 GB
• phi3:14b — ~12–20+ GB

Note: Model sizes vary significantly depending on the provider, format (FP16, INT8 quantization, etc.), and additional assets. Quantized models (e.g., INT8) can significantly reduce size, while FP32/FP16 or models with additional tokenizer/vocab files require more space. Plan for sufficient additional storage if using larger models or multiple models simultaneously.

Option 2: Manual Installation

Prerequisites: - Python 3.10+ (python.org) - Git (git-scm.com) - Ollama (ollama.ai/download)

Windows - Step by Step:

```cmd

6. Install and start Ollama

curl -fsSL https://ollama.ai/install.sh | sh ollama serve &

Option 3: Git Clone (Quick Installation)

```bash

Ollama Setup

```bash

Install Ollama

curl -fsSL https://ollama.ai/install.sh | sh # Linux/macOS

Or with setup script

run.bat # Windows ./run.sh # Linux/macOS

What is Machine Learning?

**New Commands:**

- `status` - Shows token usage and available context capacity
 

Context Usage Tracker ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Source ┃ Tokens ┃ Share ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩ │ Conversation │ 850 │ 8.5% │ │ Search Results │ 320 │ 3.2% │ │ Total Used │ 1,170 │ 11.7% │ │ Available │ 8,830 │ 88.3% │ │ Maximum │ 10,000 │ 100% │ └───────────────────┴───────────┴───────────┘

- `settings` - Interactive configuration editor
 

Displays all settings and allows: • Category selection (llm, search, rag, cache, osint, all) • Change LLM model (qwen3:8b, deepseek-r1:8b, etc.) • Adjust temperature (0.0-1.0) • Configure max tokens (now 16,000 for RTX 3080+) • Change search region (de-de, us-en, wt-wt) • Configure OSINT max results & rate limits • Enable/disable RAG • Enable/disable cache • Save changes directly to config.json • Auto-restart after saving (optional)

- `restart` - Restart agent
 

• Reloads config.json • Fully reinitializes agent • Optional session preservation • No session interruption ```

Or re-run setup

./setup.sh # or setup.bat ```

Quickstart

Usage

Note: The first start may take significantly longer than subsequent starts! Initialization, dependency installation, and model downloads may take several minutes, depending on hardware and internet connection. After the first successful start, all subsequent starts are significantly faster.

Quickstart

1. Start API Server

run_api.bat

2. Open API Documentation - Interactive Docs: http://localhost:8000/docs - ReDoc: http://localhost:8000/redoc

3. Send Query

curl -X POST http://localhost:8000/query \
 -H "X-API-Key: your-key" \
 -H "Content-Type: application/json" \
 -d '{"query": "What is Python?", "use_tools": false}'

User Guides

• Installation Guide - Detailed installation
• LangGraph Guide - Multi-hop reasoning
• Plugin Tutorial - Plugin development
• Health Monitoring - System monitoring

2. Create virtual environment

python -m venv venv venv\Scripts\activate

5. Configuration

copy .env.example .env notepad .env # Optional: Add API keys

2. Create virtual environment

python3 -m venv venv source venv/bin/activate

5. Configuration

cp .env.example .env nano .env # Optional: Add API keys

2. Virtual Environment

python -m venv venv source venv/bin/activate # Linux/macOS venv\Scripts\activate # Windows

5. Config

cp .env.example .env ```

1. Set API key in .env

CRAWLLAMA_API_KEY=your_secure_api_key_min_32_chars

3. Adjust rate limit (optional)

RATE_LIMIT=100

4. Configure CORS origins (optional)

ALLOWED_ORIGINS=http://localhost:3000,http://localhost:8080

**Usage with API Key:**

CLI Commands and Options

Basic Options

Advanced Options (v1.1)

Configuration

config.json

{
 "llm": {
 "base_url": "http://127.0.0.1:11434",
 "model": "qwen3:8b",
 "temperature": 0.7,
 "max_tokens": 10000,
 "stream": true
 },
 "search": {
 "provider": "duckduckgo",
 "max_results": 5,
 "timeout": 10
 },
 "rag": {
 "enabled": true,
 "batch_size": 100,
 "max_workers": 4
 },
 "cache": {
 "enabled": true,
 "ttl_hours": 24,
 "max_size_mb": 500,
 "clear_on_startup": false
 },
 "osint": {
 "max_results": 20,
 "email_search_limit": 50,
 "phone_search_limit": 50,
 "general_osint_limit": 100
 },
 "multihop": {
 "enabled": true,
 "max_hops": 3,
 "confidence_threshold": 0.7,
 "enable_critique": true
 },
 "plugins": {
 "example_plugin": {
 "enabled": true
 }
 },
 "security": {
 "rate_limit": 1.0,
 "max_context_length": 8000,
 "check_robots_txt": true
 }
}

Recommended max_tokens Settings:

Tip: Use the status command to monitor your token usage in real-time!

.env (Optional)

```bash

API Keys (optional)

BRAVE_API_KEY=your_brave_api_key SERPER_API_KEY=your_serper_api_key

Proxy (optional)

HTTP_PROXY=http://proxy:port HTTPS_PROXY=https://proxy:port ```

Adjust in config.json

"security": { "rate_limit": 2.0 # 2 req/s } ```

Interactive CLI Interface

CrawlLama's adaptive intelligence system with automatic agent selection and interactive commands.

1. CLI - Interactive Mode

```bash python main.py --interactive

4. CLI - Direct Queries

```bash

5. FastAPI Server

```bash

2. For local development (without API key)

CRAWLLAMA_DEV_MODE=true

With API key header

curl -X POST http://localhost:8000/query \ -H "Content-Type: application/json" \ -H "X-API-Key: your_api_key_here" \ -d '{"query": "test"}'

Or in dev mode (without API key)

export CRAWLLAMA_DEV_MODE=true python app.py

**Example Requests:**

REST API

CrawlLama provides a complete REST API for integration into custom applications.

Start API Server

Windows:

run_api.bat

Linux/macOS:

./run_api.sh

Or manually:

uvicorn app:app --host 0.0.0.0 --port 8000

Key Endpoints

• POST /query - Execute queries (with/without web search, multi-hop)
• GET /health - Health check
• GET /stats - System statistics
• POST /memory/remember - Store data (OSINT)
• GET /memory/recall/{category} - Retrieve data
• GET /plugins - Manage plugins
• POST /cache/clear - Clear cache

API Documentation

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Web Scraping

• Respects robots.txt
• Rate limiting (1 req/s default)
• Identifiable user agent
• Users are responsible for compliance with local laws

API Keys

• Brave Search API: brave.com/search/api
• Serper API: serper.dev

API rate limits

```bash

List plugins

curl http://localhost:8000/plugins

Load plugin

curl -X POST http://localhost:8000/plugins/example_plugin/load ```

Plugin Development

Creating a Simple Plugin

```python

plugins/my_plugin.py

from core.plugin_manager import Plugin, PluginMetadata

class MyPlugin(Plugin): def get_metadata(self) -> PluginMetadata: return PluginMetadata( name="MyPlugin", version="1.0.0", description="My custom plugin", author="Your Name", dependencies=[] )

def get_tools(self): return [self.my_tool]

def my_tool(self, input: str) -> str: return f"Processed: {input}" ```

See: Plugin Tutorial for details

Alternative: deepseek-r1:8b, llama3:7b, mistral

```

Troubleshooting