# ALMA — Autonomous Learning of Memory Algorithms

An example that evolves memory strategies for autonomous agents using an LLM-driven evolutionary loop. ALMA discovers PTC-Lisp programs that help agents learn from experience across pluggable environments (GraphWorld, ALFWorld).

## How It Works

```
  ┌──────────────┐
  │   Baseline   │ ← evaluate null design (no memory) to establish score floor
  └──────┬───────┘
         ▼
  ┌─────────────┐
  │   Archive   │ ← scored designs with source strings + runtime logs
  └──────┬──────┘
         │ sample parents
         ▼
  ┌─────────────┐
  │ DebugAgent  │ ← SubAgent greps runtime logs (prints, tool traces, errors)
  └──────┬──────┘
         │ evidence-based critique
         ▼
  ┌─────────────┐
  │  MetaAgent  │ ← LLM SubAgent defn's live closures
  └──────┬──────┘
         │ namespace = step.memory
         ▼
    ┌────┴────┐
    ▼         ▼
 Collection  Deployment
 (learn)     (evaluate)
    │         │
    └────┬────┘
         ▼
  Score − Baseline → Archive
```

ALMA maintains an **archive** of memory designs — PTC-Lisp namespaces containing functions and data that store and retrieve knowledge across episodes. Each iteration:

1. **Baseline** — evaluate a no-memory design to establish the score floor
2. **Sample** parent designs from the archive (weighted by score, novelty, and exploration)
3. **Debug** — a DebugAgent (SubAgent with `grep`) searches through runtime logs from parent evaluations, finding evidence of recall failures, store misuse, errors, and debug output
4. **Generate** a new design via a multi-turn PTC-Lisp SubAgent, guided by the debug agent's critique
5. **Collection phase** — run tasks sequentially, updating memory after each episode; runtime logs (prints, tool call traces) are captured for each episode
6. **Deployment phase** — run new tasks with frozen memory to test generalization
7. **Archive** the design with score normalized against the baseline, along with runtime logs for future debugging

The key insight: deployment score (not collection) determines fitness, and normalizing against a no-memory baseline ensures scores reflect the actual value of memory rather than task difficulty.

### Runtime Log Capture

Every call to `mem-update` and `recall` captures a runtime log containing:
- **Prints** — output from `(println ...)` calls in the design code
- **Tool calls** — name, arguments, and result for every tool invocation (e.g., `store-obs`, `find-similar`, `graph-path`)

These logs are attached to each episode result and formatted into greppable text by `DebugLog`, enabling the DebugAgent to search for specific patterns like empty recall results, failed tool calls, or missing graph paths.

### DebugAgent vs Analyst

The previous Analyst was a single-shot LLM call that read source code and metrics but had no visibility into what actually happened at runtime. The DebugAgent replaces it with a SubAgent that can `grep` through structured runtime logs:

```
[recall] TOOL find-similar: k=3 query="flask" -> []
[recall] PRINT: no results found for flask
[mem-update] TOOL store-obs: text="visited room_A" -> "stored:1"
[mem-update] TOOL graph-update: edges=[["room_A","room_B"]] -> "ok"
```

This gives the meta-agent evidence-based feedback: "recall returned empty results because nothing was stored with collection namespacing" rather than guessing from source code alone.

## Differences from the Original Paper

This implementation is inspired by [ALMA (Automated meta-Learning of Memory designs for Agentic systems)](https://github.com/zksha/alma) ([paper](https://arxiv.org/abs/2602.07755)), but takes a fundamentally different approach to how memory designs are represented and executed.

### What we keep from the paper

- **Evolutionary archive** with quality-diversity sampling (score + novelty + exploration)
- **Two-phase evaluation**: collection (memory builds up) then deployment (frozen memory, unseen tasks) — deployment score as fitness
- **Meta-agent** that samples parent designs and proposes new ones
- **Memory as retrieve + update**: two functions that store experience and provide advice

### What we do differently

| Aspect | Original (Python) | This implementation (PTC-Lisp) |
|--------|-------------------|-------------------------------|
| **Design representation** | Python code strings generated by LLM, stored as text, `exec()`'d at runtime | Live PTC-Lisp closures in memory — no string eval |
| **Meta-agent** | Multi-step LLM pipeline: ideate, program, verify, debug (up to 3 retries) | DebugAgent greps runtime logs for evidence, then multi-turn PTC-Lisp SubAgent `defn`s functions — validation is built into the interpreter |
| **Verification** | Trial runs + LLM-powered debugging loop | Static analysis + sandboxed execution at the language level — no external debug loop needed |
| **Design structure** | Modular sub-modules with dedicated databases, hierarchical `update()`/`retrieve()` chains | Flat namespace of pure functions + `def`'d state, backed by namespaced vector store, graph store, and LLM analysis tools |
| **Recall output** | Arbitrary structured data injected into prompt context | Text string — the `recall` function returns human-readable advice |
| **Observation data** | Raw text strings (`"You are in room_A. You see: key..."`) — memory code must parse with string splitting/regex | Structured maps via `data/` context (`(:location (:result obs))`) — no parsing needed, less error-prone |
| **Persistence** | Python code strings stored directly | Closures stored as `(fn ...)` strings via `CoreToSource.serialize_closure/1`; hydrated by running through `Lisp.run` |
| **Safety** | Docker containers for sandboxing `exec()`'d code | BEAM process sandbox with timeout (1s) and memory limits (10MB) — no containers needed |
| **Environments** | ALFWorld, TextWorld, Baba Is AI, MiniHack | GraphWorld + ALFWorld via pluggable `Environment` behaviour |

### Why PTC-Lisp instead of Python strings?

The original paper generates Python code as strings and uses `exec()` to run it. This works but has downsides:

1. **Fragile generation** — LLMs must produce syntactically correct Python as a string inside a larger prompt response. Escaping, indentation, and import management are error-prone.
2. **Debug loops** — broken code requires a separate LLM-powered fix cycle (up to 3 retries in the paper).
3. **No static guarantees** — errors are only caught at runtime during trial runs.

Our approach uses PTC-Lisp, a sandboxed Clojure-like DSL where the LLM generates code that is immediately parsed, statically analyzed, and executed within a BEAM process. The MetaAgent is itself a SubAgent that writes `defn`'s interactively — if a function doesn't work, the SubAgent can fix it in the next turn, using the same mechanism as any other multi-turn agent. No separate debug infrastructure needed.

The closures produced are live values in memory, not strings. They carry their Core AST and can be serialized back to source when needed (for archiving or novelty comparison), but during execution there is zero parsing overhead.

## Architecture

```
Alma.run/1                        Entry point — orchestrates the loop
  │
  ├── Alma.Loop                   Single iteration logic + telemetry
  │     ├── DebugAgent.analyze    SubAgent greps runtime logs for evidence
  │     ├── DebugLog              Formats runtime logs into greppable text
  │     ├── MetaAgent.generate    LLM SubAgent defn's closures directly
  │     ├── MemoryHarness         Calls live closures with injected tools
  │     │     ├── VectorStore     N-gram embedding + cosine similarity (with collections)
  │     │     ├── GraphStore      Undirected graph + BFS pathfinding
  │     │     └── TaskAgent.run   Runs a single task via env-provided tools
  │     └── Analysis              Trajectory metrics and compression for MetaAgent
  │
  ├── Alma.Archive                Evolutionary archive with weighted sampling
  │                               AST-based novelty, JSON persistence
  │                               Seeds: null baseline + env-specific baseline
  │
  ├── Alma.Trace                  Telemetry spans for ptc_viewer tracing
  │
  ├── Alma.Environment            Behaviour for pluggable environments
  │
  ├── Environments.GraphWorld     Procedurally generated room navigation
  │     └── Generator             Seeded task/graph generation
  │
  └── Environments.ALFWorld       Text-based household tasks (via Python bridge)
        └── Port                  GenServer wrapping Elixir Port for Python
```

| Module | Purpose |
|--------|---------|
| `Alma` | Entry point, `run/1` |
| `Alma.Loop` | Single iteration: debug, generate, evaluate, archive |
| `Alma.DebugAgent` | SubAgent with `grep` that analyzes parent runtime logs for evidence-based critique |
| `Alma.DebugLog` | Formats runtime logs (prints, tool traces) into greppable text |
| `Alma.MetaAgent` | Multi-turn PTC-Lisp SubAgent that produces live closures |
| `Alma.MemoryHarness` | Calls design closures with namespace merge, tool injection, and runtime log capture |
| `Alma.VectorStore` | Pure Elixir vector store with character n-gram cosine similarity and collection namespacing |
| `Alma.GraphStore` | Undirected graph adjacency map with BFS shortest-path |
| `Alma.TaskAgent` | Runs one task via SubAgent with environment-provided tools |
| `Alma.Archive` | Evolutionary archive with AST-based novelty and weighted sampling |
| `Alma.Analysis` | Trajectory metrics and compressed episode summaries for MetaAgent context |
| `Alma.Trace` | Telemetry spans for ptc_viewer tracing |
| `Alma.Environment` | Behaviour defining the pluggable environment interface |
| `Alma.Environments.GraphWorld` | Room navigation environment |
| `Alma.Environments.GraphWorld.Generator` | Seeded procedural task generation |
| `Alma.Environments.ALFWorld` | ALFWorld household environment (Python bridge) |
| `Alma.Environments.ALFWorld.Port` | GenServer wrapping Elixir Port for Python bridge |

### Lisp-Native Designs

A design is a **PTC-Lisp namespace** — the full `step.memory` produced by the MetaAgent SubAgent. It contains live closures and any helper functions or constants they depend on.

The two key closures are:

- **`mem-update`** — called after each episode with episode data in context. Uses `def` to persist state across episodes.
- **`recall`** — called before each task with task info in context. Returns a text string of advice for the task agent.

Designs can also include **helper functions** (e.g. `success-rate`, `format-stats`) that both closures share. The namespace carries everything together, solving the "vanishing helpers" problem where isolated closures lose access to sibling functions.

```clojure
;; A complete design namespace — the MetaAgent discovers what to store and retrieve
(defonce episode-count 0)

(defn mem-update []
  (def episode-count (inc episode-count))
  ;; Process data/observation_log, data/success, data/actions, data/task
  ;; Use tool/store-obs (with collections), tool/graph-update, tool/analyze, def, etc.
  (tool/graph-update {"edges" [["room_A" "room_B"]]})
  (tool/store-obs {"text" "found key in kitchen" "collection" "spatial"})
  ...)

(defn recall []
  ;; Use data/task, tool/find-similar, tool/graph-path, persisted state, etc.
  ;; Return a string with advice for the task agent
  (let [path (tool/graph-path {"from" "room_A" "to" "room_C"})]
    (str "Shortest path: " path)))
```

The MetaAgent decides autonomously what information to store and how to retrieve it. The evolutionary loop selects designs whose `recall` advice improves task agent performance.

### How Closures Execute

The `MemoryHarness` merges the design's full namespace into the Lisp runtime memory before calling each closure, and injects tools backed by mutable Agent state:

```
design.namespace ──merge──► episodic memory ──put closure──► Lisp.run("(recall)", tools: tools)
```

**Injected tools** (available in both `mem-update` and `recall`):

| Tool | Signature | Purpose |
|------|-----------|---------|
| `tool/store-obs` | `(text :string, metadata :map, collection :string) -> :string` | Store an observation with auto-embedding. Optional `collection` param for namespacing (default: `"default"`) |
| `tool/find-similar` | `(query :string, k :int, collection :string) -> [:map]` | Find top-k similar observations. Optional `collection` filters to one namespace; omit to search all |
| `tool/summarize` | `(text :string, instruction :string) -> :string` | LLM-powered text condensation/synthesis |
| `tool/analyze` | `(text :string, instruction :string, format :string) -> :any` | LLM-powered analysis. Optional `format`: `"text"` (default) or `"json"` (returns parsed map) |
| `tool/graph-update` | `(edges [[:string]]) -> :string` | Add undirected edges to a persistent graph |
| `tool/graph-neighbors` | `(node :string) -> [:string]` | Get sorted neighbors of a node |
| `tool/graph-path` | `(from :string, to :string) -> [:string]` | BFS shortest path, or `nil` if disconnected |

The **vector store** (`Alma.VectorStore`) supports both dense vectors (from real embedding models via `--embed-model`) and sparse n-gram vectors (default, no API calls needed). When `--embed-model` is set, embedding I/O happens outside the Agent lock for safe concurrent access. Collection namespacing lets designs maintain logically separate stores (e.g. `"spatial"` vs `"strategy"`) while still allowing cross-collection search when no collection is specified.

The **graph store** (`Alma.GraphStore`) provides an in-process undirected adjacency map with BFS pathfinding, equivalent to what the original Python ALMA offloads to NetworkX. Designs can build spatial maps incrementally via `tool/graph-update` and query them for neighbors or shortest paths.

Both stores live as special keys in the memory map, persisted across `mem-update` calls during collection and frozen during deployment.

Context data is accessed via `data/` prefix (`data/task`, `data/success`, etc.). Memory variables defined via `def` persist across calls as bare names (`stats`, `visits`, etc.).

### Archive Persistence

Designs are serialized for persistence using `CoreToSource.serialize_closure/1`, which converts each live closure to its `(fn ...)` source string. The archive stores `mem_update_source` and `recall_source` as separate fields:

```clojure
;; mem_update_source
(fn [] (def episode-count (inc episode-count)) (doseq [obs data/observation_log] ...))

;; recall_source
(fn [] (let [results (tool/find-similar {"query" (str (:goal data/task)) "k" 5})] ...))
```

On load, `Archive.hydrate/1` reconstructs each closure by evaluating its source string through `Lisp.run`. Novelty comparison concatenates both source strings for AST-based distance calculation.

### Pluggable Environments

ALMA uses an `Alma.Environment` behaviour that decouples the evolutionary loop from any specific domain. Each environment provides:

- `reset/1`, `step/2`, `observe/1`, `success?/1` — core simulation
- `task_prompt/0`, `task_tools/2` — SubAgent configuration for the task agent
- `generate_tasks/2`, `generate_family_tasks/2` — procedural task generation
- `summarize_observation/2`, `format_goal/1` — trajectory analysis
- `seed_design_source/0` — optional PTC-Lisp baseline for the archive
- `context_schema/0` — optional data schema for MetaAgent prompts

The loop, archive, harness, and meta-agent are all domain-blind — they work with any environment that implements the behaviour.

### GraphWorld Environment

A grid of connected rooms containing objects with meaningful names (key, book, lamp, gem, scroll, etc.). The agent has four tools (`look`, `move_to`, `pick_up`, `put_down`) and must deliver a target object to a destination room within 20 steps. Graphs are procedurally generated with configurable room count, object count, connectivity, and seed.

**Family-based generation:** When `:family` is set, topology (room connectivity) is seeded by the family seed while object/goal/agent placement is seeded by `:seed`. This means episodes within the same family share room layout, making spatial memory genuinely transferable across episodes. By default, family mode is on (family = seed).

### ALFWorld Environment

Text-based household tasks from the [ALFWorld](https://github.com/alfworld/alfworld) benchmark — the primary domain in the original ALMA paper. The agent selects from admissible text commands (e.g., "go to desk 1", "take mug 1 from desk 1", "put mug 1 in/on shelf 1") to complete household goals like cleaning, heating, or placing objects.

Communication with ALFWorld happens via a Python bridge (`priv/alfworld_bridge.py`) running as an Elixir Port. The bridge stays alive across episodes for efficient reuse.

**Prerequisites:** Python 3 with `alfworld` installed (`pip install alfworld`). ALFWorld data must be downloaded (`alfworld-download`).

```bash
mix alma.run --env alfworld                          # default settings
mix alma.run --env alfworld --python /path/to/python # custom Python
mix alma.run --env alfworld --iterations 3 --episodes 2
```

### Archive Sampling

Designs are sampled with weight = `0.5 * score + 0.3 * novelty + 0.2 * (1 / (1 + times_sampled))`, balancing exploitation (high score), diversity (AST-based novelty), and exploration (undersampled designs).

## Running

### Prerequisites

Requires [PtcRunner](../../) (the parent project) and an LLM API key.

```bash
cd examples/alma
mix deps.get
```

### Tests

```bash
mix test              # no LLM calls required
```

### Full Run (requires LLM)

The simplest way to run ALMA is via the Mix task:

```bash
mix alma.run                                    # GraphWorld, 5 iterations, 3 episodes, bedrock:haiku
mix alma.run --iterations 10 --episodes 5       # longer run
mix alma.run --model bedrock:sonnet             # use a different model
mix alma.run --meta-model openrouter:openai/gpt-5 --model bedrock:haiku  # strong meta, cheap exec
mix alma.run --embed-model openai:text-embedding-3-small  # real embeddings for VectorStore
mix alma.run --rooms 6 --seed 123               # larger world, different seed
mix alma.run --env alfworld                     # ALFWorld household tasks (requires Python)
mix alma.run --env alfworld --python python3.11 # ALFWorld with specific Python
mix alma.run --no-trace                         # disable trace file output
mix alma.run --quiet                            # suppress per-iteration output
```

**Options:**

| Flag | Default | Description |
|------|---------|-------------|
| `--env` | `graph_world` | Environment: `graph_world` or `alfworld` |
| `--iterations` | `5` | Number of evolutionary iterations |
| `--episodes` | `3` | Tasks per collection/deployment phase |
| `--rooms` | `8` | Rooms per GraphWorld environment |
| `--seed` | `42` | Random seed for reproducibility |
| `--family` | same as seed | Family seed for shared topology across episodes |
| `--no-family` | off | Disable family mode (legacy single-seed behavior) |
| `--deploy-seeds` | `3` | Seed offsets for deployment scoring (more = robust) |
| `--model` | `bedrock:haiku` | LLM model for task execution |
| `--meta-model` | same as `--model` | LLM model for meta agent and debug agent |
| `--embed-model` | `embed` (ollama:nomic-embed-text) | Embedding model for VectorStore similarity |
| `--no-embed` | off | Disable real embeddings, use n-gram fallback |
| `--no-trace` | off | Disable JSONL trace file output |
| `--quiet` | off | Suppress verbose iteration output |
| `--python` | `python3` | Python executable path (ALFWorld only) |
| `--alfworld-data` | `~/.cache/alfworld/` | ALFWorld data directory |

Tracing is enabled by default — each run writes a timestamped file to `traces/alma_<timestamp>.jsonl`.

### Tracing with ptc_viewer

View a trace in the browser:

```bash
cd ../.. && mix ptc.viewer --trace-dir examples/alma/traces
```

### Programmatic API

For scripted or iex usage:

```elixir
llm = PtcRunner.LLM.callback("bedrock:haiku")

{archive, trace_path} = Alma.run(
  llm: llm,
  iterations: 5,
  episodes: 5,
  rooms: 5,
  seed: 42,
  trace: true          # writes to traces/alma_<timestamp>.jsonl
)

best = Alma.Archive.best(archive)
IO.inspect(best.design, label: "Best memory design")
```

Pass a string to `trace:` for a custom path: `trace: "my_trace.jsonl"`.

### Telemetry

ALMA emits telemetry events for external observability:

| Event | Measurements | Metadata |
|-------|-------------|----------|
| `[:alma, :iteration, :start]` | — | `generation`, `parent_ids` |
| `[:alma, :iteration, :stop]` | `duration`, `collection_score`, `deployment_score`, `normalized_score`, `baseline_score` | `generation`, `design_name`, `archive_size` |

### Archive Persistence

Save and restore evolved designs across sessions:

```elixir
archive |> Alma.Archive.save("archive.json")
archive = Alma.Archive.load("archive.json") |> Alma.Archive.hydrate()
```

## Future Improvements

See [FUTURE.md](FUTURE.md) for planned improvements.
