# Wire Formats

knowing supports multiple wire format encodings for graph data, each optimized for a different layer of the system. All formats encode the same `Payload` structure and are selected via the `format` parameter on MCP tools and CLI commands.

## Architecture

```
                          ┌─────────────┐
                          │  Graph Store │
                          └──────┬──────┘
                                 │ Payload
                    ┌────────────┼────────────┐
                    │            │            │
              ┌─────▼─────┐ ┌───▼────┐ ┌────▼────┐
              │   binary   │ │  gcf   │ │  json   │
              │ (transport)│ │ (LLM)  │ │ (compat)│
              └─────┬─────┘ └───┬────┘ └────┬────┘
                    │            │            │
              ┌─────▼─────┐ ┌───▼────┐ ┌────▼────┐
              │  daemon ↔  │ │ agent  │ │  human  │
              │  services  │ │context │ │  debug  │
              └───────────┘ └────────┘ └─────────┘
```

| Codec | Optimizes for | Consumer | Token savings vs JSON | When to use |
|-------|---------------|----------|----------------------|-------------|
| `gcf` | Maximum compression | AI agents in tight-budget workflows | **84%** | Default for MCP tools, session dedup, repeated calls |
| `binary` | Bytes on wire, speed | Services, caches, daemon IPC | N/A (not text) | Internal transport between knowing processes |
| `json` | Compatibility | Humans, generic API consumers, debugging | 0% (baseline) | When downstream consumers need standard JSON |
| `xml` | Structured markup | XML-based toolchains | ~-20% (larger) | Legacy integrations |
| `markdown` | Human readability | Documentation, display | ~10% | Human consumption, not agent workflows |

## How to Select a Format

**CLI:**
```bash
knowing context -task "add caching" -format gcf
knowing context -task "add caching" -format json
```

**MCP tools** (via `format` parameter):
```json
{"tool": "context_for_task", "arguments": {"task_description": "add caching", "format": "gcf"}}
```

**Recommendation: use `gcf` for all agent workflows.** LLM comprehension eval
(session 27, `eval/TestLLMFormatComprehension`) proved GCF achieves 100% accuracy
on structured extraction tasks at 16% of JSON's token cost. JSON scored 66.7%
(miscounts on large payloads). The "wait for comprehension validation" caveat is
resolved: GCF is both the most compact and the most accurately comprehended format.

- Use `gcf` for all AI agent workflows. Recommended default.
- Use `json` when debugging, piping to jq, or integrating with systems that expect JSON.
- Use `xml` for human-readable output only. Does not include edge data in output.

The format only affects the output encoding. The retrieval pipeline (seed matching, RWR, HITS, RRF fusion, scoring) is identical regardless of format. With `gcf`, more symbols fit within the same token budget because each symbol costs fewer tokens.

## GCF (Graph Compact Format)

Text-only, graph-native encoding designed for LLM consumption. Exploits three properties of graph data that flat formats cannot:

1. **Referential identity.** Nodes get local IDs (`@0`, `@1`). Edges reference by ID instead of repeating full qualified names.
2. **Graph topology.** Edges encoded as `@target<@source type` instead of verbose JSON objects.
3. **Hierarchical grouping.** Distance-based sections (`## targets`, `## related`) eliminate per-row distance fields.

### Grammar

```
payload       = header { node-line | edge-line | group-line | comment } ;
header        = "GCF" SP "tool=" token { SP key-value } LF ;
group-line    = "##" SP text LF ;
node-line     = "@" id SP kind SP qname SP score SP provenance LF ;
edge-line     = "@" target "<" "@" source SP edge-type [ SP status ] LF ;
comment       = "#" SP text LF ;

id            = DIGIT { DIGIT } ;
kind          = "fn" | "type" | "method" | "iface" | "var" | "const"
              | "resource" | "table" | "class" | "selector" | "field"
              | "route" | "ext" | "file" | "pkg" | "svc" ;
qname         = non-whitespace-text ;
score         = float ;
provenance    = "ast_inferred" | "ast_resolved" | "lsp_resolved"
              | "scip_resolved" | "otel_trace" | "structural" | token ;
status        = "added" | "removed" ;
```

### Example

```
GCF tool=context_for_task budget=5000 tokens=1847 symbols=10
## targets
@0 fn github.com/blackwell-systems/knowing/internal/mcp.requireHash 0.78 lsp_resolved
@1 method github.com/blackwell-systems/knowing/internal/mcp.Server.registerTools 0.74 lsp_resolved
@2 fn github.com/blackwell-systems/knowing/internal/mcp.requireStringArg 0.67 lsp_resolved
@3 fn github.com/blackwell-systems/knowing/internal/mcp.getIntArg 0.66 lsp_resolved
## related
@4 fn github.com/blackwell-systems/knowing/internal/mcp.NewServer 0.54 lsp_resolved
@5 fn github.com/blackwell-systems/knowing/cmd/knowing.cmdServe 0.51 ast_inferred
@6 fn github.com/blackwell-systems/knowing/cmd/knowing.cmdMCP 0.46 ast_inferred
## extended
@7 type github.com/blackwell-systems/knowing/internal/mcp.Server 0.42 lsp_resolved
@8 iface github.com/blackwell-systems/knowing/internal/types.GraphStore 0.38 lsp_resolved
@9 type github.com/blackwell-systems/knowing/internal/store.SQLiteStore 0.35 lsp_resolved
## edges
@0<@4 calls
@0<@5 calls
@0<@6 calls
@1<@4 calls
@2<@5 calls
@2<@6 calls
@9<@8 implements
@4<@8 references
```

The same data in JSON: ~965 tokens. In GCF: ~233 tokens. **75.9% savings** (simple heuristic). Using the word+punctuation token estimator from FINDINGS.md: 84.0% median across 6 fixtures.

### Format Elements

**Header:** `GCF tool=<name> budget=<N> tokens=<N> symbols=<N> [pack_root=<64-char hex>]`

When a PackRoot is computed (all `context_for_task` calls), the header includes a `pack_root` field. Agents can store this value and pass it back as the `pack_root` parameter on the next call: if the result is unchanged, the server returns `"unchanged"` instead of resending the full context (93-99% byte savings).

**Group headers:** `## targets` (distance 0), `## related` (distance 1), `## extended` (distance 2+), `## edges`

**Node lines:** `@{id} {kind} {qualified_name} {score} {provenance}`

**Edge lines:** `@{target}<@{source} {edge_type} [{status}]`

The `<` arrow points toward the target. `@0<@4 calls` means "@4 calls @0."

**Kind abbreviations:**

| Short | Full |
|-------|------|
| `fn` | function |
| `type` | type |
| `method` | method |
| `iface` | interface |
| `var` | var |
| `const` | const |
| `resource` | resource |
| `table` | table |
| `class` | class |
| `selector` | selector |
| `field` | field |
| `route` | route_handler |
| `ext` | external |
| `file` | file |
| `pkg` | package |
| `svc` | service |

### Session Statefulness

Across multiple tool calls in a session, previously-transmitted nodes can be referenced without retransmission:

```
GCF tool=context_for_files tokens=800 symbols=5 session=true
## targets
@0  # previously transmitted
@7 fn github.com/blackwell-systems/knowing/internal/mcp.handleBlastRadius 0.62 lsp_resolved
## edges
@0<@7 calls
```

Multi-call workflows get progressively cheaper as the session builds a shared vocabulary of known symbols.

### Where Token Savings Come From

1. **No field names.** GCF uses positional encoding. JSON repeats 9 field names per symbol.
2. **Local ID edge references.** `@0<@4 calls` (5 tokens) vs full qualified name pairs (30+ tokens).
3. **Group headers.** One `## related` replaces N `"distance": 1` fields.
4. **Kind abbreviations.** `fn` vs `"function"`, `iface` vs `"interface"`.
5. **No structural delimiters.** No `{}`, `[]`, `:`, `","`, indentation whitespace.

---

## Binary (GCB1)

Compact binary encoding optimized for transport between services and persistent caching. Not readable by humans or LLMs directly; designed for machine-to-machine paths where byte efficiency and encode/decode speed matter.

### Wire Layout

```
[magic:4 "GCB1"][version:1]
[header]
  tool        : length-prefixed string
  tokens_used : varint
  token_budget: varint
  num_symbols : varint
  num_edges   : varint
[symbols × num_symbols]
  qname       : length-prefixed string
  kind        : uint8 (enum)
  score       : float32 (4 bytes, little-endian)
  provenance  : uint8 (enum)
  distance    : uint8
  signature   : length-prefixed string
  blast_radius: float32
  confidence  : float32
  recency     : float32
  distance_c  : float32
[edges × num_edges]
  source_idx  : varint (index into symbols array)
  target_idx  : varint (index into symbols array)
  edge_type   : uint8 (enum)
  status      : uint8 (enum)
```

### Design Choices

- **Varint encoding** for integers: small values (symbol counts, indices) use 1 byte instead of 4-8.
- **Enum IDs** for kinds, provenances, edge types, statuses: 1 byte instead of variable-length strings.
- **float32** for scores and components: 4 bytes with sufficient precision (scores only need 2 decimal places).
- **Index-based edges**: edges reference symbols by array index (varint), not by name.
- **Length-prefixed strings**: no null terminators, no escaping, no delimiters.
- **No padding or alignment**: every byte carries data.

### Enum Tables

**Kind (16 entries):** function=1, type=2, method=3, interface=4, var=5, const=6, resource=7, table=8, class=9, selector=10, field=11, route_handler=12, external=13, file=14, package=15, service=16

**Provenance (7 entries):** ast_inferred=1, ast_resolved=2, lsp_resolved=3, otel_trace=4, scip_resolved=5, runtime_observed=6, structural=7

**Edge type (38 entries):** calls=1, imports=2, implements=3, references=4, handles_route=5, depends_on=6, deploys=7, exposes=8, configures=9, extends=10, overrides=11, decorates=12, throws=13, owned_by=14, authored_by=15, tests=16, runtime_calls=17, runtime_rpc=18, runtime_produces=19, runtime_consumes=20, contains=21, member_of=22, documents=23, consumes_endpoint=24, implements_rpc=25, consumes_rpc=26, gated_by_flag=27, deployed_by=28, tested_by=29, publishes=30, subscribes=31, connects_to=32, similar_to=33, co_tested_with=34, type_hint_of=35, accesses_field=36, reads_env=37, executes_process=38

**Status:** unchanged/empty=0, added=1, removed=2

### When to Use Binary

- Daemon-to-client IPC (the daemon stores and transmits binary; the client decodes to Payload then re-encodes to GCF or JSON for its consumer)
- Response caching (binary is the most space-efficient on-disk format)
- Cross-service communication where both ends are knowing-aware
- Streaming large graph results where decode latency matters

### Tradeoffs vs GCF

| | Binary | GCF |
|--|--------|-----|
| Byte size | Smallest | ~10-15% larger (text overhead) |
| Token count | N/A (not text) | Optimized |
| LLM-readable | No | Yes |
| Human-readable | No | Yes |
| Score precision | float32 (~7 digits) | 2 decimal places |
| Components preserved | Yes (full) | No (stripped for token savings) |
| Extensibility | Requires version bump | Append new fields freely |

---

## JSON

Standard JSON serialization. Maximum compatibility, zero configuration, works with any consumer. The baseline against which other formats are measured.

```json
{
  "tool": "context_for_task",
  "tokens_used": 1847,
  "token_budget": 5000,
  "pack_root": "<64-char hex>",
  "symbols": [
    {
      "qualified_name": "github.com/blackwell-systems/knowing/internal/mcp.requireHash",
      "kind": "function",
      "score": 0.78,
      "signature": "func requireHash(args map[string]any, key string) (types.Hash, error)",
      "provenance": "lsp_resolved",
      "distance": 0,
      "components": {
        "blast_radius": 0.40,
        "confidence": 0.25,
        "recency": 0.06,
        "distance": 0.15
      }
    }
  ],
  "edges": [
    {
      "source": "github.com/blackwell-systems/knowing/internal/mcp.NewServer",
      "target": "github.com/blackwell-systems/knowing/internal/mcp.requireHash",
      "edge_type": "calls"
    }
  ]
}
```

Use JSON when: debugging, piping to `jq`, integrating with tools that expect JSON, or when the consumer is not token-constrained.

---

## Benchmark Comparison

All three codecs encoding the same 10-symbol, 8-edge payload:

| Codec | Bytes | Tokens | vs JSON (bytes) | vs JSON (tokens) |
|-------|-------|--------|-----------------|------------------|
| JSON | 4,153 | 965 | baseline | baseline |
| GCF | 1,079 | 233 | -74% | -75.9% |
| Binary | 448 | N/A | -89% | N/A |

Full scorecard across 6 fixture cases (8 to 30 symbols):

| Case | JSON(T) | GCF(T) | Token Savings |
|------|---------|--------|---------------|
| context_for_task (10 sym) | 965 | 233 | 75.9% |
| context_for_task (30 sym) | 2,968 | 649 | 78.1% |
| context_for_files (15 sym) | 1,490 | 334 | 77.6% |
| blast_radius (8 sym) | 835 | 208 | 75.1% |
| semantic_diff (12 sym) | 1,206 | 295 | 75.5% |
| graph_query (20 sym) | 2,078 | 423 | 79.6% |

**Median token savings (GCF vs JSON): 76.7%** (simple heuristic). FINDINGS.md word+punctuation estimator: **84.0% median**.

Encode p99 latency: 64 microseconds (30-symbol payload, Apple M4 Pro).

---

## Codec Registry

The `internal/wire` package provides a pluggable codec registry. Core GCF types and encoding come from the standalone [`gcf-go`](https://github.com/blackwell-systems/gcf-go) library ([spec](https://github.com/blackwell-systems/gcf)); the wire package re-exports them and adds knowing-specific codecs. Formats are selected by name at runtime:

```go
import (
    gcf "github.com/blackwell-systems/gcf-go"              // GCF types and encoding
    "github.com/blackwell-systems/knowing/internal/wire"    // registry, binary, json, bridge
)

// Encode with a named codec.
output, err := wire.EncodeWith("gcf", payload)

// Decode with a named codec.
payload, err := wire.DecodeWith("binary", input)

// List all registered codecs.
for _, c := range wire.List() {
    fmt.Printf("%s: %s\n", c.Name, c.Description)
}

// Register a custom codec.
wire.Register(&wire.Codec{
    Name:        "msgpack",
    Description: "MessagePack encoding for external interop",
    Encode:      myMsgpackEncoder,
    Decode:      myMsgpackDecoder,
})
```

Built-in codecs are registered at init time. Custom codecs can be added by any package that imports `internal/wire` and calls `Register()` before use.

### Adding a New Codec

Implement two functions matching these signatures:

```go
type Encoder func(p *Payload) (string, error)
type Decoder func(input string) (*Payload, error)
```

Then register:

```go
func init() {
    wire.Register(&wire.Codec{
        Name:        "my-format",
        Description: "What this format optimizes for",
        Encode:      myEncoder,
        Decode:      myDecoder,
    })
}
```

The MCP tools and CLI pass the `format` parameter directly to the registry, so new codecs become available to all consumers immediately.

---

## Usage

### CLI

```bash
knowing context -task "refactor auth" -format gcf      # LLM-optimized
knowing context -task "refactor auth" -format json     # human/debug
knowing context -task "refactor auth" -format gcb      # pipe to another service
```

### MCP Tools

```json
{
  "name": "context_for_task",
  "arguments": {
    "task_description": "refactor auth middleware",
    "token_budget": 5000,
    "format": "gcf"
  }
}
```

Default is `xml` for MCP tools (human-readable structured output). CLI default is also `xml`. Agents that understand GCF should request it explicitly for 84%+ token savings.

---

## Format Comprehension Eval

### Token Cost Benchmark (deterministic)

Measures token cost across formats for the same payload. No LLM involved.

Run: `GOWORK=off go test ./eval/ -run TestFormatComprehension -v`

| Format | Avg tokens | vs JSON |
|--------|-----------|---------|
| JSON | 1,818 | baseline |
| XML | 1,818 | 100% |
| GCF | 265 | **15%** |

### LLM Comprehension Benchmark (session 27)

Sends the same context payload in each format to an actual LLM and measures
whether it can answer structured questions correctly. Six questions with
objectively verifiable answers: top symbol identification, symbol count, edge
count, kind extraction, seed/related group count, and edge type enumeration.

Run: `GOWORK=off go test ./eval/ -run TestLLMFormatComprehension -v -timeout 30m`

(Uses `claude -p` by default. Set `EVAL_BACKEND=api` with `ANTHROPIC_API_KEY` for direct API calls.)

| Format | Accuracy | Avg Tokens | vs JSON |
|--------|----------|-----------|---------|
| **gcf** | **100%** (6/6) | **2,687** | **16%** |
| json | 66.7% (4/6) | 16,372 | baseline |
| xml | 66.7% (4/6) | 5,026 | 31% |

> Note: TOON was also evaluated (100% accuracy, 9,427 tokens, 58% of JSON) but was removed in v0.15.0. GCF provides better compression at the same accuracy.

**Per-question results:**

| Question | json | xml | gcf |
|----------|------|-----|-----|
| top_symbol (identify highest-scored) | PASS | PASS | PASS |
| symbol_count (count all symbols) | FAIL (113 vs 133) | PASS | PASS |
| edge_count (count all edges) | FAIL (120 vs 131) | FAIL (0 vs 131) | PASS |
| top_kind (kind of top symbol) | PASS | PASS | PASS |
| seed_count (distance-0 symbols) | PASS | PASS | PASS |
| edge_type_list (enumerate edge types) | PASS | FAIL (no edges in XML) | PASS |

**Key findings:**

1. **GCF comprehension is validated.** 100% accuracy on all 6 structured extraction tasks. The concern that LLMs couldn't parse GCF's `@N` local IDs and `@target<@source` edge notation was unfounded.

2. **JSON is the worst performer on large payloads.** At 36K tokens (symbol_count task), the LLM miscounted symbols (113 vs 133). GCF at 5K tokens counted correctly. Verbosity hurts comprehension, not just token cost.

3. **XML doesn't include edge data.** The XML formatter (`FormatContextBlock`) renders symbols but not edges, causing edge-related questions to fail. This is a format limitation, not an LLM limitation.

4. **GCF is 6.1x more token-efficient than JSON** (2,687 vs 16,372 avg tokens) with higher accuracy. There is no reason to prefer any other format for agent workflows.

**Recommendation: GCF should be the default format for all agent workflows.**

---

## Delta Encoding

When an agent passes a `pack_root` from a prior call and the current result differs, the server computes a structural diff and returns only what changed. This is a fourth level of token optimization layered on top of GCF.

```
GCF tool=context_for_task delta=true base_root=aaa111 new_root=bbb222 tokens=30 savings=81%
## removed
fn github.com/example/project.OldHandler
## added
@0 fn github.com/example/project.NewHandler 0.85 rwr
## edges_added
github.com/example/project.Router -> github.com/example/project.NewHandler calls
```

Three outcomes when `pack_root` is sent:
1. **Same root**: "unchanged" (zero tokens)
2. **Different root, prior known**: delta encoding (removed + added sections)
3. **Different root, prior unknown**: full retransmission (fallback)

Delta is only used when it saves more than 40% over full retransmission (60% threshold in `DiffPacks.IsWorthIt`). The diff operates on node hashes (set difference, O(n)).

**Benchmark (session 27):** 81.2% token savings at 96.6% symbol overlap on re-query scenarios. See [context-packing.md](context-packing.md) for the full protocol and `bench/delta-packing/` for the benchmark.

## Implementation

| File | Purpose |
|------|---------|
| [`gcf-go`](https://github.com/blackwell-systems/gcf-go) | Standalone GCF library: `Payload`/`Symbol`/`Edge`/`Components` types, `Encode`, `Decode`, `Session`, `EncodeWithSession`, `DeltaPayload`, `EncodeDelta` |
| [`gcf` spec](https://github.com/blackwell-systems/gcf) | GCF specification v1.0: grammar, encoding rules, session statefulness, delta extension |
| `internal/wire/gcf.go` | Type aliases and delegating wrappers re-exporting gcf-go |
| `internal/wire/registry.go` | Codec registry (Register, Get, List, EncodeWith, DecodeWith) |
| `internal/wire/bridge.go` | `FromContextBlock`: converts `ContextBlock` to wire `Payload` with edge discovery |
| `internal/wire/json.go` | JSON codec (encode/decode via standard library) |
| `internal/wire/binary.go` | GCB binary codec (varint + length-prefixed, 38 edge type enums) |
| `internal/context/delta.go` | `DiffPacks`: structural diff between two `ContextBlock` values |
| `internal/wire/gcf_test.go` | GCF round-trip and encoding tests |
| `internal/wire/session_test.go` | Session deduplication tests |
| `internal/wire/delta_test.go` | Delta encoding tests |
| `internal/wire/registry_test.go` | Registry and JSON codec tests |
| `internal/wire/binary_test.go` | Binary codec tests |
| `bench/wire-format/` | Benchmark harness with 6 fixture cases in `cases/` |
| `bench/wire-format/scorecard.md` | Auto-generated comparison table |
| `bench/wire-format/FINDINGS.md` | Detailed results and interpretation |
| `bench/delta-packing/` | Delta packing benchmark (cross-task + re-query simulation) |
| `eval/format_comprehension_test.go` | Token cost benchmark (deterministic, no LLM) |
| `eval/format_llm_comprehension_test.go` | LLM comprehension eval (6 questions, 4 formats, cli/api backends) |

---

## Comparison to External Approaches

### Table formats (TSV/columnar)

Column headers + TSV rows achieve ~27% savings by eliminating field name repetition. But they treat graph data as flat tables: edge references still require full identifier strings in each row. GCF's local ID system pushes savings to 75%+.

### Binary formats (protobuf, MessagePack, FlatBuffers)

Optimize for machine parsing speed and byte size. An LLM cannot read a protobuf payload directly; it must be decoded to text first, eliminating the savings at the point of consumption. knowing's binary codec fills this niche for machine-to-machine paths, while GCF serves the LLM path.

### JSON-LD / RDF

Verbose by design (full URIs, type annotations, context declarations). Optimizes for semantic interoperability across systems, not token-constrained LLM consumption.
