# Gleann Plugin Extraction Benchmark

> Auto-generated by `go test ./tests/benchmarks/ -run TestPluginBenchmark`  
> Date: 2026-05-13 09:39

## Overview

This benchmark compares gleann's 4 document extraction backends across speed,
structural quality, and content fidelity using the same test documents.

### Extraction Layers

| Layer | Backend | Type | Dependencies | When Used |
|:-----:|---------|------|--------------|----------|
| -1 | **go-native** | In-process (Go) | Zero external deps | Always available (fallback) |
| 0 | **markitdown-cli** | CLI subprocess | Python `markitdown` | When markitdown is installed |
| 1a | **plugin-docs** | HTTP server | Python markitdown + docling | When plugin is installed |
| 1b | **plugin-marker** | HTTP server | Python marker-pdf + surya OCR | When plugin is installed |

### Fallback Chain

```
gleann index build → for each file:
  1. Try plugin (Layer 1a/1b) via HTTP /convert
  2. If no plugin → try markitdown CLI (Layer 0)
  3. If no markitdown → Go-native (Layer -1)
```

This means **gleann always works** even without any plugins installed.
Plugins add higher-quality extraction for specific formats.

## Metrics

| Metric | Description | Measurement |
|--------|-------------|-------------|
| **Latency** | Wall-clock time per document conversion | `time.Since(start)` in ms |
| **Throughput** | Documents processed per second | `1000 / latency_ms` |
| **Sections** | Section nodes in output graph | Count of `_type=Section` nodes |
| **Edges** | Graph edges (section relationships) | Count of `HAS_SECTION` + `HAS_SUBSECTION` |
| **Hierarchy Accuracy** | % of edges that are parent-child | `HAS_SUBSECTION / total_edges * 100` |
| **Heading Detection** | % of known headings found in output | Case-insensitive match against ground truth |
| **Keyword Fidelity** | % of expected keywords present in text | Case-insensitive match against ground truth |
| **Max Depth** | Deepest heading level detected | Count of `#` in markdown headings (1-6) |
| **Word Count** | Total words in extracted markdown | `len(strings.Fields(md))` |
| **Table Detection** | Whether tables were found | Presence of `\|` table or `<table>` |
| **Code Blocks** | Number of fenced code blocks | Count of `` ``` `` pairs |
| **Link Count** | Hyperlinks found | `[text](url)` + bare `https://` |

## Format Support Matrix

| Format | go-native | markitdown-cli | plugin-docs | plugin-marker |
|--------|:---------:|:--------------:|:-----------:|:-------------:|
| PDF | ✅ | ✅ | ✅ | ✅ |
| DOCX | ✅ | ✅ | ✅ | ✅ |
| XLSX | ✅ | ✅ | ✅ | — |
| PPTX | ✅ | ✅ | ✅ | ✅ |
| CSV | ✅ | ✅ | ✅ | — |
| HTML | ✅ | — | — | ✅ |
| EPUB | — | — | — | ✅ |
| PNG/JPG | — | ✅ | ✅ | ✅ |
| TIFF/BMP | — | — | — | ✅ |

## Detailed Results

| File | Backend | Latency | Sections | Edges | Hier.Acc | Head.Det | Kwd.Fid | Depth | Words | Table | Links |
|------|---------|:-------:|:--------:|:-----:|:--------:|:--------:|:-------:|:-----:|:-----:|:-----:|:-----:|
| benchmark_report.pdf | go-native | 4ms | 3 | 3 | 66% | 100% | 100% | 2/3 | 366 | — | 0 |
| benchmark_report.pdf | markitdown-cli | 413ms | 0 | 0 | 0% | 100% | 100% | 0/3 | 385 | ✅ | 0 |
| benchmark_report.pdf | plugin-docs | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| benchmark_report.pdf | plugin-marker | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| technical_report.docx | go-native | 0ms | 7 | 7 | 85% | 100% | 100% | 3/0 | 194 | — | 0 |
| technical_report.docx | markitdown-cli | — | — | — | — | — | — | — | — | — | — | `markitdown-cli: markitdown failed: Tr...` |
| technical_report.docx | plugin-docs | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| technical_report.docx | plugin-marker | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| benchmark_data.xlsx | go-native | 4ms | 3 | 3 | 66% | 100% | 100% | 2/0 | 262 | ✅ | 0 |
| benchmark_data.xlsx | markitdown-cli | — | — | — | — | — | — | — | — | — | — | `markitdown-cli: markitdown failed: Tr...` |
| benchmark_data.xlsx | plugin-docs | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| research_slides.pptx | go-native | 0ms | 6 | 6 | 83% | 100% | 100% | 2/0 | 116 | — | 0 |
| research_slides.pptx | markitdown-cli | — | — | — | — | — | — | — | — | — | — | `markitdown-cli: markitdown failed: Tr...` |
| research_slides.pptx | plugin-docs | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| research_slides.pptx | plugin-marker | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| api_reference.html | go-native | 0ms | 14 | 14 | 92% | 100% | 85% | 4/3 | 243 | — | 1 |
| api_reference.html | plugin-marker | — | — | — | — | — | — | — | — | — | — | `plugin not running at http://localhos...` |
| benchmarks.csv | go-native | 0ms | 1 | 1 | 0% | 100% | 100% | 1/0 | 267 | ✅ | 0 |
| benchmarks.csv | markitdown-cli | 326ms | 0 | 0 | 0% | 100% | 100% | 0/0 | 265 | ✅ | 0 |
| benchmarks.csv | plugin-docs | — | — | — | — | — | — | — | — | — | — | `plugin request: Post "http://localhos...` |

## Per-Format Comparison

### PDF

| Backend | Latency | Sections | Head.Det | Kwd.Fid | Words | Table |
|---------|:-------:|:--------:|:--------:|:-------:|:-----:|:-----:|
| go-native | 4ms | 3 | 100% | 100% | 366 | — |
| markitdown-cli | 413ms | 0 | 100% | 100% | 385 | ✅ |
| plugin-docs | ERROR | — | — | — | — | — |
| plugin-marker | ERROR | — | — | — | — | — |

### DOCX

| Backend | Latency | Sections | Head.Det | Kwd.Fid | Words | Table |
|---------|:-------:|:--------:|:--------:|:-------:|:-----:|:-----:|
| go-native | 0ms | 7 | 100% | 100% | 194 | — |
| markitdown-cli | ERROR | — | — | — | — | — |
| plugin-docs | ERROR | — | — | — | — | — |
| plugin-marker | ERROR | — | — | — | — | — |

### XLSX

| Backend | Latency | Sections | Head.Det | Kwd.Fid | Words | Table |
|---------|:-------:|:--------:|:--------:|:-------:|:-----:|:-----:|
| go-native | 4ms | 3 | 100% | 100% | 262 | ✅ |
| markitdown-cli | ERROR | — | — | — | — | — |
| plugin-docs | ERROR | — | — | — | — | — |

### PPTX

| Backend | Latency | Sections | Head.Det | Kwd.Fid | Words | Table |
|---------|:-------:|:--------:|:--------:|:-------:|:-----:|:-----:|
| go-native | 0ms | 6 | 100% | 100% | 116 | — |
| markitdown-cli | ERROR | — | — | — | — | — |
| plugin-docs | ERROR | — | — | — | — | — |
| plugin-marker | ERROR | — | — | — | — | — |

### HTML

| Backend | Latency | Sections | Head.Det | Kwd.Fid | Words | Table |
|---------|:-------:|:--------:|:--------:|:-------:|:-----:|:-----:|
| go-native | 0ms | 14 | 100% | 85% | 243 | — |
| plugin-marker | ERROR | — | — | — | — | — |

### CSV

| Backend | Latency | Sections | Head.Det | Kwd.Fid | Words | Table |
|---------|:-------:|:--------:|:--------:|:-------:|:-----:|:-----:|
| go-native | 0ms | 1 | 100% | 100% | 267 | ✅ |
| markitdown-cli | 326ms | 0 | 100% | 100% | 265 | ✅ |
| plugin-docs | ERROR | — | — | — | — | — |

## Summary by Backend

| Backend | Avg Latency | Avg Sections | Head.Det | Kwd.Fid | Hier.Acc | Table Det. | Formats | Errors |
|---------|:-----------:|:------------:|:--------:|:-------:|:--------:|:----------:|:-------:|:------:|
| go-native | 1ms | 5.7 | 100% | 98% | 65% | 33% | 6 | 0 |
| markitdown-cli | 369ms | 0.0 | 100% | 100% | 0% | 100% | 5 | 3 |
| plugin-docs | 0ms | 0.0 | 0% | 0% | 0% | 0% | 5 | 5 |
| plugin-marker | 0ms | 0.0 | 0% | 0% | 0% | 0% | 4 | 4 |

## Analysis

### Speed

- **go-native** is the fastest backend since it runs in-process with zero network overhead.
- **markitdown-cli** adds Python subprocess startup cost (~300-500ms per file).
- **plugin-docs** amortizes startup via persistent HTTP server; docling PDF parsing adds ~500ms.
- **plugin-marker** is slowest due to deep learning model inference (surya OCR).

### Quality

- **go-native** extracts text but produces limited structural metadata (few or no headings from PDF).
- **markitdown-cli** produces markdown but often without heading markers for PDF content.
- **plugin-docs (docling)** produces the best PDF structure: deep section hierarchy, table detection.
- **plugin-marker** matches docling on section detection, adds OCR capability for scanned documents.

### Recommendations

| Use Case | Recommended Backend | Reason |
|----------|-------------------|--------|
| No dependencies available | **go-native** | Zero deps, always works |
| Lightweight install | **markitdown-cli** | `pip install markitdown`, no server |
| Production PDF workflow | **plugin-docs (docling)** | Best speed/quality balance for PDF |
| Scanned documents / OCR | **plugin-marker** | Only backend with deep-learning OCR |
| Office documents (DOCX/XLSX) | **plugin-docs** or **go-native** | Both handle Office formats well |
| HTML extraction | **go-native** or **plugin-marker** | Both support HTML natively |
| Maximum format coverage | **plugin-marker** | 15+ formats incl. EPUB, TIFF |

## Reproducing

```bash
# Prerequisites: both plugins installed and running
# Start plugins:
gleann tui  # → Plugins → install gleann-docs and gleann-marker

# Or start manually:
~/.gleann/plugins/gleann-docs/.venv/bin/python \
  ~/.gleann/plugins/_repos/gleann-plugin-docs/main.py --serve --port 8765 &
~/.gleann/plugins/gleann-marker/.venv/bin/python \
  ~/.gleann/plugins/_repos/gleann-plugin-marker/main.py --serve &

# Run benchmark
go test ./tests/benchmarks/ -run TestPluginBenchmark -v -timeout 300s

# Or via the shell script (also starts/stops plugins automatically)
./tests/e2e/plugin_benchmark.sh
```
