heripo-engine

System Requirements

• macOS (Apple Silicon or Intel)
• Node.js >= 24.0.0
• pnpm >= 10.0.0
• Python 3.9 - 3.12 (⚠️ Python 3.13+ is not supported)
• jq (JSON processing tool)
• poppler (PDF text extraction tools)

```bash

Install dependencies

pnpm install

Installation

Install Python 3.11 (recommended)

brew install python@3.11

Install jq

brew install jq

Install poppler

brew install poppler

Install Node.js and pnpm

brew install node npm install -g pnpm ```

For detailed installation guide, see @heripo/pdf-parser README.

Package Installation

```bash

Install individual packages

pnpm add @heripo/pdf-parser pnpm add @heripo/document-processor pnpm add @heripo/model pnpm add @heripo/logger

Or install all at once

pnpm add @heripo/pdf-parser @heripo/document-processor @heripo/model @heripo/logger ```

Build all

pnpm build

Build specific package

pnpm --filter @heripo/pdf-parser build

Usage Examples

Basic Usage

import type { DoclingDocument } from '@heripo/model';

import { anthropic } from '@ai-sdk/anthropic';
import { openai } from '@ai-sdk/openai';
import { DocumentProcessor } from '@heripo/document-processor';
import { Logger } from '@heripo/logger';
import { PDFParser } from '@heripo/pdf-parser';
import { readFile } from 'node:fs/promises';

const logger = new Logger({
  debug: (...args) => console.debug('[heripo]', ...args),
  info: (...args) => console.info('[heripo]', ...args),
  warn: (...args) => console.warn('[heripo]', ...args),
  error: (...args) => console.error('[heripo]', ...args),
});

// 1. PDF Parsing
const pdfParser = new PDFParser({
  port: 5001,
  logger,
});

await pdfParser.init();

const tokenUsageReport = await pdfParser.parse(
  'file:///path/to/report.pdf',
  'report-001',
  async (artifactDir) => {
    const doclingDocument = JSON.parse(
      await readFile(`${artifactDir}/result.json`, 'utf8'),
    ) as DoclingDocument;

    // 2. Document Processing (inside callback)
    const processor = new DocumentProcessor({
      logger,
      fallbackModel: anthropic('claude-opus-4-5'),
      pageRangeParserModel: openai('gpt-5.2'),
      tocExtractorModel: openai('gpt-5.1'),
      captionParserModel: openai('gpt-5-mini'),
      textCleanerBatchSize: 10,
      captionParserBatchSize: 5,
      captionValidatorBatchSize: 5,
    });

    const { document, usage } = await processor.process(
      doclingDocument,
      'report-001',
      artifactDir,
    );

    // 3. Use Results
    console.log('TOC:', document.chapters);
    console.log('Images:', document.images);
    console.log('Tables:', document.tables);
    console.log('Footnotes:', document.footnotes);
    console.log('Token Usage:', usage.total);
  },
  true, // cleanupAfterCallback
  {}, // PDFConvertOptions
);

// Cleanup
await pdfParser.dispose();

Advanced Usage

// Specify LLM models per component + fallback retry
const processor = new DocumentProcessor({
  logger,
  fallbackModel: anthropic('claude-opus-4-5'), // For retry on failure
  pageRangeParserModel: openai('gpt-5.2'),
  tocExtractorModel: openai('gpt-5.1'),
  validatorModel: openai('gpt-5.2'),
  visionTocExtractorModel: openai('gpt-5-mini'),
  captionParserModel: openai('gpt-5-nano'),
  textCleanerBatchSize: 20,
  captionParserBatchSize: 10,
  captionValidatorBatchSize: 10,
  maxRetries: 3,
  maxValidationRetries: 3,
  enableFallbackRetry: true, // Automatically retry with fallbackModel on failure (default: false)
  onTokenUsage: (report) => console.log('Token usage:', report.total),
});

Development Guidelines

• All tests must pass (pnpm test)
• 100% code coverage must be maintained
• ESLint and Prettier rules must be followed
• Commit messages must follow Conventional Commits

Demo Application

Online Demo

Try it without local installation:

🔗 https://engine-demo.heripo.org

The online demo has a daily usage limit (3 times). For full functionality, local execution is recommended.

Web Demo (Next.js)

A web application providing real-time PDF processing monitoring:

```bash cd apps/demo-web cp .env.example .env

Set LLM API keys in .env file

pnpm install pnpm dev ```

Access http://localhost:3000 in your browser

Key Features:

• PDF upload and processing option configuration
• Real-time processing status monitoring (SSE)
• Processing result visualization (TOC, images, tables)
• Job queue management

For detailed usage, see apps/demo-web/README.md.

Data Pipeline

Raw Data Extraction → Archaeological Data Ledger → Archaeological Data Standard → Domain Ontology → DB Storage

Current Implementation (v0.1.x):

• ✅ PDF parsing and OCR (Docling SDK)
• ✅ Document structure extraction (TOC, chapters/sections, page mapping)
• ✅ Image/table extraction and caption parsing

Planned Stages:

• 🔜 Immutable Ledger (universal archaeological model, concept extraction)
• 🔜 Extensible Standardization (hierarchical standard model, normalization)
• 🔜 Ontology (semantic model, knowledge graph)
• 🔜 Production Ready (performance optimization, API stability)

For a detailed roadmap, see docs/roadmap.md.

Packages

Package Documentation

• @heripo/pdf-parser
• @heripo/document-processor
• @heripo/model
• @heripo/logger

Test specific package

pnpm --filter @heripo/pdf-parser test pnpm --filter @heripo/document-processor test ```

Package-Specific Commands

```bash

Test specific package (with coverage)

pnpm --filter @heripo/pdf-parser test:coverage

Watch mode for specific package

pnpm --filter @heripo/pdf-parser dev ```