pdfmuse-core 0.1.6

Deterministic PDF/DOCX parser for RAG / LLMs — one Rust core, with Python, Node & WASM bindings that produce byte-identical output.

pdfmuse is a precision pre-layer for AI/RAG: it extracts everything a file actually contains — text with exact coordinates, fonts, vector rules, tables, links — fast, robustly, and identically across every binding. It stops cleanly at the ML boundary: OCR and visual layout inference are left to a pluggable backend, so the core stays deterministic with zero ML dependencies. It is not another probabilistic vision model.

Why pdfmuse

Performance

Two things matter for a RAG pre-layer: how fast, and whether it keeps the content.

Per-document latency — median over 200 runs, a 1-page 242 KB résumé, Apple Silicon:

For the text path use to_text() / to_markdown() — they return a string straight from the Rust core, so Python and Node keep that ~1.3 ms speed (~4× PyMuPDF). parse() returns the full IR (chars + coordinates), which adds host-side deserialization if you consume it as objects.

Across 22 real-world PDFs (resumes, reports, invoices; median of 7 runs, core-to-core, each returning a string):

Content is preserved (median 100% non-whitespace character coverage vs PyMuPDF). Numbers are hardware-dependent — reproduce with benches/ (python benches/compare.py) and eyeball fidelity with examples/visual_check.py.

Install

# Rust
cargo add pdfmuse-core
# Python (abi3 wheels)
pip install pdfmuse
# Node
npm install @pdfmuse/node   # native binding
# WASM (browser)
npm install @pdfmuse/core   # or build: wasm-pack build crates/pdfmuse-wasm --target web

Usage

CLI (debug/inspection):

pdfmuse parse report.pdf --format md      # structured Markdown (headings, tables)
pdfmuse parse report.pdf --format json    # full IR (chars, bboxes, blocks, warnings)

Rust:

let data = std::fs::read("report.pdf")?;
let doc = pdfmuse_core::parse(&data, None)?;                 // auto-detect PDF/DOCX
for page in &doc.pages {
    for ch in &page.chars { /* ch.text, ch.bbox {x0,y0,x1,y1}, ch.size */ }
}
let md = pdfmuse_core::to_markdown(&doc);
let chunks = pdfmuse_core::chunk(&doc);                      // RAG chunks + {page, bbox, heading_path}

Python:

import pdfmuse
data = open("report.pdf", "rb").read()
text = pdfmuse.to_text(data)         # plain text — fast path (~1.3ms, no full-IR json.loads)
md = pdfmuse.to_markdown(data)       # structured Markdown (headings, tables)
doc = pdfmuse.parse(data)            # full IR: doc.pages[i].chars/blocks with bboxes

Node:

const { toText, toMarkdown, parse } = require("@pdfmuse/node");
const data = fs.readFileSync("report.pdf");
const text = toText(data);           // plain text — fast path
const doc = parse(data);             // full IR (typed Document)

WASM (browser — digital PDFs; scanned pages return a NeedsOcr warning to hand off server-side):

import init, { to_text, parse } from "@pdfmuse/core";
await init();
const text = to_text(new Uint8Array(bytes));         // plain text
const doc = JSON.parse(parse(new Uint8Array(bytes))); // full IR

Integrations

• LangChain — langchain-pdfmuse: a PdfmuseLoader with single / page / elements modes. In elements mode each chunk carries section-aware metadata (heading_path, bbox, category) — reproducible chunks for RAG.

  from langchain_pdfmuse import PdfmuseLoader
  docs = PdfmuseLoader("report.pdf", mode="elements").load()
  

• LlamaIndex — llama-index-readers-pdfmuse: a PdfmuseReader with the same modes and section-aware metadata.

  from llama_index.readers.pdfmuse import PdfmuseReader
  docs = PdfmuseReader(mode="elements").load_data("report.pdf")
  

Scope boundary

In the core (deterministic): text + coordinates/font/size/color · vector rules & rects · line/paragraph/column clustering · ruled & whitespace-aligned table reconstruction · full DOCX structure · JSON / Markdown / RAG-chunk output.

Out of the core (pluggable VisionBackend): scanned-page OCR · borderless-table structure recognition · heading/body/caption classification. Text-less (scanned/image) pages are flagged NeedsOcr and left for a backend — see docs/adr/0001-pdf-engine-strategy.md.

Guarding this boundary is what keeps pdfmuse fast, stable, and distinct from vision models.

Layout

crates/
  pdfmuse-core/     pure-Rust core: PDF/DOCX → unified IR (parser, tokenizer, layout, output)
  pdfmuse-python/   PyO3 (abi3) binding
  pdfmuse-node/     napi-rs binding
  pdfmuse-wasm/     wasm-bindgen binding
  pdfmuse-cli/      debug CLI (`pdfmuse`)
tests/{corpus,snapshots}   golden corpus + insta snapshots
tests/parity/              cross-binding byte-identical gate (Python == Node == WASM)
examples/visual_check.py   render original ↔ coordinate reconstruction for QA
fuzz/                      cargo-fuzz targets (never-panic)

Testing gates

• Snapshot tests (insta + tests/corpus)
• Cross-binding parity CI — Python/Node/WASM output byte-identical (a red gate blocks merge)
• Robustness — mutated/garbage input never panics (tests/robustness.rs + fuzz/)
• CJK correctness suite

Status

Core is feature-complete (milestones M0–M4 + real-world hardening M4.5): PDF + DOCX → unified IR → JSON / Markdown / RAG chunks, three byte-identical bindings, encryption, CJK. Currently in M5 · polish & release. Roadmap and tasks live in Linear (project pdfmuse).

License

Dual-licensed under MIT or Apache-2.0, at your option.