rs-chunks

Part of chunk-engine — one Rust engine, three byte-identical SDKs (py-chunks · js-chunks · rs-chunks). Full documentation, playground and benchmarks: chunkengine.dev

The chunk-engine reference engine — a pure-Rust library that turns any of 36 file extensions (17 format families) into typed, structure-aware chunks for RAG. No ML, no external services, no separate parser step.

py-chunks and js-chunks are bindings over this crate; it is the source of truth for all chunking behavior.

Install

cargo add rs-chunks

The library import name is chunks_rs.

Quick start

use chunks_rs::{get_chunks, get_chunks_from_bytes, get_markdown};

// get_chunks(path, mode, window_size, overlap, sentences_per_chunk, paragraphs_per_page)
let chunks = get_chunks("report.docx", "semantic", 3, 1, 3, 15)?;

for c in &chunks {
    println!("[{}] {}", c.content_type, c.content);
    // c.metadata is a serde_json::Value with format-specific provenance
}

// From bytes — the filename drives dispatch by extension
let chunks = get_chunks_from_bytes(&bytes, "report.docx", "default", 3, 1, 3, 15)?;

// One-shot Markdown conversion
let md = get_markdown("deck.pptx")?;

Every chunk is Chunk { content: String, content_type: String, metadata: serde_json::Value }.

📖 Chunking modes · Supported formats · Output schema · Metadata reference

Per-format APIs

Each format family is available under chunks_rs::formats::*, exposing chunk, chunk_with_options, stream (a native Iterator), to_markdown, and — where applicable — *_with_images and *_from_bytes entry points. Use these when you need format-specific parameters the dispatcher doesn't expose:

use chunks_rs::formats::{csv, pptx};

let chunks = csv::chunk("data.csv", "row", 10, 5, 1, true, None, "utf-8", true)?;

for c in csv::stream("data.csv", "row", 10, 5, 1, true, None, "utf-8", true)? {
    let c = c?;   // streaming yields Result<Chunk>
}

// (chunks, images) — images are (name, bytes) pairs
let (chunks, images) = pptx::chunk_with_images("deck.pptx", "default", 3, 1, 3, 15)?;

Features

PDF parsing is always compiled in — it is pure Rust and builds for wasm32. The default pdf-native feature adds only page rasterisation (via the liteparse crate / PDFium), the fallback used when a scanned PDF has no embedded page image to return. Disable default features for wasm32:

rs-chunks = { version = "0.6", default-features = false }

PDFs still parse without it; a text-less one reports that it has no text rather than returning page renders.

Parity

Validated against the py-chunks reference implementation over every fixture × every mode (examples/parity_dump.rs + examples/parity_check.py):

• 2204 / 2214 chunk comparisons byte-identical (99.5%)
• 1056 / 1056 image extractions identical
• 273 / 273 markdown conversions identical

All OOXML, legacy binary (.doc/.ppt), OpenDocument, email, ebook and delimited families are byte-identical. The remaining differences are confined to semantic-mode primary_merge_reason, a tie-break the reference engine resolves via randomized HashMap iteration order.

Streaming (stream) yields the same chunks as chunk. Adversarial inputs fail with a clean ChunkError and never panic — panic-prone third-party parsers are wrapped.

Develop

cargo test
cargo run --release --example parity_dump    # parity harness

License

MIT