rs-chunks

Part of chunk-engine — one Rust engine, three byte-identical SDKs (py-chunks · js-chunks · rs-chunks). Full documentation, playground and benchmarks: chunkengine.dev

The chunk-engine reference engine — a pure-Rust library that turns any of 36 file extensions (17 format families) into typed, structure-aware chunks for RAG. No ML, no external services, no separate parser step.

py-chunks and js-chunks are bindings over this crate; it is the source of truth for all chunking behavior.

Install

cargo add rs-chunks

The library import name is chunks_rs.

Quick start

use chunks_rs::{get_chunks, get_chunks_from_bytes, get_markdown};

// get_chunks(path, mode, window_size, overlap, sentences_per_chunk, paragraphs_per_page)
let chunks = get_chunks("report.docx", "semantic", 3, 1, 3, 15)?;

for c in &chunks {
    println!("[{}] {}", c.content_type, c.content);
    // c.metadata is a serde_json::Value with format-specific provenance
}

// From bytes — the filename drives dispatch by extension
let chunks = get_chunks_from_bytes(&bytes, "report.docx", "default", 3, 1, 3, 15)?;

// One-shot Markdown conversion
let md = get_markdown("deck.pptx")?;

Every chunk is Chunk { content: String, content_type: String, metadata: serde_json::Value }.

📖 Chunking modes · Supported formats · Output schema · Metadata reference

Per-format APIs

Each format family is available under chunks_rs::formats::*, exposing chunk, chunk_with_options, stream (a native Iterator), to_markdown, and — where applicable — *_with_images and *_from_bytes entry points. Use these when you need format-specific parameters the dispatcher doesn't expose:

use chunks_rs::formats::{csv, pptx};

let chunks = csv::chunk("data.csv", "row", 10, 5, 1, true, None, "utf-8", true)?;

for c in csv::stream("data.csv", "row", 10, 5, 1, true, None, "utf-8", true)? {
    let c = c?;   // streaming yields Result<Chunk>
}

// (chunks, images) — images are (name, bytes) pairs
let (chunks, images) = pptx::chunk_with_images("deck.pptx", "default", 3, 1, 3, 15)?;

Features

PDF parsing is always compiled in — it is pure Rust and builds for wasm32. The default pdf-native feature adds only page rasterisation (via the liteparse crate / PDFium), the fallback used when a scanned PDF has no embedded page image to return. Disable default features for wasm32:

rs-chunks = { version = "0.6", default-features = false }

PDFs still parse without it; a text-less one reports that it has no text rather than returning page renders.

Parity

Validated against the py-chunks reference implementation over every fixture × every mode (examples/parity_dump.rs + examples/parity_check.py):

• 4,748 / 4,748 chunk comparisons byte-identical (100%) — last re-verified 2026-09-10

Every family — OOXML, legacy binary (.doc/.ppt), OpenDocument, email, ebook, PDF, notebook and delimited — is byte-identical, including semantic-mode primary_merge_reason: all engines share the same deterministic tie-break (sort by count descending, then key ascending), so there is no residual nondeterminism. The same harness family also checks image extraction (images_dump.rs) and markdown conversion (md_images_dump.rs) byte-for-byte.

Streaming (stream) yields the same chunks as chunk. Every dispatch entry point (get_chunks, get_markdown, and the _from_bytes / _with_images variants) runs the parse behind a catch_unwind boundary, so on a native target a panic in a third-party parser surfaces as ChunkError::Parse rather than unwinding into the caller.

Two limits on that, stated because the guarantee was previously written without them. wasm32-unknown-unknown builds with panic-strategy: abort, so on the WASM target catch_unwind cannot intercept anything and a panic aborts. And a stack overflow or an allocation failure is not a panic on any target — those abort the process, which is why the recursive walkers carry explicit depth caps and the allocating paths carry explicit size caps rather than relying on the boundary.

Develop

cargo test
cargo run --release --example parity_dump    # parity harness

License

MIT