rs-chunks

Fast, high-fidelity document chunking for RAG — a pure-Rust engine covering 36 file extensions across 17 format families (Office, OpenDocument, PDF, email, ebooks, notebooks, delimited text, and more).

rs-chunks is the standalone Rust edition of the py-chunks engine. It is validated to produce the same chunks as py-chunks across the entire fixture corpus (see Parity).

Supported formats

Usage

use chunks_rs::{get_chunks, get_markdown};

// Source-agnostic dispatch by extension. `mode` selects the strategy;
// "default" is each format's natural strategy.
let chunks = get_chunks("report.docx", "default", 3, 1, 3, 15)?;
for c in &chunks {
    println!("[{}] {}", c.content_type, c.content);
    // c.metadata is a serde_json::Value with format-specific provenance
}

// One-shot Markdown conversion.
let md = get_markdown("deck.pptx")?;

Every chunk is a Chunk { content: String, content_type: String, metadata: serde_json::Value }.

Chunking modes

Prose/binary document formats support: default, section, semantic, sentence, page_aware, sliding_window. Spreadsheets support: row (default), table, sheet, semantic, page_aware, sliding_window. Delimited text supports: row (default), sliding_window, page_aware.

Per-format APIs

Each family is also directly available under chunks_rs::formats::*, exposing chunk, chunk_with_options, stream (a native Iterator), to_markdown, and (where applicable) *_with_images entry points.

use chunks_rs::formats::csv;
let chunks = csv::chunk("data.csv", "row", 10, 5, 1, true, None, "utf-8", true)?;
for c in csv::stream("data.csv", "row", 10, 5, 1, true, None, "utf-8", true)? {
    let c = c?; // streaming yields Result<Chunk>
}

Parity

rs-chunks is checked against the py-chunks reference engine over every fixture × every mode (examples/parity_dump.rs + examples/parity_check.py):

• 2204 / 2214 comparisons identical (99.5%).
• All OOXML (Word/PowerPoint/Excel), legacy binary (.doc/.ppt), OpenDocument, email, ebook, and delimited families are byte-identical.
• The handful of differences are confined to semantic-mode primary_merge_reason, a tie-break the reference engine resolves via a randomized HashMap iteration order — i.e. py-chunks itself is non-deterministic there, and rs-chunks reproduces that behavior. The merge_reasons list (and all chunk content) is identical.

Image extraction is likewise validated against py-chunks:

• chunk_with_images vs get_chunks(list_images=True) — 1056 / 1056 identical (chunk counts, image counts, image names, and image-chunk contents), including the OOXML and legacy-binary (.doc/.ppt ODRAW) formats.
• to_markdown_with_images vs get_markdown(list_images=True) — 273 / 273 identical (markdown string + image set) across every image-bearing format.

Streaming (stream) yields the same chunks as chunk/get_chunks, matching py-chunks' stream_chunks (verified equal to its batch output).

Robustness: adversarial fixtures fail with a clean ChunkError and never panic or abort (panic-prone third-party parsers are wrapped).

License

MIT.