rs-chunks 0.6.4

Fast, high-fidelity document chunking for RAG — a pure-Rust engine covering 36 file formats (Office, OpenDocument, PDF, email, ebooks, notebooks, and more).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

//! # chunks-rs
//!
//! Fast, high-fidelity document chunking for RAG pipelines — a pure-Rust engine
//! covering 36 file formats (Office, OpenDocument, PDF, email, ebooks,
//! notebooks, delimited text, and more).
//!
//! Every format returns the same [`Chunk`] shape (`content`, `content_type`,
//! `metadata`), supports multiple chunking strategies ([`ChunkMode`]), and can
//! stream large documents chunk-by-chunk.
//!
//! ```no_run
//! use chunks_rs::formats::csv;
//! let chunks = csv::chunk("data.csv", "row", 10, 5, 1, true, None, "utf-8", true).unwrap();
//! for c in &chunks {
//!     println!("[{}] {}", c.content_type, c.content);
//! }
//! ```

// Clippy 1.98 introduced `chunks_exact_to_as_chunks`, flagging nine
// long-standing call sites the day CI first ran. `as_chunks` needs a newer
// MSRV than this crate promises, and chasing every new nightly lint under
// `-D warnings` is churn, not correctness. Revisit when the MSRV moves.
#![allow(clippy::chunks_exact_to_as_chunks)]

pub mod chunk;
pub mod dispatch;
pub mod error;
pub mod formats;
pub mod options;

pub(crate) mod entities;
pub(crate) mod image_naming;
pub(crate) mod shared;
pub(crate) mod text_encoding;

pub use chunk::{Chunk, ChunksWithImages, ExtractedImage, ExtractedImages, MarkdownWithImages};
pub use dispatch::{
    get_chunks, get_chunks_from_bytes, get_chunks_with_images_from_bytes, get_markdown,
    get_markdown_from_bytes, get_markdown_with_images_from_bytes,
};
pub use error::{ChunkError, Result};
pub use options::{ChunkMode, ChunkOptions};