Struct FixedSizeChunker 

pub struct FixedSizeChunker { /* private fields */ }

Available on crate feature rag only.

Splits text into fixed-size chunks by byte length with configurable overlap.

Each chunk holds at most chunk_size bytes of UTF-8 and starts chunk_size - chunk_overlap bytes after the previous one, rounded down to a character boundary and always at least one character further on.

Chunk IDs are generated as {document_id}_{chunk_index}. Each chunk inherits the parent document’s metadata plus a chunk_index field.

Example

use adk_rag::FixedSizeChunker;

let chunker = FixedSizeChunker::try_new(256, 50)?;
let chunks = chunker.chunk(&document);

Implementations

impl FixedSizeChunker

pub fn new(chunk_size: usize, chunk_overlap: usize) -> FixedSizeChunker

Create a new FixedSizeChunker.

Arguments

• chunk_size — maximum number of bytes per chunk
• chunk_overlap — number of bytes shared by consecutive chunks

An invalid pair is normalised rather than rejected: a zero chunk_size becomes 1 and a chunk_overlap that is not smaller than chunk_size is ignored, with a warning logged. Use try_new to reject it.

pub fn try_new( chunk_size: usize, chunk_overlap: usize, ) -> Result<FixedSizeChunker, RagError>

Create a new FixedSizeChunker, rejecting an invalid size pair.

Arguments

• chunk_size — maximum number of bytes per chunk
• chunk_overlap — number of bytes shared by consecutive chunks

Errors

Returns RagError::ConfigError when chunk_size is zero or chunk_overlap is not smaller than chunk_size.

Example

use adk_rag::FixedSizeChunker;

assert!(FixedSizeChunker::try_new(512, 100).is_ok());
assert!(FixedSizeChunker::try_new(100, 100).is_err());

Trait Implementations

impl Chunker for FixedSizeChunker

fn chunk(&self, document: &Document) -> Vec<Chunk>

Split a document into chunks.

impl Clone for FixedSizeChunker

fn clone(&self) -> FixedSizeChunker

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FixedSizeChunker

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.