Module chunking 

Structs

Chunk

ChunkerConfig

Configuration used by the TextChunker, [DfsTextChunker], [LinearChunker], and [OverlapChunker] to build chunks. Instantiated by the TextChunker on each Separator and passed to the chunkers.

ChunkerResult

TextChunker

Splits text by paragraphs, newlines, sentences, spaces, and finally graphemes, and builds chunks from the splits that are within the desired token ranges.

Enums

ChunkError

An enumeration of possible errors that may occur during chunk operations.

Constants

DEFAULT_CHUNK_SIZE

Traits

Chunker

Functions

chunk_text

An easy alternative to the TextChunker struct.