Struct TextSplitter 

pub struct TextSplitter<Sizer>
where
    Sizer: ChunkSizer,
{ /* private fields */ }

Default plain-text splitter. Recursively splits chunks into the largest semantic units that fit within the chunk size. Also will attempt to merge neighboring chunks if they can fit within the given chunk size.

Implementations

impl<Sizer> TextSplitter<Sizer>

where Sizer: ChunkSizer,

pub fn new(chunk_config: impl Into<ChunkConfig<Sizer>>) -> Self

Creates a new TextSplitter.

use text_splitter::TextSplitter;

// By default, the chunk sizer is based on characters.
let splitter = TextSplitter::new(512);

pub fn chunks<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, ) -> impl Iterator<Item = &'text str> + 'splitter

Generate a list of chunks from a given text. Each chunk will be up to the chunk_capacity.

Method

To preserve as much semantic meaning within a chunk as possible, each chunk is composed of the largest semantic units that can fit in the next given chunk. For each splitter type, there is a defined set of semantic levels. Here is an example of the steps used:

1. Split the text by a increasing semantic levels.
2. Check the first item for each level and select the highest level whose first item still fits within the chunk size.
3. Merge as many of these neighboring sections of this level or above into a chunk to maximize chunk length. Boundaries of higher semantic levels are always included when merging, so that the chunk doesn’t inadvertantly cross semantic boundaries.

The boundaries used to split the text if using the chunks method, in ascending order:

1. Characters
2. Unicode Grapheme Cluster Boundaries
3. Unicode Word Boundaries
4. Unicode Sentence Boundaries
5. Ascending sequence length of newlines. (Newline is \r\n, \n, or \r) Each unique length of consecutive newline sequences is treated as its own semantic level. So a sequence of 2 newlines is a higher level than a sequence of 1 newline, and so on.

Splitting doesn’t occur below the character level, otherwise you could get partial bytes of a char, which may not be a valid unicode str.

use text_splitter::TextSplitter;

let splitter = TextSplitter::new(10);
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunks(text).collect::<Vec<_>>();

assert_eq!(vec!["Some text", "from a", "document"], chunks);

pub fn chunk_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, ) -> impl Iterator<Item = (usize, &'text str)> + 'splitter

Returns an iterator over chunks of the text and their byte offsets. Each chunk will be up to the chunk_capacity.

See TextSplitter::chunks for more information.

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::new(10);
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunk_indices(text).collect::<Vec<_>>();

assert_eq!(vec![(0, "Some text"), (11, "from a"), (18, "document")], chunks);

pub fn chunk_char_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, ) -> impl Iterator<Item = ChunkCharIndex<'text>> + 'splitter

Returns an iterator over chunks of the text with their byte and character offsets. Each chunk will be up to the chunk_capacity.

See TextSplitter::chunks for more information.

This will be more expensive than just byte offsets, and for most usage in Rust, just having byte offsets is sufficient. But when interfacing with other languages or systems that require character offsets, this will track the character offsets for you, accounting for any trimming that may have occurred.

use text_splitter::{Characters, ChunkCharIndex, TextSplitter};

let splitter = TextSplitter::new(10);
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunk_char_indices(text).collect::<Vec<_>>();

assert_eq!(vec![ChunkCharIndex { chunk: "Some text", byte_offset: 0, char_offset: 0 }, ChunkCharIndex { chunk: "from a", byte_offset: 11, char_offset: 11 }, ChunkCharIndex { chunk: "document", byte_offset: 18, char_offset: 18 }], chunks);

Trait Implementations

impl<Sizer> Debug for TextSplitter<Sizer>

where Sizer: ChunkSizer + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.