Struct text_splitter::TextSplitter

pub struct TextSplitter<Sizer>
where
    Sizer: ChunkSizer,
{ /* private fields */ }

Default plain-text splitter. Recursively splits chunks into the largest semantic units that fit within the chunk size. Also will attempt to merge neighboring chunks if they can fit within the given chunk size.

Implementations

impl<Sizer> TextSplitter<Sizer>

where Sizer: ChunkSizer,

pub fn new(chunk_config: impl Into<ChunkConfig<Sizer>>) -> Self

Creates a new TextSplitter.

use text_splitter::TextSplitter;

// By default, the chunk sizer is based on characters.
let splitter = TextSplitter::new(512);

pub fn chunks<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str ) -> impl Iterator<Item = &'text str> + 'splitter

Generate a list of chunks from a given text. Each chunk will be up to the chunk_capacity.

Method

To preserve as much semantic meaning within a chunk as possible, each chunk is composed of the largest semantic units that can fit in the next given chunk. For each splitter type, there is a defined set of semantic levels. Here is an example of the steps used:

use text_splitter::TextSplitter;

let splitter = TextSplitter::new(10);
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunks(text).collect::<Vec<_>>();

assert_eq!(vec!["Some text", "from a", "document"], chunks);

pub fn chunk_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str ) -> impl Iterator<Item = (usize, &'text str)> + 'splitter

Returns an iterator over chunks of the text and their byte offsets. Each chunk will be up to the chunk_capacity.

See TextSplitter::chunks for more information.

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::new(10);
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunk_indices(text).collect::<Vec<_>>();

assert_eq!(vec![(0, "Some text"), (11, "from a"), (18, "document")], chunks);

Trait Implementations

impl<Sizer> Debug for TextSplitter<Sizer>

where Sizer: ChunkSizer + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.