Struct text_splitter::TextSplitter

pub struct TextSplitter<S>where
    S: ChunkSizer,{ /* private fields */ }

Default plain-text splitter. Recursively splits chunks into the largest semantic units that fit within the chunk size. Also will attempt to merge neighboring chunks if they can fit within the given chunk size.

Implementations

impl<S> TextSplitter<S>where S: ChunkSizer,

pub fn new(chunk_sizer: S) -> Self

Creates a new TextSplitter.

use text_splitter::{Characters, TextSplitter};

// Characters is the default, so you can also do `TextSplitter::default()`
let splitter = TextSplitter::new(Characters);

pub fn with_trim_chunks(self, trim_chunks: bool) -> Self

Specify whether chunks should have whitespace trimmed from the beginning and end or not.

If false (default), joining all chunks should return the original string. If true, all chunks will have whitespace removed from beginning and end.

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::default().with_trim_chunks(true);

pub fn chunks<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, chunk_capacity: impl ChunkCapacity + 'splitter ) -> impl Iterator<Item = &'text str> + 'splitter

Generate a list of chunks from a given text. Each chunk will be up to the chunk_capacity.

Method

To preserve as much semantic meaning within a chunk as possible, a recursive approach is used, starting at larger semantic units and, if that is too large, breaking it up into the next largest unit. Here is an example of the steps used:

1. Split the text by a given level
2. For each section, does it fit within the chunk size? a. Yes. Merge as many of these neighboring sections into a chunk as possible to maximize chunk length. b. No. Split by the next level and repeat.

The boundaries used to split the text if using the top-level split method, in descending length:

1. Descending sequence length of newlines. (Newline is \r\n, \n, or \r) Each unique length of consecutive newline sequences is treated as its own semantic level.
2. Unicode Sentence Boundaries
3. Unicode Word Boundaries
4. Unicode Grapheme Cluster Boundaries
5. Characters

Splitting doesn’t occur below the character level, otherwise you could get partial bytes of a char, which may not be a valid unicode str.

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::default();
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunks(text, 10).collect::<Vec<_>>();

assert_eq!(vec!["Some text", "\n\n", "from a\n", "document"], chunks);

pub fn chunk_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, chunk_capacity: impl ChunkCapacity + 'splitter ) -> impl Iterator<Item = (usize, &'text str)> + 'splitter

Returns an iterator over chunks of the text and their byte offsets. Each chunk will be up to the chunk_capacity.

See TextSplitter::chunks for more information.

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::default();
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunk_indices(text, 10).collect::<Vec<_>>();

assert_eq!(vec![(0, "Some text"), (9, "\n\n"), (11, "from a\n"), (18, "document")], chunks);

Trait Implementations

impl<S> Debug for TextSplitter<S>where S: ChunkSizer + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TextSplitter<Characters>

fn default() -> Self

Returns the “default value” for a type.