Struct text_splitter::TextSplitter

pub struct TextSplitter<S>
where
    S: ChunkSizer,
{ /* private fields */ }

Default plain-text splitter. Recursively splits chunks into the largest semantic units that fit within the chunk size. Also will attempt to merge neighboring chunks if they can fit within the given chunk size.

Implementations

impl<S> TextSplitter<S>

where S: ChunkSizer,

pub fn new(chunk_sizer: S) -> Self

Creates a new TextSplitter.

use text_splitter::{Characters, TextSplitter};

// Characters is the default, so you can also do `TextSplitter::default()`
let splitter = TextSplitter::new(Characters);

pub fn with_trim_chunks(self, trim_chunks: bool) -> Self

Specify whether chunks should have whitespace trimmed from the beginning and end or not.

If false (default), joining all chunks should return the original string. If true, all chunks will have whitespace removed from beginning and end.

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::default().with_trim_chunks(true);

pub fn chunks<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, chunk_capacity: impl ChunkCapacity + 'splitter ) -> impl Iterator<Item = &'text str> + 'splitter

Generate a list of chunks from a given text. Each chunk will be up to the chunk_capacity.

Method

To preserve as much semantic meaning within a chunk as possible, each chunk is composed of the largest semantic units that can fit in the next given chunk. For each splitter type, there is a defined set of semantic levels. Here is an example of the steps used:

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::default();
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunks(text, 10).collect::<Vec<_>>();

assert_eq!(vec!["Some text", "\n\n", "from a\n", "document"], chunks);

pub fn chunk_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, chunk_capacity: impl ChunkCapacity + 'splitter ) -> impl Iterator<Item = (usize, &'text str)> + 'splitter

Returns an iterator over chunks of the text and their byte offsets. Each chunk will be up to the chunk_capacity.

See TextSplitter::chunks for more information.

use text_splitter::{Characters, TextSplitter};

let splitter = TextSplitter::default();
let text = "Some text\n\nfrom a\ndocument";
let chunks = splitter.chunk_indices(text, 10).collect::<Vec<_>>();

assert_eq!(vec![(0, "Some text"), (9, "\n\n"), (11, "from a\n"), (18, "document")], chunks);

Trait Implementations

impl<S> Debug for TextSplitter<S>

where S: ChunkSizer + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TextSplitter<Characters>

fn default() -> Self

Returns the “default value” for a type.