Struct text_splitter::MarkdownSplitter

pub struct MarkdownSplitter<Sizer>
where
    Sizer: ChunkSizer,
{ /* private fields */ }

Available on crate feature markdown only.

Markdown splitter. Recursively splits chunks into the largest semantic units that fit within the chunk size. Also will attempt to merge neighboring chunks if they can fit within the given chunk size.

Implementations

impl<Sizer> MarkdownSplitter<Sizer>

where Sizer: ChunkSizer,

pub fn new(chunk_config: impl Into<ChunkConfig<Sizer>>) -> Self

Creates a new MarkdownSplitter.

use text_splitter::MarkdownSplitter;

// By default, the chunk sizer is based on characters.
let splitter = MarkdownSplitter::new(512);

pub fn chunks<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str ) -> impl Iterator<Item = &'text str> + 'splitter

Generate a list of chunks from a given text. Each chunk will be up to the max_chunk_size.

Method

To preserve as much semantic meaning within a chunk as possible, each chunk is composed of the largest semantic units that can fit in the next given chunk. For each splitter type, there is a defined set of semantic levels. Here is an example of the steps used:

1. Characters
2. Unicode Grapheme Cluster Boundaries
3. Unicode Word Boundaries
4. Unicode Sentence Boundaries
5. Soft line breaks (single newline) which isn’t necessarily a new element in Markdown.
6. Inline elements such as: text nodes, emphasis, strong, strikethrough, link, image, table cells, inline code, footnote references, task list markers, and inline html.
7. Block elements suce as: paragraphs, code blocks, footnote definitions, metadata. Also, a block quote or row/item within a table or list that can contain other “block” type elements, and a list or table that contains items.
8. Thematic breaks or horizontal rules.
9. Headings by level

Splitting doesn’t occur below the character level, otherwise you could get partial bytes of a char, which may not be a valid unicode str.

Markdown is parsed according to the Commonmark spec, along with some optional features such as GitHub Flavored Markdown.

use text_splitter::MarkdownSplitter;

let splitter = MarkdownSplitter::new(10);
let text = "# Header\n\nfrom a\ndocument";
let chunks = splitter.chunks(text).collect::<Vec<_>>();

assert_eq!(vec!["# Header", "from a", "document"], chunks);

pub fn chunk_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str ) -> impl Iterator<Item = (usize, &'text str)> + 'splitter

Returns an iterator over chunks of the text and their byte offsets. Each chunk will be up to the max_chunk_size.

See MarkdownSplitter::chunks for more information.

use text_splitter::MarkdownSplitter;

let splitter = MarkdownSplitter::new(10);
let text = "# Header\n\nfrom a\ndocument";
let chunks = splitter.chunk_indices(text).collect::<Vec<_>>();

assert_eq!(vec![(0, "# Header"), (10, "from a"), (17, "document")], chunks);

Trait Implementations

impl<Sizer> Debug for MarkdownSplitter<Sizer>

where Sizer: ChunkSizer + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.