Struct text_splitter::MarkdownSplitter

pub struct MarkdownSplitter<S>
where
    S: ChunkSizer,
{ /* private fields */ }

Available on crate feature markdown only.

Markdown splitter. Recursively splits chunks into the largest semantic units that fit within the chunk size. Also will attempt to merge neighboring chunks if they can fit within the given chunk size.

Implementations

impl<S> MarkdownSplitter<S>

where S: ChunkSizer,

pub fn new(chunk_sizer: S) -> Self

Creates a new MarkdownSplitter.

use text_splitter::{Characters, MarkdownSplitter};

// Characters is the default, so you can also do `MarkdownSplitter::default()`
let splitter = MarkdownSplitter::new(Characters);

pub fn with_trim_chunks(self, trim_chunks: bool) -> Self

Specify whether chunks should have whitespace trimmed from the beginning and end or not.

If false (default), joining all chunks should return the original string. If true, all chunks will have whitespace removed from beginning and end. Indentation however will be preserved if the chunk also includes multiple lines. Extra newlines are always removed, but if the text would include multiple indented list items, the indentation of the first element will also be preserved.

use text_splitter::{Characters, MarkdownSplitter};

let splitter = MarkdownSplitter::default().with_trim_chunks(true);

pub fn chunks<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, chunk_capacity: impl ChunkCapacity + 'splitter ) -> impl Iterator<Item = &'text str> + 'splitter

Generate a list of chunks from a given text. Each chunk will be up to the max_chunk_size.

Method

To preserve as much semantic meaning within a chunk as possible, each chunk is composed of the largest semantic units that can fit in the next given chunk. For each splitter type, there is a defined set of semantic levels. Here is an example of the steps used:

1. Characters
2. Unicode Grapheme Cluster Boundaries
3. Unicode Word Boundaries
4. Unicode Sentence Boundaries
5. Soft line breaks (single newline) which isn’t necessarily a new element in Markdown.
6. Inline elements such as: text nodes, emphasis, strong, strikethrough, link, image, table cells, inline code, footnote references, task list markers, and inline html.
7. Block elements suce as: paragraphs, code blocks, and footnote definitions.
8. Container blocks such as: table rows, block quotes, list items, and HTML blocks.
9. Meta containers such as: lists and tables.
10. Thematic breaks or horizontal rules.
11. Headings by level
12. Metadata at the beginning of the document

Splitting doesn’t occur below the character level, otherwise you could get partial bytes of a char, which may not be a valid unicode str.

Markdown is parsed according to the Commonmark spec, along with some optional features such as GitHub Flavored Markdown.

use text_splitter::{Characters, MarkdownSplitter};

let splitter = MarkdownSplitter::default();
let text = "# Header\n\nfrom a\ndocument";
let chunks = splitter.chunks(text, 10).collect::<Vec<_>>();

assert_eq!(vec!["# Header\n\n", "from a\n", "document"], chunks);

pub fn chunk_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str, chunk_capacity: impl ChunkCapacity + 'splitter ) -> impl Iterator<Item = (usize, &'text str)> + 'splitter

Returns an iterator over chunks of the text and their byte offsets. Each chunk will be up to the max_chunk_size.

See MarkdownSplitter::chunks for more information.

use text_splitter::{Characters, MarkdownSplitter};

let splitter = MarkdownSplitter::default();
let text = "# Header\n\nfrom a\ndocument";
let chunks = splitter.chunk_indices(text, 10).collect::<Vec<_>>();

assert_eq!(vec![(0, "# Header\n\n"), (10, "from a\n"), (17, "document")], chunks);

Trait Implementations

impl<S> Debug for MarkdownSplitter<S>

where S: ChunkSizer + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for MarkdownSplitter<Characters>

fn default() -> Self

Returns the “default value” for a type.