Struct text_splitter::MarkdownSplitter
source · pub struct MarkdownSplitter<Sizer>where
Sizer: ChunkSizer,{ /* private fields */ }
markdown
only.Expand description
Markdown splitter. Recursively splits chunks into the largest semantic units that fit within the chunk size. Also will attempt to merge neighboring chunks if they can fit within the given chunk size.
Implementations§
source§impl<Sizer> MarkdownSplitter<Sizer>where
Sizer: ChunkSizer,
impl<Sizer> MarkdownSplitter<Sizer>where
Sizer: ChunkSizer,
sourcepub fn new(chunk_config: impl Into<ChunkConfig<Sizer>>) -> Self
pub fn new(chunk_config: impl Into<ChunkConfig<Sizer>>) -> Self
Creates a new MarkdownSplitter
.
use text_splitter::MarkdownSplitter;
// By default, the chunk sizer is based on characters.
let splitter = MarkdownSplitter::new(512);
sourcepub fn chunks<'splitter, 'text: 'splitter>(
&'splitter self,
text: &'text str
) -> impl Iterator<Item = &'text str> + 'splitter
pub fn chunks<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str ) -> impl Iterator<Item = &'text str> + 'splitter
Generate a list of chunks from a given text. Each chunk will be up to
the max_chunk_size
.
§Method
To preserve as much semantic meaning within a chunk as possible, each chunk is composed of the largest semantic units that can fit in the next given chunk. For each splitter type, there is a defined set of semantic levels. Here is an example of the steps used:
- Characters
- Unicode Grapheme Cluster Boundaries
- Unicode Word Boundaries
- Unicode Sentence Boundaries
- Soft line breaks (single newline) which isn’t necessarily a new element in Markdown.
- Inline elements such as: text nodes, emphasis, strong, strikethrough, link, image, table cells, inline code, footnote references, task list markers, and inline html.
- Block elements suce as: paragraphs, code blocks, footnote definitions, metadata. Also, a block quote or row/item within a table or list that can contain other “block” type elements, and a list or table that contains items.
- Thematic breaks or horizontal rules.
- Headings by level
Splitting doesn’t occur below the character level, otherwise you could get partial bytes of a char, which may not be a valid unicode str.
Markdown is parsed according to the Commonmark spec, along with some optional features such as GitHub Flavored Markdown.
use text_splitter::MarkdownSplitter;
let splitter = MarkdownSplitter::new(10);
let text = "# Header\n\nfrom a\ndocument";
let chunks = splitter.chunks(text).collect::<Vec<_>>();
assert_eq!(vec!["# Header", "from a", "document"], chunks);
sourcepub fn chunk_indices<'splitter, 'text: 'splitter>(
&'splitter self,
text: &'text str
) -> impl Iterator<Item = (usize, &'text str)> + 'splitter
pub fn chunk_indices<'splitter, 'text: 'splitter>( &'splitter self, text: &'text str ) -> impl Iterator<Item = (usize, &'text str)> + 'splitter
Returns an iterator over chunks of the text and their byte offsets.
Each chunk will be up to the max_chunk_size
.
See MarkdownSplitter::chunks
for more information.
use text_splitter::MarkdownSplitter;
let splitter = MarkdownSplitter::new(10);
let text = "# Header\n\nfrom a\ndocument";
let chunks = splitter.chunk_indices(text).collect::<Vec<_>>();
assert_eq!(vec![(0, "# Header"), (10, "from a"), (17, "document")], chunks);
Trait Implementations§
Auto Trait Implementations§
impl<Sizer> Freeze for MarkdownSplitter<Sizer>where
Sizer: Freeze,
impl<Sizer> RefUnwindSafe for MarkdownSplitter<Sizer>where
Sizer: RefUnwindSafe,
impl<Sizer> Send for MarkdownSplitter<Sizer>where
Sizer: Send,
impl<Sizer> Sync for MarkdownSplitter<Sizer>where
Sizer: Sync,
impl<Sizer> Unpin for MarkdownSplitter<Sizer>where
Sizer: Unpin,
impl<Sizer> UnwindSafe for MarkdownSplitter<Sizer>where
Sizer: UnwindSafe,
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<T> IntoEither for T
impl<T> IntoEither for T
source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read more