Struct ChunkMarkdown

pub struct ChunkMarkdown { /* private fields */ }

A transformer that chunks markdown content into smaller pieces.

The transformer will split the markdown content into smaller pieces based on the specified max_characters or range of characters.

For further customization, you can use the builder to create a custom splitter.

Technically that might work with every splitter text_splitter provides.

Implementations

impl ChunkMarkdown

pub fn builder() -> ChunkMarkdownBuilder

pub fn from_max_characters(max_characters: usize) -> Self

Create a new transformer with a maximum number of characters per chunk.

pub fn from_chunk_range(range: Range<usize>) -> Self

Create a new transformer with a range of characters per chunk.

Chunks smaller than the range will be ignored.

pub fn with_concurrency(self, concurrency: usize) -> Self

Set the number of concurrent chunks to process.

Trait Implementations

impl ChunkerTransformer for ChunkMarkdown

fn transform_node<'life0, 'async_trait>( &'life0 self, node: Node, ) -> Pin<Box<dyn Future<Output = IndexingStream> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

fn concurrency(&self) -> Option<usize>

Overrides the default concurrency of the pipeline

fn name(&self) -> &'static str

impl Clone for ChunkMarkdown

fn clone(&self) -> ChunkMarkdown

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkMarkdown

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChunkMarkdown

fn default() -> Self

Returns the “default value” for a type.