Struct ChunkText

pub struct ChunkText { /* private fields */ }

A transformer that chunks text content into smaller pieces.

The transformer will split the text content into smaller pieces based on the specified max_characters or range of characters.

For further customization, you can use the builder to create a custom splitter. Uses text_splitter under the hood.

Technically that might work with every splitter text_splitter provides.

Implementations

impl ChunkText

pub fn builder() -> ChunkTextBuilder

pub fn from_max_characters(max_characters: usize) -> Self

Create a new transformer with a maximum number of characters per chunk.

pub fn from_chunk_range(range: Range<usize>) -> Self

Create a new transformer with a range of characters per chunk.

Chunks smaller than the range will be ignored.

pub fn with_concurrency(self, concurrency: usize) -> Self

Set the number of concurrent chunks to process.

Trait Implementations

impl ChunkerTransformer for ChunkText

fn transform_node<'life0, 'async_trait>( &'life0 self, node: Node, ) -> Pin<Box<dyn Future<Output = IndexingStream> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

fn concurrency(&self) -> Option<usize>

Overrides the default concurrency of the pipeline

fn name(&self) -> &'static str

impl Clone for ChunkText

fn clone(&self) -> ChunkText

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkText

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChunkText

fn default() -> Self

Returns the “default value” for a type.