Skip to main content

ContentChunker

embedcache::chunking

Trait ContentChunker 

Source 
pub trait ContentChunker: Send + Sync {
    // Required methods
    fn chunk<'life0, 'life1, 'async_trait>(
        &'life0 self,
        content: &'life1 str,
        size: usize,
    ) -> Pin<Box<dyn Future<Output = Vec<String>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait;
    fn name(&self) -> &str;
}
Expand description

Trait for content chunking strategies

Implementors of this trait can provide custom chunking logic for text processing. The trait is async to allow for complex chunking strategies that might require external services or intensive computation.

Required Methods§

Source

fn chunk<'life0, 'life1, 'async_trait>( &'life0 self, content: &'life1 str, size: usize, ) -> Pin<Box<dyn Future<Output = Vec<String>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Chunk the given content into smaller pieces

§Arguments
  • content - The text content to chunk
  • size - The desired chunk size (implementation dependent)
§Returns

A vector of text chunks

Source

fn name(&self) -> &str

Get the name of this chunker for identification

§Returns

A string identifier for this chunker

Implementors§