Struct ExtractTextChunksPipeline 

pub struct ExtractTextChunksPipeline { /* private fields */ }

The extract text chunks pipeline.

This pipeline handles the first two stages of cognify:

1. Document classification (text/* only)
2. Text chunking

Implementations

impl ExtractTextChunksPipeline

pub fn new(storage: Arc<dyn StorageTrait>) -> Self

pub async fn extract_chunks( &self, data_items: Vec<Data>, max_chunk_size: usize, ) -> Result<Vec<DocumentChunk>, ChunkingError>

Extract text chunks from a set of Data items.

Implements:

1. Document classification (text/* only)
2. Text chunking

Returns the generated chunks.

pub async fn extract_chunks_with_counter<C: TokenCounter>( &self, data_items: Vec<Data>, max_chunk_size: usize, counter: &C, ) -> Result<Vec<DocumentChunk>, ChunkingError>

Extract text chunks with a custom token counter.