Skip to main content

chunk_text

alith_core::chunking

Function chunk_text 

Source 
pub fn chunk_text(
    text: &str,
    max_chunk_token_size: u32,
    overlap_percent: Option<f32>,
) -> Result<Option<Vec<String>>>
Expand description

An easy alternative to the TextChunker struct.

  • text - The natural language text to chunk.
  • max_chunk_token_size - The maxium token sized to be chunked to. Inclusive.
  • overlap_percent - The percentage of overlap between chunks. Default is None.