Skip to main content

smart_chunk

Function smart_chunk 

Source
pub fn smart_chunk(
    text: &str,
    cfg: &ChunkConfig,
    backend: &Arc<Backend>,
) -> Result<Vec<Chunk>, ChipError>
Expand description

Chunk text; offsets in the result refer to the normalized text. See smart_chunk_mapped for original-text spans and ChunkConfig for semantics.