Struct TextChunker 

pub struct TextChunker {
    pub chunk_size: usize,
    pub overlap: usize,
    pub by_sentence: bool,
}

Sliding-window text chunker.

Example

use scirs2_text::segmentation::TextChunker;

let chunker = TextChunker::new(10, 2);
let chunks = chunker.chunk("Rust is fast. Rust is safe. Rust is fun.");
assert!(!chunks.is_empty());

Fields

chunk_size: usize

Number of tokens (words) per chunk.

overlap: usize

Number of tokens of overlap between consecutive chunks.

by_sentence: bool

If true, try to respect sentence boundaries.

Implementations

impl TextChunker

pub fn new(chunk_size: usize, overlap: usize) -> Self

Create a new chunker.

pub fn with_sentence_boundaries(self) -> Self

Enable sentence-boundary-respecting mode.

pub fn chunk(&self, text: &str) -> Vec<String>

Chunk text and return plain String chunks.

pub fn chunk_with_metadata(&self, text: &str) -> Vec<TextChunk>

Chunk text and return TextChunk structs with metadata.

Trait Implementations

impl Default for TextChunker

fn default() -> Self

Returns the “default value” for a type.