Struct SemanticChunker 

pub struct SemanticChunker { /* private fields */ }

Semantic chunker that respects sentence and paragraph boundaries.

This chunker produces more coherent chunks by avoiding splits in the middle of sentences or words. It uses Unicode segmentation rules for proper international text handling.

Examples

use rlm_rs::chunking::{Chunker, SemanticChunker};

let chunker = SemanticChunker::new();
let text = "Hello, world! This is a test. Another sentence here.";
let chunks = chunker.chunk(1, text, None).unwrap();

Implementations

impl SemanticChunker

pub const fn new() -> Self

Creates a new semantic chunker with default settings.

pub const fn with_size(chunk_size: usize) -> Self

Creates a semantic chunker with custom chunk size and no overlap.

pub const fn with_size_and_overlap(chunk_size: usize, overlap: usize) -> Self

Creates a semantic chunker with custom size and overlap.

pub const fn min_chunk_size(self, size: usize) -> Self

Sets the minimum chunk size.

Trait Implementations

impl Chunker for SemanticChunker

fn chunk( &self, buffer_id: i64, text: &str, metadata: Option<&ChunkMetadata>, ) -> Result<Vec<Chunk>>

Chunks the input text into segments.

fn name(&self) -> &'static str

Returns the name of the chunking strategy.

fn supports_parallel(&self) -> bool

Returns whether this chunker supports parallel processing.

fn description(&self) -> &'static str

Returns a description of the chunking strategy.

fn validate(&self, metadata: Option<&ChunkMetadata>) -> Result<()>

Validates configuration before chunking.

impl Clone for SemanticChunker

fn clone(&self) -> SemanticChunker

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SemanticChunker

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SemanticChunker

fn default() -> Self

Returns the “default value” for a type.