Struct ParallelChunker 

pub struct ParallelChunker<C: Chunker + Clone> { /* private fields */ }

Parallel chunking orchestrator.

Wraps another chunker and uses rayon for parallel processing. Useful for very large texts where chunking itself is CPU-bound.

Examples

use rlm_rs::chunking::{Chunker, ParallelChunker, SemanticChunker};

let chunker = ParallelChunker::new(SemanticChunker::new());
let text = "Hello, world! ".repeat(1000);
let chunks = chunker.chunk(1, &text, None).unwrap();

Implementations

impl<C: Chunker + Clone> ParallelChunker<C>

pub fn new(inner: C) -> Self

Creates a new parallel chunker wrapping the given chunker.

Arguments

• inner - The chunker to use for actual text processing.

pub const fn min_parallel_size(self, size: usize) -> Self

Sets the minimum text size for parallel processing.

Texts smaller than this will be processed sequentially.

pub fn num_segments(self, n: usize) -> Self

Sets the number of parallel segments.

Trait Implementations

impl<C: Chunker + Clone + Send + Sync> Chunker for ParallelChunker<C>

fn chunk( &self, buffer_id: i64, text: &str, metadata: Option<&ChunkMetadata>, ) -> Result<Vec<Chunk>>

Chunks the input text into segments.

fn name(&self) -> &'static str

Returns the name of the chunking strategy.

fn supports_parallel(&self) -> bool

Returns whether this chunker supports parallel processing.

fn description(&self) -> &'static str

Returns a description of the chunking strategy.

fn validate(&self, metadata: Option<&ChunkMetadata>) -> Result<()>

Validates configuration before chunking.

impl<C: Clone + Chunker + Clone> Clone for ParallelChunker<C>

fn clone(&self) -> ParallelChunker<C>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<C: Debug + Chunker + Clone> Debug for ParallelChunker<C>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.