Struct BatchProcessor 

pub struct BatchProcessor { /* private fields */ }

Processes multiple documents through an NlpPipeline in parallel using the scirs2-core parallel abstractions.

Example

use scirs2_text::pipeline::{BatchProcessor, PipelineBuilder, PipelineStep};

let pipeline = PipelineBuilder::new()
    .add_step(PipelineStep::Tokenize)
    .add_step(PipelineStep::Lowercase)
    .add_step(PipelineStep::RemoveStopwords)
    .build();

let docs = vec![
    "The quick brown fox",
    "A lazy dog sleeps",
    "Hello world",
];
let processor = BatchProcessor::new(pipeline);
let results = processor.process_batch(&docs).unwrap();
assert_eq!(results.len(), 3);

Implementations

impl BatchProcessor

pub fn new(pipeline: NlpPipeline) -> Self

Create a new BatchProcessor wrapping the given pipeline.

pub fn with_parallel_threshold(self, threshold: usize) -> Self

Set the minimum number of documents required before parallel processing is used. Defaults to 32.

pub fn process_batch(&self, documents: &[&str]) -> Result<Vec<Vec<String>>>

Process a slice of text documents and return one token list per document.

pub fn process_batch_tolerant( &self, documents: &[&str], ) -> Vec<Result<Vec<String>, TextError>>

Process documents and also return any per-document errors instead of short-circuiting on the first failure.

pub fn pipeline(&self) -> &NlpPipeline

Return a reference to the inner pipeline.