Trait TextPreprocessor 

pub trait TextPreprocessor: Send + Sync {
    // Required method
    fn process(&self, text: &str) -> Option<String>;
}

Trait for pluggable text preprocessors.

Implement this trait to transform or filter section text before it is tokenized and chunked. The pipeline is sequential: the output of each stage feeds the next.

Implementing

use triplets_core::TextPreprocessor;

struct UppercasePreprocessor;

impl TextPreprocessor for UppercasePreprocessor {
    fn process(&self, text: &str) -> Option<String> {
        Some(text.to_uppercase())
    }
}

Required Methods

fn process(&self, text: &str) -> Option<String>

Process a text block.

Returns Some(transformed) with the (possibly modified) text, or None to signal that the section should be discarded entirely — no chunks will be produced from it.

Implementors

impl TextPreprocessor for DenoiserPreprocessor