Struct TextPreprocessingStep 

pub struct TextPreprocessingStep { /* private fields */ }

Built-in text preprocessing step for content normalization

Applies a sequence of text transformations to clean and normalize content before further processing. Supports common operations like whitespace normalization, case conversion, and special character handling.

Supported Operations

• Whitespace Normalization: Collapse multiple spaces into single spaces
• Case Conversion: Convert text to lowercase for consistency
• Special Character Removal: Remove non-alphanumeric characters
• Regex Replacement: Custom pattern-based text replacement

Example

use rrag::prelude::*;

let step = TextPreprocessingStep::new(vec![
    TextOperation::NormalizeWhitespace,
    TextOperation::RemoveSpecialChars,
    TextOperation::ToLowercase,
]);

// Can also be built fluently
let step = TextPreprocessingStep::new(vec![])
    .with_operation(TextOperation::NormalizeWhitespace)
    .with_operation(TextOperation::RegexReplace {
        pattern: r"\d+".to_string(),
        replacement: "[NUMBER]".to_string(),
    });

Performance

• Operations are applied in sequence for predictable results
• String allocations are minimized where possible
• Regex operations are compiled once and reused
• Supports batch processing for multiple documents

Implementations

impl TextPreprocessingStep

pub fn new(operations: Vec<TextOperation>) -> Self

Create a new text preprocessing step with specified operations

Trait Implementations

impl PipelineStep for TextPreprocessingStep

fn name(&self) -> &str

Step name/identifier

fn description(&self) -> &str

Step description

fn input_types(&self) -> Vec<&'static str>

Input data types this step accepts

fn output_type(&self) -> &'static str

Output data type this step produces

fn execute<'life0, 'async_trait>( &'life0 self, context: PipelineContext, ) -> Pin<Box<dyn Future<Output = RragResult<PipelineContext>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Execute the step

fn validate_input(&self, _data: &PipelineData) -> RragResult<()>

Validate input data

fn is_parallelizable(&self) -> bool

Whether this step can run in parallel with others

fn dependencies(&self) -> Vec<&str>

Dependencies on other steps (step names)