Struct Preprocessor 

pub struct Preprocessor {
    pub normalize_unicode: bool,
    pub collapse_whitespace: bool,
    pub trim: bool,
    pub to_lowercase: bool,
    pub remove_stopwords: bool,
    pub stopwords: Vec<String>,
    pub max_length: usize,
}

Configuration for text preprocessing.

Fields

normalize_unicode: bool

Whether to normalize unicode to NFC form.

collapse_whitespace: bool

Whether to collapse whitespace to single spaces.

trim: bool

Whether to trim leading/trailing whitespace.

to_lowercase: bool

Whether to convert to lowercase.

remove_stopwords: bool

Whether to remove stop words.

stopwords: Vec<String>

Custom stop words (if empty, uses built-in default set).

max_length: usize

Maximum string length to process (0 = unlimited).

Implementations

impl Preprocessor

pub fn new() -> Self

Create a new preprocessor with default settings.

pub fn with_normalize_unicode(self, v: bool) -> Self

Set unicode normalization.

pub fn with_collapse_whitespace(self, v: bool) -> Self

Set whitespace collapsing.

pub fn with_trim(self, v: bool) -> Self

Set trimming.

pub fn with_lowercase(self, v: bool) -> Self

Set lowercase conversion.

pub fn with_remove_stopwords(self, v: bool) -> Self

Set stopword removal.

pub fn with_stopwords(self, words: Vec<String>) -> Self

Set custom stop words.

pub fn with_max_length(self, max: usize) -> Self

Set max input length.

pub fn process(&self, text: &str) -> String

Preprocess a text string.

Returns the processed string, or an error if processing fails.

Trait Implementations

impl Clone for Preprocessor

fn clone(&self) -> Preprocessor

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Preprocessor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Preprocessor

fn default() -> Self

Returns the “default value” for a type.