Struct DocumentPreprocessor 

pub struct DocumentPreprocessor {
    pub clean_whitespace: bool,
    pub normalize_unicode: bool,
    pub detect_language: bool,
    pub chunk_size: Option<usize>,
}

Document preprocessor for cleaning and normalizing text.

Fields

clean_whitespace: bool

Normalize whitespace (collapse multiple spaces, normalize line breaks)

normalize_unicode: bool

Normalize Unicode (NFC normalization)

detect_language: bool

Detect and record language

chunk_size: Option<usize>

Maximum chunk size (None = no chunking)

Implementations

impl DocumentPreprocessor

pub fn new() -> Self

Create a new preprocessor with default settings.

pub fn with_all_cleaning() -> Self

Create a preprocessor with all cleaning enabled.

pub fn prepare(&self, text: &str) -> PreparedDocument

Prepare text for entity extraction.

Trait Implementations

impl Clone for DocumentPreprocessor

fn clone(&self) -> DocumentPreprocessor

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DocumentPreprocessor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DocumentPreprocessor

fn default() -> Self

Returns the “default value” for a type.