Struct linfa_preprocessing::CountVectorizerParams

pub struct CountVectorizerParams(_);

Implementations

impl CountVectorizerParams

pub fn convert_to_lowercase(self, convert_to_lowercase: bool) -> Self

If true, all documents used for fitting will be converted to lowercase.

pub fn split_regex(self, regex_str: &str) -> Self

Sets the regex espression used to split decuments into tokens

pub fn n_gram_range(self, min_n: usize, max_n: usize) -> Self

If set to (1,1) single tokens will be candidate vocabulary entries, if (2,2) then adjacent token pairs will be considered, if (1,2) then both single tokens and adjacent token pairs will be considered, and so on. The definition of token depends on the regex used fpr splitting the documents.

min_n should not be greater than max_n

pub fn normalize(self, normalize: bool) -> Self

If true, all charachters in the documents used for fitting will be normalized according to unicode’s NFKD normalization.

pub fn document_frequency(self, min_freq: f32, max_freq: f32) -> Self

Specifies the minimum and maximum (relative) document frequencies that each vocabulary entry must satisfy. min_freq and max_freq must lie in [0;1] and min_freq should not be greater than max_freq

pub fn stopwords<T: ToString>(self, stopwords: &[T]) -> Self

List of entries to be excluded from the generated vocabulary.

impl CountVectorizerParams

pub fn fit<T: ToString + Clone, D: Data<Elem = T>>(
    &self,
    x: &ArrayBase<D, Ix1>
) -> Result<CountVectorizer>

Learns a vocabulary from the documents in x, according to the specified attributes and maps each vocabulary entry to an integer value, producing a CountVectorizer.

Returns an error if:

• one of the n_gram boundaries is set to zero or the minimum value is greater than the maximum value
• if the minimum document frequency is greater than one or than the maximum frequency, or if the maximum frequency is
  smaller than zero
• if the regex expression for the split is invalid

pub fn fit_files<P: AsRef<Path>>(
    &self,
    input: &[P],
    encoding: EncodingRef,
    trap: DecoderTrap
) -> Result<CountVectorizer>

Learns a vocabulary from the documents contained in the files in input, according to the specified attributes and maps each vocabulary entry to an integer value, producing a CountVectorizer.

The files will be read using the specified encoding, and any sequence unrecognized by the encoding will be handled according to trap.

Returns an error if:

• one of the n_gram boundaries is set to zero or the minimum value is greater than the maximum value
• if the minimum document frequency is greater than one or than the maximum frequency, or if the maximum frequency is
  smaller than zero
• if the regex expression for the split is invalid
• if one of the files couldn’t be opened
• if the trap is strict and an unrecognized sequence is encountered in one of the files

pub fn fit_vocabulary<T: ToString>(
    &self,
    words: &[T]
) -> Result<CountVectorizer>

Produces a CountVectorizer with the input vocabulary. All struct attributes are ignored in the fitting but will be used by the CountVectorizer to transform any text to be examined. As such this will return an error in the same cases as the fit method.

Trait Implementations

impl Clone for CountVectorizerParams

fn clone(&self) -> CountVectorizerParams

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CountVectorizerParams

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CountVectorizerParams

fn default() -> Self

Returns the “default value” for a type.

impl ParamGuard for CountVectorizerParams

type Checked = CountVectorizerValidParams

The checked hyperparameters

type Error = PreprocessingError

Error type resulting from failed hyperparameter checking

fn check_ref(&self) -> Result<&Self::Checked, Self::Error>

Checks the hyperparameters and returns a reference to the checked hyperparameters if successful

fn check(self) -> Result<Self::Checked, Self::Error>

Checks the hyperparameters and returns the checked hyperparameters if successful

fn check_unwrap(self) -> Self::Checked

Calls check() and unwraps the result