Struct TfIdfVectorizer 

pub struct TfIdfVectorizer {
    pub min_df: usize,
    pub max_df: f64,
    pub max_features: Option<usize>,
    pub ngram_range: (usize, usize),
    pub char_level: bool,
    pub case_sensitive: bool,
    /* private fields */
}

TF-IDF (Term Frequency-Inverse Document Frequency) preprocessor

TF-IDF is a numerical statistic that reflects how important a word is to a document in a collection of documents. It increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.

TF-IDF(t,d,D) = TF(t,d) × IDF(t,D)

where:

• TF(t,d) = (Number of times term t appears in document d) / (Total number of terms in d)
• IDF(t,D) = log(Total number of documents / Number of documents containing term t)

Fields

min_df: usize

Minimum document frequency (ignore terms that appear in fewer documents)

max_df: f64

Maximum document frequency (ignore terms that appear in more documents)

max_features: Option<usize>

Maximum number of features

ngram_range: (usize, usize)

N-gram range (min_n, max_n)

char_level: bool

Whether to use character-level n-grams

case_sensitive: bool

Case sensitivity

Implementations

impl TfIdfVectorizer

pub fn new() -> Self

Create a new TF-IDF vectorizer

pub fn with_min_df(self, min_df: usize) -> Self

Set minimum document frequency

pub fn with_max_df(self, max_df: f64) -> Self

Set maximum document frequency

pub fn with_max_features(self, max_features: Option<usize>) -> Self

Set maximum features

pub fn with_ngram_range(self, min_n: usize, max_n: usize) -> Self

Set n-gram range

pub fn with_char_level(self, char_level: bool) -> Self

Set character level

pub fn with_case_sensitive(self, case_sensitive: bool) -> Self

Set case sensitivity

pub fn fit(&mut self, documents: &[String]) -> Result<()>

Fit the vectorizer on a corpus

pub fn transform(&self, documents: &[String]) -> Result<Array2<f64>>

Transform documents to TF-IDF matrix

pub fn fit_transform(&mut self, documents: &[String]) -> Result<Array2<f64>>

Fit and transform in one step

pub fn get_feature_names(&self) -> Result<Vec<String>>

Get feature names

Trait Implementations

impl Clone for TfIdfVectorizer

fn clone(&self) -> TfIdfVectorizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TfIdfVectorizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TfIdfVectorizer

fn default() -> Self

Returns the “default value” for a type.