pub struct TfIdfVectorizer { /* private fields */ }
Expand description
Simlar to CountVectorizer
but instead of
just counting the term frequency of each vocabulary entry in each given document,
it computes the term frequecy times the inverse document frequency, thus giving more importance
to entries that appear many times but only on some documents. The weight function can be adjusted
by setting the appropriate method. This struct provides the same string
processing customizations described in CountVectorizer
.
Implementations§
source§impl TfIdfVectorizer
impl TfIdfVectorizer
sourcepub fn convert_to_lowercase(self, convert_to_lowercase: bool) -> Self
pub fn convert_to_lowercase(self, convert_to_lowercase: bool) -> Self
If true, all documents used for fitting will be converted to lowercase.
sourcepub fn split_regex(self, regex_str: &str) -> Self
pub fn split_regex(self, regex_str: &str) -> Self
Sets the regex espression used to split decuments into tokens
sourcepub fn n_gram_range(self, min_n: usize, max_n: usize) -> Self
pub fn n_gram_range(self, min_n: usize, max_n: usize) -> Self
If set to (1,1)
single tokens will be candidate vocabulary entries, if (2,2)
then adjacent token pairs will be considered,
if (1,2)
then both single tokens and adjacent token pairs will be considered, and so on. The definition of token depends on the
regex used fpr splitting the documents.
min_n
should not be greater than max_n
sourcepub fn normalize(self, normalize: bool) -> Self
pub fn normalize(self, normalize: bool) -> Self
If true, all charachters in the documents used for fitting will be normalized according to unicode’s NFKD normalization.
sourcepub fn document_frequency(self, min_freq: f32, max_freq: f32) -> Self
pub fn document_frequency(self, min_freq: f32, max_freq: f32) -> Self
Specifies the minimum and maximum (relative) document frequencies that each vocabulary entry must satisfy.
min_freq
and max_freq
must lie in 0..=1
and min_freq
should not be greater than max_freq
sourcepub fn stopwords<T: ToString>(self, stopwords: &[T]) -> Self
pub fn stopwords<T: ToString>(self, stopwords: &[T]) -> Self
List of entries to be excluded from the generated vocabulary.
sourcepub fn fit<T: ToString + Clone, D: Data<Elem = T>>(
&self,
x: &ArrayBase<D, Ix1>
) -> Result<FittedTfIdfVectorizer>
pub fn fit<T: ToString + Clone, D: Data<Elem = T>>( &self, x: &ArrayBase<D, Ix1> ) -> Result<FittedTfIdfVectorizer>
Learns a vocabulary from the texts in x
, according to the specified attributes and maps each
vocabulary entry to an integer value, producing a FittedTfIdfVectorizer.
Returns an error if:
- one of the
n_gram
boundaries is set to zero or the minimum value is greater than the maximum value - if the minimum document frequency is greater than one or than the maximum frequency, or if the maximum frequecy is
smaller than zero
sourcepub fn fit_vocabulary<T: ToString>(
&self,
words: &[T]
) -> Result<FittedTfIdfVectorizer>
pub fn fit_vocabulary<T: ToString>( &self, words: &[T] ) -> Result<FittedTfIdfVectorizer>
Produces a FittedTfIdfVectorizer with the input vocabulary.
All struct attributes are ignored in the fitting but will be used by the FittedTfIdfVectorizer
to transform any text to be examined. As such this will return an error in the same cases as the fit
method.
pub fn fit_files<P: AsRef<Path>>( &self, input: &[P], encoding: EncodingRef, trap: DecoderTrap ) -> Result<FittedTfIdfVectorizer>
Trait Implementations§
source§impl Clone for TfIdfVectorizer
impl Clone for TfIdfVectorizer
source§fn clone(&self) -> TfIdfVectorizer
fn clone(&self) -> TfIdfVectorizer
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read more