pub struct CountVectorizerValidParams { /* private fields */ }
Expand description
Count vectorizer: learns a vocabulary from a sequence of documents (or file paths) and maps each vocabulary entry to an integer value, producing a CountVectorizer that can be used to count the occurrences of each vocabulary entry in any sequence of documents. Alternatively a user-specified vocabulary can be used for fitting.
Attributes
If a user-defined vocabulary is used for fitting then the following attributes will not be considered during the fitting phase but they will still be used by the CountVectorizer to transform any text to be examined.
split_regex
: the regex espression used to split decuments into tokens. Defaults to r“\b\w\w+\b“, which selects “words”, using whitespaces and punctuation symbols as separators.convert_to_lowercase
: if true, all documents used for fitting will be converted to lowercase. Defaults totrue
.n_gram_range
: if set to(1,1)
single tokens will be candidate vocabulary entries, if(2,2)
then adjacent token pairs will be considered, if(1,2)
then both single tokens and adjacent token pairs will be considered, and so on. The definition of token depends on the regex used fpr splitting the documents. The default value is(1,1)
.normalize
: if true, all charachters in the documents used for fitting will be normalized according to unicode’s NFKD normalization. Defaults totrue
.document_frequency
: specifies the minimum and maximum (relative) document frequencies that each vocabulary entry must satisfy. Defaults to(0., 1.)
(i.e. 0% minimum and 100% maximum)stopwords
: optional list of entries to be excluded from the generated vocabulary. Defaults toNone
Implementations§
source§impl CountVectorizerValidParams
impl CountVectorizerValidParams
sourcepub fn fit<T: ToString + Clone, D: Data<Elem = T>>(
&self,
x: &ArrayBase<D, Ix1>
) -> Result<CountVectorizer>
pub fn fit<T: ToString + Clone, D: Data<Elem = T>>( &self, x: &ArrayBase<D, Ix1> ) -> Result<CountVectorizer>
Learns a vocabulary from the documents in x
, according to the specified attributes and maps each
vocabulary entry to an integer value, producing a CountVectorizer.
Returns an error if:
- one of the
n_gram
boundaries is set to zero or the minimum value is greater than the maximum value - if the minimum document frequency is greater than one or than the maximum frequency, or if the maximum frequency is
smaller than zero - if the regex expression for the split is invalid
sourcepub fn fit_files<P: AsRef<Path>>(
&self,
input: &[P],
encoding: EncodingRef,
trap: DecoderTrap
) -> Result<CountVectorizer>
pub fn fit_files<P: AsRef<Path>>( &self, input: &[P], encoding: EncodingRef, trap: DecoderTrap ) -> Result<CountVectorizer>
Learns a vocabulary from the documents contained in the files in input
, according to the specified attributes and maps each
vocabulary entry to an integer value, producing a CountVectorizer.
The files will be read using the specified encoding
, and any sequence unrecognized by the encoding will be handled
according to trap
.
Returns an error if:
- one of the
n_gram
boundaries is set to zero or the minimum value is greater than the maximum value - if the minimum document frequency is greater than one or than the maximum frequency, or if the maximum frequency is
smaller than zero - if the regex expression for the split is invalid
- if one of the files couldn’t be opened
- if the trap is strict and an unrecognized sequence is encountered in one of the files
sourcepub fn fit_vocabulary<T: ToString>(
&self,
words: &[T]
) -> Result<CountVectorizer>
pub fn fit_vocabulary<T: ToString>( &self, words: &[T] ) -> Result<CountVectorizer>
Produces a CountVectorizer with the input vocabulary.
All struct attributes are ignored in the fitting but will be used by the CountVectorizer
to transform any text to be examined. As such this will return an error in the same cases as the fit
method.
Trait Implementations§
source§impl Clone for CountVectorizerValidParams
impl Clone for CountVectorizerValidParams
source§fn clone(&self) -> CountVectorizerValidParams
fn clone(&self) -> CountVectorizerValidParams
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read more