Module linfa_preprocessing::tf_idf_vectorization

Term frequency - inverse document frequency vectorization methods

Structs

FittedTfIdfVectorizer

Counts the occurrences of each vocabulary entry, learned during fitting, in a sequence of texts and scales them by the inverse document document frequency defined by the method. Each vocabulary entry is mapped to an integer value that is used to index the count in the result.

TfIdfVectorizer

Simlar to CountVectorizer but instead of just counting the term frequency of each vocabulary entry in each given document, it computes the term frequecy times the inverse document frequency, thus giving more importance to entries that appear many times but only on some documents. The weight function can be adjusted by setting the appropriate method. This struct provides the same string processing customizations described in CountVectorizer.

Enums

TfIdfMethod

Methods for computing the inverse document frequency of a vocabulary entry