Struct CountVectorizer 

pub struct CountVectorizer {
    pub max_features: Option<usize>,
    pub min_df: usize,
    pub max_df: f64,
    pub binary: bool,
    pub lowercase: bool,
}

An unfitted count vectorizer.

Tokenizes documents by splitting on non-alphanumeric boundaries, builds a vocabulary sorted alphabetically, and transforms documents into a term-count matrix.

Examples

use ferrolearn_preprocess::count_vectorizer::{CountVectorizer, FittedCountVectorizer};

let docs = vec![
    "the cat sat".to_string(),
    "the cat sat on the mat".to_string(),
];
let cv = CountVectorizer::new();
let fitted = cv.fit(&docs).unwrap();
let counts = fitted.transform(&docs).unwrap();
assert_eq!(counts.nrows(), 2);
assert_eq!(counts.ncols(), fitted.vocabulary().len());

Fields

max_features: Option<usize>

Maximum number of features (vocabulary size). None means no limit.

min_df: usize

Minimum document frequency (absolute count) for a term to be included.

max_df: f64

Maximum document frequency as a fraction of total documents. Terms appearing in more than max_df * n_docs documents are excluded.

binary: bool

If true, all counts are clipped to 0/1 (binary occurrence).

lowercase: bool

If true, lowercase all tokens before counting.

Implementations

impl CountVectorizer

pub fn new() -> Self

Create a new CountVectorizer with default settings.

pub fn max_features(self, n: usize) -> Self

Set the maximum number of features.

pub fn min_df(self, min_df: usize) -> Self

Set the minimum document frequency.

pub fn max_df(self, max_df: f64) -> Self

Set the maximum document frequency as a fraction of total documents.

pub fn binary(self, binary: bool) -> Self

Enable or disable binary mode.

pub fn lowercase(self, lowercase: bool) -> Self

Enable or disable lowercasing.

pub fn fit(&self, docs: &[String]) -> Result<FittedCountVectorizer, FerroError>

Fit the vectorizer on a corpus of documents.

Errors

Returns FerroError::InsufficientSamples if the corpus is empty. Returns FerroError::InvalidParameter if max_df is not in (0, 1].

Trait Implementations

impl Clone for CountVectorizer

fn clone(&self) -> CountVectorizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CountVectorizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CountVectorizer

fn default() -> Self

Returns the “default value” for a type.