Struct TextFeatureSelector 

pub struct TextFeatureSelector<State = Untrained> { /* private fields */ }

Text feature selection using TF-IDF weights and linguistic analysis

This selector analyzes text features represented as term frequency matrices and applies various text-specific selection criteria:

• Document frequency filtering (min_df, max_df)
• TF-IDF scoring for term importance
• Chi-squared statistical tests with target variables
• N-gram analysis (when configured)
• Part-of-speech and syntactic features (when enabled)

Input Format

The input matrix X should be structured as:

• Rows: Documents
• Columns: Terms/features (e.g., from TF-IDF vectorization)
• Values: Term frequencies or TF-IDF scores

Examples

use sklears_feature_selection::domain_specific::text_features::TextFeatureSelector;
use sklears_core::traits::{Fit, Transform};
use scirs2_core::ndarray::{Array1, Array2};

let selector = TextFeatureSelector::new()
    .min_df(0.02)              // Minimum 2% document frequency
    .max_df(0.90)              // Maximum 90% document frequency
    .max_features(Some(500))   // Select top 500 features
    .ngram_range((1, 2))       // Include unigrams and bigrams
    .include_pos(true);        // Include part-of-speech features

let x = Array2::zeros((100, 1000)); // 100 documents, 1000 terms
let y = Array1::zeros(100);          // Document labels

let fitted_selector = selector.fit(&x, &y)?;
let transformed_x = fitted_selector.transform(&x)?;

Implementations

impl TextFeatureSelector<Untrained>

pub fn new() -> Self

pub fn min_df(self, min_df: f64) -> Self

Set the minimum document frequency threshold

Terms appearing in fewer than min_df fraction of documents will be filtered out. This helps remove very rare terms that may not be reliable predictors.

Arguments

• min_df - Fraction between 0.0 and 1.0

pub fn max_df(self, max_df: f64) -> Self

Set the maximum document frequency threshold

Terms appearing in more than max_df fraction of documents will be filtered out. This helps remove very common terms (like stop words) that may not be discriminative.

Arguments

• max_df - Fraction between 0.0 and 1.0

pub fn max_features(self, max_features: Option<usize>) -> Self

Set the maximum number of features to select

When set to Some(n), selects the top n features by combined score. When set to None, uses document frequency filtering only.

pub fn ngram_range(self, ngram_range: (usize, usize)) -> Self

Set the n-gram range for feature extraction

• (1, 1): Unigrams only
• (1, 2): Unigrams and bigrams
• (2, 3): Bigrams and trigrams
• etc.

Note: This parameter is informational for compatibility; actual n-gram extraction should be done during preprocessing.

pub fn include_pos(self, include_pos: bool) -> Self

Enable or disable part-of-speech features

When enabled, the selector will give preference to features that represent important part-of-speech categories.

Note: This requires preprocessing to extract POS features.

pub fn include_syntax(self, include_syntax: bool) -> Self

Enable or disable syntactic features

When enabled, the selector will consider syntactic relationships and dependency parsing features.

Note: This requires preprocessing to extract syntactic features.

impl TextFeatureSelector<Trained>

pub fn vocabulary(&self) -> &HashMap<String, usize>

Get the vocabulary mapping from terms to feature indices

Returns a reference to the vocabulary dictionary where keys are term names and values are their corresponding feature indices.

pub fn idf_scores(&self) -> &Array1<Float>

Get the IDF (Inverse Document Frequency) scores

Returns an array where each element is the IDF score for the corresponding selected feature.

pub fn feature_names(&self) -> &[String]

Get the names of selected features

Returns a reference to the vector of feature names that were selected.

pub fn selected_features(&self) -> &[usize]

Get the indices of selected features

Returns a reference to the vector of original feature indices that were selected during fitting.

pub fn n_features_selected(&self) -> usize

Get the number of selected features

pub fn feature_summary(&self) -> Vec<(usize, &str, Float)>

Get feature information as a structured summary

Returns a vector of tuples containing (feature_index, feature_name, idf_score) for all selected features, sorted by feature index.

Trait Implementations

impl<State: Clone> Clone for TextFeatureSelector<State>

fn clone(&self) -> TextFeatureSelector<State>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<State: Debug> Debug for TextFeatureSelector<State>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TextFeatureSelector<Untrained>

fn default() -> Self

Returns the “default value” for a type.

impl Estimator for TextFeatureSelector<Untrained>

type Config = ()

Configuration type for the estimator

type Error = SklearsError

Error type for the estimator

type Float = f64

The numeric type used by this estimator

fn config(&self) -> &Self::Config

Get estimator configuration

fn validate_config(&self) -> Result<(), SklearsError>

Validate estimator configuration with detailed error context

fn check_compatibility( &self, n_samples: usize, n_features: usize, ) -> Result<(), SklearsError>

Check if estimator is compatible with given data dimensions

fn metadata(&self) -> EstimatorMetadata

Get estimator metadata

impl Fit<ArrayBase<OwnedRepr<f64>, Dim<[usize; 2]>>, ArrayBase<OwnedRepr<f64>, Dim<[usize; 1]>>> for TextFeatureSelector<Untrained>

type Fitted = TextFeatureSelector<Trained>

The fitted model type

fn fit(self, x: &Array2<Float>, y: &Array1<Float>) -> SklResult<Self::Fitted>

Fit the model to the provided data with validation

fn fit_with_validation( self, x: &X, y: &Y, _x_val: Option<&X>, _y_val: Option<&Y>, ) -> Result<(Self::Fitted, FitMetrics), SklearsError>

where Self: Sized,

Fit with custom validation and early stopping

impl SelectorMixin for TextFeatureSelector<Trained>

fn get_support(&self) -> SklResult<Array1<bool>>

Get support mask

fn transform_features(&self, indices: &[usize]) -> SklResult<Vec<usize>>

Transform by selecting features

impl Transform<ArrayBase<OwnedRepr<f64>, Dim<[usize; 2]>>> for TextFeatureSelector<Trained>

fn transform(&self, x: &Array2<Float>) -> SklResult<Array2<Float>>

Transform the input data