Struct ConceptExtractor 

pub struct ConceptExtractor { /* private fields */ }

Extracts concepts and keywords from text using TF-IDF and frequency analysis.

The extractor maintains corpus-level document-frequency statistics so that IDF weights improve as more documents are processed.

Example

use ipfrs_semantic::concept_extractor::{ConceptExtractor, ExtractorConfig};

let config = ExtractorConfig::default();
let mut extractor = ConceptExtractor::new(config);

let concepts = extractor.extract("The quick brown fox jumps over the lazy dog.");
for c in &concepts {
    println!("{:?}: score={:.4}", c.term, c.score);
}

Implementations

impl ConceptExtractor

pub fn new(config: ExtractorConfig) -> Self

Creates a new extractor with the given configuration.

pub fn extract(&mut self, text: &str) -> Vec<Concept>

Extracts and returns the top concepts from text, updating internal corpus statistics so subsequent calls benefit from improved IDF weights.

pub fn tokenize(text: &str) -> Vec<String>

Tokenizes text into lower-cased tokens, splitting on whitespace and ASCII punctuation, dropping tokens shorter than min_term_length.

This is the public lower-cased variant used for TF/IDF computation.

pub fn compute_tf(tokens: &[String]) -> HashMap<String, u32>

Computes term frequency (raw count) for each token in tokens.

pub fn compute_tfidf(&self, term: &str, tf: u32, doc_len: usize) -> f64

Returns the TF-IDF score for a term given its raw frequency tf and the document length doc_len.

Uses augmented TF (0.5 + 0.5 × tf / max_tf) to prevent bias towards long documents, combined with smoothed IDF.

pub fn extract_ngrams(tokens: &[String], n: usize) -> Vec<String>

Extracts all n-grams of size n from tokens, joining tokens with a single space to form the phrase string.

pub fn detect_concept_type(term: &str) -> ConceptType

Classifies a term into its ConceptType.

• Entity — starts with an ASCII upper-case letter or every ASCII letter in the term is upper-case (acronym).
• Technical — contains _ (snake_case) or an interior upper-case letter (camelCase).
• Phrase — contains a space (multi-word).
• Keyword — everything else.

pub fn is_stop_word(&self, word: &str) -> bool

Returns true if word appears in the configured stop-word list (case-insensitive comparison).

pub fn update_corpus(&mut self, tokens: &[String])

Updates corpus document-frequency statistics for the unique terms found in tokens. Each unique term in the token slice counts as appearing in one additional document.

pub fn top_concepts(concepts: &mut Vec<Concept>, n: usize) -> Vec<Concept>

Returns the top n concepts from concepts sorted by descending score. Deduplicates by term (keeps the highest-scoring entry).

pub fn stats(&self) -> &ExtractorStats

Returns a reference to the cumulative extraction statistics.