Module concept_extractor 

Concept and Keyword Extraction

Provides TF-IDF-based concept and keyword extraction from text, supporting n-gram phrases, named entity detection, technical term identification, and corpus-level IDF scoring for multi-document analysis.

Overview

• TF-IDF scoring with configurable smoothing
• N-gram phrase extraction (bigrams, trigrams, etc.)
• Entity detection (capitalized / all-caps terms)
• Technical term detection (camelCase, snake_case)
• Stop-word filtering with configurable word list
• Multi-document corpus statistics for accurate IDF

Structs

Concept

A single extracted concept with scoring metadata.

ConceptExtractor

Extracts concepts and keywords from text using TF-IDF and frequency analysis.

ExtractorConfig

Configuration for ConceptExtractor.

ExtractorStats

Cumulative statistics collected by ConceptExtractor.

Enums

ConceptType

Classifies the semantic role of an extracted concept.