Module analysis 

Importance-detection pipeline (tokenizer, lexicon, scoring). Pure computation, no I/O. Enabled by the analysis feature (default). Text analysis primitives for importance detection.

This module is pure computation - no I/O, no storage, no network. It provides tokenization, stopword filtering, n-gram extraction, and term count computation.

Re-exports

pub use classification::classify_passages;

pub use classification::ClassificationConfig;

pub use classification::ClassifiedPassage;

pub use classification::ImportanceCategory;

pub use classification::PassageContext;

pub use extraction::extract_passages;

pub use extraction::ExtractedPassage;

pub use extraction::ExtractionConfig;

pub use extraction::ExtractionEntry;

pub use frequency::term_counts;

pub use frequency::term_counts_with_ngrams;

pub use injection::adjust_weights;

pub use injection::scale_budget;

pub use injection::InjectionConfig;

pub use lexicon::Lexicons;

pub use ngrams::bigrams;

pub use ngrams::extract;

pub use ngrams::trigrams;

pub use prefilter::strip_execution_artifacts;

pub use prefilter::FilterToggle;

pub use prefilter::PrefilterConfig;

pub use recurrence::compute_recurrence;

pub use recurrence::RecurrenceConfig;

pub use recurrence::RecurrenceResult;

pub use scoring::pack_segments;

pub use scoring::score_passages;

pub use scoring::ImportanceSegment;

pub use scoring::ScoringConfig;

pub use tokenizer::Tokenizer;

pub use tokenizer::TokenizerConfig;

Modules

classification

Importance classification of passages into categories.

extraction

Extraction of importance segments from scored passages.

frequency

Term frequency and session term-map construction.

injection

Detection of prompt-injection-shaped content.

lexicon

Lexicon configuration for importance scoring.

ngrams

N-gram extraction over token sequences.

prefilter

Pre-filtering of passages before classification.

recurrence

Cross-session recurrence scoring.

scoring

Importance scoring of classified passages.

tokenizer

Text tokenization and stopword filtering.

Functions

build_session_term_maps

Build per-session term-count maps by pre-filtering, tokenizing, and computing n-gram term counts for each session’s entries.