Skip to main content

Crate keyword_extraction_pipeline

Crate keyword_extraction_pipeline 

Source
Expand description

Deterministic tier-1 keyword and identifier extraction.

Raw file in, ranked keywords out. The crate is corpus-blind: every score derives from the document itself plus pinned static resources. There is no document-frequency table, no IDF, and no cross-document state anywhere, so a document’s keyword set is a pure function of (bytes, config, resources) — summarised by PipelineVersion.

Corpus-relative ranking, storage, triage scoring and blurb scheduling all belong to the consuming system.

§What this crate cannot tell you

Nothing measured here is evidence that keyword-based shortlisting reduces blurb spend. That is an extrinsic property of the consuming system, measurable only with real documents and a real query matcher. This crate is evaluated intrinsically: did it extract the right things, reproducibly.

Re-exports§

pub use config::Config;
pub use error::ExtractError;
pub use resources::Resources;
pub use types::DocumentResult;
pub use types::DocumentStatus;
pub use types::FormatHint;
pub use types::Keyword;
pub use types::Kind;
pub use types::Language;
pub use types::Origin;
pub use version::PipelineVersion;

Modules§

canonical
Canonicalisation: raw text to the single canonical form everything else indexes into.
config
Tunable parameters. Every field feeds crate::PipelineVersion in a fixed order, so changing any of them changes the version stamp and invalidates cached keyword sets.
error
language
Deterministic language detection.
parse
Format parsing: raw bytes to raw text.
prose
The deterministic prose gate.
resources
Static resources: the general English wordlist, stopword list and lemma exception list.
stages
Extraction stages.
tokenize
Tokenisation over canonical text.
types
version
The pipeline version stamp.

Functions§

canonicalise
Regenerate the canonical text a result’s offsets refer to.
extract
Extract keywords from a document.