Expand description
Deterministic tier-1 keyword and identifier extraction.
Raw file in, ranked keywords out. The crate is corpus-blind: every score derives
from the document itself plus pinned static resources. There is no document-frequency
table, no IDF, and no cross-document state anywhere, so a document’s keyword set is a
pure function of (bytes, config, resources) — summarised by PipelineVersion.
Corpus-relative ranking, storage, triage scoring and blurb scheduling all belong to the consuming system.
§What this crate cannot tell you
Nothing measured here is evidence that keyword-based shortlisting reduces blurb spend. That is an extrinsic property of the consuming system, measurable only with real documents and a real query matcher. This crate is evaluated intrinsically: did it extract the right things, reproducibly.
Re-exports§
pub use config::Config;pub use error::ExtractError;pub use resources::Resources;pub use types::DocumentResult;pub use types::DocumentStatus;pub use types::FormatHint;pub use types::Keyword;pub use types::Kind;pub use types::Language;pub use types::Origin;pub use version::PipelineVersion;
Modules§
- canonical
- Canonicalisation: raw text to the single canonical form everything else indexes into.
- config
- Tunable parameters. Every field feeds
crate::PipelineVersionin a fixed order, so changing any of them changes the version stamp and invalidates cached keyword sets. - error
- language
- Deterministic language detection.
- parse
- Format parsing: raw bytes to raw text.
- prose
- The deterministic prose gate.
- resources
- Static resources: the general English wordlist, stopword list and lemma exception list.
- stages
- Extraction stages.
- tokenize
- Tokenisation over canonical text.
- types
- version
- The pipeline version stamp.
Functions§
- canonicalise
- Regenerate the canonical text a result’s offsets refer to.
- extract
- Extract keywords from a document.