Crate language_text_analysis

Expand description

§language-text-analysis

A small, deterministic multilingual text-analysis pipeline for search indexes and document-processing applications.

The pipeline is:

normalize → tokenize → stop-word removal → optional stemming

It provides one analysis path for both indexing and querying, which keeps the terms stored in an index comparable with the terms produced for a search query.

§Behavior

Unicode lowercasing and accent stripping (Zürich becomes zurich).
Unicode word segmentation, including character-level segmentation for CJK ideographs.
Optional default stop-word removal for English and German.
Optional Snowball stemming for English, German, French, Spanish, Italian, Dutch, Portuguese, Swedish, Norwegian, Danish, Finnish, and Russian.
BCP-47 language-tag selection with a configurable fallback language.
Repeated terms remain repeated, preserving term frequency for callers that score results.

use language_text_analysis::{
    Analyzer,
    Language,
};

let analyzer = Analyzer::builder()
    .default_language(Language::English)
    .stemming(true)
    .build();

let terms = analyzer.analyze("Zürich is running", Some("en-US"));
assert_eq!(
    terms.into_iter().map(|term| term.into_inner()).collect::<Vec<_>>(),
    ["zurich", "run"],
);

Analyzer::new() provides English defaults, removes the bundled English stop-word list, and leaves stemming disabled. Applications that need maximum recall can select StopWordPolicy::None.

The analyzer returns a Vec<Term> for one input string. It does not own an index, ranking algorithm, corpus, or application-specific query language.

§License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE)
MIT license (LICENSE-MIT)

at your option.

§Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Structs§

Analyzer: The text-analysis pipeline: normalize → tokenize → stop-word removal → optional stemming.
AnalyzerBuilder: Builder for an Analyzer with a non-default configuration.
Term: A single normalized search term — the output unit of the analysis pipeline and the key under which the full-text index stores postings.

Enums§

Language: A language the analyzer can stem and stop-word-filter for.
StopWordPolicy: How the analyzer treats stop words.