language-text-analysis 0.1.3

Small, deterministic multilingual text-analysis pipeline for search and document processing
Documentation

language-text-analysis

Crates.io Documentation CI License: MIT OR Apache-2.0 Downloads

A small, deterministic multilingual text-analysis pipeline for search indexes and document-processing applications.

The pipeline is:

normalize → tokenize → stop-word removal → optional stemming

It provides one analysis path for both indexing and querying, which keeps the terms stored in an index comparable with the terms produced for a search query.

Behavior

  • Unicode lowercasing and accent stripping (Zürich becomes zurich).
  • Unicode word segmentation, including character-level segmentation for CJK ideographs.
  • Optional default stop-word removal for English and German.
  • Optional Snowball stemming for English, German, French, Spanish, Italian, Dutch, Portuguese, Swedish, Norwegian, Danish, Finnish, and Russian.
  • BCP-47 language-tag selection with a configurable fallback language.
  • Repeated terms remain repeated, preserving term frequency for callers that score results.
use language_text_analysis::{
    Analyzer,
    Language,
};

let analyzer = Analyzer::builder()
    .default_language(Language::English)
    .stemming(true)
    .build();

let terms = analyzer.analyze("Zürich is running", Some("en-US"));
assert_eq!(
    terms.into_iter().map(|term| term.into_inner()).collect::<Vec<_>>(),
    ["zurich", "run"],
);

Analyzer::new() provides English defaults, removes the bundled English stop-word list, and leaves stemming disabled. Applications that need maximum recall can select StopWordPolicy::None.

The analyzer returns a Vec<Term> for one input string. It does not own an index, ranking algorithm, corpus, or application-specific query language.

License

Copyright © 2026 DataRoad Inc, Delaware, USA, trading as Legra.

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.