laurus 0.3.1

Unified search library for lexical, vector, and semantic retrieval
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

//! Language-specific analyzers.
//!
//! This module provides analyzers optimized for specific languages, each with
//! appropriate tokenization and stop word filtering.
//!
//! # Available Languages
//!
//! - [`english`] - English text analysis with regex tokenization and English stop words
//! - [`japanese`] - Japanese text analysis with Lindera morphological analyzer
//!
//! # Examples
//!
//! ```
//! use laurus::analysis::analyzer::analyzer::Analyzer;
//! use laurus::analysis::analyzer::language::english::EnglishAnalyzer;
//!
//! let analyzer = EnglishAnalyzer::new().unwrap();
//! let tokens: Vec<_> = analyzer.analyze("Hello the world").unwrap().collect();
//!
//! // "the" is filtered as a stop word
//! assert_eq!(tokens.len(), 2);
//! ```

pub mod english;
pub mod japanese;