1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
//! Analyzer implementations that combine tokenizers and filters.
//!
//! This module provides complete text analysis pipelines that combine tokenizers
//! and token filters to process text for indexing and searching. Analyzers are
//! the main entry point for text analysis in Laurus.
//!
//! # Available Analyzers
//!
//! - [`standard::StandardAnalyzer`] - General-purpose analyzer with whitespace tokenization
//! - [`simple::SimpleAnalyzer`] - Simple lowercase + letter tokenization
//! - [`keyword::KeywordAnalyzer`] - Treats entire input as single token (for IDs, tags)
//! - [`noop::NoOpAnalyzer`] - No-op analyzer for testing
//! - [`pipeline::PipelineAnalyzer`] - Customizable analyzer with filter chain
//! - [`per_field::PerFieldAnalyzer`] - Different analyzers per field
//! - [`language`] - Language-specific analyzers (English, Japanese, etc.)
//!
//! # Architecture
//!
//! ```text
//! Text → Tokenizer → Token Filters → Analyzed Tokens
//! ```
//!
//! # Examples
//!
//! ```
//! use laurus::analysis::analyzer::analyzer::Analyzer;
//! use laurus::analysis::analyzer::standard::StandardAnalyzer;
//!
//! let analyzer = StandardAnalyzer::new().unwrap();
//! let tokens: Vec<_> = analyzer.analyze("Hello World!").unwrap().collect();
//!
//! // Tokens: ["hello", "world"]
//! assert_eq!(tokens.len(), 2);
//! ```