Skip to main content

Analyzer

laurus::analysis::analyzer::analyzer

Trait Analyzer 

Source 
pub trait Analyzer:
    Send
    + Sync
    + Debug {
    // Required methods
    fn analyze(&self, text: &str) -> Result<TokenStream>;
    fn name(&self) -> &str;
    fn as_any(&self) -> &dyn Any;
}
Expand description

Trait for analyzers that convert text into processed tokens.

This is the core trait that all analyzers must implement. Analyzers are responsible for the complete text processing pipeline, from raw text to indexed tokens.

§Thread Safety

The trait requires Send + Sync to allow analyzers to be used safely across thread boundaries, which is essential for concurrent indexing.

§Trait Methods

  • analyze - Process text into tokens
  • name - Get analyzer identifier
  • as_any - Enable downcasting to concrete types

Required Methods§

Source

fn analyze(&self, text: &str) -> Result<TokenStream>

Analyze the given text and return a stream of tokens.

This is the main method that performs the complete analysis pipeline, including tokenization and all configured filters.

§Arguments
  • text - The raw input text to analyze
§Returns

A TokenStream (boxed iterator of tokens) that can be consumed by the indexer, or an error if analysis fails.

§Examples
ⓘ
use laurus::analysis::analyzer::analyzer::Analyzer;
use laurus::analysis::analyzer::standard::StandardAnalyzer;

let analyzer = StandardAnalyzer::new().unwrap();
let tokens: Vec<_> = analyzer.analyze("The quick brown fox").await.unwrap().collect();

// "The" is removed as a stop word, others are lowercased
assert_eq!(tokens.len(), 3);
assert_eq!(tokens[0].text, "quick");
Source

fn name(&self) -> &str

Get the name of this analyzer (for debugging and configuration).

The name is used to identify the analyzer in logs, error messages, and configuration files.

§Returns

A static string representing the analyzer’s unique identifier.

§Examples
use laurus::analysis::analyzer::analyzer::Analyzer;
use laurus::analysis::analyzer::standard::StandardAnalyzer;

let analyzer = StandardAnalyzer::new().unwrap();
assert_eq!(analyzer.name(), "standard");
Source

fn as_any(&self) -> &dyn Any

Provide access to the concrete type for downcasting.

This method enables downcasting from &dyn Analyzer to a concrete analyzer type, which is useful when you need access to type-specific methods (e.g., PerFieldAnalyzer::get_analyzer).

§Returns

A reference to the analyzer as &dyn Any for downcasting.

§Examples
use laurus::analysis::analyzer::analyzer::Analyzer;
use laurus::analysis::analyzer::per_field::PerFieldAnalyzer;
use laurus::analysis::analyzer::standard::StandardAnalyzer;
use std::sync::Arc;

let per_field = PerFieldAnalyzer::new(Arc::new(StandardAnalyzer::new().unwrap()));
let analyzer: &dyn Analyzer = &per_field;

// Downcast to access PerFieldAnalyzer-specific methods
if let Some(pf) = analyzer.as_any().downcast_ref::<PerFieldAnalyzer>() {
    let field_analyzer = pf.get_analyzer("title");
    println!("Got analyzer for 'title' field");
}

Implementors§