Struct tantivy_analysis_contrib::icu::ICUTokenizer

pub struct ICUTokenizer;

Available on crate feature icu only.

ICU Tokenizer. It does not (yet ?) work as Lucene’s counterpart. Getting a tokenizer is simple :

use tantivy_analysis_contrib::icu::ICUTokenizer;

let tokenizer = ICUTokenizer;

Example

Here is an example of a tokenization result

use tantivy::tokenizer::{TextAnalyzer, Token};
use tantivy_analysis_contrib::icu::ICUTokenizer;

let mut tmp = TextAnalyzer::builder(ICUTokenizer::default()).build();
let mut token_stream = tmp.token_stream("我是中国人。 １２３４ Ｔｅｓｔｓ ");

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "我".to_string());

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "是".to_string());

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "中".to_string());

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "国".to_string());

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "人".to_string());

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "１２３４".to_string());

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "Ｔｅｓｔｓ".to_string());

assert_eq!(None, token_stream.next());

Trait Implementations

impl Clone for ICUTokenizer

fn clone(&self) -> ICUTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ICUTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ICUTokenizer

fn default() -> ICUTokenizer

Returns the “default value” for a type.

impl Tokenizer for ICUTokenizer

type TokenStream<'a> = ICUTokenizerTokenStream<'a>

The token stream returned by this Tokenizer.

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Creates a token stream for a given str.

impl Copy for ICUTokenizer