[−][src]Trait tantivy::tokenizer::TokenStream

pub trait TokenStream {
    fn advance(&mut self) -> bool;

    fn token(&self) -> &Token;

    fn token_mut(&mut self) -> &mut Token;

    fn next(&mut self) -> Option<&Token> { ... }

    fn process(&mut self, sink: &mut dyn FnMut(&Token)) -> u32 { ... }
}

TokenStream is the result of the tokenization.

It consists consumable stream of Tokens.

Example

use tantivy::tokenizer::*;

let tokenizer = TextAnalyzer::from(SimpleTokenizer)
       .filter(RemoveLongFilter::limit(40))
       .filter(LowerCaser);
let mut token_stream = tokenizer.token_stream("Hello, happy tax payer");
{
    let token = token_stream.next().unwrap();
    assert_eq!(&token.text, "hello");
    assert_eq!(token.offset_from, 0);
    assert_eq!(token.offset_to, 5);
    assert_eq!(token.position, 0);
}
{
    let token = token_stream.next().unwrap();
    assert_eq!(&token.text, "happy");
    assert_eq!(token.offset_from, 7);
    assert_eq!(token.offset_to, 12);
    assert_eq!(token.position, 1);
}

Required methods

fn advance(&mut self) -> bool

Advance to the next token

Returns false if there are no other tokens.

fn token(&self) -> &Token

Returns a reference to the current token.

fn token_mut(&mut self) -> &mut Token

Returns a mutable reference to the current token.

Loading content...

Provided methods

fn next(&mut self) -> Option<&Token>

Helper to iterate over tokens. It simply combines a call to .advance() and .token().

use tantivy::tokenizer::*;

let tokenizer = TextAnalyzer::from(SimpleTokenizer)
      .filter(RemoveLongFilter::limit(40))
      .filter(LowerCaser);
let mut token_stream = tokenizer.token_stream("Hello, happy tax payer");
while let Some(token) = token_stream.next() {
    println!("Token {:?}", token.text);
}

fn process(&mut self, sink: &mut dyn FnMut(&Token)) -> u32

Helper function to consume the entire TokenStream and push the tokens to a sink function.

Remove this.

Loading content...

Implementations on Foreign Types

impl<'a> TokenStream for Box<dyn TokenStream + 'a>[src]

Loading content...

Implementors

impl TokenStream for PreTokenizedStream[src]

Loading content...