[−][src]Trait rust_tokenizers::preprocessing::tokenizer::base_tokenizer::ConsolidatableTokens

pub trait ConsolidatableTokens<T> where
    T: TokenTrait, {
    fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<T>;
}

ConsolidatableTokens

This trait can be implemented for collections of tokens (i.e. things that implement TokenTrait) and instantiates an iterator to quickly iterate over the tokens in consolidated form, e.g. grouping subtokens into words.

use rust_tokenizers::preprocessing::tokenizer::base_tokenizer::{Token, ConsolidatableTokens};
let tokens: Vec<Token> = vec!(); //add some tokens
for (wordcount, word_tokens) in tokens.iter_consolidate_tokens().enumerate() {
      eprintln!("word #{} - {:?}", wordcount+1, word_tokens);
}