[−][src]Trait rust_tokenizers::ConsolidatableTokens

pub trait ConsolidatableTokens<T> where
    T: TokenTrait, {
    fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, T>ⓘ
Notable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
    T: TokenTrait,     type Item = &'a [T];
;
}

ConsolidatableTokens

This trait can be implemented for collections of tokens (i.e. things that implement TokenTrait) and instantiates an iterator to quickly iterate over the tokens in consolidated form, e.g. grouping subtokens into words.

use rust_tokenizers::{ConsolidatableTokens, Token};
let tokens: Vec<Token> = vec![]; //add some tokens
for (wordcount, word_tokens) in tokens.iter_consolidate_tokens().enumerate() {
    eprintln!("word #{} - {:?}", wordcount + 1, word_tokens);
}

Required methods

fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, T>ⓘ

Notable traits for ConsolidatedTokenIterator<'a, T>

impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
    T: TokenTrait,  type Item = &'a [T];
[src]

Creates an iterator from a sequence of ConsolidatableTokens.

Loading content...

Implementations on Foreign Types

impl ConsolidatableTokens<Token> for Vec<Token>[src]

impl<'a> ConsolidatableTokens<TokenRef<'a>> for Vec<TokenRef<'a>>[src]

Loading content...

Implementors

Loading content...