Trait Tokenizer 

pub trait Tokenizer<'a> {
    type Stream: TokenStream;

    // Required method
    fn token_stream(&'a self, source: &'a str) -> Self::Stream;
}

A tokenizer trait.

This trait generates a TokenStream from a source string.

Required Associated Types

type Stream: TokenStream

Required Methods

fn token_stream(&'a self, source: &'a str) -> Self::Stream

Implementors

impl<'a> Tokenizer<'a> for OokTokenizer

Available on crate feature ook only.

type Stream = OokTokenStream<'a>

impl<'a> Tokenizer<'a> for SimpleTokenizer

type Stream = SimpleTokenStream<'a>