Trait Tokenize

pub trait Tokenize: Sized {
    type CharType;

    // Required method
    fn next_token<I>(input: &mut I) -> Result<Token<Self>>
       where I: InputStream<CharType = Self::CharType>;

    // Provided method
    fn lexer<I>(input: I) -> Lexer<I, Self> { ... }
}

Trait for token kinds that can be tokenized from an input stream.

Required Associated Types

type CharType

The type of the character produced by the input stream.

Required Methods

fn next_token<I>(input: &mut I) -> Result<Token<Self>>

where I: InputStream<CharType = Self::CharType>,

Reads the next token from the given input stream.

Returns the token (Token<Self>) if successful, otherwise Err.

Provided Methods

fn lexer<I>(input: I) -> Lexer<I, Self>

Creates a lexer from the given input stream that produces the current token kind.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors