Struct piston_meta::Tokenizer

pub struct Tokenizer {
    pub tokens: Vec<(Range, MetaData)>,
}

Stores all the meta data sequentially.

Fields

tokens: Vec<(Range, MetaData)>

The read tokens.

Methods

impl Tokenizer

fn new() -> Tokenizer

Creates a new tokenizer.

fn data(&mut self, data: MetaData, state: &TokenizerState, range: Range) -> TokenizerState

Reads meta data.