Struct piston_meta::Tokenizer
[−]
[src]
pub struct Tokenizer { pub tokens: Vec<(Range, MetaData)>, }
Stores all the meta data sequentially.
Fields
tokens: Vec<(Range, MetaData)>
The read tokens.
Methods
impl Tokenizer
[src]
fn new() -> Tokenizer
Creates a new tokenizer.
fn data(&mut self, data: MetaData, state: &TokenizerState, range: Range) -> TokenizerState
Reads meta data.