Struct Tokenizer 

pub struct Tokenizer<'a> { /* private fields */ }

字句解析器。入力文字列をトークン列へ変換する。

Implementations

impl<'a> Tokenizer<'a>

pub fn new(dialect: &'a dyn Dialect, input: &'a str) -> Self

pub fn tokenize(&mut self) -> Result<Vec<TokenWithSpan>>

Trait Implementations

impl<'a> Clone for Tokenizer<'a>

fn clone(&self) -> Tokenizer<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Tokenizer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.