Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(file: &Path, text: &str) -> Result<TokenStream, ModelError>
Expand description

An editor’s byte order mark is not Rust, but the compiler skips it.