Function full_moon::tokenizer::tokens [−][src]
pub fn tokens(code: &str) -> Result<Vec<Token<'_>>, TokenizerError>
Expand description
Returns a list of tokens.
You probably want parse
instead.
Errors
If the code passed is malformed from normal Lua expectations,
a TokenizerError
will be returned.
assert!(tokens("local x = 1").is_ok()); assert!(tokens("local 4 = end").is_ok()); // tokens does *not* check validity of code, only tokenizing assert!(tokens("--[[ Unclosed comment!").is_err());