pub fn tokens(code: &str) -> Result<Vec<Token>, TokenizerError>
Expand description
Returns a list of tokens.
You probably want parse
instead.
Errors
If the code passed is malformed from normal Lua expectations,
a TokenizerError
will be returned.
assert!(tokens("local x = 1").is_ok());
assert!(tokens("local 4 = end").is_ok()); // tokens does *not* check validity of code, only tokenizing
assert!(tokens("--[[ Unclosed comment!").is_err());