[−][src]Function full_moon::tokenizer::tokens
pub fn tokens<'a>(code: &'a str) -> Result<Vec<Token<'a>>, TokenizerError>
Returns a list of Token
structs.
You probably want parse
instead.
Errors
If the code passed is malformed from normal Lua expectations,
a TokenizerError
will be returned.
assert!(tokens("local x = 1").is_ok()); assert!(tokens("local 4 = end").is_ok()); // tokens does *not* check validity of code, only tokenizing assert!(tokens("--[[ Unclosed comment!").is_err());