Function full_moon::tokenizer::tokens

pub fn tokens(code: &str) -> Result<Vec<Token>, TokenizerError>

Returns a list of tokens. You probably want parse instead.

Errors

If the code passed is malformed from normal Lua expectations, a TokenizerError will be returned.

assert!(tokens("local x = 1").is_ok());
assert!(tokens("local 4 = end").is_ok()); // tokens does *not* check validity of code, only tokenizing
assert!(tokens("--[[ Unclosed comment!").is_err());