[][src]Function full_moon::tokenizer::tokens

pub fn tokens<'a>(code: &'a str) -> Result<Vec<Token<'a>>, TokenizerError>

Returns a list of Token structs. You probably want parse instead.

Errors

If the code passed is malformed from normal Lua expectations, a TokenizerError will be returned.

assert!(tokens("local x = 1").is_ok());
assert!(tokens("local 4 = end").is_ok()); // tokens does *not* check validity of code, only tokenizing
assert!(tokens("--[[ Unclosed comment!").is_err());