Crate lua_tokenizer

let source = " <source code here> ";

let tokenizer = lua_tokenizer::Tokenizer::new(source);
// tokenizer itself is a lazy iterator.
for token in tokenizer {
    match token {
        Ok(token) => {
            // do something with token
        }
        Err(e) => {
            print!("Tokenize Error: {}", e);
        }
    }
}

Structs

• Span
  range of a token in the source code
• Token
  Token classification and metadata.
• Tokenizer
  lazy tokenize iterator.

Enums

• IntOrFloat
  Lua’s numeric representation, can be either integer or float.
• TokenType
  classifies the type of token
• TokenizeError

Type Aliases

• FloatType
  type alias for lua float type.
• IntType
  type alias for lua integer type.