Crate lua_tokenizer 

let source = " <source code here> ";

let tokenizer = lua_tokenizer::Tokenizer::new(source);
// tokenizer itself is a lazy iterator.
for token in tokenizer {
    match token {
        Ok(token) => {
            // do something with token
        }
        Err(e) => {
            print!("Tokenize Error: {}", e);
        }
    }
}

Structs

Span

range of a token in the source code

Token

Token classification and metadata.

Tokenizer

lazy tokenize iterator.

Enums

IntOrFloat

Lua’s numeric representation, can be either integer or float.

TokenType

classifies the type of token

TokenizeError

Type Aliases

FloatType

type alias for lua float type.

IntType

type alias for lua integer type.