Crate luthor

Luthor provides a collection of lexers for various formats and languages. It also exposes types that aid in building lexers of your own.

Modules

lexers

Pre-built lexers for various languages/formats.

token

Token-related types.

Structs

StateFunction

A recursive function type used by lexers to manage their state. Based on Rob Pike’s “Lexical Scanning in Go” talk, these functions are invoked in a call/return loop (letting the current function determine the next) until a None value is returned, after which lexing is complete.

Tokenizer

The Tokenizer type is used to produce and store tokens for lexers.