ruby-lexer is a library that provides the APIs needed to lex the Ruby programming language's syntax into a stream of tokens.
This library is provided as a set of parser combinator functions, powered by nom. All of the parser combinators are structured to semantically reflect the ISO Ruby specification, but minor deviations from the spec have been made when necessary (namely, re-ordering alternative parsers to consume the largest production first).
The top-level parser combinators that return tokens are publically exported within the lexers module.
Defines the functions used by the
Tracks lexer-specific metadata
Tracks location information and user-defined metadata for nom's source input.
indicates which parser returned an error
Defines something that may be interpolated
Defines the tokens that are returned as a result of lexing
Parses a ruby program
Tokenizes a ruby program
Describes a list of tokens that make up an expression
Describes the lexer's input type
Describes a list of statements