lex

This library aids in parsing programming languages.

lex = "0.15.0-rc.1"

There are no dependencies.

Lexer

The lexer converts source text into a sequence of tokens using ordered rules. Tokens are generic over a user-defined TokenKind enum. The lexer is error-free — all input is tokenizable.

use lex::lexer::matchers::{ident, whitespace};
use lex::lexer::{Lexer, Token, TokenKind};
use lex::literal;

let lexer: Lexer<Kind> = Lexer::default ()
.with_rule(Kind::Whitespace, whitespace)
.with_rule(Kind::Ident, ident)
.with_rule(Kind::Semi, literal!(";"));

let tokens: Vec<Token<Kind> > = lexer.lex("let x;");

Parser

The parser provides a token-stream cursor with skip sets, checkpoints for backtracking, multi-error recovery, and leading comment extraction.

use lex::parser::Parser;

let mut parser: Parser<Kind> = Parser::new(source, tokens)
.with_skip(Kind::Whitespace);

let token: Token<Kind> = parser.expect(Kind::Ident) ?;

Built-in Matchers

• ident — [a-zA-Z_][a-zA-Z0-9_]*
• digits — [0-9]+
• whitespace — ASCII whitespace
• literal!("...") — exact string match
• line_comment!("//") — line comment with delimiter