Crate jlek 

Generate lexer from token specifications.

JLEK is the lexer used by JJIK parser generator. Currently, JLEK cannot be used standalone without JJIK, since the generated lexer module (lexer.rs) depends on a module generated by JJIK (symbol.rs).

Usage

[dependencies]
jlek = "0.1.0"

A token specification consists of an identifier and a regular expression (see TokenSpec). Accepted regular expression syntax is given in Regular Expression Syntax.

use std::path::PathBuf;

// create a token specification for decimal numbers
let number = jlek::TokenSpec::new("Number".to_string(), "\\d\\d*".to_string());
let token_specs = vec![number];

// generate `lexer.rs` at `output_directory`
let output_directory = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("src");
jlek::generate(&token_specs, &output_directory).unwrap();

You can then use the generated lexer as follows:

mod lexer;

let lexer = lexer::Lexer::from_source_str("123");
lexer.next_token().unwrap();

Regular Expression Syntax

Character classes

\d      decimal digit (0-9).
\w      lowercase character (a-z).
\W      uppercase character (A-Z).
\       escape character for matching with special characters (`\`, `*`, `|`, `(`, `)`), e.g.
        `\*` matches with "*".

Supported operators

xy      concatenation; match with x followed by y.
x|y     disjunction; match with either x or y.
x*      kleene; match with one or more occurance x.
(x)     parenthesis; groups an expression for overriding precedence.

Structs

TokenSpec

A token specification.

Functions

generate

Generates a lexer from token specifications.