Crate erl_tokenize 

Erlang source code tokenizer.

The public entry point is the free function scan_token: given the whole source string and the current Position, it returns the next Token, or Ok(None) when the end of the source is reached. On failure, the returned Error carries a diagnostic position and a resume position that can be passed straight back into scan_token so a bad token never spins in place. To scan the whole source in one call, use scan_tokens.

Design

• One call scans one token.
• A Token does not borrow the source and does not carry any decoded value. Value extraction happens only when the caller invokes Token::value.
• The caller owns the source string and drives the current position from token to token.
• Comments and whitespace are returned as ordinary tokens. Callers that only care about grammatical tokens filter them out via TokenKind::is_lexical or TokenKind::is_hidden.
• Position does not carry a file path. Associating a scanned source with its file name or buffer identifier is the caller’s responsibility.

Examples

Tokenize the Erlang code io:format("Hello").:

let src = r#"io:format("Hello")."#;
let texts: Vec<_> = erl_tokenize::scan_tokens(src)?
    .into_iter()
    .map(|token| token.text(src))
    .collect();
assert_eq!(texts, ["io", ":", "format", "(", r#""Hello""#, ")", "."]);

Skip comments and whitespace on the caller side:

let src = "%% greeting\nhello world";
let lexical: Vec<_> = erl_tokenize::scan_tokens(src)?
    .into_iter()
    .filter(|token| token.kind().is_lexical())
    .map(|token| token.text(src))
    .collect();
assert_eq!(lexical, ["hello", "world"]);

Resume after a lexical error using Error::resume_position:

let src = "\u{2603} foo";
let mut position = erl_tokenize::Position::new();
let mut texts = Vec::new();
loop {
    match erl_tokenize::scan_token(src, position) {
        Ok(Some(token)) => {
            texts.push(token.text(src));
            position = token.end();
        }
        Ok(None) => break,
        Err(error) => {
            position = error.resume_position;
        }
    }
}
assert_eq!(texts, [" ", "foo"]);

Compatibility target

This crate aims to match the behavior of Erlang/OTP’s erl_scan module at the OTP-29.0.5 tag. CI runs the token diff against that tag’s stdlib source.

References

• erl_scan module
• Erlang Data Types

Structs

Error

Lexical error produced by the scanner.

Position

Position within a source string.

Token

A scanned token: kind and a half-open position range in the source.

Enums

ErrorKind

Classification of a lexical Error.

Keyword

Erlang reserved word.

Symbol

Punctuation or operator symbol.

TokenKind

Kind of a scanned token.

TokenValue

Decoded value of a Token, borrowing from the source where possible.

Functions

scan_token

Scans a single token from source starting at position.

scan_tokens

Scans all tokens from source until EOF.

Type Aliases

Result

This crate’s Result type.