Crate erl_tokenize

Erlang source code tokenizer.

Examples

Tokenizes the Erlang code io:format("Hello").:

use erl_tokenize::{Tokenizer, TokenKind};

let src = r#"io:format("Hello")."#;
let tokenizer = Tokenizer::new(src);
let tokens = tokenizer.collect::<Result<Vec<_>, _>>().unwrap();

assert_eq!(tokens.iter().map(|t| t.kind()).collect::<Vec<_>>(),
           [TokenKind::Atom, TokenKind::Symbol, TokenKind::Atom, TokenKind::Symbol,
            TokenKind::String, TokenKind::Symbol, TokenKind::Symbol]);

assert_eq!(tokens.iter().map(|t| t.text()).collect::<Vec<_>>(),
           ["io", ":", "format", "(", r#""Hello""#, ")", "."]);

References

• erl_scan module
• Erlang Data Types

Modules

tokens	Tokens.
values	Token values.

Structs

Tokenizer

Tokenizer.

Enums

ErrorKind	The list of the possible error kinds
Token	Token.
TokenKind	Token kind.
TokenValue	Token value.

Type Definitions

Error	This crate specific error type.
Result	This crate specific Result type.