[−][src]Crate erl_tokenize
Erlang source code tokenizer.
Examples
Tokenizes the Erlang code io:format("Hello").
:
use erl_tokenize::Tokenizer; let src = r#"io:format("Hello")."#; let tokenizer = Tokenizer::new(src); let tokens = tokenizer.collect::<Result<Vec<_>, _>>().unwrap(); assert_eq!(tokens.iter().map(|t| t.text()).collect::<Vec<_>>(), ["io", ":", "format", "(", r#""Hello""#, ")", "."]);
References
erl_scan
module- Erlang Data Types
Modules
tokens | Tokens. |
values | Token values. |
Structs
Error | This crate specific error type. |
Lexer | Lexer. |
Position | Position of token. |
Tokenizer | Tokenizer. |
Enums
ErrorKind | The list of the possible error kinds |
HiddenToken | Hidden token. |
LexicalToken | Lexical token. |
Token | Token. |
Traits
PositionRange | This trait allows to get a (half-open) range where the subject is located. |
Type Definitions
Result | This crate specific |