Crate erl_tokenize [−] [src]
Erlang source code tokenizer.
Examples
Tokenizes the Erlang code io:format("Hello").
:
use erl_tokenize::{Tokenizer, TokenKind}; let src = r#"io:format("Hello")."#; let tokenizer = Tokenizer::new(src); let tokens = tokenizer.collect::<Result<Vec<(_, _)>, _>>().unwrap(); assert_eq!(tokens.iter().map(|&(ref t, _)| t.kind()).collect::<Vec<_>>(), [TokenKind::Atom, TokenKind::Symbol, TokenKind::Atom, TokenKind::Symbol, TokenKind::String, TokenKind::Symbol, TokenKind::Symbol]); assert_eq!(tokens.iter().map(|&(ref t, _)| t.text()).collect::<Vec<_>>(), ["io", ":", "format", "(", r#""Hello""#, ")", "."]);
References
- erl_scan module
- Erlang Data Types
Modules
tokens |
Tokens. |
values |
Token values. |
Structs
Position |
Position of token. |
Tokenizer |
Tokenizer. |
Enums
ErrorKind |
The list of the possible error kinds |
Token |
Token. |
TokenKind |
Token kind. |
TokenValue |
Token value. |
Type Definitions
Error |
This crate specific error type. |
Result |
This crate specific |