Enum erl_tokenize::Token [−][src]
pub enum Token {
Atom(AtomToken),
Char(CharToken),
Comment(CommentToken),
Float(FloatToken),
Integer(IntegerToken),
Keyword(KeywordToken),
String(StringToken),
Symbol(SymbolToken),
Variable(VariableToken),
Whitespace(WhitespaceToken),
}
Expand description
Token.
Variants
Atom(AtomToken)
Tuple Fields
0: AtomToken
Char(CharToken)
Tuple Fields
0: CharToken
Comment(CommentToken)
Tuple Fields
0: CommentToken
Float(FloatToken)
Tuple Fields
0: FloatToken
Integer(IntegerToken)
Tuple Fields
0: IntegerToken
Keyword(KeywordToken)
Tuple Fields
0: KeywordToken
String(StringToken)
Tuple Fields
0: StringToken
Symbol(SymbolToken)
Tuple Fields
0: SymbolToken
Variable(VariableToken)
Tuple Fields
Whitespace(WhitespaceToken)
Tuple Fields
Implementations
Tries to convert from any prefixes of the text to a token.
Examples
use erl_tokenize::{Token, Position};
use erl_tokenize::values::Symbol;
let pos = Position::new();
// Atom
let token = Token::from_text("foo", pos.clone()).unwrap();
assert_eq!(token.as_atom_token().map(|t| t.value()), Some("foo"));
// Symbol
let token = Token::from_text("[foo]", pos.clone()).unwrap();
assert_eq!(token.as_symbol_token().map(|t| t.value()), Some(Symbol::OpenSquare));
Returns the original textual representation of this token.
Examples
use erl_tokenize::{Token, Position};
let pos = Position::new();
// Comment
assert_eq!(Token::from_text("% foo", pos.clone()).unwrap().text(), "% foo");
// Char
assert_eq!(Token::from_text(r#"$\t"#, pos.clone()).unwrap().text(), r#"$\t"#);
Returns true
if this is a lexical token, otherwise false
.
Returns true
if this is a hidden token, otherwise false
.
Tries to convert into LexicalToken
.
Tries to convert into HiddenToken
.
Tries to return the reference to the inner AtomToken
.
Tries to return the reference to the inner CharToken
.
Tries to return the reference to the inner FloatToken
.
Tries to return the reference to the inner IntegerToken
.
Tries to return the reference to the inner KeywordToken
.
Tries to return the reference to the inner StringToken
.
Tries to return the reference to the inner SymbolToken
.
Tries to return the reference to the inner VariableToken
.
Tries to return the reference to the inner CommentToken
.
Tries to return the reference to the inner WhitespaceToken
.
Tries to return the inner AtomToken
.
Tries to return the inner CharToken
.
Tries to return the inner FloatToken
.
Tries to return the inner IntegerToken
.
Tries to return the inner KeywordToken
.
Tries to return the inner StringToken
.
Tries to return the inner SymbolToken
.
Tries to return the inner VariableToken
.
Tries to return the inner CommentToken
.
Tries to return the inner WhitespaceToken
.
Trait Implementations
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Returns the (inclusive) start position of this.
Returns the (exclusive) end position of this.
Auto Trait Implementations
impl RefUnwindSafe for Token
impl UnwindSafe for Token
Blanket Implementations
Mutably borrows from an owned value. Read more