Struct lib_ruby_parser::Token
source · [−]#[repr(C)]pub struct Token {
pub token_type: i32,
pub token_value: Bytes,
pub loc: Loc,
pub lex_state_before: LexState,
pub lex_state_after: LexState,
}
Expand description
A token that is emitted by a lexer and consumed by a parser
Fields
token_type: i32
Numeric representation of the token type, e.g. 42 (for example) for tINTEGER
token_value: Bytes
Value of the token, e.g “42” for 42
loc: Loc
Location of the token
lex_state_before: LexState
Lex state before reading the token
lex_state_after: LexState
Lex state after reading the token
Implementations
sourceimpl Token
impl Token
sourcepub fn into_bytes(self) -> Vec<u8>
pub fn into_bytes(self) -> Vec<u8>
Consumes a token and returns an owned byte array of the token value
sourcepub fn as_str_lossy(&self) -> Result<&str, Utf8Error>
pub fn as_str_lossy(&self) -> Result<&str, Utf8Error>
Converts token value into &str
sourcepub fn to_string_lossy(&self) -> String
pub fn to_string_lossy(&self) -> String
Converts token to a string, replaces unknown chars to U+FFFD
sourcepub fn to_string(&self) -> Result<String, FromUtf8Error>
pub fn to_string(&self) -> Result<String, FromUtf8Error>
Converts token to a string
sourcepub fn into_string(self) -> Result<String, FromUtf8Error>
pub fn into_string(self) -> Result<String, FromUtf8Error>
Consumes a token and converts it into a string
sourcepub fn token_name(&self) -> &'static str
pub fn token_name(&self) -> &'static str
Returns name of the token
Trait Implementations
impl Eq for Token
impl StructuralEq for Token
impl StructuralPartialEq for Token
Auto Trait Implementations
impl RefUnwindSafe for Token
impl Send for Token
impl Sync for Token
impl Unpin for Token
impl UnwindSafe for Token
Blanket Implementations
sourceimpl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more