pub struct Token<'a> {
pub body: &'a str,
pub indices: Range<usize>,
pub tags: Vec<&'a str, Global>,
}
Expand description
A representation of a token. The body
field points to the underlying string
from which the characters in this token come. The indices
field contains
a range representing which contiguous characters are used. The tags
field is
a vector of borrowed strings with user-defined values.
Fields§
§body: &'a str
§indices: Range<usize>
Implementations§
source§impl<'a> Token<'a>
impl<'a> Token<'a>
sourcepub fn content(&'a self) -> &'a str
pub fn content(&'a self) -> &'a str
Gets the content of the token as a string slice from the body. Mostly used for printing tokens.
sourcepub fn has_tag(&self, tag: &str) -> bool
pub fn has_tag(&self, tag: &str) -> bool
Whether or not a token’s tags contain a certain string slice value.
sourcepub fn single_char(&self) -> Option<char>
pub fn single_char(&self) -> Option<char>
If this token is a single character long, returns that character. Otherwise, returns None.