Enum TokenText 

pub enum TokenText {
    Slice {
        source: Arc<str>,
        range: Range<usize>,
    },
    Owned(Arc<str>),
    Static(&'static str),
}

Representation for the textual payload of a Token.

Tokens may either borrow slices from the input stream or own standalone strings that were generated during macro expansion. The enum keeps track of the data’s lifetime and enables zero-copy lexing for most tokens.

Variants

Slice

Borrowed slice from an input string.

Fields

source: Arc<str>

Shared reference to the original input.

range: Range<usize>

Range of the slice within the source string.

Owned(Arc<str>)

Heap allocated string stored in an Arc.

Static(&'static str)

Static string literal.

Implementations

impl TokenText

pub const fn slice(source: Arc<str>, start: usize, end: usize) -> Self

Creates a borrowed slice from an input Arc<str>.

pub fn as_str(&self) -> &str

Returns the string slice represented by this token.

pub fn len(&self) -> usize

Returns the length in bytes of the token text.

pub fn is_empty(&self) -> bool

Returns true when the token text is empty.

pub fn to_owned_string(&self) -> String

Converts the token text into an owned String.

pub fn clone_into(&self, buf: &mut String)

Clone into a given string buffer.

Trait Implementations

impl Clone for TokenText

fn clone(&self) -> TokenText

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenText

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&'static str> for TokenText

fn from(value: &'static str) -> Self

Converts to this type from the input type.

impl From<Arc<str>> for TokenText

fn from(value: Arc<str>) -> Self

Converts to this type from the input type.

impl From<String> for TokenText

fn from(value: String) -> Self

Converts to this type from the input type.

impl PartialEq<&str> for TokenText

fn eq(&self, other: &&str) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<String> for TokenText

fn eq(&self, other: &String) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<TokenText> for &str

fn eq(&self, other: &TokenText) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq<TokenText> for String

fn eq(&self, other: &TokenText) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq for TokenText

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenText