Expand description
A single token or a delimited sequence of token trees (e.g. [1, (), ..]
).
Variants
Group(Group)
A token stream surrounded by bracket delimiters.
Ident(Ident)
An identifier.
Punct(Punct)
A single punctuation character (+
, ,
, $
, etc.).
Literal(Literal)
A literal character ('a'
), string ("hello"
), number (2.3
), etc.
Implementations
Trait Implementations
sourceimpl Display for TokenTree
impl Display for TokenTree
Prints the token tree as a string that is supposed to be losslessly
convertible back into the same token tree (modulo spans), except for
possibly TokenTree::Group
s with Delimiter::None
delimiters and negative
numeric literals.
sourceimpl Extend<TokenTree> for TokenStream
impl Extend<TokenTree> for TokenStream
sourcefn extend<I>(&mut self, streams: I)where
I: IntoIterator<Item = TokenTree>,
fn extend<I>(&mut self, streams: I)where
I: IntoIterator<Item = TokenTree>,
sourcefn extend_one(&mut self, item: A)
fn extend_one(&mut self, item: A)
extend_one
)sourcefn extend_reserve(&mut self, additional: usize)
fn extend_reserve(&mut self, additional: usize)
extend_one
)sourceimpl From<LitByteStr> for TokenTree
impl From<LitByteStr> for TokenTree
sourcefn from(lit: LitByteStr) -> TokenTree
fn from(lit: LitByteStr) -> TokenTree
sourceimpl From<TokenTree> for TokenStream
impl From<TokenTree> for TokenStream
sourcefn from(token: TokenTree) -> TokenStream
fn from(token: TokenTree) -> TokenStream
sourceimpl FromIterator<TokenTree> for TokenStream
impl FromIterator<TokenTree> for TokenStream
Collects a number of token trees into a single stream.
sourcefn from_iter<I>(streams: I) -> TokenStreamwhere
I: IntoIterator<Item = TokenTree>,
fn from_iter<I>(streams: I) -> TokenStreamwhere
I: IntoIterator<Item = TokenTree>,
sourceimpl ToTokens for TokenTree
impl ToTokens for TokenTree
sourcefn to_tokens(&self, dst: &mut TokenStream)
fn to_tokens(&self, dst: &mut TokenStream)
sourcefn to_token_stream(&self) -> TokenStream
fn to_token_stream(&self) -> TokenStream
sourcefn into_token_stream(self) -> TokenStreamwhere
Self: Sized,
fn into_token_stream(self) -> TokenStreamwhere
Self: Sized,
impl Token for TokenTree
Auto Trait Implementations
impl RefUnwindSafe for TokenTree
impl !Send for TokenTree
impl !Sync for TokenTree
impl Unpin for TokenTree
impl UnwindSafe for TokenTree
Blanket Implementations
sourceimpl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
sourceimpl<T> Spanned for Twhere
T: Spanned + ?Sized,
impl<T> Spanned for Twhere
T: Spanned + ?Sized,
sourcefn span(&self) -> Span
fn span(&self) -> Span
Span
covering the complete contents of this syntax tree
node, or Span::call_site()
if this node is empty. Read moresourceimpl<T> SpannedExt for Twhere
T: Spanned + ?Sized,
impl<T> SpannedExt for Twhere
T: Spanned + ?Sized,
sourcefn byte_range(&self, source: &str) -> Range<usize>
fn byte_range(&self, source: &str) -> Range<usize>
TODO(H2CO3): a faster, less naive implementation would be great.
We should use the byte offset of start
to compute that of end
,
sparing the double scan of the source up until the start location.
let source = r#"
-3.667
1248 "string ű literal"
"wíőzs"
"#;
let tokens: Many<Lit> = source.parse()?;
assert_eq!(tokens.len(), 4);
assert_eq!(tokens[0].byte_range(source), 4..10);
assert_eq!(tokens[1].byte_range(source), 13..17);
assert_eq!(tokens[2].byte_range(source), 19..38);
assert_eq!(tokens[3].byte_range(source), 45..54);
sourcefn char_range(&self, source: &str) -> Range<usize>
fn char_range(&self, source: &str) -> Range<usize>
TODO(H2CO3): a faster, less naive implementation would be great.
We should use the char offset of start
to compute that of end
,
sparing the double scan of the source up until the start location.
let source = r#"
-3.667
1248 "string ű literal"
"wíőzs"
"#;
let tokens: Many<Lit> = source.parse()?;
assert_eq!(tokens.len(), 4);
assert_eq!(tokens[0].char_range(source), 4..10);
assert_eq!(tokens[1].char_range(source), 13..17);
assert_eq!(tokens[2].char_range(source), 19..37);
assert_eq!(tokens[3].char_range(source), 44..51);