Expand description
A single token or a delimited sequence of token trees (e.g. [1, (), ..]
).
Variants
Group(Group)
A token stream surrounded by bracket delimiters.
Ident(Ident)
An identifier.
Punct(Punct)
A single punctuation character (+
, ,
, $
, etc.).
Literal(Literal)
A literal character ('a'
), string ("hello"
), number (2.3
), etc.
Implementations
Trait Implementations
sourceimpl Display for TokenTree
impl Display for TokenTree
Prints the token tree as a string that is supposed to be losslessly
convertible back into the same token tree (modulo spans), except for
possibly TokenTree::Group
s with Delimiter::None
delimiters and negative
numeric literals.
sourceimpl Extend<TokenTree> for TokenStream
impl Extend<TokenTree> for TokenStream
sourcefn extend<I>(&mut self, streams: I)where
I: IntoIterator<Item = TokenTree>,
fn extend<I>(&mut self, streams: I)where
I: IntoIterator<Item = TokenTree>,
sourcefn extend_one(&mut self, item: A)
fn extend_one(&mut self, item: A)
extend_one
)sourcefn extend_reserve(&mut self, additional: usize)
fn extend_reserve(&mut self, additional: usize)
extend_one
)sourceimpl From<LitByteStr> for TokenTree
impl From<LitByteStr> for TokenTree
sourcefn from(lit: LitByteStr) -> TokenTree
fn from(lit: LitByteStr) -> TokenTree
sourceimpl From<TokenTree> for TokenStream
impl From<TokenTree> for TokenStream
sourcefn from(token: TokenTree) -> TokenStream
fn from(token: TokenTree) -> TokenStream
sourceimpl FromIterator<TokenTree> for TokenStream
impl FromIterator<TokenTree> for TokenStream
Collects a number of token trees into a single stream.
sourcefn from_iter<I>(streams: I) -> TokenStreamwhere
I: IntoIterator<Item = TokenTree>,
fn from_iter<I>(streams: I) -> TokenStreamwhere
I: IntoIterator<Item = TokenTree>,
sourceimpl ToTokens for TokenTree
impl ToTokens for TokenTree
sourcefn to_tokens(&self, dst: &mut TokenStream)
fn to_tokens(&self, dst: &mut TokenStream)
sourcefn to_token_stream(&self) -> TokenStream
fn to_token_stream(&self) -> TokenStream
sourcefn into_token_stream(self) -> TokenStream
fn into_token_stream(self) -> TokenStream
impl Token for TokenTree
Auto Trait Implementations
impl RefUnwindSafe for TokenTree
impl !Send for TokenTree
impl !Sync for TokenTree
impl Unpin for TokenTree
impl UnwindSafe for TokenTree
Blanket Implementations
sourceimpl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
sourceimpl<T> Spanned for Twhere
T: Spanned + ?Sized,
impl<T> Spanned for Twhere
T: Spanned + ?Sized,
sourcefn span(&self) -> Span
fn span(&self) -> Span
Span
covering the complete contents of this syntax tree
node, or Span::call_site()
if this node is empty. Read moresourceimpl<T> SpannedExt for Twhere
T: Spanned + ?Sized,
impl<T> SpannedExt for Twhere
T: Spanned + ?Sized,
sourcefn byte_range(&self, source: &str) -> Range<usize>
fn byte_range(&self, source: &str) -> Range<usize>
TODO(H2CO3): a faster, less naive implementation would be great.
We should use the byte offset of start
to compute that of end
,
sparing the double scan of the source up until the start location.
let source = r#"
-3.667
1248 "string ű literal"
"wíőzs"
"#;
let tokens: Many<Lit> = source.parse()?;
assert_eq!(tokens.len(), 4);
assert_eq!(tokens[0].byte_range(source), 4..10);
assert_eq!(tokens[1].byte_range(source), 13..17);
assert_eq!(tokens[2].byte_range(source), 19..38);
assert_eq!(tokens[3].byte_range(source), 45..54);
sourcefn char_range(&self, source: &str) -> Range<usize>
fn char_range(&self, source: &str) -> Range<usize>
TODO(H2CO3): a faster, less naive implementation would be great.
We should use the char offset of start
to compute that of end
,
sparing the double scan of the source up until the start location.
let source = r#"
-3.667
1248 "string ű literal"
"wíőzs"
"#;
let tokens: Many<Lit> = source.parse()?;
assert_eq!(tokens.len(), 4);
assert_eq!(tokens[0].char_range(source), 4..10);
assert_eq!(tokens[1].char_range(source), 13..17);
assert_eq!(tokens[2].char_range(source), 19..37);
assert_eq!(tokens[3].char_range(source), 44..51);