Module syntax::tokenstream
[−]
[src]
Token Streams
TokenStream
s represent syntactic objects before they are converted into ASTs.
A TokenStream
is, roughly speaking, a sequence (eg stream) of TokenTree
s,
which are themselves a single Token
or a Delimited
subsequence of tokens.
Ownership
TokenStreams
are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a TokenStream
(such as slice
) produces an entirely new TokenStream
from the borrowed reference to
the original. This essentially coerces TokenStream
s into 'views' of their subparts,
and a borrowed TokenStream
is sufficient to build an owned TokenStream
without taking
ownership of the original.
Structs
Cursor | |
Delimited |
A delimited sequence of token trees |
ThinTokenStream |
The |
TokenStream |
Token Streams |
TokenStreamBuilder |
Enums
TokenTree |
When the main rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token-tree. This is a very loose structure, such that all sorts of different AST-fragments can be passed to syntax extensions using a uniform type. |