Module syntex_syntax::tokenstream

Token Streams

TokenStreams represent syntactic objects before they are converted into ASTs. A TokenStream is, roughly speaking, a sequence (eg stream) of TokenTrees, which are themselves either a single Token, a Delimited subsequence of tokens, or a SequenceRepetition specifier (for the purpose of sequence generation during macro expansion).

Ownership

TokenStreams are persistant data structures construced as ropes with reference counted-children. In general, this means that calling an operation on a TokenStream (such as slice) produces an entirely new TokenStream from the borrowed reference to the original. This essentially coerces TokenStreams into 'views' of their subparts, and a borrowed TokenStream is sufficient to build an owned TokenStream without taking ownership of the original.

Structs

Cursor
Delimited	A delimited sequence of token trees
SequenceRepetition	A sequence of token trees
TokenStream

Enums

KleeneOp	A Kleene-style repetition operator for token sequences.
TokenTree	When the main rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token-tree. This is a very loose structure, such that all sorts of different AST-fragments can be passed to syntax extensions using a uniform type.