Module syntax::tokenstream

Token Streams

TokenStreams represent syntactic objects before they are converted into ASTs. A TokenStream is, roughly speaking, a sequence (eg stream) of TokenTrees, which are themselves a single Token or a Delimited subsequence of tokens.

Ownership

TokenStreams are persistent data structures constructed as ropes with reference counted-children. In general, this means that calling an operation on a TokenStream (such as slice) produces an entirely new TokenStream from the borrowed reference to the original. This essentially coerces TokenStreams into 'views' of their subparts, and a borrowed TokenStream is sufficient to build an owned TokenStream without taking ownership of the original.

Structs

Cursor
DelimSpan
TokenStream	Token Streams
TokenStreamBuilder

Enums

IsJoint
TokenTree	When the main rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token-tree. This is a very loose structure, such that all sorts of different AST-fragments can be passed to syntax extensions using a uniform type.

Type Definitions

TreeAndJoint