Module syntax 

The Abstract Syntax Tree (AST) for Tokel.

This module contains the data structures and syn::parse::Parse implementations that represent the Tokel grammar. It is responsible for taking a raw proc_macro2::TokenStream and structuring it into a recursive AST that can be evaluated bottom-up by the engine.

Grammar Mapping

The types in this module directly correspond to the formal EBNF grammar:

• TokelStream: The root sequence of elements. Represents TokelStream ::= Element*.
• Element: An individual unit in the stream. It is either a standard Rust token tree (TokelTree) or an expansion block ([ Block ] Pipeline).
• Block: The inner contents of an expansion block (the < TokelStream > part).
• Pipeline: A sequence of one or more transformers attached to the end of an expansion block (e.g., :case[[pascal]]:prefix[[Get]]).
• Transformer: A single transformation operation (like case) and its optional arguments parsed from double brackets ([[ ... ]]).

Parsing Strategy

The parser performs a deep traversal of the incoming token stream. When it encounters standard Rust delimiters ((), {}, or [] that do not start with <), it recursively parses their inner contents as a new TokelStream.

This recursive parsing ensures that expansion blocks deeply nested inside standard Rust code, None-delimited macro capture groups, or even inside transformer arguments, are accurately located and represented in the final AST.

Structs

Block

An expansion block.

Pipe

A pass of a single transformer with the provided argument.

Pipeline

The :-punctuated sequence of Pipe, to form a complete pipeline for an expansion block.

TokelGroup

A tokel-specific TokenGroup.

TokelStream

A tokel-specific stream of source-level elements.

Enums

Element

An element inside a tokel-processed token-stream.

TokelTree

A tokel-specific TokenTree.