Crate tokel_std 

tokel-std

A standard library of minimal, composable Transformer implementations for the Tokel engine.

Available Transformers

Transformer	Description	Example
Reverse	Reverses the sequence of token trees.	[< a b c >]:reverse -> c b a
Intersperse	Inserts a token tree between each input token tree.	[< a b c >]:intersperse[[,]] -> a , b , c
PushLeft	Prepends a TokenStream to the input.	[< b c >]:push_left[[a]] -> a b c
PushRight	Appends a TokenStream to the input.	[< a b >]:push_right[[c]] -> a b c
PopLeft	Removes the first token tree from the input.	[< a b c >]:pop_left -> b c
PopRight	Removes the last token tree from the input.	[< a b c >]:pop_right -> a b
Take	Keeps the first N token trees. (Argument: syn::LitInt).	[< a b c d >]:take[[2]] -> a b
Skip	Discards the first N token trees. (Argument: syn::LitInt).	[< a b c d >]:skip[[2]] -> c d
Repeat	Repeats the input N times. (Argument: syn::LitInt).	[< a b >]:repeat[[3]] -> a b a b a b
Count	Returns the number of input token trees as an integer literal.	[< a b c >]:count -> 3
Sequence	Yields an integer sequence. (Argument: [[ start..end ]]).	[< >]:sequence[[1..=3]] -> 1 2 3
Enumerate	Stateful generator yielding an incrementing u32 literal.	[< >]:enumerate -> 0
Concatenate	Concatenates text representations of tokens into a single Ident.	[< hello _ world >]:concatenate -> hello_world
Case	Converts identifiers and strings to a target CaseStyle.	[< hello_world >]:case[[pascal]] -> HelloWorld

Notes

• Arguments: Transformers can accept other transformer expressions as arguments. Inner expressions are evaluated first.
• State: Stateful transformers (like Enumerate) keep their state for the lifetime of the registered instance.
• Details: See the module-level documentation (iter, string, structure, state) for specific argument types and behaviors.

---

The following is the main tokel workspace documentation:

Tokel

Tokel is a procedural macro library (provided as a standalone component for use in other procedural macro libraries and as its own derive crate) that exposes a centralized method to manipulate a token-stream.

Tokel employs a recursive, expansion block-based model with transformer pipelines attached to the end of each expansion block.

The Expansion Model

Tokel takes a singular top-level token-stream (a “TokelStream” in the formal grammar definition) and performs recursive descent to find and evaluate these blocks.

Expansion blocks are denoted by [< ... >]. Any tokens outside of an expansion block are ignored and emitted as-is.

When Tokel encounters an expansion block, it evaluates it bottom-up (inside-out). Any nested blocks are fully expanded before the outer block is processed. Once the inner token-stream is resolved, it is passed through the transformer pipeline attached to the block.

Transformers

Transformers are pure functions that map an input token-stream to an output token-stream. They are chained to the end of an expansion block using a colon (:).

If a transformer takes arguments, they are provided using double brackets [[ ... ]]. The arguments themselves are parsed as a standard Tokel token-stream, meaning you can nest expansion blocks inside them.

// Expands to `GetMyIdentifier`
[< my_identifier >]:case[[pascal]]:prefix[[Get]]

Because transformers only apply to the resolved output of a [< ... >] block, the syntax avoids the ambiguity of attaching transforms to individual tokens.

NOTE: For a full list of standard transformers, see the tokel-std crate.

Formal Grammar

TokelStream       ::= Element*

Element           ::= "[" Block "]" Pipeline
                    | TokelTree

(* NOTE: A `None`-delimited group is left unchanged. *)
TokelTree         ::= "(" TokelStream ")"
                    | "{" TokelStream "}"
                    | "[" TokelStream "]"
                    | Ident | Literal | Punct

Block             ::= "<" TokelStream ">"

Pipe              ::= ":" Transformer

Pipeline          ::= Pipe*

Transformer       ::= Ident ( "[[" TokelStream "]]" )?

License

Copyright (C) 2026 W. Frakchi

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

See the full license agreement for further information.

Modules

iter

Iteration-related Tokel Transformers.

state

Stateful generation Transformers.

string

String and text-manipulation Tokel Transformers.

structure

Structural boundary and grouping Tokel Transformers.

Functions

register

Registers all standard Transformers into the provided Registry.