Skip to main content

Crate tokel

Crate tokel 

Source
Expand description

§tokel

tokel is a facade crate that re-exports both tokel-engine and tokel-derive when the appropriate crate features have been selected.

  • tokel-engine: enabled via the engine feature.
  • tokel-derive: enabled via the derive feature (default). The tokel::stream! and tokel::item! macros are re-exported at the crate root for usability.

This crate is no_std by default when the engine feature is disabled.

The following is the main tokel workspace documentation:

§Tokel

Tokel is a procedural macro library (provided as a standalone component for use in other procedural macro libraries and as its own derive crate) that exposes a centralized method to manipulate a token-stream.

Tokel employs a recursive, expansion block-based model with transformer pipelines attached to the end of each expansion block.

§The Expansion Model

Tokel takes a singular top-level token-stream (a “TokelStream” in the formal grammar definition) and performs recursive descent to find and evaluate these blocks.

Expansion blocks are denoted by [< ... >]. Any tokens outside of an expansion block are ignored and emitted as-is.

When Tokel encounters an expansion block, it evaluates it bottom-up (inside-out). Any nested blocks are fully expanded before the outer block is processed. Once the inner token-stream is resolved, it is passed through the transformer pipeline attached to the block.

§Transformers

Transformers are pure functions that map an input token-stream to an output token-stream. They are chained to the end of an expansion block using a colon (:).

If a transformer takes arguments, they are provided using double brackets [[ ... ]]. The arguments themselves are parsed as a standard Tokel token-stream, meaning you can nest expansion blocks inside them.

ⓘ
// Expands to `GetMyIdentifier`
[< my_identifier >]:case[[pascal]]:prefix[[Get]]

Because transformers only apply to the resolved output of a [< ... >] block, the syntax avoids the ambiguity of attaching transforms to individual tokens.

NOTE: For a full list of standard transformers, see the tokel-std crate.

§Formal Grammar

TokelStream       ::= Element*

Element           ::= "[" Block "]" Pipeline
                    | TokelTree

(* NOTE: A `None`-delimited group is left unchanged. *)
TokelTree         ::= "(" TokelStream ")"
                    | "{" TokelStream "}"
                    | "[" TokelStream "]"
                    | Ident | Literal | Punct

Block             ::= "<" TokelStream ">"

Pipe              ::= ":" Transformer

Pipeline          ::= Pipe*

Transformer       ::= Ident ( "[[" TokelStream "]]" )?

§License

Copyright (C) 2026 W. Frakchi

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

See the full license agreement for further information.

Modules§

derive
A module to re-export all public items from the tokel-derive crate.

Macros§

stream
Evaluates and expands Tokel transformations within the provided token stream.

Attribute Macros§

attribute
Evaluates Tokel transformations strictly within the arguments of an attribute.