tokel-engine 0.2.4

The core parsing, AST, and bottom-up evaluation engine for the Tokel macro language.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

# Tokel

**Tokel** is a procedural macro library (provided as a *standalone component* for use in other procedural macro libraries and as its own *derive* crate) that exposes a centralized method to manipulate a token-stream.

**Tokel** employs a recursive, *expansion block*-based model with *transformer* pipelines attached to the end of each *expansion block*.

## The Expansion Model

*Tokel* takes a singular top-level token-stream (a "`TokelStream`" in the formal grammar definition) and performs recursive  descent to find and evaluate these blocks.

Expansion blocks are denoted by `[< ... >]`. Any tokens outside of an expansion block are ignored and emitted as-is.

When Tokel encounters an expansion block, it evaluates it **bottom-up (inside-out)**. Any nested blocks are fully expanded before the outer block is processed. Once the inner token-stream is resolved, it is passed through the transformer pipeline attached to the block.

### Transformers

Transformers are pure functions that map an input token-stream to an output token-stream. They are chained to the end of an expansion block using a colon (`:`).

If a transformer takes arguments, they are provided using double brackets `[[ ... ]]`. The arguments themselves are parsed as a standard Tokel token-stream, meaning you can nest expansion blocks inside them.

```ignore
// Expands to `GetMyIdentifier`
[< my_identifier >]:case[[pascal]]:prefix[[Get]]
```

Because transformers only apply to the resolved output of a `[< ... >]` block, the syntax avoids the ambiguity of attaching transforms to individual tokens.

**NOTE**: For a full list of standard transformers, see the `tokel-std` crate.

### Formal Grammar

```ebnf,ignore
TokelStream       ::= Element*

Element           ::= "[" Block "]" Pipeline
                    | TokelTree

(* NOTE: A `None`-delimited group is left unchanged. *)
TokelTree         ::= "(" TokelStream ")"
                    | "{" TokelStream "}"
                    | "[" TokelStream "]"
                    | Ident | Literal | Punct

Block             ::= "<" TokelStream ">"

Pipe              ::= ":" Transformer

Pipeline          ::= Pipe*

Transformer       ::= Ident ( "[[" TokelStream "]]" )?
```

# License

Copyright (C) 2026 W. Frakchi

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

See [the full license agreement](./LICENSE) for further information.