pesti-structural-tokenizer
Rust structural tokenizer for LLM code understanding — parses Rust source into AST via syn, then emits a compact stream of semantic tokens that encode syntactic structure while discarding implementation details.
What This Does
Traditional byte-pair encoding (BPE) tokenizers treat source code as opaque text. This crate takes a different approach: it uses syn to parse Rust source into a complete AST, then walks the tree emitting structural tokens that capture declarations, control flow, and expressions while eliding bodies of non-declaration functions.
The result is that LLMs get explicit signals about code structure without having to infer it from whitespace and punctuation alone — and the token count drops dramatically (57KB → 2KB structural digests). This improves code understanding, generation accuracy, and reduces hallucination of syntactically invalid code.
Folded Map Structural Folding
The folded-map approach transforms flat token streams into hierarchical spans by matching delimiter pairs ({ }). This produces a tree structure that represents the actual nesting in the source code.
Before (flat tokens):
[FN_DECL] [IDENT:"add"] [PAREN_OPEN] [PARAM_DECL] [PAREN_CLOSE] [BRACE_OPEN] [RETURN_EXPR] [BINARY_OP] [BRACE_CLOSE]
After (folded spans):
Live Demo
Run the fold demo to see structural folding in action:
Example output showing nested block folding:
Input: 18 raw tokens with nested braces
[ROOT]
KW(if) (if)
( (())
IDENT(x) (x)
) (()))
[BLOCK OPEN]
KW(let) (let)
IDENT(y) (y)
BINARY_OP (=)
INT_LIT (1)
; (;)
[BLOCK OPEN]
KW(return) (return)
IDENT(y) (y)
BINARY_OP (+)
INT_LIT (1)
; (;)
[BLOCK CLOSE]
[BLOCK CLOSE]
Folded into 3 nested span nodes
Generated API Documentation
The folded span tree can be traversed to automatically generate API documentation from source code. Run the api_docs example:
Example output for a Rust module with multiple functions:
## API Documentation
- `fn add(a: i32, b: i32) -> i32`
- `fn multiply(a: i32, b: i32) -> i32`
- `fn main()`
This demonstrates how folded spans enable automatic documentation generation from the structural understanding of code.
Why Structural Tokenization?
LLMs trained on raw BPE tokens must learn Rust's syntax implicitly. Structural tokenization makes this knowledge explicit:
FN_DECL— unambiguous function definition boundaryBRACE_OPEN/BRACE_CLOSE— explicit block delimitersIF_ELSE— control flow structure is first-classSTRUCT_FIELD— data layout boundaries are clearMATCH_EXPR— pattern matching expressions are distinct units
This matters for code generation, refactoring tools, and any agentic workflow that needs to understand Rust source.
Installation
[]
= "0.1"
Quick Start
use ;
let code = r#"
fn add(a: i32, b: i32) -> i32 {
a + b
}
"#;
let tokenizer = new;
let tokens = tokenizer.tokenize?;
// Count structural vs content tokens
let structural = tokens.iter
.filter
.count;
println!;
println!;
API Overview
Core Types
StructuralTokenizer— the tokenizer instance (usenew()orDefault::default())StructuralToken— a single token with kind, text, and source spanTokenKind— enum of ~90 distinct structural token kindsTokenizeError— parse errors from syn
Main Functions
let tokenizer = new;
// Tokenize Rust source into structural tokens
let tokens = tokenizer.tokenize?;
Token Kinds
The tokenizer produces ~90 distinct token kinds covering:
- Definitions:
FN_DECL,STRUCT_DECL,ENUM_DECL,TRAIT_DECL,IMPL_BLOCK - Control flow:
IF_ELSE,FOR_LOOP,WHILE_LOOP,MATCH_EXPR,BREAK_EXPR,CONTINUE_EXPR - Expressions:
CALL_EXPR,METHOD_CALL,FIELD_ACCESS,BINARY_OP,UNARY_OP - Literals:
INT_LIT,FLOAT_LIT,STR_LIT,CHAR_LIT,BOOL_LIT - Delimiters:
BRACE_OPEN/CLOSE,PAREN_OPEN/CLOSE,BRACKET_OPEN/CLOSE - Attributes:
MACRO_INVOCATION,ATTRIBUTE,LIFETIME,GENERIC_PARAMS
See the TokenKind enum for the complete list.
Use Cases
LLM Code Understanding (PESTI Runner)
Primary use case: feed structurally tokenized code to LLMs via PESTI's inference engine, giving models explicit syntactic context without requiring them to parse Rust grammar themselves.
Code Analysis Tools
Build static analysis tools that understand Rust structure:
// Find all function declarations
let fn_decls = tokens.iter
.filter
.;
Code Generation Pipelines
Generate Rust code with guaranteed syntactic correctness by emitting structural tokens first, then filling in content.
Educational Tools
Visualize Rust code structure for teaching/learning purposes.
Relationship to pesti-runner
This crate is a dependency of pesti-runner, PESTI's inference engine. However, it's designed to be useful independently for any tool that needs to analyze or generate Rust code with structural awareness.
Architecture
- Parsing:
syncrate parses source into complete AST (replaces hand-written recursive descent parser from v0.1.0) - Walking: Manual AST walker traverses items, statements, and expressions
- Emission: Structural tokens emitted based on node kind; function bodies elided except for declaration signatures
- Folded Map: Stack-based
{ }folding transforms flat token streams into nestedSpanobjects representing code structure
License
AGPL-3.0-or-later