Expand description
Structural tokenizer for Rust source code using syn.
Parses into AST via syn, then manually walks emitting semantic tokens.
Supports budget-aware collapsing for LLM context size management.
Structs§
- Budget
- Budget configuration for budget-aware tokenization.
- Elided
Span - Marker for a subtree that was elided due to budget constraints.
- Span
- Structural
Token - Structural token with position information
- Structural
Tokenizer - The structural tokenizer — uses syn to parse, then manually walks AST.
Enums§
- Emission
- Emission result — either a token or an elision marker.
- Node
Kind Tag - Tag for node kind used in elision summaries.
- Span
Item - Span
Kind - Token
Kind - Kinds of structural tokens we emit
- Tokenize
Error - Errors from the structural tokenizer