Skip to main content

Crate pesti_structural_tokenizer

Crate pesti_structural_tokenizer 

Source
Expand description

Structural tokenizer for Rust source code using syn. Parses into AST via syn, then manually walks emitting semantic tokens. Supports budget-aware collapsing for LLM context size management.

Structs§

Budget
Budget configuration for budget-aware tokenization.
ElidedSpan
Marker for a subtree that was elided due to budget constraints.
Span
StructuralToken
Structural token with position information
StructuralTokenizer
The structural tokenizer — uses syn to parse, then manually walks AST.

Enums§

Emission
Emission result — either a token or an elision marker.
NodeKindTag
Tag for node kind used in elision summaries.
SpanItem
SpanKind
TokenKind
Kinds of structural tokens we emit
TokenizeError
Errors from the structural tokenizer

Functions§

fold