Module logos_lexer 

SIMD-accelerated lexer using Logos.

This module provides a fast tokenizer for Beancount syntax using the Logos crate, which generates a DFA-based lexer with SIMD optimizations where available.

Structs

Span

A span in the source code (byte offsets).

Enums

Token

Token types produced by the Logos lexer.

Functions

tokenize

Tokenize source code into a vector of (Token, Span) pairs for the AST-style parser.

tokenize_lossless

Tokenize source code losslessly: every byte of source appears in exactly one emitted (Token, Span) entry. This is the input to the CST builder.