Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Tokenizer for Scheme source code

Corresponds to OpenJade’s SchemeParser lexical analysis methods.

Whitespace Handling

Critical difference from Steel: This tokenizer is fully whitespace-agnostic. Whitespace can appear anywhere between tokens without affecting parsing.

Example that breaks Steel but works here:

(let ((x 1)
      (y 2))  ; Multi-line let bindings
  (+ x y))

Implementations

impl Tokenizer

pub fn new(input: &str) -> Self

Create a new tokenizer from source code

pub fn position(&self) -> Position

Get current position (for error reporting)

pub fn next_token(&mut self) -> ParseResult<Token>

Get the next token

pub fn peek_token(&mut self) -> ParseResult<&Token>

Peek at the next token without consuming it