Skip to main content

Module cursor

Module cursor 

Source
Expand description

The token buffer the parser reads.

Design: spec/06-lexer-and-parser.md section 6.3.

The input is the slice of tokens phase 7 produced, which always ends in one TokenKind::Eof. A cursor is a position in that slice and nothing more: it holds no parser state, it builds nothing and it reports nothing. That is what makes saving one and restoring it safe, and it is why the recovery skips in crate::recover take a cursor rather than the whole parser.

§Why the lookahead is bounded

Unbounded backtracking is how a C parser becomes quadratic on the input a fuzzer eventually finds, so Cursor::peek refuses to look further than MAX_LOOKAHEAD tokens ahead and panics rather than quietly widening the window. The bound is a budget rather than a fact about the grammar: a decision that cannot be made inside it is either a save and a restore, which is deliberate and rare, or a sign that the decision is being made in the wrong place, and the panic is how that conversation starts.

Structs§

Cursor
A position in the token stream.
Mark
A saved position, taken by Cursor::save and given back to Cursor::restore.

Constants§

MAX_LOOKAHEAD
How far ahead Cursor::peek will look.