Struct StackedTransaction 

pub struct StackedTransaction<'txn, 'inp, 'closure, L, Ctx, Lang: ?Sized = (), P: DropPolicy = Rollback, Cmpl = Complete>
where
    L: Lexer<'inp>,
    L::State: Clone,
    Ctx: ParseContext<'inp, L, Lang>,
    Cmpl: Completeness,
{ /* private fields */ }

Available on crate features alloc or std only.

A scoped backtracking transaction that holds several live savepoints at once, mirroring SQL savepoint semantics.

The lean Transaction captures a single begin point; StackedTransaction adds an internal last-in, first-out stack of savepoints so a parser can keep several fallback positions live simultaneously and return to any of them. Reach for it when a single alternative is not enough — best- or longest-match selection (mark a savepoint after each parsed stage, score them, then rollback_to the winner), multi-segment speculation with fallback to any earlier boundary, or recovery scans juggling several anchor candidates. For a single speculative alternative, prefer Transaction; for closure-shaped speculation, attempt / try_attempt.

SQL savepoint semantics

The four operations map onto SQL exactly:

this type	SQL	effect
savepoint	SAVEPOINT	mark the current position, return an id
rollback_to	ROLLBACK TO	return to a mark, destroy the younger savepoints, keep the mark
release	RELEASE SAVEPOINT	forget a mark and the younger ones, keep the parsed progress
commit	COMMIT	keep everything, forget all savepoints
rollback	ROLLBACK	return to the begin point, discard everything

Rolling back to an older savepoint always destroys every newer one, so out-of-order revival is impossible by construction — the restore discipline holds because the internal stack only ever shrinks from the top. A misused SavepointId is rejected in layers: a temporally-misused id at compile time via its lifetime brand, a foreign id from another live parser and a stale id both by a runtime check in every build — see SavepointId.

commit and rollback consume the transaction and are available whatever the drop policy. What an undecided transaction does on drop is the compile-time DropPolicy P: the default Rollback (from begin_stacked) rolls back to the begin point, discarding all savepoints — the database default; Commit (from begin_stacked_with) keeps the progress. Cost when unused is low: the transaction’s own savepoint Vec never allocates until the first savepoint, and a begin captures one field address and records its base checkpoint on the input’s shared lineage stack (an amortized Vec push) — no counter, no atomic.

Mixing with raw save/restore, state surgery, and nested transactions

The guard deref-coerces to InputRef, so raw save / restore and the nested backtracking tools are all reachable through it. These rules govern how they interact with the live savepoints and the begin point:

• A raw restore below a savepoint (but above the base) invalidates the savepoint — detect-at-use. Savepoints are not pinned (only the base is), so this restore succeeds; it rolls the lineage back past the savepoint’s own checkpoint, so the savepoint is no longer on a live lineage, and rollback_to / release with it panics as stale in every build — release and no-target_has_atomic-ptr targets included. Restoring the wrong lineage is never silently honored.
• A raw restore below the begin point would tear out the whole transaction — detect-at-cause. Restoring a raw checkpoint taken before begin_stacked would pop the pinned base off the live lineage, so in allocator builds it panics at the restore itself (restore would invalidate a live transaction guard or attempt …) — refused where it is caused, before any commit/rollback decision. On allocator-less targets there is no pin set, so this is unspecified-but-bounded rather than checked; in allocator builds the older detect-at-use backstops (an explicit rollback asserting a live base, a rolling-back drop skipping a stale one) remain as defense in depth behind the pin check.
• State surgery, nested attempt / try_attempt / Transaction, and a LIFO-clean raw save/restore pair taken above the savepoints, are all legal and do not disturb the savepoints. set_state / state_mut re-key the forward-scanning facts but are transactional — a savepoint taken before the surgery stays valid, and rollback_to it undoes the surgery (the regime, boundary, watermark, and position all return). A nested speculation that saves and then restores or commits its own younger checkpoint leaves every savepoint below it untouched.

The raw-restore rules above are reachable only with the unstable-raw feature. Without it, raw save / restore are crate-internal, so a downstream crate cannot mix a raw restore into a live transaction at all — only the savepoint operations and nested guards remain, and none of those can invalidate a savepoint.

// Best-match selection across three stages: keep a fallback after each, then return
// to the highest-scoring one and resume from exactly there.
let mut txn = input.begin_stacked();

let mut best = None;
let mut best_score = i32::MIN;
for _ in 0..3 {
  let score = parse_one_stage(&mut txn);         // parse through the guard (DerefMut)
  let sp = txn.savepoint();                       // fallback point after this stage
  if score > best_score {
    best_score = score;
    best = Some(sp);
  }
}

if let Some(sp) = best {
  txn.rollback_to(sp);   // resume right after the best stage; younger savepoints die
}
txn.commit();            // keep the winning prefix

Implementations

impl<'txn, 'inp, L, Ctx, Lang: ?Sized, P: DropPolicy, Cmpl> StackedTransaction<'txn, 'inp, '_, L, Ctx, Lang, P, Cmpl>

where L: Lexer<'inp>, L::State: Clone, Ctx: ParseContext<'inp, L, Lang>, Cmpl: Completeness,

pub fn savepoint(&mut self) -> SavepointId<'txn>

Marks the current position as a savepoint and returns its id (SQL SAVEPOINT).

The returned SavepointId stays usable for rollback_to and release until an older savepoint destroys it or it is released. Its lifetime is branded to this transaction, so it cannot escape the transaction’s scope.

pub fn rollback_to(&mut self, sp: SavepointId<'txn>)

Rolls back to sp (SQL ROLLBACK TO): returns the input to sp’s position — cursor, span, lexer state, emission log, dedup watermark, and poison boundary all restored — and destroys every savepoint created after it, while keeping sp itself valid for a later rollback.

Checkpoint is single-use, so keeping sp reusable is done by restoring the stored checkpoint and immediately re-saving at the now-current position, swapping the fresh checkpoint into sp’s slot. This preserves the classic SQL loop of rolling back to the same savepoint any number of times; it costs one extra O(1) save per call on this cold path, plus one settle per destroyed savepoint (the same per-entry work release and commit already pay).

Panics

Panics if sp was issued by a different, simultaneously-live transaction (stacked transaction: savepoint belongs to a different transaction), was destroyed by an earlier rollback_to / release (stacked transaction: savepoint is stale (destroyed by an earlier rollback or release)), or had its checkpoint invalidated by a raw restore below it through the transaction (stacked transaction: savepoint is stale (invalidated by a raw restore below it)). All three checks — an address compare and two short stack scans — run in every build. (Using an id after its transaction ended is a compile error, not a panic; see SavepointId. State surgery is transactional and does not invalidate a savepoint — one taken before it stays valid, and rolling back to it undoes the surgery.)

pub fn release(&mut self, sp: SavepointId<'txn>)

Releases sp (SQL RELEASE SAVEPOINT): forgets sp and every savepoint created after it, keeping the parsed progress. The input position does not move.

Panics

Same as rollback_to: a foreign or already-destroyed id panics.

impl<'inp, L, Ctx, Lang: ?Sized, P: DropPolicy, Cmpl> StackedTransaction<'_, 'inp, '_, L, Ctx, Lang, P, Cmpl>

where L: Lexer<'inp>, L::State: Clone, Ctx: ParseContext<'inp, L, Lang>, Cmpl: Completeness,

pub fn commit(self)

Commits the whole transaction: keeps every parsed byte and forgets all savepoints and the begin point without restoring. Available whatever the drop policy.

pub fn rollback(self)

Rolls the whole transaction back to the begin point, discarding every savepoint and all parsed progress. Available whatever the drop policy (a Commit guard can still be rolled back explicitly).

Methods from Deref<Target = InputRef<'inp, 'closure, L, Ctx, Lang, Cmpl>>

pub fn consume_cached_one(&mut self) -> Option<Spanned<L::Token, L::Span>>

Consumes one token already lexed and waiting at the front of the stream, returning it if there is one; the cursor is advanced.

pub fn consume_cached_to<F>( &mut self, f: F, ) -> Option<Spanned<L::Token, L::Span>>

where F: FnMut(CachedTokenRefOf<'_, 'inp, L>) -> bool,

Consumes tokens already lexed and waiting at the front of the stream until the predicate returns true.

Advances the cursor to the end of the last consumed token. Returns the last consumed token.

pub fn consume_cached_while<F>( &mut self, f: F, ) -> Option<Spanned<L::Token, L::Span>>

where F: FnMut(CachedTokenRefOf<'_, 'inp, L>) -> bool,

Consumes tokens already lexed and waiting at the front of the stream while the predicate returns true.

Advances the cursor to the end of the last consumed token. Returns the last consumed token.

pub fn consume_all_cached(&mut self) -> Option<Spanned<L::Token, L::Span>>

Consumes every token already lexed and waiting at the front of the stream — a parked one included.

Advances the cursor to the end of the last of them. Returns the last consumed token.

Drains per token through consume_cached_to (with a never-matching predicate), so every retained token — not only the last — settles through the one commit primitive. The observable result is unchanged: the front empties, the cursor lands at the end of the last retained token with its state, and the last token is returned; but each token in the run commits individually, exactly as it would have had the caller consumed them one by one.

pub fn recursion(&self) -> &RecursionLimiter

The recursion budget this parse descends against: the live depth and the limit it may not exceed.

Read-only, deliberately. There is no recursion_mut: the cell has exactly one writer, the Descent guard descend hands out, which is what makes “every level entered is a level left” a property of the type system on every exit path rather than of caller discipline — short of leaking the guard instead of dropping it, which is Rust’s universal mem::forget caveat and is covered on Descent. Balance is the property that buys; where a level is left is the guard’s scope, and that is the caller’s to place — unless the scope is descending’s closure, which places it for you. See Descent.

Configure it with InputContext::with_recursion_limiter or ParserContext::with_recursion_limiter.

pub fn descending<F, T, E>(&mut self, f: F) -> Result<T, E>

where F: FnOnce(&mut Self) -> Result<T, E>, E: From<RecursionLimitReached<L::Offset, Lang>>,

Runs f one level of recursive descent deeper, or fails terminally if the configured limit is exceeded.

This is the form to write a recursive combinator in. The level is raised before f runs and released when f returns, propagates with ?, or unwinds — and nothing f can write releases it earlier, because f is handed the input and never the guard, and recursion is read-only. The scope of the level and the extent of the frame are therefore the same region by construction, rather than by the caller placing a binding correctly.

use tokora::{
  Emitter, InputRef, Lexer, ParseContext,
  error::RecursionLimitReached,
};

/// A recursive production that counts against the parse's shared depth budget.
fn nested<'inp, L, Ctx>(
  inp: &mut InputRef<'inp, '_, L, Ctx>,
  remaining: usize,
) -> Result<usize, <Ctx::Emitter as Emitter<'inp, L>>::Error>
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
  <Ctx::Emitter as Emitter<'inp, L>>::Error: From<RecursionLimitReached<L::Offset, ()>>,
{
  inp.descending(|inp| match remaining {
    0 => Ok(inp.recursion().depth()),
    n => nested(inp, n - 1),
  })
}

Errors thread through untouched, so ? composes

f returns the frame’s own Result<T, E> and it is returned unchanged, exactly as try_attempt threads its closure’s error. E is not tied to the emitter’s error type — the trip is returned, never emitted, so it is built directly as E through the From bound, which the frame’s error type satisfies for the same reason descend’s does.

Write the whole frame body as the closure and its returns keep their meaning: the closure returns Result<T, E> and this method returns it verbatim, so return Err(e) and return Ok(v) still leave the frame. What a closure cannot host is a return meant for an enclosing function, or a break/continue aimed at a loop outside it; a body shaped that way is what descend is still public for.

If f panics

The level is released on the unwind, by the guard’s destructor, in std and no_std alike — the same edge descend covers, and for the same reason. A host that catches the unwind is handed an input whose depth is what it was before the call.

What one level means

The same thing it means for descend: whatever the caller says it means, counted per call and shared by every parser on this input.

pub fn descend( &mut self, ) -> Result<Descent<'_, 'inp, 'closure, L, Ctx, Lang, Cmpl>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<RecursionLimitReached<L::Offset, Lang>>,

Enters one level of recursive descent, or fails terminally if the configured limit is exceeded — the low-level escape hatch under descending.

Reach for descending instead unless the frame’s body cannot be a closure — because it must return out of an enclosing function, or break/continue a loop outside it, or because it is large enough that relocating it is a change in its own right. This method hands the level back as an ordinary value, so where the level ends is caller code, and four measured spellings end it before the recursion it was taken for; only one of the four warns. The list, the measurements and the reason no rule can close it are on Descent.

The returned Descent guard is the level: it derefs to this handle, so the frame’s body runs through it, and leaving its scope — by return, by ?, or by an unwind — releases the level. There is no matching “ascend” to forget.

let mut frame = inp.descend()?;   // one level, for as long as `frame` lives
let inp = &mut *frame;            // the body below is unchanged

Bind it. There is no “ascend” to forget, but there is a scope to place wrong: inp.descend()?; as a bare statement compiles, because ? takes the Result apart and leaves the guard a temporary that dies at the semicolon — releasing the level before the recursion it was taken for, and putting the native stack back at risk with the budget reading zero. Descent is #[must_use] so that one line warns, and a warning on that one line is all it is: see what the guard does and does not enforce.

What one level means

Whatever the caller says it means — the primitive counts calls, not grammar constructs. The two Pratt engines call it once at each frame prologue, so for them the budget reads “live pratt frames on this input”, the root expression included. The budget is a property of the input session, not of a parser, so two recursive parsers composed into one grammar draw on one depth: that is what makes it a bound on native stack use rather than on any single production.

On Err, the depth is exactly what it was

The trip path lowers the depth again before it builds the error, so a caller that catches the failure and parses something else finds the cell describing only the frames that are genuinely live. Building the error is the grammar’s own From, and it runs after the decrement, so even a panicking conversion cannot leak a level.

The error is terminal, and it is returned

RecursionLimitReached is terminal for every value: no amount of further input clears a depth budget, so Recover, InplaceRecover and skip_then_retry re-raise it rather than synthesizing a node. It is returned, never emitted, so no rewind can erase it and no recording emitter can turn a tripped budget into a truncated-but-successful parse. Its offset is committed consumption at frame entry — cache-independent, so a prefilled lookahead window trips at the same place an empty one does.

Terminality is stored on the input, not in the error payload. The trip arm counts the trip on Input::resource_trips before it builds anything the grammar can see, and the three combinators above consult that cell in addition to MaybeTerminal::is_terminal on the converted value. So the re-raise holds for a grammar error that stores the value and delegates is_terminal, and equally for one whose From discards it — () included. A discarding sink loses the payload; it does not lose the stop, because the stop was never the payload’s to carry. That is invariance, not new semantics: across those three combinators it makes every supported sink behave the way a delegating one already did.

The resilient collection loops — repeated, separated and their delimited forms — read the same cell, and there it is not a second opinion but the only one: those swallow arms carry no MaybeTerminal bound, so it is the whole of what stops an element’s trip being emitted as a diagnostic and looped past. That one is not invariance — it was spent for every error type before #148 — so it changes those families’ behaviour on a trip. See Input::resource_trips and parser::many’s GATE_CENSUS.

What every one of those sites tests is a per-attempt transition, not the session fact. The cell is monotone and is never cleared, so reading it absolutely would say “this parse has tripped” forever after the first trip — and refuse recovery, and refuse emit-and-continue, for every later failure including ordinary syntax errors in unrelated constructs. Each site therefore takes a trip_snapshot before the attempt it is judging and asks tripped_during_attempt after it — the same pair a consumer outside this crate now reads. A real trip is still re-raised wherever it actually happens, including a second one after grammar code caught the first.

What is recorded is the fact that the budget was exceeded, and not the depth. A scanner limit trip latches the poison boundary because the lexer’s tally is monotone in the input; descent depth is the opposite kind of fact and is fully restored by the unwind that carries the error out, so it is not latched and must not be. That a budget was once exceeded is monotone in exactly the scanner’s sense — no unwind, no rollback and no further input can un-exceed it — so it is latched, and latched for the parse’s remaining life. The two latches differ in what they record: a scanner trip has a position, so it latches where; a descent trip has only a control stack, so it latches whether. See RecursionLimitReached.

Example

descending’s example, written out by hand — which is all the guard form is. Keep the binding and the shadowing let inp = &mut *frame; together, because that pair is what makes every line below reach the input through the level.

use tokora::{
  Emitter, InputRef, Lexer, ParseContext,
  error::RecursionLimitReached,
};

/// A recursive production that counts against the parse's shared depth budget.
fn nested<'inp, L, Ctx>(
  inp: &mut InputRef<'inp, '_, L, Ctx>,
  remaining: usize,
) -> Result<usize, <Ctx::Emitter as Emitter<'inp, L>>::Error>
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
  <Ctx::Emitter as Emitter<'inp, L>>::Error: From<RecursionLimitReached<L::Offset, ()>>,
{
  let mut frame = inp.descend()?;
  let inp = &mut *frame;
  match remaining {
    0 => Ok(inp.recursion().depth()),
    n => nested(inp, n - 1),
  }
}

pub fn trip_snapshot(&self) -> ResourceTripBaseline<'closure>

Snapshots the session’s resource-trip counter — how many times a resource budget has been exceeded in this input session so far — for an attempt-relative terminality witness.

The descent twin of the scanner’s latch_snapshot, and used exactly as scanner_trip_snapshot is: take the baseline once per attempt, hand it back to tripped_during_attempt when judging that attempt’s failure.

The counter itself is a monotone session fact, counted up by the trip arm behind descend and never lowered: a Checkpoint does not carry it, a restore does not touch it, and Descent’s Drop releases only the depth. That arm is its one writer and no method on this handle hands out a mutable route to the cell, so a consumer can read this witness and cannot forge it.

The value is a baseline, and that is all the type lets it be

ResourceTripBaseline is opaque: no accessor, no PartialEq, no constructor. Hand it back to tripped_during_attempt and nothing else is expressible — not the difference of two baselines, not a trip count, and not the session-absolute reading below. The contract used to be a sentence asking callers not to do those things; it is now the type refusing to let them.

The session-absolute reading is the one that had to become unspellable. trip_snapshot() != 0 would be “did this parse exceed a budget at all” — a true statement about the session, and deliberately not the question any consulting site has. It stays true forever once grammar code catches one trip and carries on, so a site that reads it mid-parse charges every later failure — an ordinary syntax error in an unrelated construct included — with a stop that is already over, and one deep construct early in a document suppresses every diagnostic after it. tokora/tests/root_loop_trip_witness.rs measures what that costs. The question itself is legitimate after the parse, where there is no later attempt to poison, and it has a published answer there: cst::Cst::resource_trips (feature rowan).

A baseline cannot leave the handle that issued it

It carries the handle’s 'closure, and a parser reaches its input through a universally quantified handle lifetime, so there is no region a caller can name that one could be parked in. That is the shape of carrying a baseline across a PartialSession redrive — which builds a fresh input and a fresh counter starting at zero, so the carried baseline would compare against a different cell and the next input’s first trip would read as “nothing happened”.

The control first, because a compile_fail that fails for the wrong reason proves nothing. The identical function stashing an ordinary usize read off the same handle compiles, so the scaffolding — the Cell, the bounds, the borrow of inp — is sound:

use core::cell::Cell;
use tokora::{Emitter, InputRef, Lexer, ParseContext};

fn stash_a_number<'inp, L, Ctx>(
  inp: &mut InputRef<'inp, '_, L, Ctx>,
  cell: &Cell<Option<usize>>,
) where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
{
  cell.set(Some(inp.recursion().depth()));
}

Change only the value’s type, and it stops compiling — '1 must outlive 'static:

use core::cell::Cell;
use tokora::{Emitter, InputRef, Lexer, ParseContext, input::ResourceTripBaseline};

fn stash_a_baseline<'inp, L, Ctx>(
  inp: &mut InputRef<'inp, '_, L, Ctx>,
  cell: &Cell<Option<ResourceTripBaseline<'static>>>,
) where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
{
  // error[E0521]: borrowed data escapes outside of function
  cell.set(Some(inp.trip_snapshot()));
}

The legal use is the one the loop actually needs, and it compiles:

use tokora::{Emitter, InputRef, Lexer, ParseContext};

fn judge_one_attempt<'inp, L, Ctx>(inp: &mut InputRef<'inp, '_, L, Ctx>) -> bool
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
{
  let trips = inp.trip_snapshot();
  // … the attempt this baseline judges runs here …
  inp.tripped_during_attempt(trips)
}

The realistic cross-wire is refused too, and it is the shape a driver holding two inputs alive would actually write: a parser that runs a nested parse and judges the inner input with the outer input’s baseline. Compiled and refused with E0521: borrowed data escapes outside of function. The theorem underneath both refusals is that a baseline fixed at one region cannot be handed to a frame that demands every region:

Its control first, again: the same closure capturing a region-free usize off the same handle satisfies the same bound, so the refusal below is the value’s type and not the shape.

use tokora::{InputRef, Lexer, ParseContext};

fn as_a_parser_over_a_number<'inp, 'closure, L, Ctx>(
  outer: &mut InputRef<'inp, 'closure, L, Ctx>,
) -> impl for<'c> FnMut(&mut InputRef<'inp, 'c, L, Ctx>) -> bool
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
{
  let depth: usize = outer.recursion().depth();
  move |inner: &mut InputRef<'inp, '_, L, Ctx>| inner.recursion().depth() == depth
}

use tokora::{InputRef, Lexer, ParseContext, input::ResourceTripBaseline};

/// The closure a nested parse would be handed: it must work at EVERY handle region, and the
/// captured baseline is fixed at one.
fn as_a_parser<'inp, 'closure, L, Ctx>(
  outer: &mut InputRef<'inp, 'closure, L, Ctx>,
) -> impl for<'c> FnMut(&mut InputRef<'inp, 'c, L, Ctx>) -> bool
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
{
  let base: ResourceTripBaseline<'closure> = outer.trip_snapshot();
  move |inner: &mut InputRef<'inp, '_, L, Ctx>| inner.tripped_during_attempt(base)
}

A hand-written ParseInput impl is refused too, and it is the shape a parser combinator carrying a baseline in a field would take. Its control first — the same impl carrying a region-free usize — which compiles:

use tokora::{Emitter, InputRef, Lexer, ParseContext, ParseInput};

struct Control {
  depth: usize,
}

impl<'inp, L, Ctx> ParseInput<'inp, L, bool, Ctx> for Control {
  fn parse_input(
    &mut self,
    input: &mut InputRef<'inp, '_, L, Ctx>,
  ) -> Result<bool, <Ctx::Emitter as Emitter<'inp, L>>::Error>
  where
    L: Lexer<'inp>,
    Ctx: ParseContext<'inp, L>,
  {
    Ok(input.recursion().depth() == self.depth)
  }
}

and the baseline in the same field position, which does not — the method’s handle region is fresh for every call, and the field’s is fixed by the struct:

use tokora::{Emitter, InputRef, Lexer, ParseContext, ParseInput, input::ResourceTripBaseline};

struct Probe<'closure> {
  base: ResourceTripBaseline<'closure>,
}

impl<'inp, 'closure, L, Ctx> ParseInput<'inp, L, bool, Ctx> for Probe<'closure> {
  fn parse_input(
    &mut self,
    input: &mut InputRef<'inp, '_, L, Ctx>,
  ) -> Result<bool, <Ctx::Emitter as Emitter<'inp, L>>::Error>
  where
    L: Lexer<'inp>,
    Ctx: ParseContext<'inp, L>,
  {
    // error: lifetime may not live long enough
    Ok(input.tripped_during_attempt(self.base))
  }
}

The invariance rails are different cells, because the three shapes above are refused under either variance — see ResourceTripBaseline for the measured map. Two shapes the brand alone decides. The bare coercion:

use tokora::input::ResourceTripBaseline;

// error: lifetime may not live long enough — invariant, so `'long` cannot shrink
fn shrink<'long: 'short, 'short>(
  b: ResourceTripBaseline<'long>,
) -> ResourceTripBaseline<'short> {
  b
}

and the one that makes it matter — a generic adapter holding the handle at one fixed region, so InputRef’s own invariance never enters and only the baseline is asked to coerce. Its control first, the same adapter over a region-free usize:

use tokora::{Emitter, InputRef, Lexer, ParseContext};

fn adapt_a_number<'inp, 'short, L, Ctx>(
  inp: &mut InputRef<'inp, 'short, L, Ctx>,
  depth: usize,
) -> bool
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
{
  inp.recursion().depth() == depth
}

and the baseline in the same position, which does not compile — and would compile under a covariant brand, which is the whole of why the brand is not prophylaxis:

use tokora::{Emitter, InputRef, Lexer, ParseContext, input::ResourceTripBaseline};

fn adapt<'inp, 'long: 'short, 'short, L, Ctx>(
  inp: &mut InputRef<'inp, 'short, L, Ctx>,
  base: ResourceTripBaseline<'long>,
) -> bool
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L>,
{
  // error: lifetime may not live long enough — only the baseline is coerced here, so this is
  // the brand refusing and not the handle's own invariance
  inp.tripped_during_attempt(base)
}

What the region parameter does not catch is two inputs alive at once inside this crate, where handles are minted directly and their regions unify. That is a programmer error rather than a parse outcome, and tripped_during_attempt panics on it instead of answering. ResourceTripBaseline has every half and says which was measured how.

Why a pair, and not one guard that takes the snapshot for you

A guard — one call that opens the attempt, runs it and answers — is harder to misuse, and this crate has one: parser::recovery_gate’s attempt chokepoint, which owns the closure its combinator hands it and therefore owns the unit the baselines belong to. It is not the shape this crate’s loops use. parser::many’s twelve drivers take this baseline by hand, inside their element loop, and pass it to a chokepoint that reads it — because the two baselines have opposite granularities, and neither can be derived from the other. This one is per element: hoisted out of the loop it is arithmetically the session-absolute read above. The scanner’s is per collection: taken per element it is re-read after each trip an element caught, so every later exit concludes cleanly over a budget that is spent. Both fusions are measured defects, and parser::many’s GATE_CENSUS pins both placements in both directions.

A guard that snapshots for you fixes both baselines to one unit, which is that fusion. So what this crate centralizes is the verdict, never the baseline, and the baseline is published as a value the caller places. A consumer whose unit is a closure can still build the guard on top of this pair; a consumer whose unit is a region of its own loop could not have built this pair on top of a guard.

Why it is public, and why it is inherent

tripped_during_attempt carries the first argument.

Inherent rather than an opt-in trait, deliberately. An inherent item wins the method pick over an extension trait’s, so adding these two names to a type that already shipped can take a call away from a consumer who wrote either name on InputRef — silently, where the paired let b = inp.trip_snapshot(); inp.tripped_during_attempt(b) compiles on both sides and infers the argument. A trait would avoid that and cost an import. The form is chosen anyway, because InputRef carries its whole surface as inherent methods and a witness that had to be imported would be unreadable inside a generic production that imports nothing — an opt-in trait for these two alone would be the only such accessor in the crate. The cost is paid where this crate pays it: disclosed in the CHANGELOG’s “no diagnostic at the call site” section, with the UFCS spelling that restores the consumer’s item, and classified in ci/name_collision/no_collision.txt with what that classification does not establish.

What it costs, on both word sizes

A u64 load and one address read per attempt, and nothing anywhere else: no scan, no lookahead fill and no token commit reads or writes either. The count is one machine word on a 64-bit target and two on a 32-bit one — it is u64 rather than usize for the reason input::TRIP_COUNTER_EXHAUSTED gives, and this is the price of it.

This call is on the hot path of every successful element, because the descent baseline is taken per element — so the price was measured rather than argued. Every figure below is thumbv6m-none-eabi at -O, the narrowest supported target, except where it says otherwise.

	usize	u64	delta	runs
this call	5 instr	7 instr	+2	per element
tripped_during_attempt	17 instr	22 instr	+5	0-2 per collection termination, by class — see its own table
an element loop (N of the first, one of the second)	31 instr	38 instr	+7	per collection
that loop’s stack frame	16 B	24 B	+8 B	per collection
ResourceTripBaseline itself	8 B, align 4	16 B, align 8	×2	carried per element

The layout row is the one an instruction count does not show: the baseline is Copy and passed by value, so a 32-bit element loop carries twice the value it used to. It is in the accounting because it was missed the first time this was priced.

Whole-parse, on wasm32-wasip1 — the widest 32-bit target that can be executed here, since i686 cannot link on this host — a collection-parse binary grew 33 bytes (120,421 → 120,454, +0.027%), and interleaved executed latency showed no regression in either workload shape: a long single collection at -5.8% min / -5.3% median, and many short collections — the shape the per-termination +5 makes expensive — at -0.35% min / -7.25% median, n=12 each. Both sides land at or below zero, which is an instrument that cannot resolve the change rather than a speedup.

What the executed measurement covers, stated rather than generalised: the probe is a non-delimited collection terminating by decline — one of the six classes in the table on tripped_during_attempt. Of the five it does not exercise, four are cheaper or equal per termination; which they are, and why, is that table’s to say. The one that is not cheaper has to be named here rather than deferred, because it is the bound on this measurement: a skip_then_retry workload pays both verdicts per successful sync or advance step, and that shape is unmeasured.

Accepted on that scope: two instructions and eight stack bytes per element loop, against an element that has already run a cache probe or a full lex and commit, buys a counter a 32-bit loop cannot exhaust. Keeping count at usize and moving the exhaustion distinction to a second cell would buy back the four bytes and add a load at the same sites, and the executed measurement gives it nothing to recover.

pub fn tripped_during_attempt( &self, since: ResourceTripBaseline<'closure>, ) -> bool

Whether a resource budget was exceeded during the attempt that took since as its trip_snapshot baseline.

The input-side witness for resource terminality, and the answer to “did the failure I am judging come from a tripped budget”. Read by Recover, InplaceRecover and skip_then_retry beside MaybeTerminal::is_terminal, and by the four resilient collection loops (repeated, separated and their delimited forms) on its own, since those carry no MaybeTerminal bound. Either way a grammar error type that discards RecursionLimitReached on conversion — () does — still cannot turn a tripped budget into recovery, nor into a diagnostic the collection keeps parsing past.

Attempt-relative, not session-absolute — the same discipline the scanner’s latched_during_attempt applies to its latch, and for the same reason. The cell it reads is monotone, so an absolute reading answers “has this parse ever tripped”, which is true forever once grammar code catches a trip and carries on. Every later failure in the session — an ordinary syntax error in an unrelated construct included — would then be re-raised as though the budget had stopped it, and a single deep expression early in a document would suppress every diagnostic after it. Comparing against the baseline asks the question the site actually has: this attempt, not this parse.

A count rather than a bool is what makes the comparison hold up a second time. A set-once flag compares equal to a baseline taken after an earlier caught trip, so the next genuine trip would read as “nothing happened here” — narrowing the witness into a hole. The counter changes on every trip, so != is exactly “a trip happened inside this attempt”.

The granularity floor: one attempt, and it fails closed

This answers “a trip happened while the attempt ran”. It does not answer “the Err I am holding is that trip”. The two come apart inside a single attempt: grammar code that catches a trip itself, carries on, and then fails ordinarily before the attempt ends leaves the counter moved, and the ordinary failure is re-raised as though the budget had stopped it.

So the resolution of this witness is one attempt — one speculative parse for Recover and InplaceRecover, one retry cycle for skip_then_retry, one element for the resilient collection loops. Within that unit the verdict fails closed at the recovery, failure, absence and real-closer gates: an ordinary failure sharing its unit with a caught trip is re-raised, never the reverse, and a real trip reaching one of those gates is never recovered from and never filed as a diagnostic. It says nothing about Accept — an element that catches the trip and still answers Accept spends it, for every error type, on purpose; see parser::many’s module docs, “the channel neither chokepoint closes” section. Outside the unit nothing is charged at all, which is the whole point of taking a baseline.

The strong form is not implementable at this layer. Deciding whether the error in hand is the trip means interrogating its value, and the grammar’s error type may be (), whose From discards RecursionLimitReached entirely — a sink that discards is the reason this witness lives on the input instead of in the error. Any design claiming to tell the two apart is either reading a payload that is not there, or is the escape hatch below wearing a different name.

The escape hatch, if a consumer ever needs one: an explicit rebaseline — code that deliberately catches a trip declaring it settled, so the enclosing baselines move past it and the attempt is judged only on what happens after. That is a cooperative operation, and it is the design to build if the floor ever costs somebody something real. It is not built: no consumer needs it yet, and this crate does not publish API on speculation.

tokora/tests/collection_resource_trip.rs and tokora/tests/pratt_limit_unit_sink.rs each pin one cell on this behaviour, paired against the cell that moves the catch outside the unit and gets the opposite answer.

Why it is public, and why its scanner twin is not

This crate does not publish API on speculation, and this pair was crate-private for as long as the only sites judging an attempt were its own. What changed is a consumer with the defect. smear’s GraphQL parser catches a failed definition at each document root and has to decide whether that failure ends the document (al8n/smear#169); it decided by reading the error, which is the decision this counter exists to replace. A nesting refusal that reached a root loop as an ordinary error resynchronised, re-read the abandoned nest at document level, and reported one diagnostic per remaining token — 67 for one refusal at 66 levels, 804 at 800, growing with the document.

That repair took three rounds because every round left the verdict resting on something a caller implements. MaybeTerminal::is_terminal is the grammar’s own answer, and a From that discards RecursionLimitReached — () does — answers false over a real trip. A latch of the parser’s own would have to live in L::State or beside the poison boundary, and a Checkpoint carries both, so a speculative rollback refunds it. This cell is neither: written by the trip arm before any grammar code runs, outside the rollback set, one writer, no mutable route.

The scanner twin was published in the same change and withdrawn before release, and the asymmetry is the point rather than an accident of scope. That counter has a second public contract it cannot see — set_state drops the poison boundary as the crate’s documented limit-recovery path, and never touches the counter — so a loop following its own documentation can recover, read a whole document, and still be told it was truncated. InputRef::scanner_trip_snapshot carries that measurement, and also the narrower statement that replaced an earlier overclaim: try_expect_or_stop covers the declining exits and is not a replacement for that pair, because a rejecting emitter’s trip is built and propagated from inside it. No public witness answers that path today, and none did before this change either. Nothing analogous exists here, and the reason is the channel: a descent refusal is returned by descend and never routed through an emitter, so no emitter can unmark it or convert it away from the counter that already recorded it — the_descent_witness_holds_under_a_rejecting_emitter measures exactly that. No public API clears or re-keys the descent counter either: a budget once exceeded cannot be un-exceeded by more input, by an unwind, or by a rollback, and the one cooperative escape hatch — the rebaseline above — is deliberately not built. The witness a consumer can rely on is exactly the one with no second contract to conflict with.

What is published is the reading. There is no writer here, no way to lower the counter, and no rebaseline: the granularity floor above is the floor for a consumer too. tokora/tests/root_loop_trip_witness.rs is the outside-the-crate use, including the cell that measures what the amplification costs a root loop with no witness and the cell that measures what a baseline hoisted out of the loop costs one that has it.

A baseline from another input is a panic, not a verdict

It panics if since was issued by a different InputRef, and the alternative was considered and rejected: answering true — “fail closed”, the direction every real reading of this witness fails in — is the wrong answer for this one, because the two failures do not cost the same thing. A spurious true tells a root loop its attempt tripped, and a root loop told that ends a document that was fine and discards the valid suffix. The truncated parse still returns Ok, so the mistake survives testing and points at nothing. That is the identical failure shape as the state-recovery residue that kept the scanner twin crate-internal, and it would be indefensible to build it in here on purpose.

A cross-fed baseline is a programmer error, not a parse outcome. It cannot be produced by input, hostile or otherwise — only by code that wired two inputs together — and no consumer can write it at all: the region parameter refuses every public path, measured, and Input is crate-internal so there is no other way to hold two handles whose regions unify. What the panic guards is this crate’s own sites, and any future door that hands out a handle outside a closure boundary: it announces itself at the crossing instead of silently truncating, and it is distinguishable from a real trip by not being a bool at all.

The check is one word-pair comparison on a path that already runs one, and its branch is cold.

When each verdict runs — the table, derived from every call site

This is the one copy of this model. Three previous versions of it were each written from the call sites the author happened to look at, and each was wrong in a different way: “on the failure arm only” missed the clean exits, “on every normal termination” missed that some terminations evaluate nothing and some evaluate only half, and both missed the two recovery_gate sites entirely. So this is enumerated from the code: there are exactly five places in the crate that evaluate either verdict, and the six termination classes fall out of them rather than being asserted beside them.

class	site	descent	scanner	reached when
failed element	many::file_element_failure	4th term	3rd term	an element returned Err. Both sit behind is_incomplete_error and at_committed_boundary, so either can be short-circuited away
decline or stall	many::absence_after_element	3rd term	1st term	a driver concludes absence. The scanner verdict always runs; the descent one only if the scanner and latch reads are both false
probed closer	many::close_after_element	only term	never	a probe verdict handed over a real closer. The scanner reading is deliberately absent — a pre-trip closer settles the position question and only that
direct closer	the mid-scan arms of sep/delim and sep_while/delim	never	never	a closer committed straight from the driver’s own scan. The cycle’s final trip_snapshot is paid and no verdict is
recovery failure	parser::recovery_gate::judge	3rd term	4th term	a recovery attempt returned Err. Both sit behind is_incomplete() and is_terminal()
successful recovery step	parser::recovery_gate::recovery_step	1st term	2nd term	a skip_then_retry skip or advance succeeded. The descent verdict always runs — the one place either runs after something worked

Two things the table does not contain, because they are what it is for. A verdict is absent from an accepted, progressing element — no row covers that arm, and its absence is the point of taking the baseline per element. And there is no single frequency: read the rows for what each class costs rather than taking a number from here, because anything stating one number per collection is describing one row and calling it the model.

Costs a nonce comparison and a u64 comparison. Measured on thumbv6m-none-eabi, -O: 17 instructions at usize, 22 at u64. trip_snapshot carries the whole cost table and the scope the measurement covers.

Panics

If since came from a different input than self.

pub fn scanner_trip_snapshot(&self) -> ScannerTripBaseline<'closure>

Snapshots the session’s scanner-trip counter — how many times the scanner has tripped a lexer resource limit in this input session — for a rollback-proof terminality witness.

The scanner twin of trip_snapshot, used identically: take the baseline once per collection, hand it back to scanner_stopped_during_attempt when judging that attempt. The crate’s own drivers spend it on the raw, unconditioned scanner_tripped_during_attempt instead, which stays private for the reason that verdict’s docs give.

This, and not the poison boundary beside it, is what a recovery gate judges a scanner stop with. The latch is a lineage memo: a Checkpoint carries it and a restore copies it back, so comparing it across a rollback compares a restored value against what it was restored to. Reading it inside the attempt fixes one level and the level below reopens it — grammar code that catches a stop inside an inner try_attempt has that rollback erase the latch before an outer gate looks. This counter is outside the rollback set entirely and is therefore depth-independent.

Per COLLECTION, where the descent baseline is per ELEMENT

The two facts decay differently, so their baselines belong at different units and neither can be derived from the other. A descent trip that grammar code caught and parsed past stops being true of the input; a spent scanner budget does not — the token stream ends where it ended, and every attempt after it reads a view the stop truncated. So this baseline is taken once per collection, above the loop, where hoisting the descent one would be the defect: taken per element it is re-read after the trip an earlier element caught, and element 1 tripped and accepted, element 2 declines then concludes cleanly over a spent budget. That asymmetry is why the two baselines are two types: the swap does not compile, in either direction, at any of the sites that take both.

The monotone counter alone is not publishable, and this is why

set_state and state_mut re-key the input’s forward-scanning facts, and dropping the poison boundary there is the documented limit-recovery path — swap in a fresh or bigger-budget state and scanning resumes past the old boundary. Neither touches this counter, which is monotone and never cleared. So a loop doing exactly what the section above prescribes — one baseline, taken above the loop — can trip, recover through the documented path, read the rest of the document and reach a genuine end of input with the raw event still answering true, rejecting a fully recovered parse as truncated. Measured, not reasoned about: an eight-token source under a scan budget of three, recovered with set_state, consumed all eight and the witness still said tripped.

That is correct use under two conflicting public contracts rather than caller misuse. What closes it is not a second cell ordered against the trip — the design that had to settle its own rollback behaviour at every nesting depth — but a second question, asked of the regime that is installed now: scanner_stopped_during_attempt is this event and Lexer::check still refusing, and that is the public pair. The raw event stays crate-internal because the drivers need it unconditioned: a driver judging one element must re-raise a trip it caught even where grammar code recovered the regime inside that element, which is precisely the conjunct the public verdict drops.

The two public readings beside this one, and which question each answers

at_scanner_stop is the positional reading: is a stop on record at the committed cursor, now. It takes no baseline, it goes clean across a set_state recovery because the boundary dies with the regime that owned it — and it also goes clean across a speculative wrapper’s rollback, because that restores a checkpoint predating the trip. Planting this monotone counter in its place reds a_documented_widening_leaves_the_witness_reporting_a_finished_document, at Err(Eot) for an Ok(7): a fully recovered document reported as truncated.

scanner_stopped_during_attempt is the attempt-relative reading, and it is this counter conjoined with a stop still being latched — positionally unqualified, which is how it answers the lookahead residue the positional reading is blind to. The latch supplies what the counter alone gets wrong at a recovery, since both state-surgery doors drop it. Neither half is publishable on its own, which is why the raw event is still not.

It answers the rejecting-emitter path this pair exists for, and answers it without a baseline. A rejecting (fail-fast) emitter reports a lexer-resource trip by returning the value its From<<L::Token as Token>::Error> builds, and scan_with(..)? propagates that value from inside try_expect_or_stop, before the call can reach the arm that raises a terminal stop; no care in the grammar’s MaybeTerminal repairs that, because nothing on the path is terminal-marked to delegate to. The boundary is latched anyway — inside the crate’s terminal predicate, ahead of the diagnostic ever being offered to the emitter — so the stop is on record when the rejection arrives, and the live reading finds it. tokora/tests/root_loop_trip_witness.rs section 4 is the measurement, including the growth the reading removes: without it a re-keying root loop files one diagnostic per remaining token, at three document lengths.

try_expect_or_stop still covers the declining exits and is still the primitive to build a decline on: its contract is that a terminal stop is an error and never a decline, and it reads the same live boundary. What it cannot do is speak on the path where the emitter’s Err overtakes it, which is why the verdict above exists beside it.

None of that changes this pair’s own answer or its visibility. An attempt-relative verdict is a different question from a live one — this one is what a driver judging one element or one speculative parse needs, where the live reading is what a root loop holding an Err needs — and the set_state false positive above is the reason it stays where it is.

Costs one u64 load per attempt — one machine word on a 64-bit target, two on a 32-bit one — and no scan, no lookahead fill and no token commit reads it. Unlike its descent twin this baseline is taken once per collection rather than per element, so the 32-bit cost lands once per driver rather than once per element.

pub fn scanner_stopped_during_attempt( &self, since: ScannerTripBaseline<'closure>, ) -> bool

Whether the scanner is stopped for the attempt that took since as its scanner_trip_snapshot baseline: a trip was recorded while that attempt ran, and a stop is still latched.

This is the public attempt-relative scanner verdict. at_scanner_stop beside it is the positional one — is a stop on record at the committed cursor, now — and the two are not two spellings of one question.

loop {
  let scan = inp.scanner_trip_snapshot();      // per COLLECTION, above the loop
  match definition(inp) {
    Ok(true)  => {}
    Ok(false) => return Ok(()),
    Err(e) => {
      if e.is_terminal() || inp.scanner_stopped_during_attempt(scan) {
        return Err(e);                         // the document is over
      }
      report();
    }
  }
}

The residue it closes: a stop latched AHEAD of the cursor

An element’s own lookahead (peek, peek_one) can trip, latch the frontier ahead of the committed cursor, and still return Ok with a short window. Every positional witness reads clean there while the stop is live and already diagnosed — that is the first residue at_scanner_stop names, and it stays clean as the cached pre-trip tokens drain, because draining them does not carry the cursor to the frontier. This reading is not positional: it asks whether a stop is latched at all, so it answers true from the moment the lookahead latched.

Measured as rows 1 and 2 of the_attempt_relative_verdict_answers_where_the_positional_reading_is_blind, under both emitters: (at_scanner_stop, this) == (false, true).

The conjunct is the whole design, and it is what makes publishing possible

A carrier that only records the event is a permanent false positive: the counter is monotone, nothing clears it, and a loop that recovers exactly as this crate documents then reads its own finished document as truncated. That is what kept the raw pair crate-internal, and a recovery generation — a second cell ordered against the trip — is the design that would have had to settle its own rollback behaviour at every nesting depth to fix it.

The second conjunct is no such cell. It is the state the crate already keeps, asked in the present tense, and both public state-surgery doors clear it as a side effect of what they already do.

Which carrier is still in force, and what clears each

The counter records that a scanner trip happened; it does not record which of the two carriers took it, so both are asked. They clear on opposite terms, and that asymmetry is the contract rather than an implementation detail:

• a TokenBudget refusal — TokenBudgetTally::refused_an_item — is never cleared. No Checkpoint carries the tally, no re-key touches it, and there is no token_budget_mut to lower it. For a bound placed there the verdict is durable by construction, at any depth, across every rollback;
• a lexer-side trip is carried by the poison boundary, and clears exactly where that boundary dies: set_state and state_mut, which is the documented limit-recovery path, and a Checkpoint restore whose saved boundary is None.

Why a true is sound

Both publishers of a scanner trip require an item that exists — latch_if_limit_tripped is reached from classify about an item the lexer produced, and settle_met_ceiling discards its staged stop without publishing when lexer.lex() yields nothing. So a true means a real item was refused and the record of it is still standing.

The granularity floor its raw half documents is unchanged: this witnesses that a trip happened while the attempt ran, not that the Err in hand is that trip. A unit that catches a trip, does not recover the regime, and then fails ordinarily is re-raised as a stop. It fails closed, never open.

It is the stop question, and it is NOT a sufficient root-loop guard

A trip inside a speculative wrapper is restored away together with the tally that took it. try_attempt, attempt_parse and a rollback-on-drop Transaction reinstate the checkpointed state and a pre-trip None boundary, and for a scanner bound held in the lexer state — the supported placement TokenLimiter documents — the input-side refusal bit is false too. This method then answers false, correctly: the restore reinstated the tally the checkpoint saved, which is the refund that type documents as the right answer for a bound on the committed stream, so a scan from there really does yield a token.

The caller nonetheless holds the trip-derived error with the cursor and state exactly where the attempt began, and a loop guarded on this boolean alone retries the identical scan for ever. Match scanner_outcome instead, whose Stalled arm is precisely that state; this method is its Stopped arm and nothing more.

Measured

tokora/tests/root_loop_trip_witness.rs section 5. the_attempt_relative_verdict_answers_where_the_positional_reading_is_blind is the five-point table — latched ahead, draining, at the frontier, re-keyed, recovered — under both emitters; a_baseline_taken_after_the_trip_does_not_charge_this_attempt_with_it pins that it is attempt-relative and not “is this scanner spent”; an_input_side_bound_outlives_every_one_of_the_three_rollbacks is the durable carrier through all three wrappers.

Costs

A nonce comparison and a u64 comparison — the raw event’s whole cost — then, only if those say a trip happened, a bool load and an Option::is_some. No scan, no lookahead fill, no caller code at all.

Panics

If since came from a different input than self.

pub fn scanner_attempt(&self) -> ScannerAttempt<'inp, 'closure, L>

Captures one attempt’s scanner bookkeeping — the trip baseline and the committed position — for scanner_outcome.

Taken per attempt, immediately before the thing being judged runs. scanner_trip_snapshot beside it is the driver’s value and is taken per collection; the two units are different because the two questions are. “Is the scanner spent” does not stop being true of a later element, so hoisting that baseline is right; “did this attempt trip without committing anything” is about one attempt and nothing else, so hoisting this one would compare a later failure against a position many elements ago and call every one of them progress.

Costs one L::Offset::clone on top of the baseline’s u64 load and nonce derivation. That clone is caller code and is the only step here that can unwind; nothing durable is in flight on a &self read, so there is nothing for it to tear.

pub fn scanner_outcome( &self, since: ScannerAttempt<'inp, 'closure, L>, ) -> ScannerOutcome

What the attempt that took since as its scanner_attempt capture did to the scanner — the root-loop guard, and the reading a retrying loop must match on.

loop {
  let att = inp.scanner_attempt();          // per ATTEMPT, at the top of the turn
  match definition(inp) {                    // may speculate internally
    Ok(true)  => {}
    Ok(false) => return Ok(()),
    Err(e) => match inp.scanner_outcome(&att) {
      ScannerOutcome::Stopped => return Err(e),   // the document is truncated
      ScannerOutcome::Stalled => return Err(e),   // retrying would repeat this exactly
      _ => report(),                              // an ordinary syntax error: file and go on
    },
  }
}

Why a boolean could not be this

scanner_stopped_during_attempt answers “is a stop in force”, and it is correct and not a sufficient loop guard, because the failure that costs most is one where no stop is in force. A failing try_attempt restores the checkpointed state and a pre-trip None boundary; for a scanner bound held in the lexer state — the supported placement TokenLimiter documents, not a violating shared counter — the input-side refusal bit is false as well. Every live reading of the input is then correctly clean, the caller nonetheless holds the trip-derived error, and the cursor and state are exactly where the attempt began. A loop guarded on the boolean retries the identical scan and can burn unbounded CPU on attacker-controlled input.

The facts that separate that from a real recovery are the determinism clause’s own inputs, and none of them needs a scan to read. The trip event survives the rollback because the counter is outside the rollback set; the committed offset is observable directly; and the regime has a name (Input::regime) because a State cannot be compared. A trip after which all three are unchanged is Stalled, and each way that can fail is its own arm.

The partition, and why it is exhaustive and disjoint

Four facts are read, in this order, first match winning:

1. did a trip happen inside the attempt — the session counter against the capture’s, which is outside the rollback set. no ⇒ NoTrip.
2. is a stop in force now — the input budget’s durable refusal, or a latched frontier. yes ⇒ Stopped.
3. is the lexer regime the one the capture saw — the Input::regime generation. no ⇒ ReKeyed.
4. where is the committed end relative to the capture’s — an Ord comparison, whose three outcomes are Progressed, Stalled and Rewound.

Disjoint because the arms are the branches of a single if/if/if/match chain — no input reaches two of them. Exhaustive because the first three are booleans whose false falls through and the fourth is a total trichotomy on Ord, so every combination of the four facts lands in exactly one arm.

Why Stalled is sound rather than a guess, and why it is the EQUAL arm

The Lexer contract’s determinism clause is the theorem: every scan-visible result … must derive entirely from the source, the offset being lexed at, and the lexer State. So a repeat reproduces a trip exactly when all three are unchanged, and Stalled has to test all three:

• the source never changes;

• the offset is tested for equality, not for “did not advance”. A rewind also fails to advance, and a capture describing a position the input has left supports no claim about the one it is at now — that is Rewound, and folding it into a stall would be a repetition claim with nothing behind it;

• the State cannot be compared — it is bounded Debug + Clone and nothing more — so a regime id stands in for it, and the standing-in is exact rather than approximate. Two things make it so, and each is load-bearing:

  The writers are three, and one of them names the regime. A commit threads the lexer’s post-token state forward and moves the committed span with it; a restore installs a checkpoint’s saved pair and the id that names it; a surgery (set_state / state_mut) goes through install_rekey, the sole allocation site. So equal id ∧ equal committed offset ⇒ equal State: a commit would have moved the offset, a surgery the id, and a restore carries both from one saved pair. The destructuring census in input::lineage is what keeps that trichotomy from silently gaining a fourth member.

  And no public projection reaches a live-or-installable State at all. A census of assignment sites is a claim about writes, and the conclusion needed is about values — the two coincide only for a type that cannot be mutated through a shared reference, which Debug + Clone does not give. A State holding a Cell<Mode> is perfectly valid, so any &L::State handed to safe code lets a grammar flip it with no writer involved. The enumeration below is derived from the types that hold one, not from the doors that were found:

  holder	public projection	verdict
  Input	none; fields private, not constructible while a handle lives	closed by construction
  InputRef	state	owned clone
  InputRef	state_mut, set_state	tracked — both allocate a new regime id
  InputRef	lexer	an owned L built from a clone; its Lexer::state is the caller’s own copy
  InputRef	cache, and every Cache view behind it	reaches only CachedToken, whose regime is not projectable
  InputRef	every peek* and sync_*_then_peek* result	same — they are all CachedToken
  InputRef	next, try_expect*, consume_cached_*	yield Spanned<L::Token, L::Span> — provably state-free
  ParseState	state / state_mut	forward the two rows above
  Checkpoint	state	owned clone
  Checkpoint	cursor	an offset — provably state-free
  CachedToken	state, into_components, new	crate-internal: not readable, not fabricable
  CachedToken	token, into_token, as_ref, map_token, Clone	carry the regime as an opaque payload
  PeekedTokenExt	token, span	state-free
  ThroughEntry, AtFrontier, Resume, Session	not public	closed by visibility

  The cut for the whole cache/peek family is made at CachedToken, not at each door, because every one of those doors hands out that one type — so it holds for the doors that do not exist yet as well.

  Two things sit outside this table rather than in it, both by the same rule the crate already applies to a lexer. Cache and Lexer are caller-implemented infrastructure: a cache owns the entries it stores and can hand back the wrong one, which is a cache-conformance violation, and a Clone that shares its interior mutability rather than deep-copying it is the determinism clause’s own named violation. Under either, this crate’s behaviour is unspecified-but-bounded — the same posture, and the same boundary, as everywhere else on this reading.

  The id is allocated, not derived. Its source is monotone and outside the rollback set, because deriving the next id from the checkpointed cell is not injective: save at g, install B (g + 1), roll back to g, install C (g + 1) — two regimes, one name. At REGIME_EXHAUSTED the allocator stops and that id is excluded from equality, so exhaustion forces ReKeyed rather than a false stall.

The predicate therefore matches the theorem rather than approximating it. An earlier version tested only “not advanced”, which was strictly weaker in both directions — it called a documented state_mut recovery a stall (rejecting a recoverable parse) and swallowed a rewind into the same arm.

What clears each arm

• Stopped clears where its carrier dies: a TokenBudget refusal never clears, and a lexer-side latch clears at set_state / state_mut or a restore of a None boundary. Unchanged from scanner_stopped_during_attempt, which is this arm.
• Stalled clears on either of the two inputs it tests moving: the parse commits anything past where the attempt began (Progressed), or the regime is replaced (ReKeyed). Both are what a successful recovery produces, and between them they are the only things that distinguish a recovery from a retry. It cannot be permanent: an attempt that consumes even one token, or that takes the documented recovery door, is a different arm, and an attempt with no trip in it is NoTrip whatever the position did.

It is not a substitute for placing the bound where the work is

This makes the futile retry detectable, which is what a loop needs to stop. It does not make the work bounded — that is a placement question, and TokenBudget on the input is the answer, because no rollback reaches it. A root loop that must hold against hostile input wants both: this to end the loop, and the budget to bound what any loop can spend.

Costs

A nonce comparison, a u64 comparison, and — only when those say a trip happened — a bool load, an Option::is_some, and one Offset::Ord comparison. No scan, no lookahead fill, no state clone.

Panics

If since came from a different input than self.

pub fn judge_scanner<F, T>(&mut self, f: F) -> (ScannerOutcome, T)

where F: FnOnce(&mut Self) -> T,

Judges one turn: takes the capture, runs f, and hands back its value beside the ScannerOutcome for exactly that span of the parse.

The shape a retrying root loop should reach for, because it is the one that cannot be misplaced. scanner_attempt and scanner_outcome are the lower-level surface, and the capture being affine already makes a reused one a compile error — but a capture can still be taken outside a loop and spent late, which costs one stale verdict. Here capture, judged work and verdict are bound into a single turn, so neither placement exists to get wrong.

loop {
  let (outcome, parsed) = inp.judge_scanner(|inp| definition(inp));
  match parsed {
    Ok(true)  => {}
    Ok(false) => return Ok(()),
    Err(e) => match outcome {
      ScannerOutcome::Stopped | ScannerOutcome::Stalled => return Err(e),
      _ => report(),
    },
  }
}

The outcome comes first in the tuple so a caller destructuring it cannot quietly drop it the way a trailing element invites; f’s value is returned untouched, including an Err, because judging is not deciding.

It adds no rollback and no guard of its own — f receives the same handle, and whatever it does with try_attempt or a Transaction is its own business. The only thing this owns is the pair of readings around it.

Costs exactly scanner_attempt plus scanner_outcome: one L::Offset::clone and, on an attempt that did not trip, a nonce and a u64 comparison.

pub fn emitter_ref(&self) -> &Ctx::Emitter

The parse’s emitter, by shared reference — for reading a concrete emitter’s own state while the parse is running (a collecting emitter’s recorded diagnostics, a counter, a label stack).

Shared on purpose, and the receiver is the whole of the difference: every method that records anything — on Emitter, on CstEmitter, on the atomic emitter family — and every method that captures a mark takes &mut self, so a wrapper type built around this reference can forward none of them, cannot stand in an emitter slot, and cannot mint a checkpoint the input layer never took. To emit, call the forwarding methods on this handle.

What a shared reference does not promise

Ctx::Emitter is the caller’s own type, and Rust has no bound that excludes interior mutability, so an emitter that mutates itself through &self can be driven to do so from here. Nothing the crate accounts for is reachable that way — the event log, the mark stack and the checkpoint lineage all sit behind &mut receivers — but an emitter’s own guarantees about its own state are its own to keep, in the terms HashSet uses for a key that changes while it is in the set: a logic error, with unspecified rather than undefined behavior, and deliberately not enumerated.

pub fn emit_lexer_error( &mut self, err: Spanned<<L::Token as Token<'inp>>::Error, L::Span>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

Emits a lexer error — Emitter::emit_lexer_error, forwarded.

The input layer’s own lexer-error reports are deduped against a watermark; a report raised here is not, so a caller re-reporting a region the layer already reported produces two diagnostics rather than one. Noisy, never silent.

It reports; it does not license

A recording Sink tiles a source byte no committed token covers only where a lexer error the input layer raised covers it. The span handed over here is the caller’s, with no consumption behind it, so the sink records the diagnostic and no coverage span: an uncovered byte stays uncovered and finish still refuses it (FinishError::UncoveredGap). An unconsumed region is materialized through Cst::finish_partial, which tiles it — not by reporting a lexer error over it.

pub fn emit_unexpected_token( &mut self, err: UnexpectedTokenOf<'inp, L, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

Emits an unexpected-token report — Emitter::emit_unexpected_token, forwarded.

This does not publish the front-report watermark the input layer maintains for the token at the stream front, so a later close-miss report about the same token is not suppressed by it. Same direction as above: an extra diagnostic, never a missing one.

pub fn emit_error( &mut self, err: Spanned<<Ctx::Emitter as Emitter<'inp, L, Lang>>::Error, L::Span>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

Emits an application error — Emitter::emit_error, forwarded.

pub fn emit_warning( &mut self, warning: Spanned<<Ctx::Emitter as Emitter<'inp, L, Lang>>::Error, L::Span>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

Emits a warning — Emitter::emit_warning, forwarded.

pub fn emit_skipped_region( &mut self, span: L::Span, skipped: usize, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

Emits a recovery-hole note — Emitter::emit_skipped_region, forwarded.

sync_balanced raises exactly one of these per hole it skips; a caller running its own recovery loop is the reason this is reachable at all.

Span semantics. Under a CST sink this call also has a structural effect: the hole’s buffered tokens are bracketed in an error node. That bracket covers only the hole tokens that settled within the transaction reporting the hole — at or above the youngest live checkpoint. A recovery loop that widens span backward over tokens it already committed still gets the report forwarded verbatim, but the error node stops at the transaction boundary: checkpoint marks are event-log positions, and a node spliced beneath one would rename it. sync_balanced’s own spans postdate every live capture, so this bound never narrows the crate’s own recovery.

pub fn enter_label(&mut self, label: &'static str)

Pushes a diagnostic label — Emitter::enter_label, forwarded.

Pairs with exit_label; prefer labelled, which pairs them through a drop guard.

pub fn exit_label(&mut self)

Pops the innermost diagnostic label — Emitter::exit_label, forwarded.

pub fn emitter_bound_source(&self) -> Option<SourceIdentity>

The source the emitter is bound to, if any — Emitter::bound_source, forwarded.

A query, not an emission: it answers for anyone who can reach the emitter, which is why no sink-side witness built on it can encode who asked.

One of the two &self readers that do not delegate to EmitterView: a view is built from a &mut borrow, which a shared method has not got. The view carries the same method under the emitter’s own name, bound_source.

pub fn cst_start(&mut self, kind: u16) -> EventMark

where Ctx::Emitter: CstEmitter<'inp, L, Lang>,

Opens a CST node of kind — CstEmitter::cst_start, forwarded.

Raw transport: pair it with cst_finish through a both-exits bracket, or use the node-shaped combinators. The returned mark is the failing exit’s handle — spend it on cst_demote.

pub fn cst_finish(&mut self, kind: u16)

where Ctx::Emitter: CstEmitter<'inp, L, Lang>,

Closes the innermost open CST node — CstEmitter::cst_finish, forwarded.

pub fn cst_demote(&mut self, mark: EventMark, kind: u16)

where Ctx::Emitter: CstEmitter<'inp, L, Lang>,

Un-opens the node started at mark — CstEmitter::cst_demote, forwarded.

pub fn cst_mark(&mut self) -> EventMark

where Ctx::Emitter: CstEmitter<'inp, L, Lang>,

Appends a retro-wrap anchor — CstEmitter::cst_mark, forwarded.

pub fn cst_start_at(&mut self, mark: EventMark, kind: u16)

where Ctx::Emitter: CstEmitter<'inp, L, Lang>,

Retro-opens a node of kind at mark — CstEmitter::cst_start_at, forwarded.

pub fn emit_too_few( &mut self, err: TooFew<L::Span, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: TooFewEmitter<'inp, L, Lang>,

TooFewEmitter::emit_too_few, forwarded.

pub fn emit_too_many( &mut self, err: TooMany<L::Span, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: TooManyEmitter<'inp, L, Lang>,

TooManyEmitter::emit_too_many, forwarded.

pub fn emit_full_container( &mut self, err: FullContainer<L::Span, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: FullContainerEmitter<'inp, L, Lang>,

FullContainerEmitter::emit_full_container, forwarded.

pub fn emit_missing_separator( &mut self, name: CowStr, err: MissingTokenOf<'inp, L, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: SeparatedEmitter<'inp, L, Lang>,

SeparatedEmitter::emit_missing_separator, forwarded.

pub fn emit_missing_element( &mut self, err: MissingSyntaxOf<'inp, L, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: SeparatedEmitter<'inp, L, Lang>,

SeparatedEmitter::emit_missing_element, forwarded.

pub fn emit_missing_leading_separator( &mut self, name: CowStr, err: MissingTokenOf<'inp, L, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: MissingLeadingSeparatorEmitter<'inp, L, Lang>,

MissingLeadingSeparatorEmitter::emit_missing_leading_separator, forwarded.

pub fn emit_missing_trailing_separator( &mut self, name: CowStr, err: MissingTokenOf<'inp, L, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: MissingTrailingSeparatorEmitter<'inp, L, Lang>,

MissingTrailingSeparatorEmitter::emit_missing_trailing_separator, forwarded.

pub fn emit_unexpected_leading_separator( &mut self, name: CowStr, err: UnexpectedTokenOf<'inp, L, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: UnexpectedLeadingSeparatorEmitter<'inp, L, Lang>,

UnexpectedLeadingSeparatorEmitter::emit_unexpected_leading_separator, forwarded.

pub fn emit_unexpected_trailing_separator( &mut self, name: CowStr, err: UnexpectedTokenOf<'inp, L, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: UnexpectedTrailingSeparatorEmitter<'inp, L, Lang>,

UnexpectedTrailingSeparatorEmitter::emit_unexpected_trailing_separator, forwarded.

pub fn emit_unclosed<Delimiter>( &mut self, err: Unclosed<Delimiter, L::Span, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: UnclosedEmitter<'inp, L, Lang>,

UnclosedEmitter::emit_unclosed, forwarded.

pub fn emit_unexpected_end_of_lhs( &mut self, err: UnexpectedEoLhs<L::Offset, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: PrattEmitter<'inp, L, Lang>,

Available on crate feature pratt only.

PrattEmitter::emit_unexpected_end_of_lhs, forwarded.

pub fn emit_unexpected_end_of_rhs( &mut self, err: UnexpectedEoRhs<L::Offset, Lang>, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Ctx::Emitter: PrattEmitter<'inp, L, Lang>,

Available on crate feature pratt only.

PrattEmitter::emit_unexpected_end_of_rhs, forwarded.

pub fn fold<O, Pred, Init, Op>( &mut self, pred: Pred, init: Init, op: Op, ) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Init: FnOnce() -> O, Op: FnMut(O, Spanned<L::Token, L::Span>) -> O, Pred: FnMut(Spanned<&L::Token, &L::Span>) -> bool,

Folds over the input tokens using the provided accumulator function.

pub fn foldn<O, Init, Op>( &mut self, init: Init, op: Op, num: usize, ) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Init: FnOnce() -> O, Op: FnMut(O, Spanned<L::Token, L::Span>) -> O,

Folds at most n tokens over the input using the provided accumulator function.

pub fn foldr_within<O, W, Pred, Init, Op>( &mut self, pred: Pred, init: Init, op: Op, ) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Init: FnOnce() -> O, Op: FnMut(O, Spanned<L::Token, L::Span>) -> O, W: Window, Pred: FnMut(Spanned<&L::Token, &L::Span>) -> bool,

Right-folds over the input tokens using the provided accumulator function.

The maximum number of tokens folded is determined by the capacity of the specified W.

See also foldrn.

pub fn foldrn<O, Init, Op>( &mut self, init: Init, op: Op, num: usize, ) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Init: FnOnce() -> O, Op: FnMut(O, Spanned<L::Token, L::Span>) -> O,

Right-folds over the input tokens using the provided accumulator function.

This method folds up to num tokens, and this will lead to implicit allocation.

See also foldr_within.

pub fn peek_one( &mut self, ) -> Result<Option<MaybeRefCachedTokenOf<'_, 'inp, L>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

Peeks the next token without advancing the cursor.

A token already waiting at the front of the stream — parked or cached — is served without touching the lexer.

It folds a terminal stop into Ok(None)

This is the raw read: a resource-limit trip or a latched poison boundary is indistinguishable here from genuine end of input. A production that decides on the answer — “is there a { here?” — will read a halt as a grammar fact and keep going. Prefer peek_kind, head_satisfies or peek_head_map, which raise on a terminal stop and reserve Ok(None) for the real end of input.

pub fn peek<'p, W>( &'p mut self, ) -> Result<Peeked<'p, 'inp, L, W>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where W: Window,

Peeks tokens to fill the provided buffer.

If not enough tokens are cached, lexes more tokens to fill the buffer. The returned deque contains references to peeked tokens.

Partial mode: a short window, but never a hidden trip

On a non-final Partial input the fill stops at the frontier — a token the lexer decided by reading as far as the buffer end (read_frontier, floored at the item’s own span end) never enters the cache — so a peek there simply returns a shorter window than asked for. How much shorter is the lexer’s to say, not the span’s: a lookahead lexer holds back items whose spans sit well behind the buffer end, and one reporting Unbounded caches nothing at all until the stream is sealed. The Incomplete surfaces when a consume path reaches the same frontier. A terminal condition is not held back that way: a limit trip during the fill emits its diagnostic and latches the poison boundary before the holdback is consulted, so a peek can no more hide a tripped limit than a consume can. See terminal beats incomplete.

What that leaves: the return value cannot tell the three apart

The paragraph above is about the diagnostic, and it is the whole truth about the diagnostic. It is not a statement about this return type. A full window, a genuinely short one, a partial-input frontier holdback and a terminal stop all arrive here as Ok holding a window, and the last two are the same length for the same reason nothing distinguishes them: the short window is the value. A production that decides on the width — “is the second token a (?” — therefore reads a halted scanner as a grammar fact and picks the other production, which is peek_one’s Ok(None) fold one width up.

With a fatal emitter that is invisible, because the trip’s own diagnostic ends the parse either way. With a non-fatal emitter that accepts it, the caller is handed Ok with fewer tokens than it asked for and no way to ask why — a silently different parse rather than a stopped one.

Two things can express it. peek_with_emitter_terminal reports it as a flag beside the window, for a caller that wants the short window and the reason; peek_map puts it in the error arm, reserving a short Ok window for a genuine end of input, which is what peek_head_map, peek_kind and head_satisfies do at the head. Prefer one of them wherever the window’s length decides a production.

Stack footprint: one window, cache hit or miss

The window this returns is the only W::CAPACITY-sized owned token storage a peek reserves, and its worst case is the whole array live at once:

W::CAPACITY × size_of::<Maybe<CachedToken<&Token, &State, &Span>,
                             CachedToken<Token, State, Span>>>()

which for every realistic type is W::CAPACITY × (size_of::<Token>() + size_of::<State>() + size_of::<Span>()) plus per-entry padding and a discriminant. A cache miss costs no second window: tokens lexed past the cache are staged in this same buffer and rotated into place, so the miss path reserves exactly what the hit path does. (Through 0.7.3 the miss path staged them in a separate W::CAPACITY-slot array, doubling the figure above.)

Everything else in the frame is O(1) in the window width: single-entry temporaries (one CachedToken in flight to the cache, the deque’s own push and rotate temporaries), one clone of the lexer — size_of::<L>(), which contains State — and a small fixed part. Nothing here is heap-allocated, and nothing scales with the input.

Token and State are unconstrained in size, so the bound is linear in both and in the window width — with a coefficient of one, not two. A grammar carrying a large token payload or a large lexer state pays W::CAPACITY times it for the width it asks for: prefer the narrowest window that decides the production, and peek_kind or head_satisfies — which run at U1 — for a head test.

Panics

On a Cache that breaks its own contract, and only then, on the fill path that reaches the lexer. The fill reserves the window’s cache region from Cache::len before it stages anything past it, so a len that is not the resident count mis-sizes the room Cache::peek is then given. The exit that hands such a window back checks the copy that landed in it — against the room the fill left, and against the cache’s own front and back, which an inexact len cannot move — and panics rather than return a window that is wrong about the stream. Both checks run in release, because the window a broken len produces there is not a short one but a hole. Every cache this crate ships conforms, and the cache conformance kit checks a downstream one.

pub fn peek_with_emitter<'p, W>( &'p mut self, ) -> Result<(Peeked<'p, 'inp, L, W>, EmitterView<'p, 'inp, L, Ctx::Emitter, Lang>), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where W: Window,

Peeks tokens to fill the provided buffer and returns the emitter’s operations.

The second half is an EmitterView, not the emitter: the value a *_while condition is handed. Returning &mut Ctx::Emitter here would be the same door InputRef::emitter is crate-private to shut — see EmitterView for the class.

Reserves the same one owned window as peek, cache hit or miss, and panics on the same broken-Cache condition — see its stack-footprint and panics sections.

pub fn peek_with_emitter_terminal<'p, W>( &'p mut self, ) -> Result<(Peeked<'p, 'inp, L, W>, bool, EmitterView<'p, 'inp, L, Ctx::Emitter, Lang>), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where W: Window,

Peeks tokens to fill the window and reports whether the fill was cut short by a terminal scanner stop — a fresh resource-limit trip during the fill, or an already-latched poison boundary at the cursor.

The peek contract is that a short window is not itself an error (it may be a genuine end of input, or a partial-input frontier). The returned flag lets a decision-window combinator tell the one case that class hides apart: a window truncated by a terminal stop is not evidence the construct ended, so such a combinator surfaces the committed end-of-input error rather than reading the short window as a decline. The flag is true only when the window came back shorter than requested because of a terminal stop; a full window, a genuine end of input, and a partial-input frontier holdback all report false.

Callers must consult the flag immediately, before any fallible or emitting call (decide, element handlers, close probes): an ordinary error raised in between preempts the terminal stop.

Rides the same fill as peek — the terminal flag is an extra out-parameter on it, not a second code path — so it reserves the same one owned window, cache hit or miss, and panics on the same broken-Cache condition. See its stack-footprint and panics sections.

pub fn peek_map<'p, W, O, F>( &'p mut self, f: F, ) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where W: Window, F: FnOnce(Peeked<'p, 'inp, L, W>) -> O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

Windowed observation in grammar vocabulary, terminal-aware by construction — the peek::<W> analogue of peek_head_map.

f sees the filled window and its value is returned. A window that came back short because the input genuinely ended — or because a non-final Partial frontier withheld the rest — is handed to f like any other, and is Ok. A window cut short by a terminal stop — a resource-limit trip during the fill, or a latched poison boundary at the cursor — raises the same terminal end-of-input error the _or_stop family raises, and f does not run.

That distinction is the whole of what this adds to peek::<W>, and it is not something a caller can recover afterwards. peek returns Ok in both cases, holding a window of the same length, so a production that decides on the width — “is the second token a (?” — reads a halted scanner as a grammar fact and picks the other production. The diagnostic is not lost either way (see peek’s Partial-mode section); what is lost is the return value’s ability to say which of the two happened, and with a non-fatal emitter that accepts the diagnostic, the difference is a silently different parse rather than a stopped one.

The mark carries the same qualification peek_head_map’s does: it is what an accepting emitter earns, after the trip’s own diagnostic has gone to it. A fatal emitter’s rejection of that diagnostic still propagates — from the fill here rather than from a scan — but as that emitter’s value, converted from the lexer error, so it carries no terminal mark. (A fatal emitter is blind to the difference only on the emitting path: at an already-latched boundary the fill emits nothing, so even there the short window is raised rather than returned.)

The one contract difference from peek_head_map

peek_head_map answers Ok(None) at a genuine end of input and does not run f; here f runs on the window whatever its length, including an empty one. A head read has two lengths and can lift the empty one into None; a W-wide window has W + 1, and folding every short one into a single None would throw away the tokens that are there — which for a two-token decision is the head the caller already committed to. So the Option belongs to the caller’s own projection, not to this return type:

// "the second token's kind, if there is a second token"
inp.peek_map::<U2, _, _>(|w| w.iter().nth(1).map(|t| t.token().kind()))
//  Ok(Some(kind)) — a second token
//  Ok(None)       — the input genuinely ends after the head
//  Err(..)        — the scanner stopped while the window was filling

f may also hand the window straight back — peek_map::<W, _, _>(|w| w) is exactly peek::<W> with the terminal stop moved into the error arm — so nothing the unmapped form can express is lost. Taking f rather than returning the window is what lets O be free of the borrow: the window borrows the cache for as long as it lives, and a grammar that only wants a kind or a boolean out of it can go on using the input immediately.

What is guaranteed to f, and the condition on the caller

f runs exactly once when it runs at all, is handed the window this call filled, and nothing is consumed, committed or latched on its behalf. There is one route here and no fast path, so the two-route reconciliation under “what is guaranteed identical” on peek_head_map has no counterpart — but the second clause of its condition on the caller applies verbatim, and for the same mechanism: this call takes self.span().end() before the fill, for the terminal end-of-input error, and peek::<W> does not. That is one caller-supplied L::Offset::clone that the unmapped read never runs. An f that answers from the values of the window it is handed cannot see it; an f that measures the input layer can, and is asking which primitive this crate reached the window through rather than what the window holds.

Footprint and panics

Rides the same fill as peek, through peek_with_emitter_terminal: it reserves the same one owned window, cache hit or miss, and panics on the same broken-Cache condition. See peek’s stack-footprint and panics sections.

pub fn peek_head_map<O, F>( &mut self, f: F, ) -> Result<Option<O>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnOnce(Spanned<&L::Token, &L::Span>) -> O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

Width-1 head observation in grammar vocabulary, terminal-aware by construction.

The head-only sibling of peek_map, which is the same treatment at an arbitrary window width.

f sees the head as Spanned<&Token, &Span> and its value is returned. Ok(None) is genuine end of input; a terminal stop — a resource-limit trip or a latched poison boundary — raises the same terminal end-of-input error the _or_stop family raises, never a silent None. That distinction is the point: a consumer that reads a halt as “no head here” builds a value out of an input the scanner already gave up on.

The mark carries the same qualification the _or_stop family’s does: it is what an accepting emitter earns, after the trip’s own diagnostic has gone to it. A fatal emitter’s rejection of that diagnostic still propagates — from the fill here rather than from a scan — but as that emitter’s value, converted from the lexer error, so it carries no terminal mark: no UnexpectedEot is built on that path for into_terminal to raise a flag on. The arm of your error type holding a lexer error is what answers for it; see MaybeTerminal.

Rides the terminal-aware cache read (peek_with_emitter_terminal) rather than the try_expect scan, so the head is served from the front slot with no pop/hold round-trip.

A head already at the front of the stream is read where it lies

This is a width-1 read, and on a grammar’s decision points the token it wants is almost always already there: measured on a GraphQL parse, 17,028 of 17,029 times. So the front of the stream is probed first, and a head that is there is handed straight to f.

That is the same head the fill gives, by the fill’s own construction — and therefore the same answer, for a caller who meets the condition below. With one token at the front — parked or cached — the window request is already met, so the fill takes its want == 0 arm: it heads the window with the parked token if there is one and lets the cache fill in behind it, then returns. Nothing is lexed, nothing is committed, the terminal flag stays false (that arm returns before the boundary probe, so a resident head is served whatever the poison latch says), and Cache::peek is a &self read the trait documents as logically pure. All the shared route adds under that condition is arithmetic over the window slots, an overflow guard that stages nothing, and a copy of the entry into a GenericArrayDeque that is popped straight back out — and the token that copy denotes is the one handed over here.

The end-of-input offset the fill’s None arms need is not read on this route because those arms are unreachable with a head in hand; the trace hook the fill opens with is emitted here instead, so a trace build sees the same one event per call either way.

What is guaranteed identical, and the condition that makes it so

The same condition skip_while states applies here, and the difference it covers is much the smaller of the two: a peek commits nothing, so there is no frontier to clone and no mark to take on either route. What differs is confined to the cache surface and one offset read. Measured, on the same stream in the same residency, by the effect ledger in fast_path_tests:

caller-supplied step, for one width-1 read	this route	the general route
Cache	one front	one len, one peek — the fill’s want == 0 arm
L::Offset::clone	0	1 — the committed span’s end, hoisted here above the fill
L::Span::clone, L::State::clone	0	0
Emitter::checkpoint / release / any emission	none	none

As on skip_while, that table is a measurement and not a boundary: it holds the steps the ledger was built to watch, and the clause below is about every caller-supplied operation whether a table names it or not.

What this route does differently — the whole of it

Three clauses, meant as exhaustive. Against the general route, for every call it answers, this route:

1. omits caller-supplied steps and adds none. The hoisted L::Offset::clone is the one the ledger sees; as with clause (1) on skip_while, which steps is not part of the clause — it is a subset relation, not a list;
2. substitutes one Cache::front for the fill’s len + peek. All three are &self reads the cache contract defines as changing no observable, so they are the same read of the same head;
3. hands f the same token and the same span, once.

The condition on the caller

What follows holds for a caller who meets both clauses of the condition on skip_while, read here with f in the predicate’s place:

• your input-layer callbacks are inert — every caller-supplied operation this crate can reach through the input layer does only what its own contract says, and always returns normally: no unwind, no divergence. All of it, not a list. Likely to be yours: the Clone, Drop, Ord and Hash of L::Offset and L::Span, the Clone and Drop of L::State, every Cache method, the Emitter, the Lexer and its Source — named as the ones you are likely to write, not as the edge of the clause;
• f is a function of what it is handed — it answers from the values of the Spanned<&L::Token, &L::Span> it receives, not from state another callback wrote and not from the addresses those references carry.

The closure argument is the same one, and it is why this is a condition rather than a list: the three clauses above say the crate hands f the same values, so any difference must come from caller-supplied code; the first clause makes all of that code invisible whether it runs or not, and the second stops f reading which route produced its argument. F: FnOnce(..) -> O may capture whatever it likes and L::Offset is your type, so neither clause is a formality — but both are properties of your own types, checkable once.

For such a caller, guaranteed identical

The value handed to f, and therefore the value returned; that f runs exactly once; that nothing is consumed, committed, emitted or latched; that a resident head is served at a latched poison boundary and a non-resident one still raises the terminal end-of-input error.

And for a caller who does not meet it

Both of these are one clause failing, and both are measured. Neither is the list of ways to fail — a caller who breaks a clause some other way gets the same answer:

• an f that reads the input layer — the second clause. The offset row above is the whole mechanism: the general route takes self.span().end() before the fill, for a terminal end-of-input error a resident head makes unreachable, and this route does not, so the returned O differs between them. An O that differs is a parse that differs — head_satisfies and peek_kind ride this call, so the value in question is routinely a grammar decision. (an_offset_clone_counting_f_can_change_the_value_peek_head_map_returns)
• an L::Offset::clone that unwinds — the first clause, and again the sharper case, because such a caller’s f may observe nothing whatsoever. That hoisted clone is the general route’s first caller step: arm it to panic and the general route unwinds before f, where this route never reads the offset and returns Ok(Some(_)) with f run once. Whether f ran at all is then decided by which route answered. (an_unwinding_offset_clone_decides_whether_peek_head_maps_closure_runs_at_all)

An f whose answer — or whose reachability — depends on how the input layer got the head to it is not asking about the head; it is asking which route this crate took, and that is a choice this crate makes and may change in any release. Answer out of the Spanned<&L::Token, &L::Span> you were handed, from types that clone by copying, and both clauses hold by construction — as they do for every f in this crate, its tests and its examples.

pub fn head_satisfies<F>( &mut self, pred: F, ) -> Result<bool, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnOnce(&L::Token) -> bool, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

Does the head satisfy pred?

false at genuine end of input; a terminal stop is an error. Replaces the consumer-side always-decline try_expect hack, which answered false for both.

pub fn peek_kind( &mut self, ) -> Result<Option<<L::Token as Token<'inp>>::Kind>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

The head’s kind, on the same terminal-aware read as peek_head_map.

The method form of the free peek_kind. Note the contract difference from a hand-rolled peek::<U1>() fork: that discards the terminal flag, so a latched boundary reads as Ok(None); this raises.

pub fn pratt<FoldPrefix, FoldInfix, FoldPostfix, Expr, Power>( &mut self, fold_prefix: FoldPrefix, fold_infix: FoldInfix, fold_postfix: FoldPostfix, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PrattToken<'inp, Expr, Power>, Ctx::Emitter: PrattEmitter<'inp, L, Lang>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<RecursionLimitReached<L::Offset, Lang>> + From<NonAssociativeChain<L::Offset, Lang>>, Power: PrattPower, FoldPrefix: PrattFoldTokenPrefix<'inp, Power, L, Ctx, Lang>, FoldInfix: PrattFoldTokenInfix<'inp, Power, L, Ctx, Lang>, FoldPostfix: PrattFoldTokenPostfix<'inp, Power, L, Ctx, Lang>,

Available on crate feature pratt only.

Runs a token-level Pratt expression parse over this input.

This is the low-level, token-centric Pratt API. It requires the token type to implement PrattToken, which classifies each token as an operand, prefix, infix, or postfix operator. The fold closures receive raw Spanned tokens rather than typed AST nodes.

Equivalent to calling pratt_with_min_precedence with Power::default() as the minimum binding power.

For a more ergonomic higher-level API that works with any AST node type, prefer the pratt free function instead.

CST-unsupported

This token-level API folds expressions into synthetic tokens — spans covering already-folded regions with no node-kind seam to classify — so it carries no CST hook in this version. A parse that should build a syntax tree uses the typed driver and its with_cst_kinds classifier instead; the committed tokens this API consumes still auto-flow to a recording sink, but no expression nodes are recorded around them.

Parameters

• fold_prefix – called with (operator_tok, operand_tok, emitter) when a prefix operator and its operand have been successfully parsed.
• fold_infix – called with (lhs_tok, rhs_tok, operator_tok, emitter) when an infix operator and both operands have been parsed.
• fold_postfix – called with (operand_tok, operator_tok, emitter) when a postfix operator has been applied.

Returns

Ok(Some(tok)) with the combined expression token on success, Ok(None) if the input cursor did not see an LHS token, or Err(e) on a fatal emitter error.

Two failures that are not the emitter’s

Both are returned, never emitted, so no recording emitter and no rewind can turn either into a truncated-but-successful parse:

• RecursionLimitReached — the parse’s shared depth budget (see descend) ran out at this frame’s prologue. Terminal, and terminal independently of the grammar’s error type: the trip latches the input session, so a grammar whose error discards the value on conversion (() included) loses the payload and still gets the re-raise — see RecursionLimitReached’s own docs for what a discarding sink costs and does not cost.
• NonAssociativeChain — a second same-power PrattInfix::Neither operator appeared in one chain. The operator is left on the input, unconsumed. Not terminal: recovery may spend it.

pub fn pratt_with_min_precedence<FoldPrefix, FoldInfix, FoldPostfix, Expr, Power>( &mut self, fold_prefix: FoldPrefix, fold_infix: FoldInfix, fold_postfix: FoldPostfix, min_precedence: Power, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PrattToken<'inp, Expr, Power>, Ctx::Emitter: PrattEmitter<'inp, L, Lang>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<RecursionLimitReached<L::Offset, Lang>> + From<NonAssociativeChain<L::Offset, Lang>>, Power: PrattPower, FoldPrefix: PrattFoldTokenPrefix<'inp, Power, L, Ctx, Lang>, FoldInfix: PrattFoldTokenInfix<'inp, Power, L, Ctx, Lang>, FoldPostfix: PrattFoldTokenPostfix<'inp, Power, L, Ctx, Lang>,

Available on crate feature pratt only.

Runs a token-level Pratt expression parse over this input starting at a given minimum binding power.

This is the low-level, token-centric Pratt API. It requires the token type to implement PrattToken, which classifies each token as an operand, prefix, infix, or postfix operator. The fold closures receive raw Spanned tokens rather than typed AST nodes.

Only operators whose binding power is greater than or equal to min_precedence will be consumed. Operators below the threshold are left in the input for the surrounding context to handle. This is useful when embedding a Pratt expression inside a larger grammar — for example, parsing only the right-hand side of an infix operator at a specific precedence level.

Use pratt instead when you want to parse a full expression starting from Power::default().

Parameters

• fold_prefix – called with (operator_tok, operand_tok, emitter) when a prefix operator and its operand have been successfully parsed.
• fold_infix – called with (lhs_tok, rhs_tok, operator_tok, emitter) when an infix operator and both operands have been parsed.
• fold_postfix – called with (operand_tok, operator_tok, emitter) when a postfix operator has been applied.
• min_precedence – the minimum binding power; operators strictly below this level are not consumed.

Returns

Ok(Some(tok)) with the combined expression token on success, Ok(None) if the input cursor did not see an LHS token, or Err(e) on a fatal emitter error.

Plus the two returned-not-emitted failures pratt documents: RecursionLimitReached and NonAssociativeChain.

pub fn skip_while<F>( &mut self, pred: F, ) -> Result<(), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool,

Consumes consecutive tokens matching pred without reporting them.

Advances the cursor past every leading token for which pred returns true, stopping before the first token for which it returns false (that token is left unconsumed) or at end of input.

Unlike sync_to, the skipped tokens are not reported through emit_unexpected_token: they are expected and simply dropped. Genuine lexer errors encountered while skipping are still emitted, so a fatal emitter can abort on a malformed token. Already-cached (peeked) tokens are drained identically to freshly-lexed ones.

This is the primitive used to skip trivia (whitespace, comments) in the padded, padded_left, and padded_right combinators, where trivia must be consumed but must never surface as an error.

The token that stops the skip is left at the cache front

A token this call examined but did not consume is unconsumed: it is put back at the front of the peek cache — where try_expect puts the token its predicate declined, and the one place cursor reads. So the resume cursor after a skip is the stopping token’s start, whether that token had been peeked into the cache beforehand or this call lexed it a moment ago, and the next read serves it without re-lexing. The cache is an invisible optimization here as everywhere: nothing a caller can observe about a skip_while — the committed span and lexer state, the cursor, the diagnostics, the poison boundary, the dedup watermark, the tokens read next — depends on how deep it had peeked. The cache_transparency_matrix tests in src/input/input_ref/tests.rs pin that across this method and padded. That promise, and every value promise on this method, is made to callers who meet one two-clause condition — input-layer callbacks that are inert, and a predicate that is a function of what it is handed. It is stated in full under What is guaranteed identical below, and it is what makes “invisible” true rather than merely usually true.

A stopping token this call found already in the stream is not even removed: the complete-input route below judges each token where it lies. “Left at the cache front” is therefore satisfied by doing nothing at all, rather than by a pop and a matching push.

Partial mode: an Incomplete exit leaves no trace

Under Partial, a non-final buffer can end mid-scan and this method surfaces Incomplete. That exit commits nothing, so it keeps nothing: the position, the lexer state, the dedup watermark and every emission the aborted attempt made — each skipped token’s commit_token event included — are restored to the call’s entry. Refill and call again and the retry is idempotent; nothing accumulates per attempt.

Panic unwind

A panic out of the predicate, the emitter, or the lexer anywhere but the end-of-input settle is an exit too, and it settles with this method’s own posture: sync_to’s commit-and-keep, reporting nothing. This mode holds no pre-call snapshot, so every unwind keeps — there is no earlier state left to restore to, only the progress already committed — and its settle stops before the stopping token, exactly as sync_to’s does, which is why the frontier is what an interrupted stop still has to commit. REPORT_SKIPPED is false for this mode, so the prefix an unwind keeps is skipped, not diagnosed: no unexpected-token report is ever built for a token this call skips, so a panic has nothing to lose on that axis — only the genuine lexer errors crossed along the way are ever reported, and those settle and unwind exactly as any other committed emission does. No token leaves the stream, and no emitter mark is stranded. The two routes reach that by different means, and the difference is the whole of what Two routes, one skip below is about — the complete-input route holds no uncommitted position at any point where caller code runs, and holds a token out of the stream at exactly one call, which LexedFront owns across; the partial-input route keeps the shared scanner’s ScanScope, whose Drop puts the in-flight token back, commits the frontier and settles the entry mark.

That one call is the predicate, over a token the lexing run has just produced, and it is worth stating why the resident run needs no such guard while this does: a resident token is judged where it lies, so an unwinding predicate there has nothing to repair, while a lexed one exists only in this loop’s hand until it is committed or put back. Left unowned, an unwinding predicate dropped it — the complete input then re-lexed it from the previous committed span where the sealed-Partial input, whose scope holds the same token, resumed with it resident. The two answers are within one cursor range of each other, which is why the panic sweep passed over it; they are not the same observation, and the_two_completeness_routes_observe_the_same_unwound_skip now compares them exactly.

The one exit the two routes answer differently: an unwind inside the end-of-input settle

The exclusion in the first sentence above is not a hedge, and this is the whole of it. When a skip reaches end of input both routes finish the same way — read Lexer::span, take Lexer::into_state, write the pair — and on the scanned route those two calls run after ScanScope has been disarmed, so the frontier the settle was about to commit is dropped along with the unwind. This route has no frontier to drop: it committed each token as it crossed it, and those commits stand.

Measured over "ab cd ef gh" with a predicate that accepts everything and into_state armed to panic — the_two_completeness_routes_are_pinned_apart_on_an_interrupted_eof_settle in src/input/input_ref/tests.rs, which pins both columns as deliberate rather than asserting they match:

after the unwind	Complete, this route	sealed Partial, the scanner
committed span, and the tally of the state beside it	(9, 11), 4	(0, 0), 0
resume cursor	11	0
what the next reads yield	nothing; the input is spent	all four tokens, lexed again
commit_token notifications the interrupted call made	4	4

The claim is narrowed rather than the behaviour changed, and the last row is why. Both routes told the side channel that four tokens were consumed. This route’s committed position says the same thing; the scanner’s says the call never happened, so a recording sink is left holding four settles for tokens the input then serves a second time. Keeping the progress a call actually made is the better of the two answers, and it is the one the per-token commit produces by construction — the design of this route, not an accident of where the unwind landed. Degrading it to agree would cost the whole of what Two routes, one skip measures, and it would buy agreement on an unwind only a lexer, source, span or offset callback can raise: exactly the code The condition on the caller below already requires to be inert, and never the predicate, which is the one callback that condition does not cover. That asymmetry is the reason the predicate divergence directly above was fixed and this one is stated.

And it is this window and no wider. Measured rather than asserted, arming one step at a time over the same source and comparing the two routes’ whole residue: Lexer::lex at each of its five calls, Lexer::span at each of its five reads but the settle’s, the predicate at every call, the committed-token observer, and the emitter’s diagnostic path over a crossed limit trip all leave the two routes identical — as does every run in which nothing unwinds at all. The cell named above ships the no-unwind and emitter controls beside the divergence; the predicate sweep is the_two_completeness_routes_observe_the_same_unwound_skip. What is not claimed either way is a caller Clone or Drop that unwinds: the two routes run different numbers of those by construction, which is the first clause of the condition failing and is listed under And for a caller who does not meet it below.

Two routes, one skip

The grammar shape this primitive exists for — skip trivia, then look — asks for a skip at every decision point, and on a GraphQL parse those skips are 39,057 calls for 13,014 trivia tokens: 22,033 of them have nothing to skip at all. Routing each of them through the shared four-mode scanner costs a ScanScope, a clone of the frontier pair, a take-test-put-back of the token the skip stops on, and a commit of a position that in the common case did not move.

So a Complete input does not use the scanner here. It runs a loop of its own, in two phases:

1. the resident run — while a token is at the front of the stream, judge it where it lies through front. A rejection returns immediately, having removed nothing; an acceptance consumes it through commit_front, the settle that clamps the position before the token leaves the stream;
2. the lexing run — once the stream is empty, lex through scan_with exactly as the scanner does, holding each lexed token in a LexedFront across the predicate, committing it there if the predicate takes it and putting it back at the front of the stream if it does not.

A Partial input keeps the scanner. That is a typestate split, not a second fast path: the partial-input route owes an Incomplete exit that restores five facts and settles an emitter mark on every exit including an unwind, which is precisely what ScanScope exists to own, and the measured cost above is a complete-input parse. Neither route carries the other’s machinery, and no route carries both.

What this route keeps of the scan scope, and what it does not

Not “it needs no scope”. It needs the scope’s token slot, in the lexing run and only there, and nothing else the scope carries — and the heading says so because the earlier one said the opposite over a body that was already correctly qualified, which is the half a reader retains.

Under Complete the scope’s Drop does exactly two things — put the in-flight token back at the front of the stream, and commit the frontier the loop accumulated — and this route makes one of them unnecessary rather than skipping it, and keeps the other where it is actually owed:

what the scope settles	what this route does instead
the in-flight token, resident run	nothing: a resident token is judged where it lies and never leaves the stream until the settle that accounts for it; the clamp — the settle’s one fallible step — runs while it is still there
the in-flight token, lexing run	LexedFront, the scope’s token slot and nothing else of it: a token this loop lexed is owned across the predicate and put back by the guard’s Drop, so the stop and an unwinding predicate take the same exit
the uncommitted frontier	there is none: every token this route crosses is committed as it crosses it, so the committed position is the frontier at every instant
the entry mark	Complete takes none — the capture is behind Cmpl::PARTIAL and never monomorphizes

The scope’s own HOLDS_ENTRY = false for SkipWhile, so its unwind edge always keeps: the disposition this route lands on by construction is the one the scope was going to choose.

The split in the first two rows is the whole of what the measurement bought, and it is the reason the guard is where it is. The census that motivates this route is 22,033 skips that skip nothing out of 39,057 — every one of them answered by the resident run, which constructs no guard, and none of them by the lexing run, which is entered once per non-resident token and has already paid for a lex by the time it gets there.

Where the limit trip latches

A resource trip latches the poison boundary at the durable frontier — the offset up to which what the scan already passed stays reproducible. The scanner latches it through AtFrontier, its deferred, uncommitted frontier; this route latches it through AtCursor, the committed cursor. They are the same offset, and for the same reason the frontier commit disappears: this route commits each skipped token as it crosses it, so with the stream empty — the only state in which it lexes — offset reads span().end(), which is the end of the last skipped token, which is exactly what the scanner’s frontier holds. Everything downstream of the latch is unchanged, because it all lives inside scan_with: the trip is diagnosed there, deduplicated there, and the fatal-emitter exit is taken there. This route only decides what to do afterwards, and the answer is nothing — the progress a trip keeps is already committed.

The pre-lex probe is the other half: once the lex position has reached a latched boundary there is no token left to scan, so this route stops without rebuilding a lexer, exactly as the scanner does — and, again, with nothing left to commit.

What is guaranteed identical, and the condition that makes it so

Producing the same values is not the same as running the same code, and in a generic library the difference is not academic: L::Span, L::State and L::Offset are the caller’s own types, so every operation the input layer performs on one of them — clone, drop, compare, hash, format — is caller code, as are Emitter::checkpoint and release, every Cache method, and the Lexer with its Source. This route runs a different set of them. Some are measured, on the same stream in the same residency, by the effect ledger in fast_path_tests, for the call that dominates the census — a skip that skips nothing:

caller-supplied step, for one no-op skip	this route	the scan it replaces
L::Span::clone	0	1 — 2 under Partial
L::State::clone	0	1 — 2 under Partial
L::Offset::clone, taken directly	0	0 — 2 under Partial
Emitter::checkpoint, then release	none	none — one of each under Partial
Cache	one front	one pop_front, one push_front
the predicate	1	1

The offset row counts only the clones the input layer takes itself — the entry capture’s dedup watermark and rewind offset. A caller’s own L::Span::clone may clone offsets on top of that, so a span type built from two clonable offsets sees two more per span clone; the in-tree witness measures 2 under Complete and 6 under Partial on that shape, against 0 on this route.

A skip that does skip goes the other way on some rows, and that is stated rather than hidden: the scanner clones the frontier pair once and clamps once at the end, while this route runs the clamp — a Source::len, an L::Offset comparison and an L::Span::clone — once per token. It also asks the cache for its front once per token where the scan pops once per token. Fewer caller steps for the 56% of calls that skip nothing, more for the ones that skip a run, and the condition below is what makes the direction of the difference not matter.

That table is a measurement, not a boundary, and the distinction is the whole lesson of how this contract was arrived at. The table carries the steps the ledger was built to watch. The scan’s closing commit also runs a Source::len and one L::Offset comparison — the clamp inside commit_position, measured once each on the scan and zero times here for a no-op skip — and those are simply two the table never had. Successive readings of this method have each found an operation the previous naming did not contain, so the condition below is quantified over all caller-supplied code and every list inside it, this table included, is illustration. What fast_path_tests pins is the emptiness of this route’s side for a no-op skip, not an inventory of the scan’s: an inventory would be one review round out of date.

What this route does differently — the whole of it

Four clauses, meant as exhaustive and not as illustration. Against the scan it replaces, for every call it answers, this route:

1. runs a different multiset of caller-supplied steps — it may omit one, run one the scan never runs, or run one a different number of times. Which steps is deliberately not part of the clause: the table above measures some, the elided drops and the per-token clamp are more, and the clause is about all of them. It was a subset relation while this route answered only the skips that skip nothing; it is not one now, and saying so is cheaper than maintaining the list that made it true;
2. substitutes a Cache::front for the pop_front + push_front pair the scan uses to look at the same head. The cache laws make that pair an identity on the entry it came from, so the two are the same read of the same token;
3. reorders: pred runs before the steps of (1) that the scan takes ahead of it — the entry capture, the frontier clone — rather than after them;
4. hands pred the same token and the same span, once per token, in the same order — which the residency matrix pins directly.

A step this route does not take is also a value it does not produce, and therefore one it does not drop: the scan runs an L::Span::drop and an L::State::drop on frontier clones that this route never creates.

The condition on the caller

Everything guaranteed below holds for a caller who meets both of these. They are properties you check once, about your own types — not a list of differences to keep up with:

• your input-layer callbacks are inert. Every caller-supplied operation this crate can reach through the input layer does only what its own contract says it does, and always returns normally: it does not unwind, and it does not diverge. Then running one, running it twice, running it a hundred times and not running it at all are the same thing to everyone. That is the clause, and it is not a list. The ones you are likely to be the author of: the Clone and Drop of L::Span, L::State and L::Offset, and the Ord and Hash the bounds also ask of the span and the offset; Emitter::checkpoint and release; every Cache method; the Lexer and its Source. Those are named because they are the ones you are likely to write, not because they are the boundary — the boundary is the sentence above them, and each time this contract named a set instead, the next reading found something outside it;
• pred is a function of what it is handed. It answers from the values of the Spanned<&L::Token, &L::Span> it receives — not from state some other callback wrote, and not from the addresses those two references carry (the scan asks about a token it has moved out of the cache; this route asks about it where it lies). Recording that a call happened, in your own state, and reading the record after the call returns, is fine: the call sequence is itself guaranteed identical below.

Why those two are the whole condition. Any difference between the routes has to be produced by something. Clauses (1)–(4) say the crate’s own contribution is the same values in the same order, so the only remaining producer is caller-supplied code; the first clause covers all of that code and makes omitting it, adding one, running it a different number of times, and running it in a different order produce nothing; the second says your own predicate cannot read which route produced its argument. Nothing is left over. That closure is the point of stating a condition rather than listing exclusions — a list has to anticipate every way a caller might differ, and every way found so far is an instance of one clause rather than a new entry: a Clone that keeps state, a Clone that panics, a Drop elided along with its clone, the Source::len and L::Offset comparison a clamp runs, and a pred that reads its argument’s address. The first three arrived as three separate escalations of a list that was each time believed complete; the last two arrived after the clause replaced it, and needed no change to it — nor did the per-token clamp, which is the first difference that runs more caller code rather than less.

Two things the condition does not cover, because no fast path could: time and stack. This route is quicker and shallower, which is the whole reason it exists.

For such a caller, guaranteed identical

Pinned by the residency matrix: the parse result; the tokens read next and the order they arrive in; the resume cursor; the committed span and lexer state; the diagnostics; the poison boundary and the dedup watermark; the tokens the predicate is asked about, in order, and how many times it is asked; and the emitter’s outstanding-mark count. Pinned across the typestate split as well: a sealed Partial input takes every decision a Complete one takes, so the two routes are held to the same observation over the same program, source and cache-capacity sweep — with one exit excluded, which a caller who meets the condition above cannot reach: an unwind inside the end-of-input settle leaves this route holding the prefix it crossed and the scanner holding nothing. Panic unwind above states that difference, measures it and bounds it; every callback that can raise it is one the first clause requires to be inert.

And for a caller who does not meet it

Not a second list to maintain, and not one that has to be complete — each of these is one of the two clauses failing, and each is measured in fast_path_tests, so the condition is a fact rather than a caveat. A caller who breaks a clause in a way no bullet here describes gets the same answer: the guarantees are not made to them.

• a Clone that keeps state, and a pred that reads it — the second clause. The table above is the entire mechanism: the scan clones the frontier pair before it asks, this route asks first and clones nothing, so a predicate keyed on that counter answers one way here and the other way there. The answer to a skip predicate is the skip, so what differs is not a count but the parse — the committed cursor, the tokens consumed, everything downstream — and it differs in both directions: such a caller can make this route stop where the scan skips the whole stream, and consume the head where the scan consumes nothing. (a_clone_counting_predicate_can_change_the_skip_decision)
• a Clone that unwinds — the first clause, and the sharper case, because the caller here satisfies the second one completely. A pred that records nothing but its own calls observes no input-layer side effect at all, and still cannot be promised the same call sequence: with a head this route accepts, pred runs once and then the clamp’s L::Span::clone panics, where the scan panics before asking anything — one call against none, measured. Catch the unwind and the two routes have left your predicate in different states. And a no-op skip that would have unwound returns Ok(()) instead. (an_unwinding_caller_clone_leaves_the_predicate_with_a_different_call_count, a_no_op_skip_over_a_resident_head_reaches_no_panicking_caller_clone)
• an emitter or a cache that counts — the first clause, with the second deciding whether it matters. Under Partial a mark-keyed emitter sees one fewer complete, empty checkpoint/release cycle per no-op skip, and a counting cache sees one front where the scan makes a pop_front and a push_front. Neither can see a difference in what it holds: the cycle is empty and balanced and release is documented advisory, and the pop/push pair leaves the cache with the contents front read. The count becomes a parse only once it reaches pred.
• a Drop that is not inert — the first clause again, and the one a list of clone counts would have missed: a frontier clone this route never takes is a value it never drops. Measured over one no-op skip: one L::State::drop and three L::Offset::drops on the scan under Complete, two and seven under Partial, against none of either here. (a_no_op_skip_runs_no_caller_drop_where_the_scan_runs_one_per_frontier_clone)
• a pred that reads its argument’s address — the second clause, in the half that says values. The scan moves the head out of the cache into the scan scope and asks about it there; this route asks about it where it lies. A predicate that compares the address it is handed against the cache’s own front entry therefore answers one way here and the other there, and the answer to a skip predicate is the skip: one token consumed against none. This one is not even new to the fast paths — the cache has always been an invisible optimization, and how deep a caller peeked has always decided where a token sits — which is exactly why it belongs in a condition on the caller rather than in a list of this route’s differences. (an_address_reading_predicate_can_change_the_skip_decision)

The condition is reasonable, not merely convenient. A predicate that answers differently depending on how the input layer got the token to it is not asking a question about the input at all — it is asking about this library’s internal route, and which route answers a given call is a choice this crate makes and may change in any release. A callback that unwinds asks the same question in control flow: it turns which steps ran into an outcome, and which steps ran is not part of the contract either.

The crate meets its own condition and holds itself to it. Every span, state and offset it ships clones by copying fields and compares and hashes by derive over integers, with no side effect and no panic path — surveyed impl by impl — so the first clause holds for every in-tree lexer by inspection, and holds for the operations no table names as readily as for the ones it does. That is what checking a clause instead of a list buys. Every skip_while predicate it writes, in its own combinators, its tests, its benches, its examples and its conformance kit, answers out of the token it was handed, and the adversarial fixtures that do count clones read the counter in an assertion after the call, never inside the predicate. It is all written down because it is the honest boundary of the claim above.

pub fn sync_balanced<D, F>( &mut self, classifier: D, pred: F, ) -> Result<Option<Hole<L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where D: DelimClass<<L::Token as Token<'inp>>::Kind>, F: FnMut(Spanned<&L::Token, &L::Span>) -> bool,

Skip tokens, nesting-aware, until pred matches at delimiter depth zero; stops before the matching token and returns the Hole describing the skipped region.

classifier names which token kinds open and close pairs (DelimClass / Balance); pred is the depth-0 sync predicate. Each scanned token is decided in this order:

• at depth zero, pred is consulted first — so an opener or a stray closer that is itself a sync point (the classic } recovery target) syncs rather than counting;
• otherwise the token is skipped into the hole: an Balance::Open kind increments the depth, a Balance::Close kind decrements it saturating at zero (a stray closer at depth zero is plain garbage — skipped, never driving the depth negative), and a Balance::Neutral kind leaves it unchanged. Strictly above depth zero pred is never consulted, so garbage containing balanced pairs skips over enclosed sync-set tokens.

Depth counting is token-level, which leans on a lexer-contract clause: a composite token (a block string, a raw literal) is one token whose lexer already swallowed any delimiter characters inside it, so nothing inside a token can affect the depth. The count is also pair-blind: a closer closes the innermost open pair regardless of its Balance::pair identity — inside garbage, mismatched pairs are part of what is being skipped, and the parse that resumes at the sync point decides what they meant.

One diagnostic per hole

The skipped tokens are not reported individually. A successful sync that skipped at least one token reports the whole region exactly once through Emitter::emit_skipped_region, with the hole’s span and count; a zero-skip success (the sync point was the very next token) emits nothing. Genuine lexer errors crossed while skipping are still emitted (deduplicated) along the way, and they are not counted into skipped — the count covers valid tokens only. A fatal emitter rejection mid-skip follows the sync family’s fatal-exit discipline: the error token is committed and the error propagates, exactly as in sync_through.

Diagnostics travel with progress

A match commits the skipped prefix — the cursor stops before the sync token — and its hole diagnostic persists; the emission is rewind-safe by construction, because a skip is committed forward progress and an enclosing rollback unwinds the emission with the log like any other entry. A resource-limit trip mid-skip commits the skipped prefix at the durable frontier and returns Ok(None) — committed progress, but a failed sync, so no hole diagnostic is emitted for it. A no-match run to end of input commits nothing and returns Ok(None), leaving no trace: the cursor stays at the pre-call position, the emissions made during the failed scan (the lexer errors it crossed) are unwound, and the lexer-error deduplication watermark is restored, so a later genuine consume of the same region reports its errors exactly once. One diagnostic per hole means no diagnostic for a failed hole. As in sync_through, this holds even when the caller had prefilled the cache with peeked lookahead: a failed sync rewinds the drained cache prefix too, at the cost of re-lexing those tokens on the next read.

Partial mode: an Incomplete exit leaves no trace

Under Partial, a non-final buffer can end mid-scan and this method surfaces Incomplete. That exit commits nothing, so it keeps nothing: the position, the lexer state, the dedup watermark and every emission the aborted attempt made — each skipped token’s commit_token event included — are restored to the call’s entry. Refill and call again and the retry is idempotent; nothing accumulates per attempt.

Panic unwind

A panic out of the predicate, the classifier, the lexer or the emitter anywhere but the end-of-input settle is an exit too, and it settles with this method’s own posture — which is neither of the other two’s, because this is the method where the family’s two axes cross. It stops before the stopping token and commits the skipped prefix, exactly as sync_to does; it rewinds the full pre-call state at a no-match end of input, exactly as sync_through does. So the disposition of an unwind belongs to the exit, not to the method, and branching on the method alone is a defect this crate has already paid for:

• once the predicate has answered stop, in the stop settle itself, the scan keeps the prefix its own stop keeps: the position stands at the last skipped token. The in-flight token has been handed to that settle by then, so it is not put back — the put-back is precisely the step the unwind interrupted — and the retained stream is cleared against the committed position instead, leaving a catching host’s retry to re-lex the swallowed stopper. (sync_to states the held-versus-handed-over split in full; this method’s keeping arm sits entirely on the handed-over side of it.) r9_balanced_stop_exit_panic_keeps_the_prefix_like_its_own_stop_does (src/input/input_ref/tests.rs) reads a committed (3, 5) over "ab cd ef gh", with four lex calls and nothing left live;
• anywhere else — mid-scan, and equally once the frontier has left the scope for the very commit_at that would record it — the scan abandons, to the same restore-to-entry its no-match end of input performs: the stream is cleared and the position, the lexer state, the dedup watermark and every emission the scan made come back off. r9_frontier_commit_interrupted_abandons_rather_than_half_keeping reads ((0, 0), 0, 0, 0) — keeping there instead would leave the per-token settles describing a prefix the position does not cover, which a retry duplicates.

Either way the committed position never advances past a token nothing recorded, so no token is lost and no emitter mark is stranded. Note that the prefix in question is skipped, not diagnosed: this scan makes no per-token report, so the emissions an unwind decides the fate of are the genuine lexer errors it crossed — the one hole diagnostic is emitted only after a successful sync returns, which an unwind never reaches.

The exclusion is the one skip_while states, and it costs this method nothing: the scope is disarmed before the end-of-input settle runs, but that settle is this method’s rewind, and a scan that reaches it has committed no progress for a dropped frontier to strand — the position was never off the call’s entry. It is the stop exit above, not this one, where the frontier is load-bearing.

pub fn sync_through<F, Exp>( &mut self, pred: F, exp: Exp, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool, Exp: FnMut() -> Option<Expected<'inp, <L::Token as Token<'inp>>::Kind>>,

Skip tokens until the predicate matches, emitting lexer errors along the way.

If the predicate matches, the matching token is consumed and returned.

Diagnostics travel with progress: a match (or a resource-limit trip) commits the skipped prefix, so the diagnostics describing it persist. A no-match run to end of input commits nothing — the cursor stays at the pre-call position — and leaves no trace: the emissions made during the failed scan are unwound and the lexer-error deduplication watermark is restored, so a later genuine consume of the same region reports its errors exactly once.

This holds even when the caller had prefilled the cache with peeked lookahead: a failed sync rewinds the drained cache prefix too, restoring the pre-call position, at the cost of re-lexing those formerly-cached tokens on the next read.

The fatal exit commits, and the cache never changes it

A fatal emitter rejection mid-skip follows the sync family’s fatal-exit discipline: the token that trips the emitter — a skipped token whose unexpected-token diagnostic the emitter rejected, or a lexer error it rejected — is committed, and the error propagates. A caller that catches it therefore resumes after the reported token, and never re-reads or re-reports it.

Whether that token had already been peeked into the cache makes no difference: the cache is an invisible optimization, so every observable of a sync call — its return, the committed position and lexer state, the diagnostics it emits, the poison boundary, and the lexer-error dedup watermark — is a function of the token stream alone, never of how much of it had been prefetched. (The one thing a caller can see is that a peek emits the lexer errors it crosses when it crosses them, so prefetching moves such a diagnostic earlier in the log; the dedup watermark still reports it exactly once.) The cache_transparency_matrix tests in src/input/input_ref/tests.rs pin this across the whole family.

Partial mode: an Incomplete exit leaves no trace

Under Partial, a non-final buffer can end mid-scan and this method surfaces Incomplete. That exit commits nothing, so it keeps nothing: the position, the lexer state, the dedup watermark and every emission the aborted attempt made — each skipped token’s commit_token event included — are restored to the call’s entry. Refill and call again and the retry is idempotent; nothing accumulates per attempt.

Panic unwind

A panic out of the predicate, the expected-tokens closure, the lexer or the emitter anywhere but the end-of-input settle is an exit too, and it settles with this method’s own posture — which is not sync_to’s to-shaped commit. This scan consumes the stopping token, committing at its own span, and rewinds the full pre-call state at a no-match end of input, and its unwind edge follows that same split, decided by the exit reached rather than by the method:

• before the predicate has answered stop — the whole of the scan, in practice — an unwind takes the no-match exit’s posture: the retained stream is cleared and the position, the lexer state, the dedup watermark and every emission the abandoned scan made are restored to the call’s entry. sync_through_unwind_restores_emissions (src/input/input_ref/tests.rs) reads a resume cursor of 0 with all four tokens of "ab cd ef gh" still reachable and 0 emissions surviving. Restoring at the panic edge carries a price a true end of input does not — there the cache is empty by construction, here it can still hold an untouched suffix, which re-lexes — and that price is pinned rather than left as prose, at seven scans of the four-token source, by sync_through_warm_unwind_prices_its_re_lex;
• once it has, which is inside the stop settle itself, the scan keeps: the diagnosed prefix is committed at the frontier — the end of the last skipped token, since the commit at the matching token’s own span is the very step the unwind interrupted — and the stream is cleared against it. Cleared rather than put back, because by then the token has been handed to that settle: sync_to states the held-versus-handed-over split in full, and this method reaches its keeping arm only on the handed-over side of it.

Either way the committed position never advances past a token nothing recorded, so no token is lost and no emitter mark is stranded.

The exclusion is the one skip_while states, and it costs this method nothing: the scope is disarmed before the end-of-input settle runs, but for a rewinding scan that settle is the restore, and the scan holds every byte of its progress in an uncommitted frontier — the position was never advanced off the call’s entry for a dropped frontier to strand it away from. What remains at stake there is the rest of the restore, and it is measured rather than argued: r9_restore_entry_is_atomic_at_every_offset_clone (src/input/input_ref/tests.rs) sweeps every L::Offset clone the exit performs, panicking at each in turn, and demands the same three readings at all — the entry position, no stranded emitter mark, and the abandoned scan’s diagnostics rewound away rather than merely released.

pub fn sync_through_then_peek<'p, F, Exp, W>( &'p mut self, pred: F, exp: Exp, ) -> Result<(Option<Spanned<L::Token, L::Span>>, Peeked<'p, 'inp, L, W>), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool, Exp: FnMut() -> Option<Expected<'inp, <L::Token as Token<'inp>>::Kind>>, W: Window,

Skip tokens until the predicate matches, emitting lexer errors along the way.

If the predicate matches, the matching token is consumed and returned with the tokens peeked after it.

Diagnostics travel with progress, exactly as in sync_through: a match commits the skipped prefix, so the diagnostics describing it persist. A no-match run to end of input commits nothing — the cursor stays at the pre-call position — and leaves no trace: the failed scan’s emissions are unwound and the lexer-error deduplication watermark is restored, so a later genuine consume of the same region reports its errors exactly once. The returned peek is then empty. As in sync_through, the pre-call position is restored even when the caller had prefilled the cache with peeked lookahead — the drained cache prefix is rewound too, at the cost of re-lexing those tokens on the next read. A fatal emitter rejection mid-skip commits the token that tripped it, and the cache does not change that either (see sync_through).

pub fn sync_through_then_peek_with_emitter<'p, F, Exp, W>( &'p mut self, pred: F, exp: Exp, ) -> Result<(Option<Spanned<L::Token, L::Span>>, Peeked<'p, 'inp, L, W>, EmitterView<'p, 'inp, L, Ctx::Emitter, Lang>), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool, Exp: FnMut() -> Option<Expected<'inp, <L::Token as Token<'inp>>::Kind>>, W: Window,

Skip tokens until the predicate matches, emitting lexer errors along the way.

Returns the matched token, peeked tokens, and the emitter’s operations (an EmitterView — never the emitter; see that type for why).

Diagnostics travel with progress, exactly as in sync_through: a match commits the skipped prefix, so its diagnostics persist. A no-match run to end of input commits nothing — the cursor stays at the pre-call position — and leaves no trace: the failed scan’s emissions are unwound and the lexer-error deduplication watermark is restored, so a later genuine consume of the same region reports its errors exactly once. The returned peek is then empty. As in sync_through, the pre-call position is restored even when the caller had prefilled the cache with peeked lookahead — the drained cache prefix is rewound too, at the cost of re-lexing those tokens on the next read — and a fatal emitter rejection mid-skip commits the token that tripped it, cached or not.

pub fn sync_to<F, Exp>( &mut self, pred: F, exp: Exp, ) -> Result<Option<MaybeRefCachedTokenOf<'_, 'inp, L, L::Token>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool, Exp: FnMut() -> Option<Expected<'inp, <L::Token as Token<'inp>>::Kind>>,

Skip tokens until the predicate matches, emitting lexer errors along the way.

Advances through the stream, emitting each lexer error via the emitter. Stops before the first token for which pred returns true and returns it (without consuming). Non-matching non-error tokens are skipped but also reported via emit_unexpected_token.

The fatal exit commits

A fatal emitter rejection mid-skip follows the sync family’s fatal-exit discipline: the token that trips the emitter is committed and the error propagates, so a caller that catches it resumes after the reported token. This does not depend on whether the token was already in the peek cache — the cache is an invisible optimization (see sync_through).

Partial mode: an Incomplete exit leaves no trace

Under Partial, a non-final buffer can end mid-scan and this method surfaces Incomplete. That exit commits nothing, so it keeps nothing: the position, the lexer state, the dedup watermark and every emission the aborted attempt made — each skipped token’s commit_token event included — are restored to the call’s entry. Refill and call again and the retry is idempotent; nothing accumulates per attempt.

Panic unwind

A panic out of the predicate, the expected-tokens closure, the lexer or the emitter anywhere but the end-of-input settle is an exit too, and it settles with this method’s own posture, which is the to-shaped commit: every unwind keeps. This mode holds no pre-call snapshot — its end of input commits at the lexer’s end rather than rewinding to one — so an unwind has nothing to restore to, and the diagnosed prefix simply stands, committed at the frontier, the end of the last skipped token. Its stop settle stops before the stopping token, which is why the frontier is what an interrupted stop still has to commit. That is what makes this the one member of the family whose unwind disposition needs no case analysis; the two rewinding scans settle by their own postures, which are not this one, and each states its own — see sync_through and sync_balanced.

What does vary is the in-flight token, and it turns on whether the scan had let go of it yet — a distinction the scanner’s own TokenSlot makes, because the two states need opposite repairs:

• held — the token is out of the stream and not yet recorded, which is true across the predicate and nowhere else. A panic there puts it back at the front of the stream, so the stream is adjacent to the committed position again and nothing re-lexes;
• handed over — the token has gone to the stop settle, whose put-back is the step the panic interrupted. It cannot be put back, because putting it back is what failed. The retained stream is cleared instead, and the region re-lexes from the committed position, which reproduces that token and everything after it; what is lost is the cache’s memo of it, not a token;
• recorded — everywhere else, including the expected-tokens closure and the emitter’s own report, the skip has already adopted the token behind the frontier (its first act), so nothing is out of the stream and the frontier commit covers it.

So the committed position never advances past a token nothing recorded: no token is lost and no emitter mark is stranded. Measured on the handed-over case, at the stop settle, with the Cache::push_front it reaches armed to panic: r9_stop_exit_panic_still_commits_the_diagnosed_prefix (src/input/input_ref/tests.rs) reads a committed (3, 5) with both skip diagnostics standing, and the catching host’s retry resumes past them — re-lexing the swallowed stopper rather than re-diagnosing the prefix.

The exclusion is the one skip_while states, and it is the same settle: a committing scan accumulates the skipped run in an uncommitted frontier and its scope is disarmed before the end-of-input settle runs, so an unwind inside that settle drops the frontier and the position stays at the call’s entry — the diagnosed prefix is not kept there, only the diagnostics already emitted for it are. Measured with Lexer::into_state armed to panic over "ab cd ef gh": eof_commit_interrupted(true) in r9_committing_eof_commit_is_atomic_in_span_and_state (src/input/input_ref/tests.rs) reads ((0, 0), 0), the entry position beside the entry state. That cell drives the scanner at skip_while’s mode, whose end-of-input settle is this one’s, verbatim.

pub fn sync_to_then_peek_with_emitter<'p, F, Exp, W>( &'p mut self, pred: F, exp: Exp, ) -> Result<(Peeked<'p, 'inp, L, W>, EmitterView<'p, 'inp, L, Ctx::Emitter, Lang>), <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool, Exp: FnMut() -> Option<Expected<'inp, <L::Token as Token<'inp>>::Kind>>, W: Window,

Skip tokens until the predicate matches, emitting lexer errors along the way.

Returns peeked tokens and the emitter’s operations (an EmitterView — never the emitter; see that type for why). A fatal emitter rejection mid-skip commits the token that tripped it, exactly as in sync_to.

pub fn try_expect_open_angle( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is open_angle (<). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_open_angle( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be open_angle (<).

pub fn try_expect_less_than( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is less_than (<). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_less_than( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be less_than (<).

pub fn try_expect_close_angle( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is close_angle (>). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_close_angle( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be close_angle (>).

pub fn try_expect_greater_than( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is greater_than (>). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_greater_than( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be greater_than (>).

pub fn try_expect_open_brace( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is open_brace ({). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_open_brace( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be open_brace ({).

pub fn try_expect_close_brace( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is close_brace (}). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_close_brace( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be close_brace (}).

pub fn try_expect_open_paren( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is open_paren ((). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_open_paren( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be open_paren (().

pub fn try_expect_close_paren( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is close_paren ()). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_close_paren( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be close_paren ()).

pub fn try_expect_open_bracket( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is open_bracket ([). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_open_bracket( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be open_bracket ([).

pub fn try_expect_close_bracket( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is close_bracket (]). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_close_bracket( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be close_bracket (]).

pub fn try_expect_at( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is at (@). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_at( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be at (@).

pub fn try_expect_asterisk( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is asterisk (*). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_asterisk( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be asterisk (*).

pub fn try_expect_ampersand( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is ampersand (&). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_ampersand( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be ampersand (&).

pub fn try_expect_apostrophe( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is apostrophe (’). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_apostrophe( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be apostrophe (’).

pub fn try_expect_backtick( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is backtick (`). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_backtick( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be backtick (`).

pub fn try_expect_backslash( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is backslash (). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_backslash( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be backslash ().

pub fn try_expect_caret( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is caret (^). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_caret( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be caret (^).

pub fn try_expect_comma( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is comma (,). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_comma( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be comma (,).

pub fn try_expect_colon( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is colon (:). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_colon( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be colon (:).

pub fn try_expect_dot( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is dot (.). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_dot( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be dot (.).

pub fn try_expect_dollar( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is dollar ($). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_dollar( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be dollar ($).

pub fn try_expect_double_quote( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is double_quote (“). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_double_quote( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be double_quote (“).

pub fn try_expect_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is equal (=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be equal (=).

pub fn try_expect_exclamation( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is exclamation (!). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_exclamation( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be exclamation (!).

pub fn try_expect_bang( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is bang (!). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_bang( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be bang (!).

pub fn try_expect_hash( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is hash (#). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_hash( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be hash (#).

pub fn try_expect_hyphen( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is hyphen (-). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_hyphen( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be hyphen (-).

pub fn try_expect_minus( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is minus (-). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_minus( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be minus (-).

pub fn try_expect_pipe( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is pipe (|). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_pipe( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be pipe (|).

pub fn try_expect_plus( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is plus (+). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_plus( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be plus (+).

pub fn try_expect_percent( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is percent (%). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_percent( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be percent (%).

pub fn try_expect_question( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is question (?). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_question( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be question (?).

pub fn try_expect_slash( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is slash (/). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_slash( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be slash (/).

pub fn try_expect_semicolon( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is semicolon (;). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_semicolon( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be semicolon (;).

pub fn try_expect_tilde( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is tilde (~). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_tilde( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be tilde (~).

pub fn try_expect_underscore( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is underscore (_). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_underscore( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be underscore (_).

pub fn try_expect_arrow( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is arrow (->). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_arrow( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be arrow (->).

pub fn try_expect_thin_arrow( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is thin_arrow (->). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_thin_arrow( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be thin_arrow (->).

pub fn try_expect_fat_arrow( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is fat_arrow (=>). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_fat_arrow( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be fat_arrow (=>).

pub fn try_expect_pipe_arrow( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is pipe_arrow (|>). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_pipe_arrow( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be pipe_arrow (|>).

pub fn try_expect_pipe_forward( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is pipe_forward (|>). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_pipe_forward( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be pipe_forward (|>).

pub fn try_expect_colon_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is colon_equal (:=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_colon_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be colon_equal (:=).

pub fn try_expect_colon_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is colon_assign (:=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_colon_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be colon_assign (:=).

pub fn try_expect_logical_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is logical_equal (==). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_logical_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be logical_equal (==).

pub fn try_expect_logical_not_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is logical_not_equal (!=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_logical_not_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be logical_not_equal (!=).

pub fn try_expect_strict_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is strict_equal (===). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_strict_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be strict_equal (===).

pub fn try_expect_strict_not_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is strict_not_equal (!==). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_strict_not_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be strict_not_equal (!==).

pub fn try_expect_less_than_or_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is less_than_or_equal (<=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_less_than_or_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be less_than_or_equal (<=).

pub fn try_expect_greater_than_or_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is greater_than_or_equal (>=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_greater_than_or_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be greater_than_or_equal (>=).

pub fn try_expect_strict_less_than_or_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is strict_less_than_or_equal (<==). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_strict_less_than_or_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be strict_less_than_or_equal (<==).

pub fn try_expect_strict_greater_than_or_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is strict_greater_than_or_equal (>==). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_strict_greater_than_or_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be strict_greater_than_or_equal (>==).

pub fn try_expect_plus_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is plus_equal (+=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_plus_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be plus_equal (+=).

pub fn try_expect_add_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is add_assign (+=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_add_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be add_assign (+=).

pub fn try_expect_hyphen_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is hyphen_equal (-=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_hyphen_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be hyphen_equal (-=).

pub fn try_expect_sub_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is sub_assign (-=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_sub_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be sub_assign (-=).

pub fn try_expect_asterisk_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is asterisk_equal (*=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_asterisk_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be asterisk_equal (*=).

pub fn try_expect_mul_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is mul_assign (*=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_mul_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be mul_assign (*=).

pub fn try_expect_exponentiation_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is exponentiation_equal (**=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_exponentiation_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be exponentiation_equal (**=).

pub fn try_expect_exponentiation_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is exponentiation_assign (**=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_exponentiation_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be exponentiation_assign (**=).

pub fn try_expect_slash_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is slash_equal (/=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_slash_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be slash_equal (/=).

pub fn try_expect_div_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is div_assign (/=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_div_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be div_assign (/=).

pub fn try_expect_backslash_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is backslash_equal (=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_backslash_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be backslash_equal (=).

pub fn try_expect_percent_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is percent_equal (%=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_percent_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be percent_equal (%=).

pub fn try_expect_rem_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is rem_assign (%=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_rem_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be rem_assign (%=).

pub fn try_expect_ampersand_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is ampersand_equal (&=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_ampersand_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be ampersand_equal (&=).

pub fn try_expect_bitand_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is bitand_assign (&=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_bitand_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be bitand_assign (&=).

pub fn try_expect_pipe_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is pipe_equal (|=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_pipe_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be pipe_equal (|=).

pub fn try_expect_bitor_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is bitor_assign (|=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_bitor_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be bitor_assign (|=).

pub fn try_expect_caret_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is caret_equal (^=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_caret_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be caret_equal (^=).

pub fn try_expect_xor_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is xor_assign (^=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_xor_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be xor_assign (^=).

pub fn try_expect_shl_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is shl_equal (<<=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_shl_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be shl_equal (<<=).

pub fn try_expect_shl_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is shl_assign (<<=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_shl_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be shl_assign (<<=).

pub fn try_expect_shr_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is shr_equal (>>=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_shr_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be shr_equal (>>=).

pub fn try_expect_shr_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is shr_assign (>>=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_shr_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be shr_assign (>>=).

pub fn try_expect_sar_equal( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is sar_equal (>>>=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_sar_equal( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be sar_equal (>>>=).

pub fn try_expect_sar_assign( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is sar_assign (>>>=). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_sar_assign( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be sar_assign (>>>=).

pub fn try_expect_shl( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is shl (<<). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_shl( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be shl (<<).

pub fn try_expect_shr( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is shr (>>). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_shr( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be shr (>>).

pub fn try_expect_sar( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is sar (>>>). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_sar( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be sar (>>>).

pub fn try_expect_increment( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is increment (++). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_increment( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be increment (++).

pub fn try_expect_decrement( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is decrement (–). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_decrement( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be decrement (–).

pub fn try_expect_exponentiation( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is exponentiation (**). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_exponentiation( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be exponentiation (**).

pub fn try_expect_logical_and( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is logical_and (&&). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_logical_and( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be logical_and (&&).

pub fn try_expect_logical_or( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is logical_or (||). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_logical_or( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be logical_or (||).

pub fn try_expect_double_colon( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is double_colon (::). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_double_colon( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be double_colon (::).

pub fn try_expect_spread( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is spread (…). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_spread( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be spread (…).

pub fn try_expect_null_coalesce( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is null_coalesce (??). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_null_coalesce( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be null_coalesce (??).

pub fn try_expect_optional_chain( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is optional_chain (?.). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_optional_chain( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be optional_chain (?.).

pub fn try_expect_tab( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is tab ( ). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_tab( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be tab ( ).

pub fn try_expect_newline( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is newline ( ). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_newline( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be newline ( ).

pub fn try_expect_carriage_return( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is carriage_return ( ). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_carriage_return( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be carriage_return ( ).

pub fn try_expect_crlf( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is crlf ( ). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_crlf( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be crlf ( ).

pub fn try_expect_space( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>,

Tries to advance to the next valid token if it is space ( ). Otherwise leaves the input unchanged.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn expect_space( &mut self, ) -> Result<Spanned<L::Token, L::Span>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where L::Token: PunctuatorToken<'inp>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>> + From<UnexpectedToken<'inp, L::Token, <L::Token as Token<'inp>>::Kind, L::Span, Lang>>,

Advances to the next valid token and expects it to be space ( ).

pub fn try_expect<F>( &mut self, pred: F, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool,

Advances to the next valid token and expects it to satisfy the predicate.

Emits any lexer errors encountered. If a valid token is found, calls pred. If pred returns true, the token is consumed and returned. Otherwise, the token remains in the cache and Ok(None) is returned.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn try_expect_or_stop<F>( &mut self, pred: F, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> bool, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

Tries to advance to the next valid token if it satisfies pred — like try_expect, except that a terminal stop is an error, never a decline.

Ok(None) here means the thing attempted is definitely absent: the next valid token failed pred (it stays unconsumed, at the cache front), or the input has genuinely ended. A terminal stop — a resource-limit trip on this scan, or an already-latched poison boundary — is not evidence of absence, so it surfaces as the same end-of-input error the committed expect_* forms raise there, after the trip’s own diagnostic has gone to the emitter (deduplicated; a fatal emitter’s rejection still propagates from the scan itself). This is the primitive an attempt/decline caller should build on when a decline commits it to a different parse — see the try_* delimited shapes.

The one exit this cannot mark, and what reads it

The parenthesis above is a gap, not a footnote. A rejecting (fail-fast) Emitter reports the trip by returning the value its From<<L::Token as Token>::Error> builds — that Err is the report — and the scan propagates it before this body can reach the arm that raises the terminal end-of-input. The caller then holds an ordinary-looking grammar error over an exhausted scanner, and no delegation in its MaybeTerminal recovers the fact, because nothing on that path is terminal-marked.

The stop is on record anyway — the boundary is latched inside the crate’s terminal predicate, ahead of the diagnostic ever being offered to the emitter — so at_scanner_stop answers there. A root loop deciding “does this failure end the document” reads it beside is_terminal; every other caller of this method can keep reading the error value alone.

pub fn try_expect_map<O, F>( &mut self, pred: F, ) -> Result<Option<(O, Spanned<L::Token, L::Span>)>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> Option<O>,

Advances to the next valid token and expects it to satisfy the predicate.

Emits any lexer errors encountered. If a valid token is found, calls pred. If pred returns Some(output), the token is consumed and (output, token) is returned. If pred returns None, the token remains in the cache and Ok(None) is returned.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn try_expect_map_or_stop<O, F>( &mut self, pred: F, ) -> Result<Option<(O, Spanned<L::Token, L::Span>)>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> Option<O>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

Tries to advance to the next valid token, mapping it through pred — like try_expect_map, except that a terminal stop is an error, never a decline.

This is the map-shaped twin of try_expect_or_stop: Ok(None) means the thing attempted is definitely absent (the next valid token mapped to None and stays at the cache front, or the input has genuinely ended), while a terminal stop — a resource-limit trip on this scan, or an already-latched poison boundary — surfaces as the same terminal-marked end-of-input error the committed forms raise, after the trip’s own diagnostic has gone to the emitter (deduplicated). A fatal emitter’s rejection of that diagnostic still propagates from the scan itself — but as that emitter’s value, converted from the lexer error, so it carries no terminal mark: no UnexpectedEnd is built on that path for into_terminal to raise a flag on. The arm of your error type holding a lexer error is what answers for it; see MaybeTerminal. It is the primitive a map-shaped attempt (the token-pratt LHS/RHS classifier) should build on when a decline commits it to a different parse.

pub fn try_expect_and_then<O, F>( &mut self, pred: F, ) -> Result<Option<(O, Spanned<L::Token, L::Span>)>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where F: FnMut(Spanned<&L::Token, &L::Span>) -> Option<Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>>,

Advances to the next valid token and expects it to satisfy the predicate.

Emits any lexer errors encountered. If a valid token is found, calls pred. If pred returns Some(Ok(output)), the token is consumed and (output, token) is returned. If pred returns Some(Err(error)), the token is consumed and Err(error) is returned. If pred returns None, the token remains in the cache and Ok(None) is returned.

Ok(None) also covers a terminal stop (limit trip / latched poison boundary); when a decline commits the caller to a different parse, use try_expect_or_stop.

pub fn try_expect_take<O, C, P>( &mut self, classify: C, project: P, ) -> Result<Option<O>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where C: FnMut(Spanned<&L::Token, &L::Span>) -> bool, P: FnOnce(Spanned<L::Token, L::Span>) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>,

Classify the head, then project it by value — one named operation.

classify sees the head by reference, so the classification runs once; on accept the token is committed — the emitter’s committed-token hook runs — and the token then moves by value into project, which extracts the payload without a clone. project’s error is a real error, never a decline: by the time it runs, classify has already accepted and the token is committed.

Ok(None) follows try_expect: definite absence (the head stays at the cache front, unconsumed) or a terminal stop. When a decline commits the caller to a different parse, use try_expect_take_or_stop, which raises on a terminal stop instead of folding it into absence.

Panics

Nothing here catches a panic out of project. If it panics, the token is gone with the closure frame — but the commit has already happened and no transaction is open, so the input’s structural state is settled exactly as it is for a caller of try_expect that panics after taking the returned token.

pub fn try_expect_take_or_stop<O, C, P>( &mut self, classify: C, project: P, ) -> Result<Option<O>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where C: FnMut(Spanned<&L::Token, &L::Span>) -> bool, P: FnOnce(Spanned<L::Token, L::Span>) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

The or_stop twin of try_expect_take — a terminal stop is an error, never a decline.

Ok(None) means the head is definitely absent (it stays at the cache front, unconsumed) or the input genuinely ended. A resource-limit trip or a latched poison boundary raises the terminal end-of-input error instead, so a caller that commits to a different parse on a decline never mistakes a halt for a grammar choice.

The classification is try_expect_or_stop’s, so the mark is too: it is what an accepting emitter earns, after the trip’s own diagnostic has gone to it. A fatal emitter’s rejection of that diagnostic still propagates from the scan itself — but as that emitter’s value, converted from the lexer error, so it carries no terminal mark, and the arm of your error type holding a lexer error is what answers for it; see MaybeTerminal.

Panics

Same posture as try_expect_take.

pub fn cache(&self) -> &Ctx::Cache

Returns a reference to the tokenizer’s cache.

The cache stores peeked tokens that have been lexed but not yet consumed. This can be useful for inspecting the cache state or implementing custom lookahead logic.

pub fn source(&self) -> &'inp L::Source

Returns a reference to the underlying input source.

This allows access to the raw source being tokenized, which is typically a &str or &[u8] depending on your Logos token definition.

pub fn state(&self) -> L::State

Returns the current lexer state (extras), by value.

It hands out a clone, and that is a wall rather than a convenience

A shared reference to the live state would be a public path to scan-visible state that skips every door this crate tracks. State is bounded Debug + Clone and nothing more, so a perfectly valid one may hold interior mutability — a Cell<Mode> a grammar flips to change how the region ahead lexes. Through a &L::State, safe code could flip it without set_state or state_mut, and the input would be scanning under a regime nothing on it records. scanner_outcome would then report Stalled — repeating reproduces the trip — over an input where repeating can now succeed, and reject a recoverable parse.

Cloning removes the capability rather than asking a State implementor not to use it: the value handed back is the caller’s own, and mutating it cannot reach the input. That is what lets the regime generation stand for State identity — with this door closed, the only writers of the live state are a commit, a restore, and the two surgery methods, and the last two are exactly install_rekey, which stamps a fresh id. See scanner_outcome for the whole argument.

The one shape it does not cover is a State whose Clone shares its interior mutability rather than deep-copying it — an Rc<Cell<_>> tally and its family. That is already the Lexer determinism clause’s own named violation (“a shared counter”), and the input layer’s behaviour under one is unspecified-but-bounded, here as everywhere else.

Costs one L::State::clone. To change the state, use state_mut, which re-keys eagerly, or set_state.

pub fn is_final(&self) -> bool

Returns whether this input is final — the last chunk of a stream, or a Complete input (always final).

A Partial input reports the flag the driver stated (parse_partial’s is_final argument); a Complete input is final by definition, so this returns true and the partial-input frontier rules are inert.

Read-only, and constant for this handle’s life

There is no set_final on an InputRef, and that absence is a law, not an omission. is_final is a fact about the world — the caller has told us no more bytes are coming — and a parser combinator cannot possibly know it. Only the code that owns the byte buffer can.

So the sole writer is the owning input’s seal, which takes &mut Input — and this handle mutably borrows that input for its entire life. A parser therefore cannot end a stream, at any depth, inside any speculative branch. Nor can it un-end one: the seal is monotone and has no inverse anywhere in the crate.

That is what keeps finality safely out of the rollback set. It cannot change while this handle lives, so no rollback can observe it change — a Checkpoint has nothing to save, and a restore has nothing to undo. The two laws this pins:

• a failed speculative branch can never cost the frontier holdback (it could not have touched finality to begin with);
• a rollback can never un-end a stream the driver already ended (the hang that “roll finality back too” would introduce).

A parser reaching for the flag does not compile — through the handle, or through a guard’s DerefMut:

use tokora::{InputRef, Lexer, ParseContext, Partial};

fn end_the_stream<'inp, L, Ctx>(inp: &mut InputRef<'inp, '_, L, Ctx, (), Partial>)
where
  L: Lexer<'inp>,
  L::State: Clone,
  Ctx: ParseContext<'inp, L>,
{
  inp.set_final(true); // error: no method named `set_final` — finality is the driver's
}

Enforcing tests (in src/input/input_ref/partial_tests.rs): speculation_cannot_end_the_stream and rollback_cannot_un_end_a_sealed_stream.

pub fn state_mut(&mut self) -> &mut L::State

Returns a mutable reference to the current lexer state (extras).

State replacement re-keys the input’s forward-scanning facts

Mutating the state through the returned reference can change how the region ahead of the cursor lexes, so this call eagerly re-keys every offset-dependent fact that governs forward scanning: the token cache is cleared (its entries were lexed under the old state and those offsets may lex differently now), the poison boundary is dropped, and the lexer-error dedup watermark is reset to the current committed cursor. The re-key runs before this returns, so it applies whether or not the caller ends up mutating through the &mut.

Speculative peek-ahead diagnostics emitted under the old state for the region beyond the cursor stay in the emitter log, and the watermark reset makes that same region re-reportable once it re-lexes under the new state: state surgery with outstanding speculative diagnostics may re-report the re-lexed region under the new regime, so callers should complete or roll back speculation before replacing state.

Transactional: checkpoints survive state surgery

The re-key is itself transactional, not invalidating. A Checkpoint pure-copies every fact the re-key touches — regime, poison boundary, dedup watermark, cursor/span, and the cache-push counter — so restoring one saved before the surgery simply undoes it: the pre-surgery regime, boundary, watermark, and position all return, and the cache re-lexes under the restored regime. Outstanding checkpoints therefore remain valid across state surgery — a raw restore, an attempt rollback, and a StackedTransaction savepoint taken before the surgery all roll back across it cleanly.

pub fn set_state(&mut self, state: L::State)

Manually sets the lexer state (for context-sensitive lexing).

State replacement re-keys the input’s forward-scanning facts

Replacing the state can change how the region ahead of the cursor lexes, so this call re-keys every offset-dependent fact that governs forward scanning: the token cache is cleared (its entries were lexed under the old state and those offsets may lex differently now), the poison boundary is dropped, and the lexer-error dedup watermark is reset to the current committed cursor. Dropping the poison boundary is the documented limit-recovery path — swap in a fresh or bigger-budget state and scanning resumes past the old boundary.

Speculative peek-ahead diagnostics emitted under the old state for the region beyond the cursor stay in the emitter log, and the watermark reset makes that same region re-reportable once it re-lexes under the new state: state surgery with outstanding speculative diagnostics may re-report the re-lexed region under the new regime, so callers should complete or roll back speculation before replacing state.

Transactional: checkpoints survive state surgery

The re-key is itself transactional, not invalidating. A Checkpoint pure-copies every fact the re-key touches — regime, poison boundary, dedup watermark, cursor/span, and the cache-push counter — so restoring one saved before the surgery simply undoes it: the pre-surgery regime, boundary, watermark, and position all return, and the cache re-lexes under the restored regime. Outstanding checkpoints therefore remain valid across state surgery — a raw restore, an attempt rollback, and a StackedTransaction savepoint taken before the surgery all roll back across it cleanly.

pub fn at_scanner_stop(&self) -> bool

Returns whether a terminal scanner stop is in force at the committed cursor — the input has spent a scanner resource budget it cannot get past, and the parse has consumed up to it.

This is the public answer to “did this failure end the document, or is it an ordinary grammar error?” for the scanner half of terminality. Its descent half is tripped_during_attempt, and neither subsumes the other: a descent trip latches no boundary — it has a control stack rather than a position — so this reads false for one, exactly as the crate’s own positional scanner witness does.

loop {
  let trips = inp.trip_snapshot();
  match definition(inp) {
    Ok(true)  => {}
    Ok(false) => return Ok(()),
    Err(e) => {
      if e.is_terminal() || inp.tripped_during_attempt(trips) || inp.at_scanner_stop() {
        return Err(e);                 // the document is over
      }
      report();                        // an ordinary syntax error: file it and carry on
    }
  }
}

The exit no other public reading covers

try_expect_or_stop raises a terminal-marked UnexpectedEot on the declining exits, and that is the whole answer wherever the caller reaches one. A rejecting (fail-fast) Emitter does not let it: it reports a lexer-resource trip by returning the value its From<<L::Token as Token>::Error> builds — that Err is the report, not a refusal to make one — and the scan propagates it from inside that very call, before the arm that would raise the terminal stop can run. The caller receives an ordinary grammar error over an exhausted scanner, and no care in the grammar’s MaybeTerminal repairs it, because nothing on that path is terminal-marked to delegate to.

The stop is nonetheless already recorded when that Err arrives: the boundary is latched inside the crate’s terminal predicate, ahead of the diagnostic ever being offered to the emitter, precisely so that the emitter’s answer cannot decide whether the input remembers its own stop. This reads that record.

It is a reading of NOW, which is what makes it right across the recovery path

It takes no baseline and witnesses no event. It answers “is a stop on record at the committed cursor?” — the same two facts the crate’s own terminal gate inside try_expect_or_stop consults, read without consuming.

It is not, however, “what that call would do next”. That gate drains a cached token first and only then consults these facts, and the budget half below is durable and non-positional — so with a lookahead’s tokens still waiting at the front, this answers true while try_expect_or_stop hands one of them out. Measured, not reasoned about: a TokenBudget of two and a peek::<U4> over eight tokens leaves this reading true and the very next try_expect_or_stop returning Ok(Some(_)) (a_refusal_on_record_still_has_cached_tokens_in_front_of_it). A loop that treats a true as “stop draining” rather than as “the stream is truncated” therefore drops tokens that are already lexed and paid for. The two facts are the two the scanner itself refuses on:

• TokenBudgetTally::refused_an_item — the input-layer budget’s recorded refusal. No Checkpoint carries it, no re-key clears it and no token_budget_mut exists, so once an item has been refused it is refused for the life of the input;
• the poison boundary, positionally, at the committed cursor — the lexer’s own limit trip, which lives in L::State and therefore travels with the regime that produced it.

Reading the boundary live rather than against a baseline is the whole of why this can be published where the session’s scanner-trip counter cannot. set_state / state_mut drop the boundary, and dropping it is the documented limit-recovery path — swap in a fresh or bigger-budget state and scanning resumes past the old boundary. A monotone counter is never cleared by either, so a loop that recovers exactly as documented reads the rest of the document and a counter-based verdict still answers “truncated”: measured at eight tokens under a scan budget of three, recovered with set_state, all eight consumed and the witness still true. This reading goes false the moment the regime that owned the stop dies, because there is nothing left to read.

For the same reason it needs no baseline at any nesting depth. A witness that compares a rollbackable cell across an attempt compares a restored value against what it was restored to; this one is not a comparison, so there is no placement of a baseline for a caller to get wrong.

A ROLLBACK restores the record, and it restores it in both directions

try_attempt, attempt_parse, a rollback-on-drop Transaction and a raw restore all reinstate the Checkpoint’s saved cursor, saved lexer state and saved boundary. A checkpoint taken after a trip puts a live stop back and this reports it. A checkpoint taken before one — which is every speculative wrapper’s own begin point — puts a None boundary back at a rewound cursor, so an attempt that tripped inside propagates the rejecting emitter’s ordinary-looking Err with nothing left on the input to read, and this answers false.

That false is not wrong about the input — it is the wrong question for the caller. The Lexer contract requires limit accounting to derive entirely from source, offset and State, and TokenLimiter documents the consequence from the other side: a restore reinstalls the saved tally, everything spent after it is given back, because an abandoned speculation’s tokens are not in the committed stream. So after that rollback a scan from the restored cursor really does yield a token, and this reading says so correctly. What the caller has lost is not the state of the input but the history of the attempt: the wrapper’s next turn re-derives the identical trip, and a root loop reading only this one files a report every turn without ever advancing the cursor.

A loop that keeps redoing refundable work is detected by scanner_outcome’s Stalled arm — a trip after which none of the determinism clause’s three inputs moved — and bounded by putting the budget on the input. What this method does not answer, and scanner_stopped_during_attempt does, is the other residue in this list: a stop latched ahead of the committed cursor by a lookahead, which every positional reading — this one included — is blind to. This method stays the narrow live query: is a stop on record here, now.

One bound belongs on neither of them but on the caller’s choice of where to put it. A TokenBudget on the input is outside every Checkpoint, so its refusal — and this reading of it — survives any rollback; that is the placement for a bound on work performed, which is where TokenLimiter’s own documentation sends it. A tally the state merely points at — a shared counter, an ambient global — is not a third option: the Lexer determinism clause names it as a contract violation, and the input layer’s behaviour under one is unspecified-but-bounded rather than a case either reading owes an answer to.

Measured in tests/root_loop_trip_witness.rs section 5: every_speculative_wrapper_restores_a_checkpoint_that_predates_the_trip drives all three wrappers over a conforming limiter, an_input_side_bound_outlives_every_one_of_the_three_rollbacks is the TokenBudget row, the_attempt_relative_verdict_answers_where_the_positional_reading_is_blind is the five-point table the two readings differ in, and the_stall_outcome_ends_a_speculating_loop_the_boolean_cannot measures the loop no boolean ends and the outcome does.

What a false does not promise

Two more residues, named rather than implied:

• the cursor rewound behind a live boundary. An element may latch a trip, open an attempt, consume the cached pre-trip tokens and decline; the restore rewinds the cursor behind the boundary while the latch survives. Every positional witness reads clean there. Lexing strictly before the frontier still proceeds, so a loop that carries on re-parses a prefix that really is there and re-reaches the stop — but what bounds it is the loop keeping that prefix. One whose retry is itself inside a rolling-back wrapper keeps nothing and consumes nothing, and then does not terminate at all — which is the shape scanner_stopped_during_attempt exists for, and which a root loop otherwise needs its own no-progress guard against, for the same reason every collection driver in this crate has one;
• a met ceiling with the one-shot probe unspent. A budget of N over a document of N items has met its ceiling and refused nothing, and “would an item be refused?” is not “is there an item?” — only running the lexer separates them, which is what the probe is for. Until it has run, this reads false; the entry that spends it records the refusal and this reads true from then on. Answering true at the met ceiling instead would report a fully-parsed document as truncated, which is the defect the probe exists to avoid.

Costs a bool load, and — only if that is false — one Offset::Ord comparison against a latch that is None on every input that has never tripped.

pub fn token_budget(&self) -> &TokenBudgetTally

The token budget this parse’s lexer produces items against: the ceiling, and what has been charged toward it.

Read-only, deliberately. There is no token_budget_mut — the cell is written only by the crate-internal lexing chokepoint (lex_within_boundary’s charge and settle_met_ceiling’s one-shot latch), which is on the driver’s side of the seam, so a budget cannot be lowered, refunded or re-seeded by grammar code. The same absence recursion relies on.

Configure it with InputContext::with_token_budget or ParserContext::with_token_budget.

What comes back is this input’s tally, and it is not a value: it is neither Clone nor Copy, so it cannot be read out here and installed as another parse’s starting state. That door was open while the spend rode in the copyable TokenBudget, and it fabricated refusals in inputs that never refused anything — see TokenBudgetTally.

pub fn is_eoi(&self) -> bool

Returns true if reached the end of input.

Prefer is_exhausted in a loop gate

This is a frontier question — has the scanner reached the end of the buffer? — and it answers true the moment any lookahead lexes through the end, while the tokens that lookahead produced are still sitting unconsumed in front of the caller. A driver loop gated on it therefore stops early exactly when someone peeked far enough, which makes the parse a function of the caller’s lookahead history rather than of the token stream.

pub fn is_exhausted(&self) -> bool

Returns true if the input is exhausted for a consumer: no lexed token is waiting and the lexer frontier has reached the end of the buffer.

This is the predicate a driver loop wants, and is_eoi is not it — see its docs for why a frontier question makes a loop stop as a function of how deep someone peeked.

It is also independent of the cache implementation: a capacity that retains a token answers false because that token is waiting; a capacity that retains nothing answers false because its lex frontier is still behind that token’s start.

false does not promise a token

The frontier this reads is the end of the newest item the input committed or retained, and a plain next drain never commits past the last token’s end — so over a source with trailing lexer-skipped bytes this stays false after the stream is fully drained, in every capacity. The scans that settle at exhaustion (skip_while and the sync family, and therefore the padded combinators) do commit the lexer’s end and do reach true. So a consume’s own outcome is the authoritative end-of-stream signal and this predicate is the gate: it never turns true early, and its residual false is broken by the loop’s own handling of an empty consume.

Partial mode

On a non-final Partial input this is the end of the buffer, not the end of the stream: true here means a consume would surface an Incomplete, not None. A refill driver must treat it as “ask for more bytes”, never as “the construct ended”.

Fuzz coverage

In the fuzz alphabet as Op::IsExhausted; see OP_SURFACE_CENSUS in src/fuzz/ops.rs.

pub fn lexer(&self) -> L

where L::State: Clone,

Creates a lexer resuming at the lookahead frontier — the end of the newest token the consumer has not yet consumed, under the state that produced it — or, with nothing retained, at the committed position under the committed state.

The pair is read from ONE value

A retained token carries its own post-token state (a CachedToken is exactly that pair), and that is the state the byte after it must be lexed under. Reading the offset from the retained token and the state from the committed field would resume at the right byte under a state from before the retained run: a widening lookahead then lexes token k + 1 under the state from before token 1, a by-value Lexer::State limiter under-counts by the whole retained run, and the same grammar over the same input parses differently depending on how deep the caller peeked.

pub fn attempt<F, R>(&mut self, f: F) -> Option<R>

where F: FnOnce(&mut Self) -> Option<R>,

Attempts to parse with the given function, rolling back on failure.

A checkpoint is saved before f runs. If f returns Some, its progress is kept. If it returns None, the input rolls back to the checkpoint — position, lexer state, diagnostics emitted inside the attempt, the dedup watermark, and the poison boundary all return to their pre-attempt values.

This is the recommended way to backtrack: the save/restore pair is scoped to the closure, so the last-in, first-out discipline documented on restore holds by construction, even under nesting.

For a three-way flow — accept, decline, or a real error — reach for attempt_parse, which speaks the crate’s ParseAttempt vocabulary instead of making a decline borrow Option’s or Result’s.

Contract: the closure owns its span of the timeline

The attempt saves at entry and settles at exit — commit-shaped on Some, restore-shaped on None — so the last-in, first-out law holds structurally and a declined attempt leaves no trace (the rewind story above: position, lexer state, emissions, watermark, poison boundary). One violation remains expressible, only under unstable-raw: a raw restore inside f to a checkpoint saved before the attempt began would tear out the attempt’s own begin point (it pops it off the live lineage). Allocator builds pin that begin point, so such a restore panics at the restore — its message names a live transaction guard or attempt — rather than letting f continue on a torn foundation and detecting it only at the decline. A LIFO-clean raw save/restore pair taken and released entirely inside f, above the attempt’s checkpoint, is unaffected. Allocator-less targets keep no pin set, so this mixing is unspecified-but-bounded there. Enforcing tests (in src/input/input_ref/tests.rs): attempt_inner_raw_restore_below_checkpoint_panics_at_restore, attempt_inner_lifo_clean_raw_pair_is_legal, and attempt_backtrack_over_trip_reemits_diagnostic_exactly_once.

A session point f opened and abandoned is legal, and the decline reconciles it

f is caller-supplied code holding a whole InputRef, so it may open a session point and leave it open. That is a liberty begin_point grants — an abandoned point keeps its progress and is released with the handle — not a bug this method may re-classify. So the decline settles through the reconciling rollback_abandoning_points: every point younger than the attempt’s base is abandoned (unpinned, lineage entry dropped, emitter mark released) and the rewind then subsumes its progress.

It is not merely the kinder of two answers, it is the only consistent one. An abandoned point pins its base above the attempt’s, and the checked rollback refuses to rewind across a live pin — a release panic, in every allocator build, raised before anything is restored, leaving the speculative progress committed for a host that catches. The attempt’s other settling exit for the same decision, an unwind out of f, has always reconciled instead: it is the guard’s rolling-back Drop, and a Drop that may run mid-unwind can refuse nothing. Spelling the decline with the checked verb therefore made the same legal history panic when f returned and settle cleanly when f panicked. Both exits now restore the same thing.

What this does not do is settle points for you. Only a rollback that reaches below a point abandons it, because only that destroys the lineage the point describes; an accepted attempt commits, and a point f left open is still open and still settleable afterwards — the non-lexical property session points exist for. Enforcing cells (in src/input/input_ref/session_tests.rs): attempt_declining_across_an_abandoned_point_reconciles, attempt_panic_with_an_open_point_leaves_no_stranded_point, and attempt_accepting_leaves_the_closure_s_point_open_and_its_progress_kept.

If the closure panics

The begin point is held by a Transaction for the whole span of f, so an unwind out of f settles it exactly as a decline does — the guard’s Drop rolls back to the begin point and releases its pin and its lineage id. A host that catches the unwind (catch_unwind: a test harness, a fuzzer, an editor server) is therefore handed an input that is still consistent and still usable, with nothing pinned on its behalf.

For fallible closures that carry an error value, see try_attempt.

pub fn try_attempt<F, T, E>(&mut self, f: F) -> Result<T, E>

where F: FnOnce(&mut Self) -> Result<T, E>,

Attempts to parse with a fallible function, rolling back on error.

The Result-shaped sibling of attempt, for recovery- and pratt-style flows that need the failure value. A checkpoint is saved before f runs.

• If f returns Ok, its progress is kept and the value is returned.
• If f returns Err, the input rolls back to the checkpoint and the error is returned to the caller. Everything the attempt touched returns to its pre-attempt value: the position, the lexer state, the diagnostics emitted inside the attempt, the dedup watermark, and the poison boundary.

Like attempt, this is a structural way to backtrack: the save/restore pair is scoped to the closure, so the last-in, first-out discipline documented on restore holds by construction, even under nesting.

Contract: the closure owns its span of the timeline

Exactly attempt’s contract with Err as the declining shape: the last-in, first-out law holds structurally, a failed attempt leaves no trace, and the one remaining violation — a raw restore inside f to a checkpoint saved before the attempt (unstable-raw only) — panics at the restore in allocator builds, which pin the attempt’s begin point, rather than letting f continue on a torn foundation. Allocator-less targets are unspecified-but-bounded there. Enforcing tests (in src/input/input_ref/tests.rs): try_attempt_err_rolls_back_everything, try_attempt_nested_lifo, and try_attempt_inner_raw_restore_below_checkpoint_panics_at_restore.

The session-point clause carries over verbatim as well: f may open a point and abandon it, and the Err arm settles through the reconciling rollback_abandoning_points so that it restores exactly what an unwind out of f restores. See attempt’s “A session point f opened and abandoned is legal” section for why the checked verb is the wrong one here. Enforcing cell (in src/input/input_ref/session_tests.rs): try_attempt_erring_across_an_abandoned_point_reconciles.

If the closure panics

Exactly attempt’s guarantee: the begin point rides in a Transaction for the whole span of f, so an unwind settles it like a decline — roll back, unpin, release the lineage id — and a host that catches the panic keeps a consistent input with nothing pinned on its behalf.

pub fn attempt_parse<F, T, E>(&mut self, f: F) -> Result<ParseAttempt<T>, E>

where F: FnOnce(&mut Self) -> Result<ParseAttempt<T>, E>,

Speculation in the crate’s own three-way vocabulary.

• Ok(Accept(v)) → progress kept (commit).
• Ok(Decline) → rollback, no trace — a benign decline needs no fabricated error, which is the whole reason this exists beside try_attempt: a try_* production that wants to speculate otherwise has to invent an error value to decline with and then unwrap it again.
• Err(e) → rollback, and the error propagates untouched.

The guard plumbing is exactly try_attempt’s, so every one of its guarantees carries over verbatim: the last-in, first-out law holds structurally, a rolled-back attempt leaves no trace, the begin point rides a rollback-on-drop Transaction for the whole span of f, and both restoring arms settle through the reconciling rollback_abandoning_points, so a session point f opened and abandoned is reconciled rather than refused — see attempt’s “A session point f opened and abandoned is legal” section. Enforcing cells (in src/input/input_ref/session_tests.rs): attempt_parse_declining_across_an_abandoned_point_reconciles and attempt_parse_erring_across_an_abandoned_point_reconciles.

If the closure panics

The unwind settles the transaction as a decline — roll back, unpin, release the lineage id — so a host that catches the panic keeps a consistent input with nothing pinned on its behalf.

pub fn spanning<F, T, E>(&mut self, f: F) -> Result<(L::Span, T), E>

where F: FnOnce(&mut Self) -> Result<T, E>,

Closure-scoped span capture — the imperative twin of spanned, for hand-sequenced productions.

Returns the span covering exactly what f consumed, alongside f’s value. The bracket is spanned’s own — cursor before, f, then span_since — so the two spellings cannot disagree about where a construct starts or ends.

f’s error propagates unchanged and no span is produced: a failed production has no extent to report.

pub fn begin( &mut self, ) -> Transaction<'_, 'inp, 'closure, L, Ctx, Lang, Rollback, Cmpl>

Starts a transaction: a scoped, compile-time-safe form of save and restore.

The returned Transaction guard mutably borrows this input; parse through the guard (it dereferences to InputRef), then decide with commit (keep the progress) or rollback (return to the begin point). Dropping the guard without deciding rolls back — uncommitted speculative work is discarded, as in a database transaction. For a guard that instead keeps progress on drop (commit-by-default), use begin_with::<Commit>.

Prefer this for imperative flows with several exits (loops, match arms); attempt/try_attempt for single-closure speculation; raw save/restore (feature unstable-raw) only where no guard shape fits.

pub fn begin_with<D: DropPolicy>( &mut self, ) -> Transaction<'_, 'inp, 'closure, L, Ctx, Lang, D, Cmpl>

Starts a transaction with an explicit DropPolicy — the canonical generic form of begin.

The type parameter D fixes what an undecided guard does on drop:

• Rollback — restore to the begin point (the speculative default that begin selects; drop discards the speculative work);
• Commit — keep the progress (commit-by-default, the dual a Pratt-style operator loop wants: keep progress on every success and every ?-propagation, and roll back explicitly only on the branches that back out).

commit and rollback are available on either flavour; only the drop behaviour differs.

pub fn begin_stacked( &mut self, ) -> StackedTransaction<'_, 'inp, 'closure, L, Ctx, Lang, Rollback, Cmpl>

Starts a transaction that can hold several internal savepoints at once — the multi-fallback-point form of begin.

savepoint marks a position; rollback_to returns to a mark, destroying every younger savepoint while the mark itself stays valid; release forgets savepoints while keeping the parsed progress; commit / rollback decide the whole transaction. Savepoints follow SQL database semantics: rolling back to an older savepoint always destroys the newer ones — out-of-order revival is impossible by construction. A misused SavepointId is caught in layers: a temporally-misused id (kept past its transaction) at compile time via its lifetime brand, and a foreign or a stale id by a runtime check in every build; see SavepointId.

Raw save / restore, state replacement, and nested transactions are all reachable through the guard’s deref; see the mixing rules on StackedTransaction for the one combination that invalidates a savepoint (a raw restore below it — it panics as stale in every build) and which are always legal (state surgery, nested speculation, and a LIFO-clean raw pair above the savepoints).

Reach for the backtracking tools in order of shape:

• begin / Transaction — a single speculative alternative with several imperative exits (loops, match arms);
• begin_stacked / StackedTransaction — several live fallback points at once (best/longest-match selection: a savepoint after each parsed stage, then rollback_to the best-scoring one);
• attempt / try_attempt — closure-shaped speculation;
• begin_with::<Commit> — commit-by-default flows where progress is kept on most exits;
• begin_point session points — non-lexical speculation a driver opens in one call and settles in a later one (the shape a borrowing guard cannot express).

Raw save / restore sit beneath all of these as the unstable-raw escape hatch — reachable only with that feature, for the rare shape no guard or session point fits.

Dropping an undecided guard rolls back to the begin point; for a stacked guard that instead keeps its progress on drop, use begin_stacked_with::<Commit>.

pub fn begin_stacked_with<D: DropPolicy>( &mut self, ) -> StackedTransaction<'_, 'inp, 'closure, L, Ctx, Lang, D, Cmpl>

Starts a stacked transaction with an explicit DropPolicy — the canonical generic form of begin_stacked (see begin_with for the policy meanings).

D fixes what an undecided guard does on drop: Rollback rolls back to the begin point, discarding all savepoints (the default begin_stacked selects); Commit keeps the parsed progress. The savepoint operations and commit/rollback are identical for either flavour.

pub fn begin_point(&mut self) -> SessionPointId<'closure>

Opens a session point: saves a checkpoint of the current position onto the input’s internal point stack and pins its lineage id, exactly as a transaction guard pins its begin point. Returns nothing — and that is the whole feature.

The shape the guards cannot express

Every guard (begin, begin_stacked) and both attempts are lexical: the guard is a borrow of this input, so while one is alive the input is not, and the speculative scope can only end where the borrow does — inside one expression, one block, one call. A driver that is stepped across separate method calls — a REPL, an IDE that parses a fragment, speculates, and decides on a later call — cannot hold a guard beside the input it borrows: that value would be self-referential.

A session point is a value on the input, not a borrow of it. begin_point takes &mut self, pushes, and returns; the borrow ends with the call, so the whole consume surface (next, peek, try_expect, any parser you hand this input to) stays callable with the point still open, in this call and in later ones:

let p = inp.begin_point();  // mark — nothing is borrowed afterwards
let t = inp.next()?;        // parse, in this call or a later one
let u = inp.next()?;        // …and again
inp.rollback_point(p);      // unmark: cursor, span, state, cache, diagnostics all return

Settle the point with commit_point (keep the progress) or rollback_point (return to it), naming it by the SessionPointId this returns. The stack is the last-in, first-out order — points settle newest-first — so nesting stays structural; the id is what stops a settle from naming a shifted target, and what makes a stale one a refusal rather than a silent settle of whatever happens to be newest. points is the live depth.

A point pins its base

A session point is the base of a speculative scope, so it carries the same hazard a guard base does until it is settled: a rewind reaching below it would tear its foundation out. Two ways to reach it, both caller bugs, and each answered where it can be:

• a checked rewind below the point — a raw restore (reachable only under unstable-raw) or Transaction::rollback — is refused outright: the pin makes it panic where it is requested rather than corrupt the timeline silently;
• a reconciling rewind cannot refuse anything, so it abandons instead: every point younger than its base is unpinned, its lineage entry dropped and its emitter mark released — before the rewind, exactly as dropping the handle abandons a point still open. The point’s progress is not rolled back separately; the rewind subsumes it. Two shapes reach it: a guard’s or an attempt’s rollback on drop, which may not refuse (a Drop may run while already unwinding, where panicking is forbidden), and the explicit Transaction::rollback_abandoning_points, which chooses not to.

Those are the two answers, and which one a given rollback should be is a question about who owns the points — that verb’s docs answer it. A scope that owns every point opened inside it keeps the refusal, because there a point still open above the base means the code that opened it lost track of its own speculation. A scope spanning foreign code does not: attempt, try_attempt and attempt_parse hand a whole handle to a caller-supplied closure, and the typed pratt driver hands one to grammar hooks, so all of their explicit rollbacks reconcile — matching what their own unwind edge has always done. Abandoning is a liberty this method grants (see the drop section below); a rollback that answered it with a release panic would be re-classifying that liberty as a bug from the wrong side of the seam.

Settle your points before the scope that opened them ends and neither arises. What never happens either way is the third outcome: a point left on the stack describing a lineage the rewind destroyed.

One thing a point deliberately cannot reach is the interior of a node bracket a caller is already inside: settling a point from an enclosing frame there would rewind below that bracket’s start and stale the mark its failing exit must spend, so the invariant 'closure brand on the id refuses it at compile time — the wall is the brand, not a runtime check. See A rollback below the start cannot happen mid-frame in the node module docs.

Contract: a point is scoped to this handle, and never outlives it

A session point is non-lexical — it outlives the call that opened it — but it is not unbounded: it lives on this InputRef and dies with it. It cannot be carried to another handle, not even one taken from the same input, and this is a law, not a convention.

The reason is what a Checkpoint carries. Among its facts is the emitter’s emission mark — an index into the log of the emitter this handle borrows (Emitter::checkpoint), which rollback_point replays into Emitter::rewind. A point saved while emitter A was borrowed and settled while emitter B is would truncate B’s log at A’s mark: a diagnostic count from one timeline, applied to another. So a checkpoint is only meaningful within the one emitter borrow that produced it, and a session point — a checkpoint held across calls — must be scoped to that borrow.

That scope is this handle: as_ref takes the emitter borrow, the handle holds it, and the borrow ends when the handle dies. The type system enforces it — the 'closure brand on Checkpoint (and Cursor) is invariant in the emitter-borrow lifetime, so a checkpoint cannot even be held across the moment a second handle is taken from the same input; the attempt is a borrow error, not a runtime surprise. The point stack is therefore a field of the handle rather than of the input, on purpose.

Dropping the handle with points open: pins released, progress kept, nothing rewound

Unlike a guard — whose drop rolls back (or, under Commit, keeps) its undecided scope — dropping the handle with live session points performs no rollback. Their speculative work is kept: every token consumed, every diagnostic emitted, and every state change made through an open point stands, exactly as if each had been committed. A session ends explicitly; rolling an abandoned one back implicitly would silently paper over a driver that lost track of its own points — the deliberate opposite of a guard’s drop policy — so the end is left explicit to surface that bug instead.

It is not merely the chosen policy — it is the only buildable one

The deferral that carried this to the end of the campaign was framed as a policy decision between commit, rollback and a hybrid. Two of those three are not available from this drop site, which turns the decision into a structural fact plus a naming duty.

Session::drop holds the lineage and the emitter, so it can release marks and pins. It does not hold the input itself: it cannot restore the span, the lexer state or the cursor. So “rollback on abandon” is not a posture this crate declined to take — it is unwritable from here. The one thing that is buildable, rewinding the emitter while keeping the position, is the hybrid restore the crate already refuses elsewhere, because a position without its matching emissions is the torn state the whole settle discipline exists to prevent.

A debug_assert!(points.is_empty()) on drop is also rejected, and for a reason worth stating rather than leaving as taste: abandonment via ? through an enclosing rollback is a legal history, and an assert that fires on legal histories is exactly the class this crate has spent the campaign narrowing.

So: an abandoned session point commits. Handle death keeps committed progress and releases the point’s mark; a point you need rolled back must be settled by rollback_point before the handle dies, because the drop site cannot reach the input to restore it. (Same argument family as “Checkpoint deliberately has no Drop”.)

What the drop does do is release the bookkeeping: each remaining point’s pin and its live-checkpoint lineage entry are dropped from the input’s lineage memos, and its emitter mark is released (see the session cell’s Drop in input_ref::session). It has to, precisely because the point is split across lifetimes — the Checkpoint dies with the handle, but the pin lives on the input and the mark-keyed bookkeeping in the emitter, and both outlive it. A pin left behind would stand for a point nobody can ever settle, so the pin set would no longer hold exactly the live begin points and would grow for the life of the input — and a mark never released would strand one row of an event sink’s checkpoint stack per abandoned point, the same leak one layer up. Enforcing tests: dropping_the_handle_releases_the_open_points, dropping_the_handle_keeps_the_progress_of_the_open_points, and a_second_handle_rewinds_across_an_abandoned_point (in src/input/input_ref/session_tests.rs), and abandoned_session_points_release_their_emitter_marks (in src/cst/sink/tests.rs).

The SessionPointId does not change this: an id merely dropped is not a signal, so a point whose id went out of scope is abandoned exactly like one whose driver forgot it, and its progress is still kept. The id makes a settle exact; keep-on-abandon is a separate, and deliberate, policy choice. #[must_use] is the one nudge available at the type level.

Fuzz coverage

The abandon path is in the fuzz alphabet as Op::SessionAbandon (session.abandon(drop)); see OP_SURFACE_CENSUS in src/fuzz/ops.rs.

pub fn commit_point(&mut self, point: SessionPointId<'closure>)

Settles the session point point names by committing it: pops it off the internal stack, releases its pin, and keeps every bit of progress made since it opened — the consuming commit that releases the checkpoint’s lineage entry.

Panics

Panics with a message prefixed no live session point when nothing is open, and refuses a point that belongs to another input (foreign session point), is not the newest open one (session point settled out of order), or is no longer open at all (stale session point) — see SessionPointId.

pub fn rollback_point(&mut self, point: SessionPointId<'closure>)

Settles the session point point names by rolling back to it: pops it off the internal stack, releases its pin first — so restoring to the point does not trip its own pin, mirroring the guards’ settle ordering — then performs the checked restore. Position, span, lexer state, token cache, emission log, dedup watermark, and poison boundary all return to where the point opened.

Panics

Panics with a message prefixed no live session point when nothing is open, and refuses a point that belongs to another input (foreign session point), is not the newest open one (session point settled out of order), or is no longer open at all (stale session point) — see SessionPointId.

pub fn points(&self) -> usize

The number of live session points — the depth of the speculation stack begin_point pushes onto, for a driver tracking where it sits in a nested speculation.

pub fn slice(&self) -> <L::Source as Source<L::Offset>>::Slice<'inp>

Returns a slice of the current token from the input source.

pub fn slice_since( &self, cursor: &Cursor<'inp, 'closure, L>, ) -> Option<<L::Source as Source<L::Offset>>::Slice<'inp>>

Returns a slice of the input source from the given cursor to the current position.

pub fn slice_from( &self, cursor: &Cursor<'inp, 'closure, L>, ) -> Option<<L::Source as Source<L::Offset>>::Slice<'inp>>

Returns a slice of the input source from the given cursor to the end of the input.

pub fn slice_range<'r, R>( &self, range: R, ) -> Option<<L::Source as Source<L::Offset>>::Slice<'inp>>

where R: RangeBounds<&'r Cursor<'inp, 'closure, L>>, 'closure: 'r,

Returns a slice of the input source for the given cursor range.

pub fn span(&self) -> &L::Span

Returns the span of the current position.

pub fn span_since(&self, cursor: &Cursor<'inp, 'closure, L>) -> L::Span

Returns a span from the given cursor to the current position.

pub fn span_from(&self, cursor: &Cursor<'inp, 'closure, L>) -> L::Span

Returns a span from the given cursor to the end of the input.

pub fn span_range(&self, range: Range<&Cursor<'inp, 'closure, L>>) -> L::Span

Returns a span for the given cursor range.

pub fn save(&mut self) -> Checkpoint<'inp, 'closure, L>

Available on crate feature unstable-raw only.

Saves the current state as a Checkpoint for backtracking.

Unstable: feature-gated raw API

save is one third of the raw checkpoint triple (save / restore / commit) and is public only under the unstable-raw feature; without it the method is crate-internal, so a Checkpoint can be neither obtained nor consumed from another crate. The supported backtracking surface is the transaction guards (begin / begin_stacked), the session points, and attempt/try_attempt — together these cover every legal backtracking shape. The last-in, first-out / lineage contract documented here and on restore governs the raw triple unchanged whenever the feature is on.

The checkpoint captures the cursor, the last-consumed span, the lexer state, the emitter’s emission mark, the lexer-error dedup watermark, and the poison boundary — everything restore needs to make this exact moment the live state again.

Saving is amortized O(1): it clones the lexer state and a few offsets, and — in allocator builds — records the checkpoint’s id on the input’s live-checkpoint lineage stack (one Vec push) so restore ordering and savepoint validity can be tracked in every build; allocator-less builds allocate nothing. Saving never invalidates other checkpoints; only restoring does (see Checkpoint’s validity section).

Every checkpoint save returns should end in exactly one of restore (abandon this branch and rewind) or commit (keep this branch’s progress and release the checkpoint’s lineage entry); a checkpoint merely dropped keeps its progress but strands that lineage entry until an older restore pops through it.

Prefer attempt/try_attempt when the save/restore pair brackets a single speculative computation — they enforce the restore discipline by construction.

pub fn cursor(&self) -> &Cursor<'inp, 'closure, L>

Returns the current cursor position.

If there are cached tokens, the cursor points to the start of the first cached token; otherwise, it points to the current position.

This is the lookahead (cache-front) position: a peek or a scan decline moves it across skipped bytes without committing anything. It is not a progress metric — for committed progress compare span().end().

pub fn offset(&self) -> &L::Offset

Returns the current offset of the tokenizer.

This is the end of the last lexed token (cached or otherwise).

pub fn restore(&mut self, checkpoint: Checkpoint<'inp, 'closure, L>)

Available on crate feature unstable-raw only.

Rewinds the input to checkpoint’s save point.

Unstable: feature-gated raw API

restore is part of the raw checkpoint triple (save / restore / commit) and is public only under the unstable-raw feature; without it the method is crate-internal. The supported backtracking surface is the transaction guards (begin / begin_stacked), the session points, and attempt/try_attempt; each enforces the last-in, first-out discipline below by construction. That contract applies to the raw triple unchanged whenever the feature is on.

After a restore, the input behaves exactly as it did the moment the checkpoint was taken:

• the cursor, last-consumed span, and lexer state are restored; consuming resumes from the saved position. Cached tokens appended after the save belong to the abandoned continuation and are dropped so their region re-lexes (re-emitting any lexer error it held); tokens cached before the save re-lex identically — this includes a pre-save cached token the abandoned branch already consumed out of the cache: it is re-lexed on demand after the restore. By the Lexer determinism contract that replay is identical (the same token and span, its diagnostics exactly once, an in-State limiter recounting the same), while scan-count instrumentation held outside the lexer state will observe the additional scans;
• diagnostics emitted after the save are rolled back — the emitter’s emission log is truncated to the saved mark (see Emitter::rewind);
• the lexer-error dedup watermark returns to its saved value: an error whose emission was just rolled back becomes re-emittable — exactly once — if the resumed parse reaches it again, while errors retained from before the save stay deduplicated;
• the poison boundary returns to its saved value: an input unpoisoned at save time is unpoisoned again (a rolled-back limit trip re-trips and re-diagnoses if re-reached); an input poisoned at save time gets the saved boundary and its retained diagnostic back, still paired.

A checkpoint restores only into the handle that saved it

A Checkpoint is branded with the 'closure lifetime of the handle that saved it, and that brand is invariant, so restore (and commit) accept only a checkpoint carrying this handle’s own brand. Every handle a parser receives arrives through the closure that produced it (apply hands it a for<'closure> borrow), so any two handles carry rigidly distinct brands that cannot unify. Restoring a checkpoint that a different handle produced — even a second handle over the same source, reached through a nested parse — is therefore a compile error, not a runtime check. A debug assert additionally re-checks input identity as a backstop (see Debug builds).

use tokora::{InputRef, Lexer, ParseContext};

// Two handles of the same input carry distinct, unrelated `'closure` brands, so a
// checkpoint saved on `a` cannot be restored into `b`.
fn foreign_restore<'inp, L, Ctx>(
  a: &mut InputRef<'inp, '_, L, Ctx>,
  b: &mut InputRef<'inp, '_, L, Ctx>,
) where
  L: Lexer<'inp>,
  L::State: Clone,
  Ctx: ParseContext<'inp, L>,
{
  let ckp = a.save();
  b.restore(ckp); // error: the two handles' `'closure` brands cannot unify
}

Contract: restores are last-in, first-out

Restoring this checkpoint invalidates every checkpoint saved after it. Equivalently: with several live checkpoints, always restore the youngest one you intend to return to; never restore a checkpoint after restoring one older than it.

Both of these are fine:

// Nested speculation — inner ended before outer (each ends in commit or restore):
let outer = input.save();
let inner = input.save();
if try_variant_a(input) { input.commit(inner) } else { input.restore(inner) } // youngest first
if try_variant_b(input) { input.commit(outer) } else { input.restore(outer) } // then the older

// Retry loop — a fresh checkpoint per iteration:
loop {
  let ckp = input.save();
  match try_parse(input) {
    Ok(v) => { input.commit(ckp); break v }          // success: keep progress, release the id
    Err(_) => input.restore(ckp),                    // failure: the youngest live one
  }
}

This is a contract violation:

let a = input.save();
let b = input.save();   // b is younger than a
input.restore(a);       // rolls history back past b's save point:
                        // b now refers to a lineage that no longer exists
input.restore(b);       // ✗ contract violation

The reason is structural, not stylistic: restoring a truncated the emission log below b’s mark and un-lexed the tokens b’s position depends on. A truncated log cannot be rebuilt, so there is no correct state the second restore could produce.

Debug builds

Debug builds track live checkpoints exactly and panic on any out-of-order restore (message begins non-LIFO checkpoint restore). cargo test compiles with debug assertions by default, so exercising your parser’s backtracking paths in tests surfaces violations immediately.

A debug assert also re-checks that the checkpoint belongs to this input — a backstop for the one construction the 'closure brand cannot catch: two inputs borrowed in a single scope, where the compiler is free to unify their brands. Through the public closure API the brand already makes every foreign restore a compile error, so this assert is defense in depth; it is compiled out entirely in release, where it costs nothing.

Release builds

Release builds do not check. An out-of-order restore leaves the input in an unspecified but bounded state. Even then, all of the following still hold: no undefined behavior, no leak, no panic originating in the input layer, every scan terminates (the resource-limiter state travels inside the checkpoint, so a re-reached limit re-trips instead of rescanning without bound), and the input remains usable.

What is not guaranteed after a violation: diagnostics may be missing or attributed to the wrong branch, and the replayed token stream may differ from what was visible at the save. The only well-specified use of a checkpoint is restoring it while it is still valid.

The attached emitter keeps its own posture, and one of them is louder than this. The no-panic clause above is the input layer speaking for itself; a violation still reaches Emitter::rewind, and an emitter that can detect the resulting unpaired settle is permitted to report it (see that method’s mid-unwind contract). The recording CST Sink (the rowan feature) does: a stale restore that lands on a mid-log mark whose row was already spent panics there in every build, release included, rather than shear the event log from the diagnostic log. Every built-in diagnostics emitter (Verbose, Fatal, Silent) has nothing to detect and stays silent as before.

A restore the emitter refuses is not rolled back either. The emitter’s own state is left exactly as it was — the Sink decides before it mutates — but this method is not transactional across that panic: it raises from the middle of the rollback, so the lineage has already been popped through the target while the position and the error-reporting witnesses have not been restored. That is inside the “unspecified but bounded” envelope above and is the cost of being told at all; the input stays usable, and the only way to reach it is the violation being reported.

pub fn commit(&mut self, checkpoint: Checkpoint<'inp, 'closure, L>)

Available on crate feature unstable-raw only.

Commits checkpoint: keeps every bit of progress made since its save and releases the checkpoint’s lineage entry. This is the success-path counterpart to restore — the verb for a speculative branch that worked out.

Unstable: feature-gated raw API

commit is part of the raw checkpoint triple (save / restore / commit) and is public only under the unstable-raw feature; without it the method is crate-internal. The supported backtracking surface is the transaction guards (begin / begin_stacked), the session points, and attempt/try_attempt; the lineage contract below applies to the raw triple unchanged whenever the feature is on.

Like restore, commit accepts only a checkpoint carrying this handle’s own invariant 'closure brand; committing one a different handle saved is a compile error.

Contract: end each checkpoint in exactly one of restore or commit

A saved Checkpoint should end its life in exactly one of two ways: hand it to restore to abandon the branch and rewind, or hand it to commit to keep the branch’s progress. A checkpoint that is merely dropped keeps the progress too — dropping rewinds nothing — but in allocator builds its id lingers on the input’s live-checkpoint lineage stack until an older restore happens to pop through it. Repeated successful speculation that drops rather than commits therefore grows that stack for the life of the input; commit is what keeps it bounded. (The stranded ids are inert lineage bookkeeping, not unsafety: every restore still replays its lineage exactly.)

A retry loop keeps its progress by committing the youngest live checkpoint on success:

loop {
  let ckp = input.save();
  match try_parse(input) {
    Ok(v) => { input.commit(ckp); break v }   // success: keep progress, release the id
    Err(_) => input.restore(ckp),             // failure: rewind to the save
  }
}

Releasing is O(1) when checkpoint is the youngest live checkpoint — the common retry-loop case — and a linear removal otherwise (e.g. a younger raw checkpoint was dropped above it); the rest of the stack keeps its order either way, so an older restore still pops cleanly through the gap. Committing an already-invalidated checkpoint — one an older restore already popped off the lineage — is a harmless no-op: its id is simply absent, so nothing is released and no state changes (no panic, in any build).

Allocator-less builds keep no lineage stack, so commit there merely drops the checkpoint; the growth it prevents cannot arise without a stack to grow.

pub fn next( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Cmpl: SurfaceIncomplete<'inp, L, Ctx, Lang>,

Advances the cursor and returns the next valid token, emitting errors encountered on the way.

Skips over lexer errors, emitting them through the provided emitter. Non-fatal errors are emitted and the method continues to the next token.

Partial-input frontier (Partial, non-final)

On a Partial input that is not yet final (is_final == false), three conservative rules keep a construct that later input could still extend from being mistaken for a finished one — each surfaces an Incomplete on the Err channel instead:

1. Frontier holdback — a token the lexer decided by reading as far as the buffer end is not yielded; it may be a prefix of a longer token once more input arrives.
2. Frontier error — a non-terminal lexer error decided the same way is not emitted; it may be a truncation artifact.
3. Non-final EOF — lexer exhaustion is not treated as genuine end of input; more may come.

“Read as far as the buffer end” is the fact the lexer reports through read_frontier, floored at the item’s own span end — not the item’s span reaching the end, which is the pre-0.10.0 proxy this replaced. The two coincide only for a lexer that never reads past what it emits (SpanEnd). A lexer that probes ahead and backtracks is held back while its span sits behind the end — that is the case the proxy got wrong — and one reporting Unbounded is held back everywhere.

A terminal trip outranks all three

Every rule above says “more input may change this” — so none of them may apply to a condition no input can change. A limit trip (and the poison boundary it latches) is exactly that: it emits its diagnostic and yields Ok(None) even when the tripping token ends on the buffer end, because a limiter’s tally is monotone and no refill can un-trip it. Terminal beats incomplete, always — see the law, the dual of the crate’s never-recoverable law.

With is_final == true, or on a Complete input, all three rules are off and next behaves identically to before this typestate existed (the checks are eliminated at monomorphization). The frontier holdback means a token the lexer decided by reading to the end becomes visible only after more input arrives or the input is marked final — a latency that is correct by construction. It is one token for a SpanEnd lexer and more for a lookahead one, up to every token for Unbounded; see the input module docs for how far back it reaches, and for the Sans-I/O resumption loop.

pub fn next_or_stop( &mut self, ) -> Result<Option<Spanned<L::Token, L::Span>>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>

where Cmpl: SurfaceIncomplete<'inp, L, Ctx, Lang>, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error: From<UnexpectedEot<L::Offset, Lang>>,

Consumes the next valid token like next, except that a terminal stop is an error, never a silent end of input — the committed-consume sibling of try_expect_or_stop.

next folds three distinct outcomes into Ok(None): a genuine end of input, a fresh resource-limit trip (a Scan::Tripped), and an already-latched poison boundary at the cursor. For a committed leaf — one that turns next() == None into an UnexpectedEot — that fold is a false negative: a fresh trip becomes a plain, recoverable end-of-input error, so Recover synthesizes a value and re-enters the scanner, re-tripping the same limit instead of re-raising it. No input ever clears a tripped limit, so a terminal stop must be re-raised untouched exactly as an Incomplete is — see the never-recoverable dual.

So this draws the same split the attempt/decline primitives draw:

• Ok(Some(tok)) — a real consumed token;
• Ok(None) — a genuine end of input; the caller builds its plain UnexpectedEot, exactly as it did off next() == None;
• Err(..) — a terminal stop (a fresh trip, or the poison boundary it latches), surfaced as the committed form’s end-of-input error already marked terminal via into_terminal, so recovery re-raises it. A fatal emitter’s rejection of the trip diagnostic still propagates from the scan itself — but as that emitter’s value, converted from the lexer error, so it carries no terminal mark: no UnexpectedEnd is built on that path for into_terminal to raise a flag on. The arm of your error type holding a lexer error is what answers for it; see MaybeTerminal.

Zero-cost on the success path

The terminal classification lives only on the cold exhaustion arms — the pre-latched-boundary short-circuit and the Tripped/Eof outcomes of the one scan. A cache hit and a Scan::Token return the token with no terminal work, so this is next plus a single boundary compare on the end-of-input arm. The terminal signal rides inside the UnexpectedEot value, so no MaybeTerminal bound reaches the caller — the same boundary-witness discipline the resilient collection loops gate on.

Trait Implementations

impl<'inp, 'closure, L, Ctx, Lang: ?Sized, P: DropPolicy, Cmpl> Deref for StackedTransaction<'_, 'inp, 'closure, L, Ctx, Lang, P, Cmpl>

where L: Lexer<'inp>, L::State: Clone, Ctx: ParseContext<'inp, L, Lang>, Cmpl: Completeness,

type Target = InputRef<'inp, 'closure, L, Ctx, Lang, Cmpl>

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl<'inp, L, Ctx, Lang: ?Sized, P: DropPolicy, Cmpl> DerefMut for StackedTransaction<'_, 'inp, '_, L, Ctx, Lang, P, Cmpl>

where L: Lexer<'inp>, L::State: Clone, Ctx: ParseContext<'inp, L, Lang>, Cmpl: Completeness,

fn deref_mut(&mut self) -> &mut Self::Target

Mutably dereferences the value.

impl<'inp, L, Ctx, Lang: ?Sized, P: DropPolicy, Cmpl> Drop for StackedTransaction<'_, 'inp, '_, L, Ctx, Lang, P, Cmpl>

where L: Lexer<'inp>, L::State: Clone, Ctx: ParseContext<'inp, L, Lang>, Cmpl: Completeness,

fn drop(&mut self)

Decides an undecided transaction according to its DropPolicy. After commit / rollback the base and savepoints are already taken, so this is a no-op whatever the policy.

• Rollback: roll back to the begin point (the database default, all savepoints and progress discarded). Every savepoint is settled first, youngest first, then the base is restored — a savepoint’s emitter mark can read the same value as the base’s, so the one rewind cannot settle it (see rollback_to).
• Commit: keep the progress, forgetting every savepoint id (youngest first) then the base — the same lineage-id hygiene as commit.

An undecided guard dropped while the thread is unwinding takes the rollback arm whatever its policy (std builds) — a panic aborts the region rather than completing it, and the base and every savepoint settle through the same funnel either way. See Commit for the posture and the no_std divergence.

P::ROLLBACK_ON_DROP is a compile-time constant, so the Rollback policy monomorphizes to one arm with the other eliminated; Commit reads the unwind fact once per undecided drop. The rollback arm is silent (unchecked): Drop may run while already unwinding, where no_std has no thread::panicking() to guard a drop-bomb. Both arms first unpin the base (exception-safe). The pin check makes a raw restore below the base panic at that restore, so the base cannot go stale while the guard is live and the rollback arm normally just rewinds; the stale-base skip it still performs is a backstop (defense in depth, and the behavior for allocator-less builds).

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.