Struct AtomStream 

pub struct AtomStream { /* private fields */ }

A parse source over an eagerly lexed token vector, which remembers how far the parse ever got and stops it if it goes on too long.

Backtracking runs through syan’s Tape, which owns the pushback and the checkpoint scopes, so the forwarding half of this is a thin shim; the mark and the budget are the parts that are not.

Implementations

impl AtomStream

pub fn new(atoms: Vec<Atom>) -> Self

Wrap an eagerly lexed atom vector, with the budget Budget calibrates for its size.

pub fn with_budget(atoms: Vec<Atom>, budget: Budget) -> Self

Self::new with the budget chosen by the caller.

pub fn furthest(&self) -> usize

End byte of the furthest atom the parser ever consumed; 0 if it consumed nothing.

Consumed, not peeked: a lookahead that rejects a token has not made progress through it, and counting it would push every diagnostic one token to the right.

pub fn furthest_span(&self) -> Option<Span>

The span of the atom that set Self::furthest — the token the parse stopped at.

It is the token ending at the mark, not the one starting after it: the generated leaf parsers are next() → match → push()-back-on- mismatch (see crate::leaf), so the offending token has already been pulled through the stream by the time the leaf rejects it. Reporting the token after it would put every diagnostic one token to the right.

pub fn exhausted(&self) -> bool

Whether the parse hit its budget rather than reaching a real verdict.

When this is true, the failure the parser reported means only “the stream ended”, which is this type’s doing and not the source’s — so the caller must not dress it up as a claim about the source. crate::parse_error::locate does not.

pub fn served(&self) -> u64

How many atoms have been served, counting every re-read. Exposed for calibrating Budget against a real corpus.

Trait Implementations

impl ParseStream for AtomStream

type Atom = WithSpan<Token, Span>

The unit of input this stream serves.

type Error = Infallible

What the underlying source reports when it fails; see get_error.

fn next(&mut self) -> Option<Self::Atom>

Consume and return the next atom, or None at end of input.

fn peek(&mut self) -> Option<&Self::Atom>

The atom next would return, without consuming it.

fn push(&mut self, atom: Self::Atom)

Hand an atom back; it is served before anything still unread.

fn checkpoint_raw(&mut self) -> u64

Open a transaction; the returned token identifies it.

fn rollback_raw(&mut self, raw: u64)

Undo everything consumed since raw was taken, and close that scope.

fn commit_raw(&mut self, raw: u64)

Keep everything consumed since raw was taken, and close that scope.

fn get_error(&mut self) -> Result<(), Self::Error>

Report an error the source has accumulated (a lexer that failed mid-stream, say), as opposed to a parse failure. Ok(()) for a source that cannot fail.

fn skip_sep(&mut self) -> bool

Skip the separator atoms if exists. Returns whether we skipped some separators.

fn validate_spacing<S>(&mut self, is_joint: bool) -> Result<(), ParseError<S>>

where S: Span, Self: Sized, Self::Atom: Spanned<Span = S>,

Check the spacing before the next atom: is_joint demands no separator here, false demands one. Fails with ParseError::Spacing if the input disagrees.

fn dup<T, E, F>(&mut self, f: F) -> Result<T, E>

where F: FnOnce(&mut Self) -> Result<T, E>, Self: Sized,

Run a sub-parser as a transaction on this stream.