Struct miden_parsing::Scanner
source · pub struct Scanner<S> { /* private fields */ }
Expand description
Scanner handles the low-level details of reading characters from a raw input stream of bytes. It decodes those bytes into UTF-8 characters, and associates each character with the SourceIndex at which it occurs.
The Scanner is intended to be consumed by a lexer, which handles converting the stream of characters into a token stream for use by the parser.
Scanner Lifecycle
The following illustrates how content flows from the raw input stream through the scanner.
lexer <- (peek) <- pending <- source
<- (pop) <- current <- pending <- source
As shown above, the lexer is “pulling” characters from the scanner.
When “peeking” a character, we return the character currently in the
pending
field, but if pending
is empty, we read enough bytes from
the source to construct a UTF-8 character, and store it as pending
,
as well as returning it to the lexer.
When “popping” a character (i.e. we are advancing the scanner in the
input), we are returing the character in the current
field, and then
moving the character in pending
into current
. Accordingly, if any
of those fields is empty, we must pull from the next field in the chain,
reading bytes from the input as we go.
Implementations§
source§impl<S> Scanner<S>where
S: Source,
impl<S> Scanner<S>where S: Source,
sourcepub fn start(&self) -> SourceIndex
pub fn start(&self) -> SourceIndex
Returns a SourceIndex representing the start of the source
sourcepub fn advance(&mut self)
pub fn advance(&mut self)
Advance scanner pipeline by a single character.
pending
becomes current
, and bytes are read from the input
to repopulate pending
.
sourcepub fn pop(&mut self) -> (SourceIndex, char)
pub fn pop(&mut self) -> (SourceIndex, char)
Return the current character and advance our position in the source
sourcepub fn peek(&self) -> (SourceIndex, char)
pub fn peek(&self) -> (SourceIndex, char)
Return the next character in the input, but do not advance.
sourcepub fn peek_next(&mut self) -> (SourceIndex, char)
pub fn peek_next(&mut self) -> (SourceIndex, char)
Return the character after the next character in the input, but do not advance.
sourcepub fn read(&self) -> (SourceIndex, char)
pub fn read(&self) -> (SourceIndex, char)
Get current character in the input.