Struct LexerOfStr

pub struct LexerOfStr<'a, P, T, E>
where
    P: PosnInCharStream,
{ /* private fields */ }

A Lexer of a str, using an arbitrary stream position type, lexer token, and lexer error.

This provides implementations of Lexer and CharStream.

The Lexer implementation means that a LexerOfStr has a ‘parse’ method that can be invoked to parse a single token at a position within the str, and another ‘iter’ method that can be invoked to generate an iterator that returns all the tokens in the str

If the iterator or parser return an Err, the that error is of the generic type ‘E’ supplied to the LexerOfStr which must implement LexerError of the generic position ‘P’ - so a failure to parse a character in the string can be indicated at a particular location (byte offset, with line number and column potentially).

The actual parsing of tokens is supported through the Lexer trait for both the ‘parser’ and ‘iter’ trait methods using a &BoxDynLexerParseFn. These must be boxed functions with the signature like:

   fn parse(stream: &LexerOfStr<P, T, E>, pos:P, ch:char) ->
              LexerParseResult<P, T, E>

where

   LexerParseResult<P, T, E> = Result<Option<P, T>, E>

See the Lexer trait for more details on these parse functions

The LexerOfStr also provides a CharStream implementation, which provides methods that are can be used by the parse functions.

This provides methods to match strings, get

Implementations

impl<'a, P, T, E> LexerOfStr<'a, P, T, E>

where P: PosnInCharStream,

pub fn new(text: &'a str) -> Self

Create a new LexerOfStr by borrowing a str

Trait Implementations

impl<'a, P, T, E> CharStream<P> for LexerOfStr<'a, P, T, E>

where P: PosnInCharStream,

fn range_as_bytes(&self, ofs: usize, n: usize) -> &[u8]

Borrow some bytes of the stream from an offset

Return None if the bytes are out of range

fn get_text_span(&self, span: &StreamCharSpan<P>) -> &str

Get the text of a StreamCharSpan provided by a parser

Safety

The StreamCharSpan must have been provided by a parser and so the byte offsets are indeed utf8 character boundaries

fn get_text(&self, start: P, end: P) -> &str

Get the text between two crate::StreamCharPos provided by a parser

Safety

The crate::StreamCharPos must have been provided by a parser and so the byte offsets are indeed utf8 character boundaries

fn peek_at(&self, state: &P) -> Option<char>

Get the utf8 chararacter at the byte offset, or None at the end of a string

Safety

‘state’ is maintained as a utf8 character point boundary within or at the end of the ‘str’ borrowed by Self

fn matches_bytes(&self, state: &P, s: &[u8]) -> bool

Match the text at the offset with a str

fn matches_str(&self, pos: &P, pat: &str) -> bool

Match the text at the offset with a str

fn consumed(&self, state: P, n: usize) -> P

Move the stream state forward by the specified number of characters

fn do_while<F: Fn(usize, char) -> bool>( &self, state: P, ch: char, f: &F, ) -> (P, Option<(P, usize)>)

Steps along the stream starting at the provided state (and character) while the provided function returns true; the function is provided with the index and character (starting at 0 / ch), and it returns true if the token continues, otherwise false

fn fold<T, F: Fn(&Self, T, &P, usize, char) -> (T, Option<P>)>( &self, state: P, ch: char, acc: T, f: &F, ) -> (P, Option<(P, usize, T)>)

Steps along the stream starting at the provided state, character and accumulator value while the provided function returns (true, new accumulator); the function is provided with the latest accumulator, index, character (starting at 0 / ch), and it returns true and a new accumulator if the token continues, otherwise false and the final accumulator value

fn consumed_char(&self, state: P, ch: char) -> P

where P: PosnInCharStream,

Get a stream state after consuming the specified character at its current state

unsafe fn consumed_newline(&self, state: P, num_bytes: usize) -> P

where P: PosnInCharStream,

Get a stream state after consuming a newline at its current state

fn consumed_ascii_str(&self, state: P, s: &str) -> P

where P: PosnInCharStream,

Get the state after consuming a particular ascii string without newlines

unsafe fn consumed_chars( &self, state: P, num_bytes: usize, num_chars: usize, ) -> P

where P: PosnInCharStream,

Become the span after consuming a particular string of known character length

fn commit_consumed(&self, _up_to: &P)

Invoked by the Lexer to indicate that the stream has been consumed up to a certain point, and that (for parsing) no state earlier in the stream will be requested in the future

impl<'a, P, T, E> Clone for LexerOfStr<'a, P, T, E>

where P: PosnInCharStream,

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a, P, T: Debug, E: Debug> Debug for LexerOfStr<'a, P, T, E>

where P: PosnInCharStream + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a, P, T, E> Lexer for LexerOfStr<'a, P, T, E>

where P: PosnInCharStream, T: Debug + Clone, E: LexerError<P>,

type Token = T

The Token type is the type of the token to be returned by the Lexer; it is used as part of the result of the Lexer parse functions.

type Error = E

The error type returned by the parser functions in the lexical analyzer

type State = P

The State of the stream that is used and returned by the parse functions; it must be copy as it is replicated constantly throughout the parsing process.

fn parse<'iter>( &'iter self, state: Self::State, parsers: &[BoxDynLexerParseFn<'iter, Self>], ) -> LexerParseResult<Self::State, Self::Token, Self::Error>

This attempts to parse the next token found at the state of the Lexer stream, by applying the parsers in order.

fn iter<'iter>( &'iter self, parsers: &'iter [BoxDynLexerParseFn<'iter, Self>], ) -> Box<dyn Iterator<Item = Result<T, E>> + 'iter>

This creates an iterator over all of the tokens in the Lexer stream, by applying the parsers in order at the current stream position whenever the ‘next’ method is invoked.

impl<'a, P, T, E> Copy for LexerOfStr<'a, P, T, E>

where P: PosnInCharStream,