Skip to main content

Lexer

Struct Lexer 

Source
pub struct Lexer<'a> { /* private fields */ }

Implementations§

Source§

impl<'a> Lexer<'a>

Source

pub fn new(src: &'a str, options: Options) -> Self

Source

pub fn windowed(&self) -> bool

Whether the lexer reads a window of its source.

Source

pub fn window(&self) -> &str

The text from the cursor to the end of what the lexer holds: in a parse from a reader, the end of the window. A matcher in a grammar that declares lex.window reads the source through this and Lexer::fill, never past them.

Source

pub fn fill(&mut self, ahead: usize) -> bool

Read the source until ahead characters lie past the cursor, or it ends, and say whether they do. A whole source is all read, so this only answers. In a parse from a reader the window may drop text behind the oldest token the parser holds, never the token being cut, and a source that fails to read ends where it failed, the parse failing with it at the next token.

Source

pub fn source(&self) -> &str

Full immutable source supplied to this lexer; in a parse from a reader, the window the lexer holds.

Source

pub fn remaining(&self) -> &str

Source remaining at the live cursor; in a parse from a reader, to the window’s end.

Source

pub fn forward(&self, max_chars: usize) -> &str

Return at most max_chars Unicode scalar values from the live cursor. This is the Rust counterpart of the public lex.fwd/Lex.Fwd helper.

Source

pub fn point(&self) -> Point

Snapshot the live cursor for token construction.

Source

pub fn advance_chars(&mut self, count: usize) -> bool

Advance by Unicode scalar values. Returns false without moving when the requested count extends beyond end-of-source.

Source

pub fn token( &self, name: impl AsRef<str>, tin: Tin, value: Value, source: impl Into<TokenText>, point: Point, ) -> Token

Construct a token from a point captured before cursor advancement.

Source

pub fn token_tin(&mut self, name: impl Into<String>) -> Tin

Resolve or allocate a token identity in this lexer’s configuration.

Source

pub fn token_name(&self, tin: Tin) -> String

Resolve a token identity back to its configured name.

Source

pub fn alt_enabled(&self, alt: &AltSpec) -> bool

Whether the options this lexer runs under enable alt: not when rule.exclude names one of its groups, nor when rule.include lists groups and it declares none of them. The parser skips such an alternate, and TypeScript removes it from the rule spec (filterRules) before anything reads the spec, so a custom matcher that reads a rule’s alternates, to tell a key position from a value one, asks this to see the same alternates TypeScript does.

Source

pub fn bad(&self, why: impl Into<String>) -> Token

Construct a bad token at the current cursor.

A custom matcher that returns it reports a lexer fault, which the parser handles as it handles its own lexer’s: a rule’s fetch raises it at once with why as the code, at this position, or, under recovery, records it, coalescing a run of them into one error, and moves the cursor past its source; relexing leaves it for an alternate to re-cut. The cursor need not be advanced here.

Source

pub fn bad_span( &self, why: impl Into<String>, start: usize, end: usize, ) -> Token

Construct a bad token whose displayed source is a scalar-indexed span. As in TypeScript, the diagnostic point remains the live cursor. Returned from a matcher it is handled as Lexer::bad describes; recovery steps past the span, counted from that cursor.

Source

pub fn next_token(&mut self) -> Result<Token, TabnasError>

Fetches the next non-IGNORE token (skipping spaces, lines, comments).

Source

pub fn next_raw_token(&mut self) -> Result<Token, TabnasError>

Fetch the next token without discarding whitespace, line, or comment tokens.

Source

pub fn next_raw_for_rule( &mut self, rule: &mut Rule, context: &mut Context, ) -> Result<Token, Box<TabnasError>>

Fetch one token for an imperative parser callback, preserving ignored space/line/comment tokens just like TypeScript’s public lex.next. Replayed tokens produced by Context::rewind are served first.

Source

pub fn next_for_rule( &mut self, rule: &mut Rule, context: &mut Context, ) -> Result<Token, Box<TabnasError>>

Fetch the next non-ignored token for an imperative parser callback.

Source

pub fn relex_for_rule( &mut self, from: &Token, wanted: &[Tin], rule: &mut Rule, context: &mut Context, ) -> Option<(Token, RelexCheckpoint)>

Public negotiated-relex entry point for native parser callbacks. A successful recut commits the lexer cursor and returns an opaque undo checkpoint; a failed recut restores all lexer state before returning.

Source

pub fn unrelex(&mut self, checkpoint: RelexCheckpoint, context: &mut Context)

Undo a committed Lexer::relex_for_rule operation, including the pending tokens hidden while the replacement cut was negotiated.

Auto Trait Implementations§

§

impl<'a> !Freeze for Lexer<'a>

§

impl<'a> !RefUnwindSafe for Lexer<'a>

§

impl<'a> !Send for Lexer<'a>

§

impl<'a> !Sync for Lexer<'a>

§

impl<'a> !UnwindSafe for Lexer<'a>

§

impl<'a> Forget for Lexer<'a>

§

impl<'a> Move for Lexer<'a>

§

impl<'a> Unpin for Lexer<'a>

§

impl<'a> UnsafeUnpin for Lexer<'a>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.