Struct Lexer 

pub struct Lexer {
    pub line: usize,
    pub token_start_line: usize,
    pub suppress_m_regex: u32,
    pub last_was_bare_positional: bool,
    pub bare_positional_indices: HashSet<usize>,
    /* private fields */
}

Fields

line: usize

token_start_line: usize

Line where the most recently-returned token starts. Set by Self::next_token right after its leading skip_whitespace_and_comments call so the value reflects the emitted token’s source position even when the call recursed through a POD / heredoc skip (which advances self.line many lines before producing the real token). Read by Self::tokenize.

suppress_m_regex: u32

When > 0, the lexer treats m followed by / as a plain identifier instead of m// regex syntax. Used in thread/pipeline stages where /m/ should be a regex grep filter, not m//.

last_was_bare_positional: bool

Set true by Self::next_token right before returning a token that originated from a bare positional alias (_, _0, _1, …) — i.e. without a leading $ sigil. Read by Self::tokenize to record the emitted token’s index in Self::bare_positional_indices. Reset to false at the top of every next_token call.

bare_positional_indices: HashSet<usize>

Indices into the token vector returned by Self::tokenize for every bare-positional token. Used by the parser’s my $X = EXPR rule to auto-wrap an RHS that contains free positional aliases into an implicit zero-arg coderef (so my $f = _ * 2 ≡ my $f = fn { _ * 2 }).

Implementations

impl Lexer

pub fn new(input: &str) -> Self

pub fn new_with_file(input: &str, file: impl Into<String>) -> Self

pub fn next_token(&mut self) -> StrykeResult<Token>

pub fn tokenize(&mut self) -> StrykeResult<Vec<(Token, usize)>>

Tokenize entire input.