Struct ZshLexer 

pub struct ZshLexer<'a> {

    pub tokstr: Option<String>,
    pub tok: LexTok,
    pub tokfd: i32,
    pub toklineno: u64,
    pub lineno: u64,
    pub lexstop: bool,
    pub incmdpos: bool,
    pub incond: i32,
    pub incondpat: bool,
    pub incasepat: i32,
    pub inredir: bool,
    pub oldpos: bool,
    pub infor: i32,
    pub intypeset: bool,
    pub noaliases: bool,
    pub nocorrect: i32,
    pub nocomments: bool,
    pub lexflags: LexFlags,
    pub isfirstln: bool,
    pub heredocs: Vec<HereDoc>,
    pub isnewlin: i32,
    pub error: Option<String>,
    pub lex_add_raw: i32,
    /* private fields */
}

The Zsh Lexer

Fields

tokstr: Option<String>

Current token string

tok: LexTok

Current token type

tokfd: i32

File descriptor for redirections (e.g., 2> means fd=2)

toklineno: u64

Line number at start of current token

lineno: u64

Current line number

lexstop: bool

Lexer has stopped (EOF or error)

incmdpos: bool

In command position (can accept reserved words)

incond: i32

In condition [[ … ]]

incondpat: bool

In pattern context (RHS of == != =~ in [[ ]])

incasepat: i32

In case pattern

inredir: bool

In redirection

oldpos: bool

Saved incmdpos from before a redirop / for / foreach / select — restored on the NEXT non-redir token. Mirrors static int oldpos in C zsh’s ctxtlex (lex.c:319). Required for cases like for x ( ... ) where ( after the var name should tokenize as INPAR — that depends on incmdpos being restored to 1 from before FOR was lexed, which in turn depends on this saved value.

infor: i32

After ‘for’ keyword

intypeset: bool

Parsing typeset arguments

noaliases: bool

Disable alias expansion

nocorrect: i32

Disable spelling correction

nocomments: bool

Disable comment recognition

lexflags: LexFlags

Lexer flags

isfirstln: bool

Whether this is the first line

heredocs: Vec<HereDoc>

Pending here-documents

isnewlin: i32

After newline

error: Option<String>

Error message if any

lex_add_raw: i32

Raw-input capture flag — when nonzero, every char read through hgetc is also appended to tokstr_raw via zshlex_raw_add. Direct mirror of zsh/Src/lex.c:161 lex_add_raw. Used by skipcomm (lex.c:2082) to preserve the literal text of $(...) command substitutions for re-execution / display.

Implementations

impl<'a> ZshLexer<'a>

pub fn new(input: &'a str) -> Self

Create a new lexer for the given input

pub fn zshlex_raw_add(&mut self, c: char)

Append a char to the raw-input capture buffer. Direct port of zsh/Src/lex.c:2024-2039 zshlex_raw_add. Called from hgetc when lex_add_raw is nonzero so cmd-sub bodies ($(...), <(...), >(...)) can be replayed verbatim without re-lexing.

pub fn exalias<R: AliasResolver>(&mut self, resolver: &mut R) -> bool

Run alias / reserved-word expansion on the just-lexed token. Direct port of zsh/Src/lex.c:1949-2021 exalias. Returns true if an alias was injected (the caller’s loop should re-run gettok to consume the injected text).

C source flow:

1. Spell-correct (lex.c:1958-1962) — disabled in zshrs.
2. If tokstr is None: set lextext from tokstrings[tok] and checkalias against that (lex.c:1964-1969).
3. Otherwise: untokenize tokstr into a working copy (lex.c: 1971-1980).
4. ZLE word-tracking: call gotword() if LEXFLAGS_ZLE (lex.c:1982-1991).
5. STRING tokens: try checkalias, then reservation lookup (lex.c:1993-2015).
6. Clear inalmore (lex.c:2016).

Takes an AliasResolver trait object so the lexer doesn’t hard-depend on the executor’s alias-table types. zshrs callers implement AliasResolver over their alias hash tables.

pub fn zshlex_raw_back(&mut self)

Pop the last char from the raw-input capture buffer. Direct port of zsh/Src/lex.c:2042-2049 zshlex_raw_back. Called when the lexer ungets a char that was just captured raw — the raw buffer must mirror the live input so this undoes the last add.

pub fn zshlex_raw_mark(&self, offset: i64) -> i64

Mark the current raw-buffer offset (for restore later). Direct port of zsh/Src/lex.c:2052-2058 zshlex_raw_mark. Returns len + offset so callers can restore via back_to_mark.

pub fn zshlex_raw_back_to_mark(&mut self, mark: i64)

Restore raw-buffer offset to a previously-saved mark. Direct port of zsh/Src/lex.c:2061-2068 zshlex_raw_back_to_mark. Truncates the raw buffer to mark bytes — undoes any captures since the mark was taken (used when a speculative parse fails and the lexer rolls back).

pub fn take_raw_buf(&mut self) -> String

Take the captured raw-input buffer, clearing it. Useful for callers that need the literal command-sub body after lexing (e.g. compile-time string capture for $(...)).

pub fn lex_context_save(&mut self, ls: &mut LexStack)

Save lexical context onto a LexStack. Direct port of zsh/Src/lex.c:215-239 lex_context_save. After save, the lexer is in a clean state suitable for parsing a nested input (command substitution body, here-doc terminator, eval’d string).

pub fn lex_context_restore(&mut self, ls: &mut LexStack)

Restore lexical context from a LexStack. Direct port of zsh/Src/lex.c:244-262 lex_context_restore. Inverse of lex_context_save. Called after the nested parse completes.

pub fn lexinit(&mut self)

Initialize lexical state. Direct port of zsh/Src/lex.c:440-445 lexinit. Resets dbparens / nocorrect / lexstop and sets tok to ENDINPUT so the next gettok starts from a known baseline. Note: the constructor Self::new already sets equivalent defaults; this method exists for the rare case a caller wants to recycle a ZshLexer across multiple input strings.

pub fn zshlex(&mut self)

Main lexer entry point — fetch the next token. Direct port of zsh/Src/lex.c:265-313 zshlex. Loop body matches the C source do { ... } while (tok != ENDINPUT && exalias()) at lex.c:270-276, followed by here-doc draining (lex.c:278-306), newline tracking (lex.c:307-310), and SEMI/NEWLIN→SEPER folding (lex.c:311-312).

zshrs port note: exalias() (lex.c:1953) is not yet wired into the loop. The C source iterates as long as exalias keeps re-injecting alias text into the input buffer; zshrs’s alias expansion happens post-lex in exec.rs. The loop body therefore runs once and breaks unconditionally — documented divergence.

pub fn ctxtlex(&mut self)

Lex next token AND update per-context flags. Direct port of zsh/Src/lex.c:316-369 ctxtlex. The post-token state machine at lex.c:322-358 sets incmdpos based on the token shape: list separators / pipes / control keywords reset to cmd-pos; word-shaped tokens leave cmd-pos. Redirections (lex.c:361-368) stash prior incmdpos and force the redir target to non-cmd-pos.

pub fn gotword(&mut self)

Mark the current word as the one ZLE was looking for. Direct port of zsh/Src/lex.c:1881-1897 gotword. Only meaningful when the lexer was started with LEXFLAGS_ZLE for completion; after this call lexflags is cleared so subsequent tokens don’t re-trigger word tracking.

zshrs port note: zsh’s gotword updates wb/we (word begin/ end positions) based on zlemetacs (cursor pos), zlemetall (line length), inbufct, and addedx — all live in zsh’s input.c globals which zshrs hasn’t wired through the lexer. Only the lexflags = 0 side-effect at lex.c:1895 is reproducible without that integration.

pub fn register_heredoc(&mut self, terminator: String, strip_tabs: bool)

Register a heredoc to be processed at next newline

pub fn check_reserved_word(&mut self) -> bool

Check for reserved word — mirrors lex.c:2002-2015 in exalias, but reachable from the bare zshlex path (without an AliasResolver). Promotes STRING tokens to keyword tokens when:

• incmdpos is set (or text is } ending a brace block)
• text is ]] and we’re inside [[ ]] (incond > 0)
• text is bare ! and we’re at the start of a cond (incond == 1)