Crate lua_lex 

Lexical analyzer — port of llex.c + llex.h.

Provides the Lua 5.4 lexer: character-by-character scanning of a ZIO input stream into Token values, with one-token lookahead. The llex.h header is merged here per PORTING.md §1.

C source files

• reference/lua-5.4.7/src/llex.c (581 lines, 24 functions)
• reference/lua-5.4.7/src/llex.h (91 lines; merged here)

Design notes

• LexState.L (back-pointer to lua_State) is removed. All functions that need LuaState receive it as state: &mut LuaState.
• Token.token is i32 in Phase A (matching the C int token field). Single-byte tokens are their ASCII values; reserved-word tokens start at FIRST_RESERVED (257). A proper TokenKind enum is deferred to Phase B.
• save / save_and_next are now fallible (Result<(), LuaError>); the ? operator replaces the C noreturn lexerror call on buffer overflow.
• The goto read_save / only_save / no_save pattern in read_string is translated via the local EscapeResult enum.

Structs

LexBuffer

Placeholder for LexBuffer from lua_vm::zio. TODO(port): replace with use lua_vm::zio::LexBuffer in Phase B. types.tsv: Mbuffer → LexBuffer

LexState

Per-chunk lexer (and shared parser) state.

LuaState

Per-thread Lua execution state.

LuaString

LuaTable

A Lua table: hybrid array + hash map.

Token

A single lexed token with its semantic payload.

ZIO

Placeholder for ZIO from lua_vm::zio. TODO(port): replace with use lua_vm::zio::ZIO in Phase B. types.tsv: Zio → ZIO

Enums

LuaError

The Lua error type. Carries a LuaValue payload because Lua errors can be any value (typically a string).

TokenValue

Semantic payload carried by a token.

Constants

EOZ

End-of-stream sentinel returned by ZIO::getc.

FIRST_RESERVED

First token kind value that is not a single-byte character. Single-byte tokens are represented by their ASCII value (0-255).

LUA_ENV

Name of the global environment upvalue.

NUM_RESERVED

Number of reserved words (keywords).

TK_AND

and

TK_BREAK

break

TK_CONCAT

.. (concatenation)

TK_DBCOLON

::

TK_DO

do

TK_DOTS

... (vararg)

TK_ELSE

else

TK_ELSEIF

elseif

TK_END

end

TK_EOS

<eof>

TK_EQ

==

TK_FALSE

false

TK_FLT

<number> (float literal)

TK_FOR

for

TK_FUNCTION

function

TK_GE

>=

TK_GOTO

goto

TK_IDIV

// (floor division)

TK_IF

if

TK_IN

in

TK_INT

<integer> (integer literal)

TK_LE

<=

TK_LOCAL

local

TK_NAME

<name> (identifier)

TK_NE

~=

TK_NIL

nil

TK_NOT

not

TK_OR

or

TK_REPEAT

repeat

TK_RETURN

return

TK_SHL

<<

TK_SHR

>>

TK_STRING

<string> (string literal)

TK_THEN

then

TK_TRUE

true

TK_UNTIL

until

TK_WHILE

while (last keyword; NUM_RESERVED = TK_WHILE - FIRST_RESERVED + 1 = 22)

Statics

LUAX_TOKENS

Display strings for tokens, indexed by token - FIRST_RESERVED.

Functions

init

Initialise the lexer subsystem: intern all reserved words and fix them in the GC so they are never collected.

lex_error

Build a syntax error, optionally annotated with the offending token text.

lookahead

Peek at the next token without consuming the current one.

next

Consume the current token; load the next one from the stream.

set_input

Initialise ls for lexing a new chunk from stream z.

syntax_error

Report a syntax error at the current token.

token2str

Produce a human-readable token description (for error messages and the parser).