Module input 

The input stack and the tokenizer (TeX: The Program §300–§365).

TeX reads from a stack of input levels. Each level is either a text source read line by line (a file, the terminal, a \read stream, a \scantokens pseudo-file or lines printed from Lua) or a token list (a macro body, a parameter, backed-up tokens, \everypar, …). get_next produces the next token from the top level, converting characters into tokens according to their category codes.

Structs

InputState

The input stack.

TextLevel

A text input level.

TokLevel

A token-list input level.

Enums

CatMode

The category-code regime of a line printed from Lua.

Level

An input level.

LexState

The scanner state of a text level (state).

LineSource

Where the lines of a text level come from.

LuaItem

One item printed from Lua during a \directlua call.

LuaLineKind

How a line printed from Lua is fed to the scanner.

ScannerStatus

What the scanner is doing, for runaway diagnostics (scanner_status).

TokKind

The kind of a token-list input level (token_type).

Constants

NO_EXPAND_FLAG

no_expand_flag: the modifier of the special \relax produced by \noexpand (TeX uses 257; LuaTeX uses a value outside the char range).

Functions

decode_line

Decodes a line of input into characters, removing trailing spaces (TeX’s input_ln; tabs are kept, as in LuaTeX).

next_line

Splits the next line off data[pos..]. Lines end at LF, CR or CR LF. Returns the line bytes and the new position, or None at end of data.