Module token 

Tokens and token lists.

A token is a u32:

• a character token packs a command code and a character: (cmd << 21) | chr, with chr ≤ 0x1F_FFFF. The command is usually a category code (≤ 15), but LuaTeX’s token.create(chr, cmd) and token.new make such tokens for any command (e.g. char_given);
• a control-sequence token is CS_TOKEN_FLAG + cs, where cs is the id of the control sequence in the hash table.

Inside macro definitions TeX also uses match (13) and end_match (14) tokens in the parameter text and out_param (5) tokens in the body.

Structs

Macro

A macro definition.

Token

A TeX token.

Constants

CS_TOKEN_FLAG

The offset of control-sequence tokens.

Functions

empty_list

The empty token list.

Type Aliases

TokenList

An immutable, shared token list (a macro body, the contents of a token register, …).