Module utf8x 

UTF-8 encoding for TeX characters.

TeX characters are code points 0–0x10FFFF, including surrogates, which LuaTeX encodes like any other value (a superset of standard UTF-8, as in WTF-8). Strings of TeX characters are therefore kept as byte vectors.

Constants

RAW_BYTE

Marks, in a decoded input line, a byte that does not start a valid UTF-8 sequence: RAW_BYTE | b. The tokenizer reports it (LuaTeX’s “String contains an invalid utf-8 sequence”) and error contexts show the byte itself.

Functions

chars

Decodes a whole byte string into characters.

chars_marking_invalid

Decodes an input line like chars, but marks the bytes of invalid sequences with RAW_BYTE.

decode

Decodes the character starting at bytes[i], returning it and its length. Malformed sequences decode as their first byte (length 1), which is how LuaTeX treats invalid input bytes.

encode

The encoding of a single character.

is_raw

Whether c is a RAW_BYTE marker.

len

Number of bytes in the encoding of c.

push

Appends the encoding of c to out.

push_entry

Appends the bytes of input-line entry c to out: a RAW_BYTE marker stands for its byte, any other value for its encoding.

single_char

If bytes encodes exactly one character, returns it.