Module utf8 

UTF-8 decode helpers, ported from include/hermes/Support/UTF8.h (decode side).

These mirror the inline classifiers and _decodeUTF8SlowPath/decodeUTF8 from Support/UTF8.h. The C++ uses an advancing const char *&from; here we use a slice plus a &mut usize index, so the raw-pointer parity lives only in cursor.rs. The lexer always passes the NUL-terminated buffer, so an out-of-range continuation read sees 0x00 (a non-continuation byte) and is correctly rejected; we also guard indexes against bytes.len() defensively.

Constants

UTF8_LINE_TERMINATOR_CHAR0

First byte of the UTF-8 encoding of U+2028/U+2029 (e2 80 a8/a9).

Functions

append_unicode_to_storage

Encode cp into storage like the lexer’s appendUnicodeToStorage (JSLexer.h:1125-1143): code points above 0xFFFF are first split into a UTF-16 surrogate pair, and each surrogate is encoded individually into UTF-8 (technically invalid UTF-8 / WTF-8, which JS string & identifier storage allows).

convert_utf8_with_surrogates_to_utf16

Decode a UTF-8 sequence, which is assumed to be valid, but may possibly contain explicitly encoded surrogate pairs, into a UTF-16 sequence. Port of convertUTF8WithSurrogatesToUTF16 (UTF8.h:216-225).

convert_utf16_to_utf8_with_replacements

Convert a UTF-16 encoded string u16s to valid UTF-8, combining surrogate pairs into supplementary-plane characters and replacing unpaired surrogates with U+FFFD. Port of convertUTF16ToUTF8WithReplacements (UTF8.cpp:99-133), dropping the maxCharacters parameter (the lexer always passes 0/unbounded).

decode_utf8

Decode a sequence of UTF8 encoded bytes into a Unicode codepoint, ASCII fast path. Port of decodeUTF8 (UTF8.h:187-193). In case of decoding errors, the provided callback is invoked with an appropriate message and UNICODE_REPLACEMENT_CHARACTER is returned.

decode_utf8_slow_path

Decode a sequence of UTF8 encoded bytes when it is known that the first byte is a start of a UTF8 sequence. Port of _decodeUTF8SlowPath (UTF8.h:77-162), reading bytes from *i and advancing *i past the consumed bytes. On malformed input it invokes error and returns the replacement character.

encode_utf8

Encode a Unicode code point as UTF-8 (up to the legacy 6-byte form, matching encodeUTF8), appending the bytes to out. Port of UTF8.cpp:encodeUTF8.

encode_utf16

Encode a 32-bit value into UTF-16, appending to out. If the value is a part of a surrogate pair, it is encoded without any conversion. Port of encodeUTF16 (UTF8.h:197-210).

is_utf8_continuation_byte

\return true if this is a UTF-8 continuation byte, or in other words, this is a byte in the “middle” of a UTF-8 codepoint.

is_utf8_leading_byte

\return true if this is a UTF-8 leading byte.

is_utf8_start

Check whether a byte is a regular ASCII or a UTF8 starting byte. \return true if it is UTF8 starting byte.

match_unicode_line_terminator_offset1

\return true if bytes starts with the UTF-8 encoding of U+2028 or U+2029. bytes[0] is assumed to be UTF8_LINE_TERMINATOR_CHAR0 (the caller checked).