Module utf8 

WTF-8 / UTF-8 → UTF-16 codec helpers, faithfully copied from the subset of hermes_parser::utf8 (itself ported from include/hermes/Support/UTF8.h) that is needed by JSONEmitter and the forthcoming AST-dumper port.

Keeping this copy in support means json_emitter and the AST-dumper can use it without taking a dependency on the parser crate, and without duplicating logic. The module is zero-unsafe (the support crate forbids unsafe_code).

Functions

convert_utf8_with_surrogates_to_utf16

Decode a UTF-8 sequence, which is assumed to be valid, but may possibly contain explicitly encoded surrogate pairs, into a UTF-16 sequence. Port of convertUTF8WithSurrogatesToUTF16 (UTF8.h:216-225).

decode_utf8

Decode a sequence of UTF8 encoded bytes into a Unicode codepoint, ASCII fast path. Port of decodeUTF8 (UTF8.h:187-193). In case of decoding errors, the provided callback is invoked with an appropriate message and UNICODE_REPLACEMENT_CHARACTER is returned.

decode_utf8_slow_path

Decode a sequence of UTF8 encoded bytes when it is known that the first byte is a start of a UTF8 sequence. Port of _decodeUTF8SlowPath (UTF8.h:77-162), reading bytes from *i and advancing *i past the consumed bytes. On malformed input it invokes error and returns the replacement character.

encode_utf16

Encode a 32-bit value into UTF-16, appending to out. If the value is a part of a surrogate pair, it is encoded without any conversion. Port of encodeUTF16 (UTF8.h:197-210).

is_utf8_start

Check whether a byte is a regular ASCII or a UTF8 starting byte. \return true if it is UTF8 starting byte.