Crate charcode

Expand description

Character encoding conversion, implementing the WHATWG Encoding Standard.

charcode converts between UTF-8 and the encodings the web actually uses: UTF-16, the 28 legacy single-byte encodings, and the legacy Chinese, Japanese and Korean multi-byte encodings. It also carries the charsets those labels actually name, which is not always the same thing — see Two lookups. It has no dependencies outside the standard library, contains no unsafe code, and works on no_std targets that have an allocator.

§Converting a whole buffer

Encoding::decode is the usual entry point. It sniffs for a byte order mark, falls back to the encoding it was called on, and substitutes U+FFFD for malformed sequences the way a browser does:

use charcode::{Encoding, WINDOWS_1252};

let (text, encoding, tally) = WINDOWS_1252.decode(b"caf\xE9");
assert_eq!(text, "caf\u{E9}");
assert_eq!(encoding, WINDOWS_1252);
assert!(tally.is_lossless());

// A byte order mark wins over the encoding you name.
let (text, encoding, _) = WINDOWS_1252.decode(b"\xEF\xBB\xBFcaf\xC3\xA9");
assert_eq!(text, "caf\u{E9}");
assert_eq!(encoding.name(), "UTF-8");

§Two lookups

Encoding::for_label answers with the charset a label names. Encoding::for_whatwg_label answers with what the WHATWG Encoding Standard resolves it to, which for 52 of its labels is a different charset — usually a superset a browser is more likely to have meant.

use charcode::{Encoding, ISO_8859_1, WINDOWS_1252};

// `iso-8859-1` names ISO-8859-1.
assert_eq!(Encoding::for_label(b"iso-8859-1"), Some(ISO_8859_1));
assert_eq!(ISO_8859_1.decode(b"\x80").0, "\u{80}");

// The standard sends it to windows-1252, where 0x80 is a euro sign.
assert_eq!(Encoding::for_whatwg_label(b"iso-8859-1"), Some(WINDOWS_1252));
assert_eq!(WINDOWS_1252.decode(b"\x80").0, "\u{20AC}");

Where the standard names an encoding after a charset whose table it does not carry, the name here is the one that fits what it holds — its Big5 is "Big5-HKSCS", its Shift_JIS is "windows-31j" — so that the charset itself can keep its own name. Every replacement is a label the standard defines, so Encoding::for_whatwg_label still finds them.

Use the second for anything that came off the network — a Content-Type charset parameter, an HTML <meta charset> — where matching browsers is the point. Use the first everywhere else:

use charcode::Encoding;

assert_eq!(Encoding::for_whatwg_label(b"latin1").unwrap().name(), "windows-1252");
assert_eq!(Encoding::for_whatwg_label(b"  Shift-JIS ").unwrap().name(), "windows-31j");
assert!(Encoding::for_whatwg_label(b"not-an-encoding").is_none());

Encoding goes the other way, and stops at the first character the target cannot represent — silently mangling text is never the default:

use charcode::{EncodeOptions, EUC_KR, Unmappable};

let (bytes, encoding, _) = EUC_KR.encode("\u{D55C}").unwrap();
assert_eq!(&bytes[..], b"\xC7\xD1");
assert_eq!(encoding, EUC_KR);

// An emoji is not in EUC-KR, so say what should happen to it.
assert!(EUC_KR.encode("\u{D55C}\u{1F600}").is_err());
let options = EncodeOptions::new().unmappable(Unmappable::Replace('?'));
let (bytes, _, tally) = EUC_KR.encode_with("\u{D55C}\u{1F600}", options).unwrap();
assert_eq!(&bytes[..], b"\xC7\xD1?");
assert_eq!(tally.errors, 1);

§Error policies

DecodeOptions and EncodeOptions say what to do about input that does not decode and characters the target cannot represent. Decoding substitutes U+FFFD by default, as the standard requires; encoding fails by default, because every alternative changes the text:

	`Malformed`	`Unmappable`
stop and report	`Fail`	`Fail` (default)
drop it	`Omit`	`Omit`
write a character	`Replace(c)` (default U+FFFD)	`Replace(c)`
`一`		`Html`
`\u4e00`		`JsonEscape`

The two escaping policies also rewrite their own introducer — & becomes &, \ becomes \\ — so what they write reads back unambiguously. With the translit feature, EncodeOptions::transliterate tries a close ASCII equivalent first and falls through to the policy above.

Both return a Cow, borrowed when the input is already the answer, so passing ASCII through an ASCII-compatible encoding costs nothing but the scan.

§Converting a stream

For input that arrives in pieces, Encoding::new_decoder and Encoding::new_encoder give state machines that carry a partial sequence from one buffer to the next. Pass last = true with the final buffer so that a truncated sequence is reported rather than dropped:

use charcode::BIG5;

let mut decoder = BIG5.new_decoder();
let mut text = String::new();
decoder.decode_to_string(&[0xA4], &mut text, false).unwrap();
decoder.decode_to_string(&[0x40], &mut text, true).unwrap();
assert_eq!(text, "\u{4E00}");

Decoder::decode_to_utf8 and Encoder::encode_from_utf8 are the allocation-free forms, writing into a caller-provided &mut [u8].

§Errors

§Features

Capabilities:

std (default): implements std::error::Error for the error types. Implies alloc.
alloc (default, through std): the conveniences that hand back an owned String, Vec or Cow — Encoding::decode and Encoding::encode, Decoder::decode_to_string, Encoder::encode_from_str and their variants. Without it the crate never allocates: what remains is encoding lookup plus the streaming API, which converts into buffers the caller provides.
serde: serializes an encoding as its name. Needs neither std nor alloc.
cli: builds the charcode command-line tool. Needs std.

The encodings themselves come in table groups, all of which are independent:

whatwg (default): the standard’s 40 encodings and whatwg-aliases.
single-byte: the standard’s 28 legacy single-byte encodings.
big5, euc-jp, euc-kr, gb18030, iso-2022-jp, shift-jis: one per legacy multi-byte encoding, because theirs are the large tables. Each also brings the charset the standard names it after: gb18030 provides GBK and GB2312, big5 provides BIG5, and the three Japanese groups share one delta back to JIS X 0208.
iso-2022-kr: ISO_2022_KR (RFC 1557), on the EUC-KR table.
iso-2022-cn: ISO_2022_CN (RFC 1922), with CNS 11643 planes 1 and 2 of its own.
iso-2022-jp-2: ISO_2022_JP_2 (RFC 1554), which brings no tables of its own — it designates six sets the groups above already carry.
scsu: SCSU (UTS #6), a compression scheme rather than a charset. No tables and no dependencies; eight windows move over Unicode instead.
extras: everything below at once.
dos: IBM PC / OEM code pages — 437, 737, 775, 850, 852, 855, 856, 857, 860 to 865, 869, 1006.
ebcdic: IBM mainframe code pages — 037, 424, 500, 875, 1026.
mac: Apple’s regional variants of Mac OS Roman.
misc: Atari ST and KZ-1048.
unicode-extras: UTF-32BE/LE and UTF-7. No tables; these are algorithmic.

UTF-8, UTF-16BE/LE, ISO_8859_1, US_ASCII, replacement and x-user-defined need no tables and are always present; the first three are what byte order mark sniffing resolves to. A build with no table group at all is about 4 KiB of static data; the whole standard is about 640 KiB, and every charset here is about 750 KiB.

Separately, whatwg-aliases adds Encoding::for_whatwg_label, the standard’s get an encoding. It is independent of which tables you take, and it never answers with a charset from outside the standard, so adding dos or ebcdic for local use cannot widen what a label off the network can select:

# Japanese and Unicode, with the standard's naming for them, and the DOS
# code pages available locally but not selectable by a remote label.
charcode = { version = "0.1", default-features = false, features = [
    "std", "whatwg-aliases", "shift-jis", "euc-jp", "iso-2022-jp", "dos",
] }

Structs§

DecodeOptions: How to decode.
Decoder: A streaming decoder for one byte stream.
EncodeOptions: How to encode.
Encoder: A streaming encoder for one text stream.
Encoding: A character encoding.
MalformedError: A malformed byte sequence found by a decode that does not substitute errors.
Tally: How much a conversion had to substitute or drop.
UnmappableError: A character that the target encoding cannot represent.

Enums§

Bom: What to do with a byte order mark at the start of the input.
DecoderResult: Why a decode that reports errors stopped.
EncoderResult: Why an encode that reports errors stopped.
Malformed: What to do with a byte sequence that does not decode.
Unmappable: What to do with a character the target encoding cannot represent.

Constants§

DECODER_MIN_BUFFER: The smallest output buffer Decoder::decode_to_utf8 can make progress with.
ENCODER_MIN_BUFFER: The smallest output buffer Encoder::encode_from_utf8 can make progress with.

Statics§

ATARI_ST: Atari-ST
BIG5: Big5, as the Unicode Consortium maps it in BIG5.TXT.
BIG5_HKSCS: The standard’s Big5: Big5 plus the Hong Kong Supplementary Character Set and other common extensions. For Big5 itself see BIG5.
EUC_JP: EUC-JP: ASCII, JIS X 0201’s katakana, JIS X 0208 and JIS X 0212.
EUC_KR: EUC-KR, in practice the Unified Hangul Code (Windows codepage 949).
GB2312: GB 2312-80, in its EUC-CN form.
GB18030: gb18030, the Chinese national standard, which covers all of Unicode.
GBK: GBK, which decodes as GB18030 but whose encoder emits only two-byte sequences.
IBM037: IBM037
IBM424: IBM424
IBM437: IBM437
IBM500: IBM500
IBM737: IBM737
IBM775: IBM775
IBM850: IBM850
IBM852: IBM852
IBM855: IBM855
IBM856: IBM856
IBM857: IBM857
IBM860: IBM860
IBM861: IBM861
IBM862: IBM862
IBM863: IBM863
IBM864: IBM864
IBM865: IBM865
IBM866: IBM866.
IBM869: IBM869
IBM875: IBM875
IBM1006: IBM1006
IBM1026: IBM1026
ISO_2022_CN: ISO-2022-CN (RFC 1922): ASCII, GB 2312 and CNS 11643 planes 1 and 2.
ISO_2022_JP: ISO-2022-JP, as RFC 1468 defines it: ASCII, JIS X 0201’s Roman set and JIS X 0208.
ISO_2022_JP_2: ISO-2022-JP-2 (RFC 1554), which designates six sets rather than three.
ISO_2022_KR: ISO-2022-KR (RFC 1557), which shifts between ASCII and KS X 1001.
ISO_8859_1: ISO-8859-1
ISO_8859_2: ISO-8859-2.
ISO_8859_3: ISO-8859-3.
ISO_8859_4: ISO-8859-4.
ISO_8859_5: ISO-8859-5.
ISO_8859_6: ISO-8859-6.
ISO_8859_7: ISO-8859-7.
ISO_8859_8: ISO-8859-8.
ISO_8859_9: ISO-8859-9
ISO_8859_8_I: ISO-8859-8-I, identical to ISO_8859_8 except in its name and implied text direction.
ISO_8859_10: ISO-8859-10.
ISO_8859_11: ISO-8859-11
ISO_8859_13: ISO-8859-13.
ISO_8859_14: ISO-8859-14.
ISO_8859_15: ISO-8859-15.
ISO_8859_16: ISO-8859-16.
KOI8_R: KOI8-R.
KOI8_U: KOI8-U.
KZ_1048: KZ-1048
MACINTOSH: macintosh.
REPLACEMENT: The replacement encoding, which decodes any non-empty input to a single U+FFFD.
SCSU: The Standard Compression Scheme for Unicode (UTS #6).
SHIFT_JIS: Shift_JIS, as JIS X 0208:1997 Annex 1 defines it.
US_ASCII: US-ASCII
UTF_7: UTF-7
UTF_8: UTF-8, the encoding every new format should use.
UTF_16BE: UTF-16BE. Decode only; encoding to it yields UTF-8, per get an output encoding.
UTF_16LE: UTF-16LE. Decode only; encoding to it yields UTF-8, per get an output encoding.
UTF_32BE: UTF-32BE
UTF_32LE: UTF-32LE
WINDOWS_31J: The standard’s Shift_JIS, which is Windows codepage 932: the NEC and IBM extension rows, the end-user defined area, and seven of JIS X 0208’s pointers remapped. For Shift_JIS itself see SHIFT_JIS.
WINDOWS_874: windows-874.
WINDOWS_1250: windows-1250.
WINDOWS_1251: windows-1251.
WINDOWS_1252: windows-1252.
WINDOWS_1253: windows-1253.
WINDOWS_1254: windows-1254.
WINDOWS_1255: windows-1255.
WINDOWS_1256: windows-1256.
WINDOWS_1257: windows-1257.
WINDOWS_1258: windows-1258.
X_MAC_ARABIC: x-mac-arabic
X_MAC_CELTIC: x-mac-celtic
X_MAC_CENTRALEURROMAN: x-mac-centraleurroman
X_MAC_CROATIAN: x-mac-croatian
X_MAC_CYRILLIC: x-mac-cyrillic.
X_MAC_FARSI: x-mac-farsi
X_MAC_GAELIC: x-mac-gaelic
X_MAC_GREEK: x-mac-greek
X_MAC_ICELANDIC: x-mac-icelandic
X_MAC_ROMANIAN: x-mac-romanian
X_MAC_TURKISH: x-mac-turkish
X_USER_DEFINED: x-user-defined, which maps bytes 0x80 to 0xFF into the private use area.
X_WHATWG_EUC_JP: The standard’s EUC-JP, which folds the NEC and IBM extension rows into the JIS X 0208 plane and remaps six of its pointers. Its decoder accepts JIS X 0212 via the 0x8F prefix; its encoder never writes one. For EUC-JP itself see EUC_JP.
X_WHATWG_ISO_2022_JP: The standard’s ISO-2022-JP, which adds a half-width katakana mode behind ESC ( I, folds the NEC and IBM extension rows into the JIS X 0208 plane, and remaps six of its pointers. For ISO-2022-JP itself see ISO_2022_JP.

Crate charcode

Crate charcode Copy item path

§Converting a whole buffer

§Two lookups

§Error policies

§Converting a stream

§Errors

§Features

Structs§

Enums§

Constants§

Statics§

Crate charcode