Crate charset_norm

Expand description

Universal character encoding detector.

charset-norm is a Rust implementation of charset_normalizer. Give it bytes of unknown origin and it tells you which encodings they were plausibly written in, which language the text is in, and how confident it is. It knows 98 code pages, from UTF-8 and the Windows/ISO-8859 families to the CJK and ISO-2022 encodings.

Decoding reproduces CPython’s codecs byte for byte, so verdicts agree with the Python package.

§Quick start

let results = charset_norm::from_bytes("Ça va très bien, merci.".as_bytes());
let best = results.best().expect("text payload");
assert_eq!(best.encoding(), "utf_8");
assert_eq!(best.decoded().unwrap(), "Ça va très bien, merci.");

from_bytes returns CharsetMatches, the plausible encodings best first. It is empty when nothing fits, which usually means the payload is binary. Each CharsetMatch exposes the encoding, the decoded text, the detected languages and the scores behind the verdict:

use charset_norm::codecs;

let text = "Всеки човек има право на образование. Образованието трябва да бъде безплатно.";
let payload = codecs::encode(text, "cp1251").unwrap();

let results = charset_norm::from_bytes(&payload);
let best = results.best().unwrap();
assert_eq!(best.encoding(), "cp1251");
assert!(best.chaos() < 0.1);

// Languages come ranked by coherence. On one short sentence, Cyrillic
// languages are hard to tell apart, so look at the whole list.
assert!(best.languages().contains(&"Bulgarian"));

// Transcode to UTF-8 (in-document charset declarations are rewritten too).
assert_eq!(best.output("utf_8").unwrap(), text.as_bytes());

Files are read with from_path; is_binary answers the text vs. binary question alone.

§Tuning detection

from_bytes_with takes DetectionOptions to restrict the candidate encodings, change how much of the payload is sampled, or set how noisy a decoding may look before it is rejected. The defaults match the Python package.

use charset_norm::{DetectionOptions, NoLogger, codecs, from_bytes_with};

let payload = codecs::encode("Příliš žluťoučký kůň úpěl ďábelské ódy.", "cp1250").unwrap();
let options = DetectionOptions {
    cp_isolation: vec!["cp1250".into(), "cp1252".into()],
    ..DetectionOptions::default()
};
let results = from_bytes_with(&payload, &options, &NoLogger);
assert_eq!(results.best().unwrap().encoding(), "cp1250");

§How it works

Candidates are tried in order: the encoding announced by a byte order mark, the one declared in the document (<meta charset>, # coding:), ASCII, UTF-8, then every other code page. For each one, the detector:

Skips it when the payload cannot be decoded with it, or when a near-identical code page (see encoding::is_cp_similar) was already rejected.
Decodes steps chunks of chunk_size bytes spread over the payload.
Scores each chunk’s chaos with mess::mess_ratio: suspicious character successions, unexpected accents, unprintable characters and so on. Candidates whose mean chaos reaches threshold are dropped.
Scores the survivors’ coherence with coherence::coherence_ratio: how well letter frequencies match known languages.
Ranks the matches: lower chaos first, then higher coherence. Encodings that produce the very same text become submatches of one another.

Detection stops early once a preferred candidate (declared, ASCII or UTF-8) decodes cleanly. When no candidate survives, ASCII, UTF-8 or a declared encoding may still be returned as a fallback (see enable_fallback).

§Modules

The top-level functions cover most needs. The building blocks are public for tools that want them:

Module	Contents
`codecs`	`CPython`-compatible decoders and encoders
`encoding`	Code page names and aliases, BOMs, declared charsets, code page languages
`mess`	Chaos (noise) scoring of decoded text
`coherence`	Language detection from letter frequencies
`unicode`	Character properties (`unicodedata` semantics)
`chunks`	Sampling of decoded chunks, as the detector does it
`log`	Diagnostics emitted while detecting

§Encoding names

Encodings are reported by their canonical CPython names: "utf_8", "cp1252", "shift_jis", "iso8859_5". Wherever a name is accepted, aliases such as "UTF-8", "windows-1252" or "latin-1" work too; see encoding::iana_name and encoding::supported_encodings.

§Diagnostics

Detection reports what it tries and why it rejects candidates through a Logger. NoLogger discards everything; implement the trait to collect messages yourself, or enable the log feature and pass log::LogCrate to forward them to the log crate.

§Feature flags

log (off by default): adds log::LogCrate, a Logger that forwards diagnostics to the log crate under the charset_norm target.

§Minimum supported Rust version

Rust 1.98.

Re-exports§

pub use log::Level;
pub use log::Logger;
pub use log::NoLogger;

Modules§

chunks: Sampling of decoded chunks, the unit the detector measures.
codecs: Native implementations of the CPython codecs charset-normalizer probes.
coherence: Language coherence: how well decoded text matches known letter frequencies.
encoding: Code page names, signatures and the languages a code page can express.
log: Diagnostics emitted while detecting.
mess: Mess (noise) detection: how implausible decoded text looks.
unicode: Character properties computed natively (no unicodedata round-trips).

Structs§

CharsetMatch: One plausible encoding for a payload.
CharsetMatches: Plausible encodings for a payload, best first.
DetectionOptions: Tuning knobs of from_bytes_with and detect.

Enums§

Error: Errors reported by the lookup helpers of this crate, such as encoding::iana_name or coherence::get_target_features.
OutputError: Why CharsetMatch::output or CharsetMatch::output_text failed.

Constants§

TOO_BIG_SEQUENCE: Payloads of at least this many bytes are sampled rather than fully decoded.
TOO_SMALL_SEQUENCE: Payloads shorter than this many bytes are considered too small for reliable detection.

Functions§

detect: from_bytes_with for a payload that is already shared: the matches keep a reference to it instead of a copy.
from_bytes: Detect the plausible encodings of payload with default options.
from_bytes_with: Detect the plausible encodings of payload with custom options, reporting progress to logger.
from_path: Read a file and detect its plausible encodings with default options.
is_binary: Whether payload looks like binary data rather than text.
sort_by_rank: Sort items by a “ranks before” predicate, reproducing CPython’s list.sort() order for fewer than 64 items (the order is observable because the predicate is not transitive). Larger inputs use a stable binary insertion sort.