pub struct DetectionOptions {
pub steps: usize,
pub chunk_size: usize,
pub threshold: f64,
pub cp_isolation: Vec<String>,
pub cp_exclusion: Vec<String>,
pub preemptive_behaviour: bool,
pub explain: bool,
pub language_threshold: f64,
pub enable_fallback: bool,
}Expand description
Tuning knobs of from_bytes_with and detect.
The Default values match the Python package’s from_bytes. Override
only the fields you need with struct update syntax:
use charset_norm::DetectionOptions;
let options = DetectionOptions {
// Never consider these code pages, whatever the payload looks like.
cp_exclusion: vec!["utf_7".into(), "hz".into()],
// Look at more of the payload.
steps: 10,
..DetectionOptions::default()
};
assert_eq!(options.chunk_size, 512);Fields§
§steps: usizeNumber of chunks sampled from the payload. Default: 5.
Chunks are spread evenly over the payload. When the payload is not
longer than steps * chunk_size bytes, it is analysed whole, as a
single chunk.
chunk_size: usizeSize of each sampled chunk, in bytes. Default: 512.
Shrunk automatically when the payload is too short to hold steps
chunks of this size.
threshold: f64Maximum acceptable mean chaos (see mess::mess_ratio) for a
candidate encoding. Default: 0.2.
Lower values reject more candidates; raise it to accept noisier text.
cp_isolation: Vec<String>When not empty, only these encodings are tried. Default: empty.
Names may be any alias ("windows-1252", "UTF-8", …).
cp_exclusion: Vec<String>Encodings never tried, by any alias. Default: empty.
preemptive_behaviour: boolFavour an encoding declared inside the payload, such as
<meta charset="..."> in HTML or a # coding: ... comment in Python
sources. Default: true.
Only the first 8 KiB are searched. A declared encoding is tried right after any BOM-announced one, and detection stops there when it decodes the payload without any chaos.
explain: boolLog the chaos breakdown of every sampled chunk at Level::Trace.
Default: false.
Only takes effect when cp_isolation names one
or two encodings, to keep the output readable.
language_threshold: f64Minimum coherence (in [0, 1]) for a language to be reported.
Default: 0.1.
enable_fallback: boolFall back on ASCII, UTF-8 or a declared encoding when every candidate
is rejected but one of those still decodes the payload. Default:
true.
Without the fallback, text that is merely noisy can come back as no
match at all (see is_binary).