pub struct DenoiseConfig {
pub min_block_len: u32,
pub max_link_density: f32,
pub min_text_ratio: f32,
pub drop_chrome: bool,
pub drop_by_class: bool,
pub keep_tables: bool,
}Expand description
Tuning knobs for the denoiser.
Fields§
§min_block_len: u32A block shorter than this (in grapheme clusters) is dropped unless it is a heading, list item or table cell.
max_link_density: f32Blocks above this anchor-text fraction are treated as navigation.
min_text_ratio: f32Blocks below this text-to-markup ratio are treated as widgets.
drop_chrome: boolDrop <nav>, <footer>, <aside> and friends outright.
drop_by_class: boolAlso drop nodes whose class/id matches the boilerplate vocabulary.
keep_tables: boolKeep <table> content as Markdown tables rather than dropping them.
Trait Implementations§
Source§impl Clone for DenoiseConfig
impl Clone for DenoiseConfig
Source§impl Debug for DenoiseConfig
impl Debug for DenoiseConfig
Source§impl Default for DenoiseConfig
impl Default for DenoiseConfig
Source§fn default() -> Self
fn default() -> Self
Tuned against benches/quality.py, which scores extraction on 23 real
pages two ways: five-word shingle overlap with the article container
(recall and precision), and a count of site furniture that leaked in.
Both are needed. Judged on overlap alone the right answer is to switch
every threshold off – it scores best – but that quadruples the
furniture, because the article container the score compares against
never contained the navigation in the first place.
From the defaults these replace, F1 goes 59.1% to 63.1% with furniture unchanged at 4 occurrences over 23 pages.
Auto Trait Implementations§
impl Forget for DenoiseConfig
impl Freeze for DenoiseConfig
impl Move for DenoiseConfig
impl RefUnwindSafe for DenoiseConfig
impl Send for DenoiseConfig
impl Sync for DenoiseConfig
impl Unpin for DenoiseConfig
impl UnsafeUnpin for DenoiseConfig
impl UnwindSafe for DenoiseConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more