Struct Deduplicator 

pub struct Deduplicator { /* private fields */ }

LRU-based content deduplicator.

Tracks seen text fragments to detect duplicates during processing.

Example

use html_cleaning::dedup::Deduplicator;

let mut dedup = Deduplicator::new(100);

assert!(!dedup.is_duplicate("first occurrence"));
assert!(!dedup.is_duplicate("first occurrence")); // seen once
assert!(!dedup.is_duplicate("first occurrence")); // seen twice
assert!(dedup.is_duplicate("first occurrence"));  // now duplicate (>2)

Implementations

impl Deduplicator

pub fn new(capacity: usize) -> Self

Create with specified capacity.

Uses default threshold of 2 (text seen more than 2 times is duplicate).

Arguments

• capacity - Maximum number of entries to track

pub fn with_threshold(capacity: usize, threshold: i32) -> Self

Create with capacity and custom duplicate threshold.

Arguments

• capacity - Maximum entries
• threshold - Number of times text can appear before considered duplicate

pub fn is_duplicate(&mut self, text: &str) -> bool

Check if text is duplicate, adding to cache.

Returns true if text has been seen more than threshold times.

pub fn check(&self, text: &str) -> bool

Check without adding to cache.

pub fn has_seen(&self, text: &str) -> bool

Check if text has been seen (regardless of count).

pub fn clear(&mut self)

Clear the cache.

pub fn len(&self) -> usize

Get current cache size.

pub fn is_empty(&self) -> bool

Check if cache is empty.