Struct DedupTransformer 

pub struct DedupTransformer {
    pub streaming: bool,
    /* private fields */
}

A transform that deduplicates samples using MinHash + LSH.

This transform buffers samples to compute signatures and find duplicates. It can operate in two modes:

• Streaming: Process samples as they arrive, outputting non-duplicates immediately
• Batch: Buffer all samples, then output deduplicated set

Example

use dedup::{Config, DedupTransformer};
use tenshift_core::sample::Sample;
use tenshift_core::transform::Transform;

let config = Config::default()
    .with_similarity_threshold(0.9);

let mut dedup = DedupTransformer::new(config).unwrap();

Fields

streaming: bool

Whether we’re in streaming mode.

Implementations

impl DedupTransformer

pub fn new(config: Config) -> Result<Self>

Create a new deduplication transformer.

Errors

Returns an error if the configuration is invalid.

pub fn with_text_field(self, field: impl Into<String>) -> Self

Set the field name containing text to deduplicate on.

pub fn with_streaming(self, enabled: bool) -> Self

Enable streaming mode (output non-duplicates immediately).

pub fn with_mark_duplicates(self, enabled: bool) -> Self

Enable marking duplicates instead of filtering them.

When enabled, duplicates are tagged with a is_duplicate field instead of being removed from the output.

pub fn process_sample(&mut self, sample: &Sample) -> Result<bool>

Process a single sample.

Computes the MinHash signature and adds to the LSH index. Returns true if the sample is unique, false if it’s a duplicate.

Errors

Returns an error if signature computation fails.

pub fn push(&mut self, sample: Sample)

Add a sample to the buffer for batch processing, or process immediately if streaming mode is enabled.

pub fn drain_streaming(&mut self) -> Vec<Sample>

Drain any pending output samples produced in streaming mode.

pub fn finish_batch(&mut self) -> Vec<Sample>

Process all buffered samples and return deduplicated results.

This computes signatures for all samples, builds the LSH index, finds clusters, and returns only unique samples.

pub fn clusters(&mut self) -> &[DuplicateCluster]

Get duplicate clusters.

Returns all detected duplicate clusters. Call after finish_batch() for complete results.

pub fn stats(&self) -> LshStats

Get statistics about the deduplication process.

pub fn unique_count(&self) -> usize

Get the number of unique documents found.

pub fn duplicate_count(&self) -> usize

Get the number of duplicate documents found.

pub fn reset(&mut self)

Reset the transformer state.