Struct StatefulDedupTransform 

pub struct StatefulDedupTransform { /* private fields */ }

A stateful deduplication transform that buffers samples.

This is the recommended transform for deduplication as it properly handles the stateful nature of duplicate detection.

Implementations

impl StatefulDedupTransform

pub fn new(config: Config) -> Result<Self>

Create a new stateful deduplication transform.

Errors

Returns an error if the configuration is invalid.

pub fn with_text_field(self, field: impl Into<String>) -> Self

Set the text field name.

pub fn with_mark_duplicates(self, enabled: bool) -> Self

Enable marking duplicates.

pub fn clusters(&mut self) -> &[DuplicateCluster]

Get duplicate clusters.

pub fn stats(&self) -> LshStats

Get statistics.

Trait Implementations

impl StatefulTransform for StatefulDedupTransform

fn push(&mut self, sample: Sample) -> Vec<Sample>

Push a sample into this transform. Returns any immediately available outputs.

fn finish(&mut self) -> Vec<Sample>

Signal that no more input is coming. Returns any buffered samples.

fn name(&self) -> &str

Human-readable name for logging.