Skip to main content

Crate incrededup

Crate incrededup 

Source
Expand description

incrededup: Performant, disk-based, incremental deduplication using MinHash LSH

This library provides:

  • Rensa-derived R-MinHash signature computation
  • LSH indexing, with Rensa-derived in-memory banding and disk-backed sidecars
  • PostgreSQL integration for document fetching and result storage
  • Parallel processing with rayon

§Example

ⓘ
use incrededup::{DbConfig, DedupeConfig, run_dedupe};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let db_config = DbConfig::from_env()?.with_table("documents");

    let dedupe_config = DedupeConfig {
        threshold: 0.8,
        ..Default::default()
    };

    let stats = run_dedupe(db_config, dedupe_config).await?;
    println!("Found {} duplicates", stats.duplicates_found);
    Ok(())
}

Re-exports§

pub use db::DbConfig;
pub use db::DbPool;
pub use db::Document;
pub use db::DupeMatch;
pub use dedupe::is_boilerplate;
pub use dedupe::load_junk_patterns;
pub use dedupe::resolve_transitivity;
pub use dedupe::run_dedupe;
pub use dedupe::run_dedupe_with_source;
pub use dedupe::DedupeConfig;
pub use dedupe::DedupeStats;
pub use dedupe::EdgeLookupMode;
pub use dedupe::InMemoryDeduplicator;
pub use dedupe::IndexBuilder;
pub use cleanup::find_pathological_clusters;
pub use cleanup::run_cleanup;
pub use cleanup::run_phase_1_5;
pub use cleanup::CleanupAction;
pub use cleanup::CleanupStats;
pub use cleanup::PathologicalCluster;
pub use cli::Args;
pub use cli::EdgeLookupArg;
pub use disk_dedupe::run_disk_dedupe;
pub use disk_dedupe::run_disk_dedupe_with_options;
pub use disk_dedupe::DiskDedupeRunOptions;
pub use disk_dedupe::DiskDedupeStats;
pub use disk_dedupe::DiskDeduplicator;
pub use lsh::DiskLSH;
pub use lsh::InMemoryLSH;
pub use minhash::calculate_band_hash;
pub use minhash::compute_band_hashes;
pub use minhash::jaccard_from_signatures;
pub use minhash::try_compute_band_hashes;
pub use minhash::try_jaccard_from_signatures;
pub use minhash::RMinHash;
pub use minhash::NUM_BANDS;
pub use minhash::NUM_PERM;
pub use minhash::ROWS_PER_BAND;
pub use sources::DocumentSource;
pub use sources::FileSystemSource;
pub use sources::PostgresSource;
pub use sources::SourceDocument;
pub use sources::SourceDupeMatch;
pub use sources::SqliteSource;
pub use storage::DatasetStorage;
pub use storage::FilteredParentStore;
pub use storage::MatchRecord;
pub use storage::MatchStore;
pub use storage::StateStore;
pub use storage::SyncProgress;
pub use storage::SyncStep;
pub use tokenizer::compute_signature;
pub use tokenizer::compute_signature_with_ngram_width;
pub use tokenizer::normalize_content;
pub use tokenizer::tokenize;
pub use tokenizer::tokenize_with_ngram_width;
pub use tokenizer::DEFAULT_CHAR_NGRAM_WIDTH;
pub use tokenizer::TOKENIZER_VERSION;
pub use union_find::UnionFind;

Modules§

cleanup
Cleanup mode for finding and handling pathological clusters.
cli
db
PostgreSQL database integration for document deduplication.
dedupe
Document deduplication orchestrator.
disk_dedupe
Disk-based deduplication with resumable state.
lsh
LSH (Locality-Sensitive Hashing) implementation with disk-backed storage.
minhash
R-MinHash implementation derived from Rensa (https://github.com/beowolx/rensa).
sources
Document source abstractions for pluggable data connectors.
storage
Canonical local storage for deduplication results.
tokenizer
Text normalization and tokenization for MinHash signatures.
union_find
Union-Find (Disjoint Set Union) data structure for transitivity resolution.

Type Aliases§

DiskIndexBuilderDeprecated
Deprecated compatibility alias for the Phase 1 index builder.