Expand description
incrededup: Performant, disk-based, incremental deduplication using MinHash LSH
This library provides:
- Rensa-derived R-MinHash signature computation
- LSH indexing, with Rensa-derived in-memory banding and disk-backed sidecars
- PostgreSQL integration for document fetching and result storage
- Parallel processing with rayon
§Example
ⓘ
use incrededup::{DbConfig, DedupeConfig, run_dedupe};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let db_config = DbConfig::from_env()?.with_table("documents");
let dedupe_config = DedupeConfig {
threshold: 0.8,
..Default::default()
};
let stats = run_dedupe(db_config, dedupe_config).await?;
println!("Found {} duplicates", stats.duplicates_found);
Ok(())
}Re-exports§
pub use db::DbConfig;pub use db::DbPool;pub use db::Document;pub use db::DupeMatch;pub use dedupe::is_boilerplate;pub use dedupe::load_junk_patterns;pub use dedupe::resolve_transitivity;pub use dedupe::run_dedupe;pub use dedupe::run_dedupe_with_source;pub use dedupe::DedupeConfig;pub use dedupe::DedupeStats;pub use dedupe::EdgeLookupMode;pub use dedupe::InMemoryDeduplicator;pub use dedupe::IndexBuilder;pub use cleanup::find_pathological_clusters;pub use cleanup::run_cleanup;pub use cleanup::run_phase_1_5;pub use cleanup::CleanupAction;pub use cleanup::CleanupStats;pub use cleanup::PathologicalCluster;pub use cli::Args;pub use cli::EdgeLookupArg;pub use disk_dedupe::run_disk_dedupe;pub use disk_dedupe::run_disk_dedupe_with_options;pub use disk_dedupe::DiskDedupeRunOptions;pub use disk_dedupe::DiskDedupeStats;pub use disk_dedupe::DiskDeduplicator;pub use lsh::DiskLSH;pub use lsh::InMemoryLSH;pub use minhash::calculate_band_hash;pub use minhash::compute_band_hashes;pub use minhash::jaccard_from_signatures;pub use minhash::try_compute_band_hashes;pub use minhash::try_jaccard_from_signatures;pub use minhash::RMinHash;pub use minhash::NUM_BANDS;pub use minhash::NUM_PERM;pub use minhash::ROWS_PER_BAND;pub use sources::DocumentSource;pub use sources::FileSystemSource;pub use sources::PostgresSource;pub use sources::SourceDocument;pub use sources::SourceDupeMatch;pub use sources::SqliteSource;pub use storage::DatasetStorage;pub use storage::FilteredParentStore;pub use storage::MatchRecord;pub use storage::MatchStore;pub use storage::StateStore;pub use storage::SyncProgress;pub use storage::SyncStep;pub use tokenizer::compute_signature;pub use tokenizer::compute_signature_with_ngram_width;pub use tokenizer::normalize_content;pub use tokenizer::tokenize;pub use tokenizer::tokenize_with_ngram_width;pub use tokenizer::DEFAULT_CHAR_NGRAM_WIDTH;pub use tokenizer::TOKENIZER_VERSION;pub use union_find::UnionFind;
Modules§
- cleanup
- Cleanup mode for finding and handling pathological clusters.
- cli
- db
- PostgreSQL database integration for document deduplication.
- dedupe
- Document deduplication orchestrator.
- disk_
dedupe - Disk-based deduplication with resumable state.
- lsh
- LSH (Locality-Sensitive Hashing) implementation with disk-backed storage.
- minhash
- R-MinHash implementation derived from Rensa (https://github.com/beowolx/rensa).
- sources
- Document source abstractions for pluggable data connectors.
- storage
- Canonical local storage for deduplication results.
- tokenizer
- Text normalization and tokenization for MinHash signatures.
- union_
find - Union-Find (Disjoint Set Union) data structure for transitivity resolution.
Type Aliases§
- Disk
Index Builder Deprecated - Deprecated compatibility alias for the Phase 1 index builder.