sketchir 0.3.0

Sketching primitives for IR: minhash/simhash/LSH-style signatures.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# sketchir

[![crates.io](https://img.shields.io/crates/v/sketchir.svg)](https://crates.io/crates/sketchir)
[![Documentation](https://docs.rs/sketchir/badge.svg)](https://docs.rs/sketchir)
[![CI](https://github.com/arclabs561/sketchir/actions/workflows/ci.yml/badge.svg)](https://github.com/arclabs561/sketchir/actions/workflows/ci.yml)

Sketching primitives for IR: MinHash/SimHash/LSH-style signatures.

## What it is

`sketchir` is the index-only layer for:
- near-duplicate detection (shingles + MinHash)
- text fingerprinting (SimHash)
- approximate similarity candidate generation (LSH)

## Best starting points

- **Near-duplicate detection**: `MinHashTextLSH` + `BlockingConfig`
- **SimHash fingerprints**: `SimHashFingerprint` / `SimHashLSH`
- **Dense-vector LSH (batch)**: `LSHIndex` -- multi-table random projection, add/build/search lifecycle
- **Dense-vector LSH (incremental)**: `DenseSimHashLSH` -- SimHash + Hamming-1 probing, no build step

## Tuning knobs (`BlockingConfig`)

| Param | Typical | Tradeoff |
|---|---|---|
| `ngram_size` | 3-9 chars | Smaller = more sensitive to noise; Larger = stricter. |
| `num_bands * num_hashes_per_band` | 100-256 | More = better Jaccard estimation, higher storage. |
| `num_bands` | 20-50 | Controls recall/precision curve (S-curve). |

## Example (MinHash blocking)

```rust
use sketchir::{BlockingConfig, MinHashTextLSH};

let cfg = BlockingConfig::default();
let mut index = MinHashTextLSH::new(cfg).unwrap();
index.insert_text("a", "hello world");
index.insert_text("b", "hello  world!");

let pairs = index.candidate_pairs();
assert!(!pairs.is_empty());
```

## License

MIT OR Apache-2.0