probabilistic-rs 0.6.3

Probabilistic data structures in Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

# probabilistic-rs

[![Crates.io](https://img.shields.io/crates/v/probabilistic-rs.svg)](https://crates.io/crates/probabilistic-rs)
[![PyPI](https://img.shields.io/pypi/v/probabilistic-rs.svg)](https://pypi.org/project/probabilistic-rs/)
[![Documentation](https://docs.rs/probabilistic-rs/badge.svg)](https://docs.rs/probabilistic-rs)
[![codecov](https://codecov.io/gh/oiwn/probabilistic-rs/graph/badge.svg?token=5JMM0V5RFO)](https://codecov.io/gh/oiwn/probabilistic-rs)

Probabilistic data structures in Rust with Python bindings and HTTP API.

## Features

- **Bloom Filter** — fast membership testing with bulk ops and optional Fjall persistence
- **Expiring Bloom Filter** — auto-expires elements via sliding time windows
- **HTTP API** — REST server with Swagger UI, managing multiple named filters
- **Python Bindings** — native wheels via PyO3/maturin
- **CLI + TUI** — interactive terminal interface

![TUI Screenshot](tui.png)

## Installation

**Rust:**
```bash
cargo add probabilistic-rs
```

**Python:**
```bash
pip install probabilistic-rs
```

## Quick Start

### Rust — Bloom Filter

```rust
use probabilistic_rs::bloom::{BloomFilter, BloomFilterConfigBuilder, BloomFilterOps};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = BloomFilterConfigBuilder::default()
        .capacity(10000)
        .false_positive_rate(0.01)
        .build()?;

    let filter = BloomFilter::create(config).await?;
    filter.insert(b"item1")?;
    assert!(filter.contains(b"item1")?);
    Ok(())
}
```

### Rust — Expiring Bloom Filter

```rust
use probabilistic_rs::ebloom::{
    ExpiringBloomFilter, ExpiringFilterConfigBuilder, ExpiringBloomFilterOps,
};
use std::time::Duration;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ExpiringFilterConfigBuilder::default()
        .capacity(1000)
        .false_positive_rate(0.01)
        .level_duration(Duration::from_secs(60))
        .max_levels(3)
        .build()?;

    let mut filter = ExpiringBloomFilter::new(config)?;
    filter.insert(b"test_item")?;
    assert!(filter.query(b"test_item")?);
    Ok(())
}
```

### Python

```python
from probabilistic_rs import BloomFilter, ExpiringBloomFilter

bf = BloomFilter(capacity=10000, false_positive_rate=0.01)
bf.insert(b"item1")
assert bf.contains(b"item1")

ebf = ExpiringBloomFilter(capacity=1000, false_positive_rate=0.01, ttl_seconds=60)
ebf.insert(b"temp_item")
assert ebf.query(b"temp_item")
```

## HTTP API

```bash
# Start server (default: localhost:3000)
probabilistic-server

# Create a filter, insert, query
curl -X POST http://localhost:3000/api/v1/bloom/create \
  -H "Content-Type: application/json" \
  -d '{"name":"my-filter","capacity":10000,"false_positive_rate":0.01}'

curl -X POST http://localhost:3000/api/v1/bloom/insert \
  -H "Content-Type: application/json" \
  -d '{"name":"my-filter","item":"hello"}'

curl -X POST http://localhost:3000/api/v1/bloom/contains \
  -H "Content-Type: application/json" \
  -d '{"name":"my-filter","item":"hello"}'
```

Endpoints: `create`, `delete`, `insert`, `contains`, `bulk_insert`, `bulk_contains`, `clear`, `stats`, `list` — available for both `/api/v1/bloom` and `/api/v1/ebloom`. Swagger UI at `/swagger-ui`.

## CLI

```bash
# Create filter
expblf create --db-path myfilter.fjall --capacity 10000 --fpr 0.01

# Operations
expblf load --db-path myfilter.fjall insert --element "key"
expblf load --db-path myfilter.fjall check --element "key"

# Interactive TUI
expblf tui --db-path myfilter.fjall
```

## Benchmarks

Measured on Apple M-series via `cargo bench` (criterion, 100 samples). Times are total for N operations.

**Bloom Filter (in-memory)**

| Operation | 1K elements | 100K elements | 1M elements |
|-----------|-------------|---------------|-------------|
| Insert    | 60.2 µs     | 6.15 ms       | 64.1 ms     |
| Query     | 61.3 µs     | 6.17 ms       | 63.1 ms     |

**Expiring Bloom Filter (in-memory, 3 levels — 5 levels nearly identical)**

| Operation    | 1K elements | 100K elements | 1M elements |
|--------------|-------------|---------------|-------------|
| Insert       | 63.8 µs     | 6.61 ms       | 68.0 ms     |
| Query        | 63.4 µs     | 6.52 ms       | 67.0 ms     |
| Bulk insert  | 59.7 µs     | 6.16 ms       | 63.8 ms     |
| Bulk query   | 62.3 µs     | 6.41 ms       | 65.8 ms     |
| Level rotate | 224 µs      | 255 µs        | —           |

Both filters sustain ~15–17M ops/s (60–65 ns/op) across all dataset sizes. The expiring filter adds ~5–10% overhead over plain bloom due to multi-level bookkeeping. Bulk operations match or slightly outperform single-item ops. Level rotation (TTL expiry) takes ~250 µs regardless of filter size.

## Configuration

| Parameter | Description | Default |
|-----------|-------------|---------|
| `capacity` | Max elements | 1,000,000 |
| `false_positive_rate` | Desired FPR | 0.01 |
| `level_duration` | TTL per level (expiring) | 60s |
| `max_levels` | Filter levels (expiring) | 3 |

## License

MIT