Struct ofilter::Stream

pub struct Stream<T> { /* private fields */ }

Streaming Bloom filter.

This is based on a pair of Bloom filters, and is designed to serve as a “streaming” filter. What does this mean?

In a regular Bloom filter, after “some time” which corresponds to nb_items different items set info the filter, the filter stops working well.

In this implementation, we use a pair of Bloom filters, once they are “wasted” because they have too many items, we drop one of them, and only keep the other. This way, we keep a trace of what was in the filter, but avoid growing infinitely and reach a state where any item would pass the filter.

In practice it is much more imprecise than a regular Bloom filter and the false positive rate is quite high. However it never reaches the point where it is totally unusable, it always refuses some items, and keeps the set within a reasonable limit.

Examples

use ofilter::Stream;
use std::hash::Hash;

#[derive(Hash)]
struct Obj {
    some: usize,
    thing: usize,
}

let mut filter: Stream<Obj> = Stream::new(100);
let obj = Obj{ some: 1, thing: 2};

assert!(!filter.check(&obj)); // object is not in filter
filter.set(&obj);             // put object in filter
assert!(filter.check(&obj));  // object is in filter

Implementations

impl<T> Stream<T>

pub fn new(capacity: usize) -> Self

Create a new streaming Bloom filter, with given capacity.

All other parameters are set to defaults, or aligned to match capacity.

This is different from a classic Bloom filter, it maintains a set of items, in this set you can be sure that the last capacity items are always kept in the filter. In other words, it works like a Bloom filter for the most recent items. Ancient items are progressively removed from the set.

Examples

use ofilter::Stream;

let filter: Stream<usize> = Stream::new(100);
assert_eq!(100, filter.capacity());

pub fn new_with_params(params: Params) -> Self

Create a new streaming Bloom filter, with specific parameters.

Examples

use ofilter::{Stream, Params};

let filter: Stream<usize> = Stream::new_with_params(Params::with_nb_items_and_fp_rate(100, 0.1));
assert_eq!(100, filter.capacity());

pub fn params(&self) -> Params

Get filter params.

This can be useful because when creating the filter, the .adjust() func is called, and may decide to fine-tune some parameters. With this, one can know exactly what is used by the filter.

Examples

use ofilter::Stream;

let filter: Stream<usize> = Stream::new(100);
println!("{}", filter.params());

pub fn capacity(&self) -> usize

Get filter capacity.

Returns the value of params.nb_items, that is the number of items the filter is designed for.

use ofilter::{Stream, Params};

let filter: Stream<usize> = Stream::new_with_params(Params::with_bit_len(1_000_000));
assert_eq!(52681, filter.capacity());

pub fn clear(&mut self)

Clear the filter.

Clears the bit vector, but keeps parameters.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
filter.set(&10);
assert!(filter.check(&10));
filter.clear();
assert!(!filter.check(&10));

pub fn is_empty(&self) -> bool

Returns true if filter is empty.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
assert!(filter.is_empty());
filter.set(&10);
assert!(!filter.is_empty());

pub fn max_fp_rate(&self) -> f64

Returns the current max false positive rate.

In theory the false positive rate fp_rate is known at filter creation. But that, is the theoretical fp_rate that the filter reaches when it is “wasted” because it has too many entries. Until then, it performs better than that, statistically.

As this filter uses two filters under the hood, there are technically 2 values for the actual false positive rate.

This function returns the max value, which is usually the most significant, as it’s reflecting the current behavior of the filter.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
assert_eq!(0.0, filter.max_fp_rate());
filter.set(&10);
assert!(filter.max_fp_rate() > 0.0); // will be params.fp_rate when filter is full

pub fn fp_rates(&self) -> (f64, f64)

Returns the current false positive rates.

In theory the false positive rate fp_rate is known at filter creation. But that, is the theoretical fp_rate that the filter reaches when it is “wasted” because it has too many entries. Until then, it performs better than that, statistically.

As this filter uses two filters under the hood, there are technically 2 values for the actual false positive rate.

This function returns both values, starting with the highest one, which is usually the most significant as it’s reflecting the current behavior of the filter.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
assert_eq!((0.0, 0.0), filter.fp_rates());
filter.set(&10);
let fp_rates = filter.fp_rates();
assert!(fp_rates.0 > 0.0); // will be greater than params.fp_rate when filter is full
assert!(fp_rates.1 > 0.0); // will be params.fp_rate when filter is full

pub fn min_level(&self) -> f64

Returns the current ratios between real min fp_rate, and theoretical fp_rate..

This is a helper to quickly compare the real min fp_rate with the theoretical fp_rate. It compares the min value and not the max, as it is when the min value reaches 1.0 that the filters are swapped after one is cleared.

When this value greater than 1.0, the filter should soon “age” and clear the most filled underlying Bloom filter.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
assert_eq!((0.0, 0.0), filter.levels());
filter.set(&10);
let levels = filter.levels();
assert!(levels.0 > 0.0); // will be greater than 1.0 when filter is full
assert!(levels.1 > 0.0); // will be 1.0 when filter is full

pub fn levels(&self) -> (f64, f64)

Returns the current ratios between real fp_rates, and theoretical fp_rates.

This is a helper to quickly compare the real fp_rates with the theoretical fp_rates.

When both of these values are greater than 1.0, the filter should soon “age” and clear the most filled underlying Bloom filter.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
assert_eq!((0.0, 0.0), filter.levels());
filter.set(&10);
let levels = filter.levels();
assert!(levels.0 > 0.0); // will be greater than 1.0 when filter is full
assert!(levels.1 > 0.0); // will be 1.0 when filter is full

pub fn age(&self) -> usize

Returns the age of the filter.

The age is the number of times the underlying Bloom filters have been cleared and swapped.

In the beginning this is 0, then when filters are full, one of them is cleared, and age becomes one. Everytime this happens again, age increases.

As a consequence, the age gives a raw estimation of how many different items have been through the cache. With a given capacity, the number of different items the cache has seen is roughly age * capacity. This is just an approximation.

use ofilter::{Stream, Params};

let params = Params{
    nb_hash: 2,
    bit_len: 0,
    nb_items: 1_000,
    fp_rate: 0.1,
    predict: true,
};
let mut filter: Stream<usize> = Stream::new_with_params(params);
assert_eq!(0, filter.age());
for i in 0..100_000 {
    filter.set(&i);
}
println!("{}", filter.age());
assert!(filter.age() >= 90);
assert!(filter.age() <= 100);

pub fn resize(&mut self, capacity: usize)

Resize the filter.

As the underlying Bloom filters are recycled in a streaming filter, it is possible to resize it.

What it means is that next time the filter ages, the new created Bloom filter will have the new size. It will keep the same number of hash nb_hash and false positive rate fp_rate. Only the number of bits bit_len and number of items nb_items change.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
filter.set(&1);
filter.resize(100);
assert_eq!(100, filter.capacity());
assert!(filter.check(&1));

impl<T> Stream<T>

where T: Hash,

pub fn set(&mut self, item: &T)

Record an item in the set.

Once this has been called, any call to check() will return true, as there are no false negatives. However some other items may test positive as a consequence of recording this one.

However, after some time, the item will ultimately disappear from the set and be replaced by new entries. Only the most recent entries are guaranteed to be here.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
filter.set(&42);

pub fn check(&self, item: &T) -> bool

Guess whether an item is likely to be in the set.

If set() has been called before with value, then this returns true, as there are no false negatives. However it may respond true even if the item has never been recorded in the set.

However, if set() has been called too long ago and too many items have been added since, it will test negative anyway.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
filter.set(&42);
assert!(filter.check(&42));

pub fn check_and_set(&mut self, item: &T) -> bool

Record an item in the set and returns its previous value.

Equivalent to calling get() then set() but performs hash lookup only once so it’s a bit more efficient.

use ofilter::Stream;

let mut filter: Stream<usize> = Stream::new(1_000);
assert!(!filter.check_and_set(&42));
assert!(filter.check(&42));

Trait Implementations

impl<T: Clone> Clone for Stream<T>

fn clone(&self) -> Stream<T>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Debug> Debug for Stream<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de, T> Deserialize<'de> for Stream<T>

where T: Deserialize<'de>,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl<T> Display for Stream<T>

Pretty-print the filter.

Examples

use ofilter::Stream;

let filter: Stream<usize> = Stream::new(100);

assert_eq!("{ age: 0, fp_rates: (0.000000, 0.000000), params: { nb_hash: 2, bit_len: 1899, nb_items: 100, fp_rate: 0.009992, predict: false } }" , format!("{}", filter));

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T> Serialize for Stream<T>

where T: Serialize,

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.