bbqvec 0.0.10

Scalable, embeddable, vector storage for approximate K-nearest-neighbors (AKNN)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

![BBQvec Logo](.github/bbqvec.png)

![Status](https://img.shields.io/badge/status-beta-blue)
[![license](https://img.shields.io/github/license/daxe-ai/bbqvec)](LICENSE)
[![GoDoc](https://godoc.org/github.com/daxe-ai/bbqvec?status.svg)](https://godoc.org/github.com/daxe-ai/bbqvec)
[![Crates.io](https://img.shields.io/crates/v/bbqvec)](https://crates.io/crates/bbqvec)
[![Go CI](https://github.com/daxe-ai/bbqvec/actions/workflows/go.yml/badge.svg)](https://github.com/daxe-ai/bbqvec/actions/workflows/go.yml)
[![Rust CI](https://github.com/daxe-ai/bbqvec/actions/workflows/rust.yml/badge.svg)](https://github.com/daxe-ai/bbqvec/actions/workflows/rust.yml)

BBQvec is an open-source, embedded vector database index for Go and Rust, providing approximate K-nearest-neighbors (aKNN).

[Read more about the algorithm on our blog!](https://open.substack.com/pub/daxeblog/p/bbqvec-a-scalable-vector-search-library?r=4gk2js&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true)

# Getting Started

## Go

```go
package main

import (
  "fmt"

  bbq "github.com/daxe-ai/bbqvec"
)

func main() {
  // Declare store parameters
  dimensions := 200
  nBasis := 10

  // Initialize the store
  backend := bbq.NewMemoryBackend(dimensions)
  datastore, _ := bbq.NewVectorStore(backend, nBasis)

  // Create some test data, 100K random vectors
  vecs := bbq.NewRandVectorSet(100_000, dimensions, nil)
  datastore.AddVectorsWithOffset(0, vecs)
  /*
  Equivalent to:
  for i, v := range vecs {
  datastore.AddVector(bbq.ID(i), v)
  }
  */

  // Run a query
  targetVec := bbq.NewRandVector(dimensions, nil)
  results, _ := datastore.FindNearest(targetVec, 10, 1000, 1)

  // Inspect the results
  top := results.ToSlice()[0]
  vec, _ := backend.GetVector(top.ID)
  fmt.Println(top.ID, vec, top.Similarity)
}
```

## Rust

```rust
use bbqvec::IndexIDIterator;

fn main() -> Result<()> {
  // Declare store parameters
  let dimensions = 200;
  let n_basis = 10;

  // Initialize the store
  let mem = bbqvec::MemoryBackend::new(dimensions, n_basis)?;
  let mut store = bbqvec::VectorStore::new(mem)?;

  // Create some test data, 100K random vectors
  let vecs = bbqvec::create_vector_set(dimensions, 100000);
  store.add_vector_iter(vecs.enumerate_ids())?;

  // Run a query
  let target = bbqvec::create_random_vector(dimensions);
  let results = store.find_nearest(&target, 10, 1000, 1)?;

  // Inspect the results
  for res in results.iter_results() {
    println!("{} {}", res.id, res.similarity)
  }
}

```

# TODOs

We're still early; Go is the more tried-and-true and suits the beta use-case, but Rust is a good deal faster. We welcome contributions.

## Go
- [ ] More benchmarks
- [ ] New Quantizations
  - [ ] Hamming Distance (single-bit vectors)
  - [ ] Novel quantizations
## Rust
- [ ] Finish disk backend to match Go (in progress, shortly)
- [ ] New Quantizations


### Acknowledgements
Thank you to MariaLetta for the [free-gophers-pack](https://github.com/MariaLetta/free-gophers-pack) and to [rustacean.net](https://rustacean.net) for the CC0 logo characters.