Crate ngt

source ·
Expand description

ngt-rs

crate doc

Rust wrappers for NGT, which provides high-speed approximate nearest neighbor searches against a large volume of data in high dimensional vector data space (several ten to several thousand dimensions). The vector data can be f32, u8, or f16.

This crate provides the following indexes:

Both quantized indexes are available through the quantized Cargo feature. Note that they rely on BLAS and LAPACK which thus have to be installed locally. Furthermore, QgIndex performances can be improved by using the qg_optim Cargo feature.

The NgtIndex default implementation is an ANNG. It can be optimized3 or converted to an ONNG through the optim module.

By default ngt-rs will be built dynamically, which requires CMake to build NGT. This means that you’ll have to make the build artifact libngt.so available to your final binary (see an example in the CI). However the static feature will build and link NGT statically. Note that OpenMP will also be linked statically. If the quantized feature is used, then BLAS and LAPACK libraries will also be linked statically.

NGT’s shared memory and large dataset features are available through the Cargo features shared_mem and large_data respectively.

Usage

Graph and tree based index (NGT Index)

Defining the properties of a new NGT index:

use ngt::{NgtProperties, NgtDistance};

// Defaut properties with vectors of dimension 3
let prop = NgtProperties::<f32>::dimension(3)?;

// Or customize values (here are the defaults)
let prop = NgtProperties::<f32>::dimension(3)?
    .creation_edge_size(10)?
    .search_edge_size(40)?
    .distance_type(NgtDistance::L2)?;

Creating/Opening a NGT index and using it:

use ngt::{NgtIndex, NgtProperties, NgtQuery};

// Create a new index
let prop = NgtProperties::dimension(3)?;
let index: NgtIndex<f32> = NgtIndex::create("target/path/to/ngt_index/dir", prop)?;

// Open an existing index
let mut index = NgtIndex::open("target/path/to/ngt_index/dir")?;

// Insert two vectors and get their id
let vec1 = vec![1.0, 2.0, 3.0];
let vec2 = vec![4.0, 5.0, 6.0];
let id1 = index.insert(vec1)?;
let id2 = index.insert(vec2)?;

// Build the index in RAM (not yet persisted on disk)
// This is required in order to be able to search vectors
index.build(2)?;

// Perform a vector search (with 1 result)
let res = index.search(&vec![1.1, 2.1, 3.1], 1, ngt::EPSILON)?;
assert_eq!(res[0].id, id1);
assert_eq!(index.get_vec(id1)?, vec![1.0, 2.0, 3.0]);

// Remove a vector and check that it is not present anymore
index.remove(id1)?;
let res = index.get_vec(id1);
assert!(res.is_err());

// Verify that now our search result is different (using the NgtQuery API)
let query = vec![1.1, 2.1, 3.1];
let res = index.search_query(NgtQuery::new(&query))?;
assert_eq!(res[0].id, id2);
assert_eq!(index.get_vec(id2)?, vec![4.0, 5.0, 6.0]);

// Persist index on disk
index.persist()?;

  1. Graph and tree based method explanation ↩

  2. Quantized graph based method explanation ↩

  3. NGT index optimizations in Python ↩

Re-exports

Modules

  • optim
    Functions aimed at optimizing NgtIndex
  • qbgquantized
    Quantized blob graph index (QBG Index)
  • qgquantized
    Quantized graph index (QG Index)

Structs

Enums

Constants

Type Aliases