Trait VectorStore

pub trait VectorStore:
    Send
    + Sync
    + Sized
    + Clone {
    type DistanceCalculator<'a>: DistCalculator
       where Self: 'a;

    // Required methods
    fn try_from_batch(
        batch: RecordBatch,
        distance_type: DistanceType,
    ) -> Result<Self>;
    fn as_any(&self) -> &dyn Any;
    fn schema(&self) -> &SchemaRef;
    fn to_batches(&self) -> Result<impl Iterator<Item = RecordBatch>>;
    fn len(&self) -> usize;
    fn distance_type(&self) -> DistanceType;
    fn row_id(&self, id: u32) -> u64;
    fn row_ids(&self) -> impl Iterator<Item = &u64>;
    fn append_batch(
        &self,
        batch: RecordBatch,
        vector_column: &str,
    ) -> Result<Self>;
    fn dist_calculator(&self, query: ArrayRef) -> Self::DistanceCalculator<'_>;
    fn dist_calculator_from_id(&self, id: u32) -> Self::DistanceCalculator<'_>;
    fn distance_between(&self, a: u32, b: u32) -> f32;

    // Provided methods
    fn is_empty(&self) -> bool { ... }
    fn dist_calculator_from_native(
        &self,
        _query: ArrayRef,
    ) -> Self::DistanceCalculator<'_> { ... }
}

Vector Storage is the abstraction to store the vectors.

It can be in-memory or on-disk, raw vector or quantized vectors.

It abstracts away the logic to compute the distance between vectors.

TODO: should we rename this to “VectorDistance”?;

Internal API

API stability is not guaranteed

Required Associated Types

type DistanceCalculator<'a>: DistCalculator where Self: 'a

Required Methods

fn try_from_batch( batch: RecordBatch, distance_type: DistanceType, ) -> Result<Self>

fn as_any(&self) -> &dyn Any

fn schema(&self) -> &SchemaRef

fn to_batches(&self) -> Result<impl Iterator<Item = RecordBatch>>

fn len(&self) -> usize

fn distance_type(&self) -> DistanceType

Return DistanceType.

fn row_id(&self, id: u32) -> u64

Get the lance ROW ID from one vector.

fn row_ids(&self) -> impl Iterator<Item = &u64>

fn append_batch(&self, batch: RecordBatch, vector_column: &str) -> Result<Self>

Append Raw RecordBatch into the Storage. The storage implement will perform quantization if necessary.

fn dist_calculator(&self, query: ArrayRef) -> Self::DistanceCalculator<'_>

Create a DistCalculator to compute the distance between the query.

Using dist calcualtor can be more efficient as it can pre-compute some values.

fn dist_calculator_from_id(&self, id: u32) -> Self::DistanceCalculator<'_>

fn distance_between(&self, a: u32, b: u32) -> f32

Provided Methods

fn is_empty(&self) -> bool

Returns true if this graph is empty.

fn dist_calculator_from_native( &self, _query: ArrayRef, ) -> Self::DistanceCalculator<'_>

Object Safety

This trait is not object safe.

Implementors

impl VectorStore for FlatStorage

type DistanceCalculator<'a> = FlatDistanceCal<'a>

impl VectorStore for ProductQuantizationStorage

type DistanceCalculator<'a> = PQDistCalculator

impl VectorStore for ScalarQuantizationStorage

type DistanceCalculator<'a> = SQDistCalculator<'a>