Struct lance_index::vector::pq::storage::ProductQuantizationStorage

pub struct ProductQuantizationStorage { /* private fields */ }

Product Quantization Storage

It stores PQ code, as well as the row ID to the orignal vectors.

It is possible to store additonal metadata to accelerate filtering later.

TODO: support f16/f64 later.

Implementations

impl ProductQuantizationStorage

pub fn new( codebook: Arc<Float32Array>, batch: RecordBatch, num_bits: u32, num_sub_vectors: usize, dimension: usize, metric_type: MetricType ) -> Result<Self>

pub fn batch(&self) -> &RecordBatch

pub async fn build( quantizer: Arc<ProductQuantizerImpl<Float32Type>>, batch: &RecordBatch, vector_col: &str ) -> Result<Self>

Build a PQ storage from ProductQuantizer and a RecordBatch.

Parameters

quantizer: ProductQuantizer The quantizer used to transform the vectors. batch: RecordBatch The batch of vectors to be transformed. vector_col: &str The name of the column containing the vectors.

pub async fn load(object_store: &ObjectStore, path: &Path) -> Result<Self>

Load full PQ storage from disk.

Parameters

object_store: &ObjectStore The object store to load the storage from. path: &Path The path to the storage.

Returns

Self

Currently it loads everything in memory. TODO: support lazy loading later.

pub fn schema(&self) -> SchemaRef

pub fn get_row_ids(&self, ids: &[u32]) -> Vec<u64>

pub async fn write_partition( &self, writer: &mut FileWriter<ManifestDescribing> ) -> Result<usize>

Write the PQ storage as a Lance partition to disk, and returns the number of rows written.

pub async fn write_full( &self, writer: &mut FileWriter<ManifestDescribing> ) -> Result<()>

Write the PQ storage to disk.

Trait Implementations

impl Clone for ProductQuantizationStorage

fn clone(&self) -> ProductQuantizationStorage

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ProductQuantizationStorage

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for ProductQuantizationStorage

fn eq(&self, other: &Self) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl QuantizerStorage for ProductQuantizationStorage

fn load_partition<'life0, 'life1, 'async_trait>( reader: &'life0 FileReader, range: Range<usize>, metric_type: MetricType, metadata: &'life1 Self::Metadata ) -> Pin<Box<dyn Future<Output = Result<Self>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Load a partition of PQ storage from disk.

Parameters

• *reader: &FileReader

type Metadata = ProductQuantizationMetadata

impl VectorStorage for ProductQuantizationStorage

fn as_any(&self) -> &dyn Any

fn len(&self) -> usize

fn row_ids(&self) -> &[u64]

fn metric_type(&self) -> MetricType

Return the metric type of the vectors.

fn dist_calculator(&self, query: &[f32]) -> Box<dyn DistCalculator>

Create a [DistCalculator] to compute the distance between the query.

fn is_empty(&self) -> bool

Returns true if this graph is empty.