Struct DiskIndexWriter 

pub struct DiskIndexWriter { /* private fields */ }

DiskIndexWriter is used to write disk index data to the storage system. The storage system and data types are provided as parameters to methods that need them.

Implementations

impl DiskIndexWriter

pub fn new( dataset_file: String, index_path_prefix: String, associated_data_file: Option<String>, block_size: usize, ) -> ANNResult<Self>

Create DiskIndexWriter instance

pub fn dataset_file(&self) -> &String

pub fn index_path_prefix(&self) -> &String

pub fn block_size(&self) -> usize

pub fn create_disk_layout<Data, StorageProvider>( &self, storage_provider: &StorageProvider, ) -> ANNResult<()>

where Data: GraphDataType<VectorIdType = u32>, StorageProvider: StorageReadProvider + StorageWriteProvider,

Create disk layout. Block #1: GraphMetadata. Block #2..#n: num_nodes_per_block nodes in each block

GraphMetadata layout: |number_of_points (8 bytes)| dimensions (8 bytes) | medoid (8 bytes) | …| node_len (8 bytes) | num_nodes_per_sector (8 bytes) | vamana_frozen_point_num (8 bytes) | …| vamana_frozen_loc (8 bytes) | append_reorder_data (8 bytes) | disk_index_file_size (8 bytes) | …| associated_data_length (8 bytes) | block_size (8 bytes) | layout_version (8 bytes) |

The metadata layout is kept compatible with the C++ diskann codebase.

After the metadata structure, the graph is laid out as a sequence of vertices and relevant data including the vector, #out neighbors, list of out neighbors, associated_data and appropriate padding. | vector (dim * size_of<VectorDataType> bytes) | neighbor_count (4 bytes) | neighbors (neighbor_count * 4 bytes) | ... | filler | associated_data (associated_data_length * size_of<AssociatedDataType> bytes) |

The node_len and length of filler are calculated as follow：

node_length = (max_degree + 1) * size_of<u32> + dim * size_of<VectorDataType> + associated_data_length * size_of<AssociatedDataType>

filler length = node_len - (length of vector + 4 + 4 * neighbor_count + associated_data_length * size_of<AssociatedDataType>)

The filler is used to pad each node to node_len bytes on the disk, so it is possible to calculate the block number and offset of a node with its Id.

When node_len < disk block size, we pack as many nodes as possible in to the disk sector without splitting a node across a sector For example, if node_len is 600B, we can pack 6 of these on a 4KB sector, and we leave 4096-3600 = 496B unused.

When node_len > disk block size, we align start of node to block size. For example, if node_len is 6700B, then it would span two 4KB sectors beginning at the start of the first sector and end on the second sector and will be followed by 1492 bytes of padding to align the next node to block boundary.

Arguments

• storage_provider - the storage provider for I/O operations

pub fn index_build_cleanup<StorageProvider>( &self, storage_provider: &StorageProvider, ) -> ANNResult<()>

where StorageProvider: StorageReadProvider + StorageWriteProvider,

pub fn disk_index_file(&self) -> String

pub fn get_pq_pivot_file(&self) -> String

pub fn get_compressed_pq_pivot_file(&self) -> String

pub fn get_disk_index_pq_pivot_file(&self) -> String

pub fn get_disk_index_compressed_pq_file(&self) -> String

pub fn get_index_path_prefix(&self) -> String

pub fn get_dataset_file(&self) -> String

pub fn get_associated_data_file(&self) -> Option<String>

pub fn get_mem_index_file(&self) -> String

pub fn get_merged_index_prefix(&self) -> String

pub fn get_merged_index_subshard_id_map_file( prefix: &str, shard: usize, ) -> String

pub fn get_merged_index_subshard_data_file(prefix: &str, shard: usize) -> String

pub fn get_merged_index_subshard_prefix(prefix: &str, shard: usize) -> String

pub fn get_merged_index_subshard_mem_index_file( prefix: &str, shard: usize, ) -> String

pub fn get_merged_index_subshard_mem_dataset_file( subshard_mem_index_prefix: &str, ) -> String