Struct MemWalIndexDetails 

pub struct MemWalIndexDetails {
    pub snapshot_ts_millis: i64,
    pub num_shards: u32,
    pub inline_snapshots: Option<Vec<u8>>,
    pub shard_specs: Vec<ShardSpec>,
    pub maintained_indexes: Vec<String>,
    pub merged_generations: Vec<MergedGeneration>,
    pub index_catchup: Vec<IndexCatchupProgress>,
}

Index details for MemWAL Index, stored in IndexMetadata.index_details. This is the centralized structure for all MemWAL metadata:

• Configuration (shard specs, indexes to maintain)
• Merge progress (merged generations per shard)
• Shard state snapshots

Writers read this index to get configuration before writing. Readers read this index to discover shards and their state. A background process updates the index periodically to keep shard snapshots current.

Shard snapshots are stored as a Lance file with one row per shard. The schema has one column per ShardManifest field, with shard fields as columns: shard_id: fixed_size_binary(16) – UUID bytes version: uint64 shard_spec_id: uint32 writer_epoch: uint64 replay_after_wal_entry_position: uint64 wal_entry_position_last_seen: uint64 current_generation: uint64 merged_generation: uint64 flushed_generations: list<struct<generation: uint64, path: string>>

Fields

snapshot_ts_millis: i64

Snapshot timestamp (Unix timestamp in milliseconds).

num_shards: u32

Number of shards in the snapshot. Used to determine storage format without reading the snapshot data.

inline_snapshots: Option<Vec<u8>>

Inline shard snapshots for small shard counts. When num_shards <= threshold (implementation-defined, e.g., 100), snapshots are stored inline as serialized bytes. Format: Lance file bytes with the shard snapshot schema.

shard_specs: Vec<ShardSpec>

Shard specs defining how to derive shard identifiers. This configuration determines how rows are partitioned into shards.

maintained_indexes: Vec<String>

Indexes from the base table to maintain in MemTables. These are index names referencing indexes defined on the base table. The primary key btree index is always maintained implicitly and should not be listed here.

For vector indexes, MemTables inherit quantization parameters (PQ codebook, SQ params) from the base table index to ensure distance comparability.

merged_generations: Vec<MergedGeneration>

Last generation merged to base table for each shard. This is updated atomically with merge-insert data commits, enabling conflict resolution when multiple mergers operate concurrently.

Note: This is separate from shard snapshots because:

1. merged_generations is updated by mergers (atomic with data commit)
2. shard snapshots are updated by background index builder

index_catchup: Vec<IndexCatchupProgress>

Per-index catchup progress tracking. When data is merged to the base table, base table indexes are rebuilt asynchronously. This field tracks which generation each index covers.

For indexed queries, if an index’s caught_up_generation < merged_generation, readers should use flushed MemTable indexes for the gap instead of scanning unindexed data in the base table.

If an index is not present in this list, it is assumed to be fully caught up.

Implementations

impl MemWalIndexDetails

pub fn inline_snapshots(&self) -> &[u8]

Returns the value of inline_snapshots, or the default value if inline_snapshots is unset.

Trait Implementations

impl Clone for MemWalIndexDetails

fn clone(&self) -> MemWalIndexDetails

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MemWalIndexDetails

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for MemWalIndexDetails

fn default() -> Self

Returns the “default value” for a type.

impl Message for MemWalIndexDetails

fn encoded_len(&self) -> usize

Returns the encoded length of the message without a length delimiter.

fn clear(&mut self)

Clears the message, resetting all fields to their default.

fn encode(&self, buf: &mut impl BufMut) -> Result<(), EncodeError>

where Self: Sized,

Encodes the message to a buffer.

fn encode_to_vec(&self) -> Vec<u8>

where Self: Sized,

Encodes the message to a newly allocated buffer.

fn encode_length_delimited( &self, buf: &mut impl BufMut, ) -> Result<(), EncodeError>

where Self: Sized,

Encodes the message with a length-delimiter to a buffer.

fn encode_length_delimited_to_vec(&self) -> Vec<u8>

where Self: Sized,

Encodes the message with a length-delimiter to a newly allocated buffer.

fn decode(buf: impl Buf) -> Result<Self, DecodeError>

where Self: Default,

Decodes an instance of the message from a buffer.

fn decode_length_delimited(buf: impl Buf) -> Result<Self, DecodeError>

where Self: Default,

Decodes a length-delimited instance of the message from the buffer.

fn merge(&mut self, buf: impl Buf) -> Result<(), DecodeError>

where Self: Sized,

Decodes an instance of the message from a buffer, and merges it into self.

fn merge_length_delimited(&mut self, buf: impl Buf) -> Result<(), DecodeError>

where Self: Sized,

Decodes a length-delimited instance of the message from buffer, and merges it into self.

impl Name for MemWalIndexDetails

const NAME: &'static str = "MemWalIndexDetails"

Simple name for this Message. This name is the same as it appears in the source .proto file, e.g. FooBar.

const PACKAGE: &'static str = "lance.table"

Package name this message type is contained in. They are domain-like and delimited by ., e.g. google.protobuf.

fn full_name() -> String

Fully-qualified unique name for this Message. It’s prefixed with the package name and names of any parent messages, e.g. google.rpc.BadRequest.FieldViolation. By default, this is the package name followed by the message name. Fully-qualified names must be unique within a domain of Type URLs.

fn type_url() -> String

Type URL for this Message, which by default is the full name with a leading slash, but may also include a leading domain name, e.g. type.googleapis.com/google.profile.Person. This can be used when serializing into the google.protobuf.Any type.

impl PartialEq for MemWalIndexDetails

fn eq(&self, other: &MemWalIndexDetails) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for MemWalIndexDetails