Struct MemWalIndexDetails 

pub struct MemWalIndexDetails {
    pub snapshot_ts_millis: i64,
    pub num_shards: u32,
    pub inline_snapshots: Option<Vec<u8>>,
    pub sharding_specs: Vec<ShardingSpec>,
    pub maintained_indexes: Vec<String>,
    pub merged_generations: Vec<MergedGeneration>,
    pub index_catchup: Vec<IndexCatchupProgress>,
    pub writer_config_defaults: HashMap<String, String>,
}

Index details for MemWAL Index, stored in IndexMetadata.index_details. This is the centralized structure for all MemWAL metadata:

• Configuration (sharding specs, indexes to maintain)
• Merge progress (merged generations per shard)
• Shard state snapshots

Writers read this index to get configuration before writing. Readers may use shard snapshots in this index as a point-in-time optimization. Readers that need the latest shard set should list shard directories in storage and read each shard’s latest manifest. A background process updates the index periodically to keep shard snapshots current.

Shard snapshots are stored as a Lance file with one row per shard. The schema records shard discovery fields. Full mutable shard state remains authoritative in the shard manifest files. shard_id: utf8 shard_spec_id: uint32 shard_field_{field_id}: typed per the matching ShardingField.result_type

Fields

snapshot_ts_millis: i64

Snapshot timestamp (Unix timestamp in milliseconds).

num_shards: u32

Number of shards in the snapshot. Used to determine storage format without reading the snapshot data.

inline_snapshots: Option<Vec<u8>>

Inline shard snapshots for small shard counts. When num_shards <= threshold (implementation-defined, e.g., 100), snapshots are stored inline as serialized bytes. Format: Lance file bytes with the shard snapshot schema.

sharding_specs: Vec<ShardingSpec>

Sharding specs defining how to derive shard identifiers. This configuration determines how rows are partitioned into shards.

maintained_indexes: Vec<String>

Indexes from the base table to maintain in MemTables. These are index names referencing indexes defined on the base table. The primary key btree index is always maintained implicitly and should not be listed here.

For vector indexes, MemTables inherit quantization parameters (PQ codebook, SQ params) from the base table index to ensure distance comparability.

merged_generations: Vec<MergedGeneration>

Last generation merged to base table for each shard. This is updated atomically with merge-insert data commits, enabling conflict resolution when multiple mergers operate concurrently.

Note: This is separate from shard snapshots because:

1. merged_generations is updated by mergers (atomic with data commit)
2. shard snapshots are updated by background index builder

index_catchup: Vec<IndexCatchupProgress>

Per-index catchup progress tracking. When data is merged to the base table, base table indexes are rebuilt asynchronously. This field tracks which generation each index covers.

For indexed queries, if an index’s caught_up_generation < merged_generation, readers should use flushed MemTable indexes for the gap instead of scanning unindexed data in the base table.

If an index is not present in this list, it is assumed to be fully caught up.

writer_config_defaults: HashMap<String, String>

Default ShardWriter configuration values for this MemWAL index.

A free-form string map persisted so that every writer — across processes and restarts — starts from the same default writer configuration. These are defaults only: an individual writer may still override any value at runtime in its own ShardWriterConfig (which is not persisted).

Implementations

impl MemWalIndexDetails

pub fn inline_snapshots(&self) -> &[u8]

Returns the value of inline_snapshots, or the default value if inline_snapshots is unset.

Trait Implementations

impl Clone for MemWalIndexDetails

fn clone(&self) -> MemWalIndexDetails

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MemWalIndexDetails

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for MemWalIndexDetails

fn default() -> Self

Returns the “default value” for a type.

impl Message for MemWalIndexDetails

fn encoded_len(&self) -> usize

Returns the encoded length of the message without a length delimiter.

fn clear(&mut self)

Clears the message, resetting all fields to their default.

fn encode(&self, buf: &mut impl BufMut) -> Result<(), EncodeError>

where Self: Sized,

Encodes the message to a buffer.

fn encode_to_vec(&self) -> Vec<u8>

where Self: Sized,

Encodes the message to a newly allocated buffer.

fn encode_length_delimited( &self, buf: &mut impl BufMut, ) -> Result<(), EncodeError>

where Self: Sized,

Encodes the message with a length-delimiter to a buffer.

fn encode_length_delimited_to_vec(&self) -> Vec<u8>

where Self: Sized,

Encodes the message with a length-delimiter to a newly allocated buffer.

fn decode(buf: impl Buf) -> Result<Self, DecodeError>

where Self: Default,

Decodes an instance of the message from a buffer.

fn decode_length_delimited(buf: impl Buf) -> Result<Self, DecodeError>

where Self: Default,

Decodes a length-delimited instance of the message from the buffer.

fn merge(&mut self, buf: impl Buf) -> Result<(), DecodeError>

where Self: Sized,

Decodes an instance of the message from a buffer, and merges it into self.

fn merge_length_delimited(&mut self, buf: impl Buf) -> Result<(), DecodeError>

where Self: Sized,

Decodes a length-delimited instance of the message from buffer, and merges it into self.

impl Name for MemWalIndexDetails

const NAME: &'static str = "MemWalIndexDetails"

Simple name for this Message. This name is the same as it appears in the source .proto file, e.g. FooBar.

const PACKAGE: &'static str = "lance.table"

Package name this message type is contained in. They are domain-like and delimited by ., e.g. google.protobuf.

fn full_name() -> String

Fully-qualified unique name for this Message. It’s prefixed with the package name and names of any parent messages, e.g. google.rpc.BadRequest.FieldViolation. By default, this is the package name followed by the message name. Fully-qualified names must be unique within a domain of Type URLs.

fn type_url() -> String

Type URL for this Message, which by default is the full name with a leading slash, but may also include a leading domain name, e.g. type.googleapis.com/google.profile.Person. This can be used when serializing into the google.protobuf.Any type.

impl PartialEq for MemWalIndexDetails

fn eq(&self, other: &MemWalIndexDetails) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for MemWalIndexDetails