Expand description
Protobuf definitions for Lance Format
Modules§
- data_
fragment - Nested message and enum types in
DataFragment. - data_
overlay_ file - Nested message and enum types in
DataOverlayFile. - deletion_
file - Nested message and enum types in
DeletionFile. - encoded_
u64_ array - Nested message and enum types in
EncodedU64Array. - fragment_
reuse_ index_ details - Nested message and enum types in
FragmentReuseIndexDetails. - manifest
- Nested message and enum types in
Manifest. - transaction
- Nested message and enum types in
Transaction. - u64_
segment - Nested message and enum types in
U64Segment.
Structs§
- Base
Path - external dataset base path
- Compacted
SsTable - A pointer to the latest SSTable compacted for a shard.
- Data
File - Data
Fragment - A DataFragment is a set of files which represent the different columns of the same rows. If column exists in the schema of a dataset, but the file for that column does not exist within a DataFragment of that dataset, that column consists entirely of nulls.
- Data
Overlay File - An overlay file supplies new values for a subset of (row offset, field) cells within a fragment, without rewriting the fragment’s base data files. It is used for efficient updates when only a small fraction of rows and/or columns change.
- Deletion
File - Deletion File
- Encoded
U64Array - / A basic bitpacked array of u64 values.
- External
File - Field
Coverage - Per-field coverage for a sparse overlay.
- Fragment
Reuse Index Details - Index
Catchup Progress - Tracks which compacted SSTable generation a base table index has been rebuilt to cover. Used to determine whether to read from SSTable indexes or base table.
- Index
File - Metadata about a single file within an index segment.
- Index
Metadata - Metadata describing an index.
- Index
Section - Index Section, containing a list of index metadata for one dataset version.
- Manifest
- Manifest is a global section shared between all the files.
- MemWal
Index Details - Index details for MemWAL Index, stored in IndexMetadata.index_details. This is the centralized structure for all MemWAL metadata:
- RowDataset
Version Run - / A run of rows with the same version.
- RowDataset
Version Sequence - A sequence of dataset versions. Similar to RowIdSequence but tracks version runs. It uses RLE (Run-Length Encoding) to efficiently represent consecutive rows with the same version.
- RowId
Sequence - A sequence of row IDs. This is split up into one or more segments, each of which can be encoded in different ways. The encodings are optimized for values that are sorted, which will often be the case with row ids. They also have optimized forms depending on how sparse the values are.
- Shard
Field Entry - A shard field value stored as raw Arrow scalar bytes.
- Shard
Manifest - Shard manifest containing epoch-based fencing and WAL state. Each shard has exactly one active writer at any time.
- Sharding
Field - Sharding field definition.
- Sharding
Spec - Sharding spec definition.
- SsTable
- An SSTable: the immutable result of flushing a MemTable, stored as a Lance dataset.
- Transaction
- A transaction represents the changes to a dataset.
- U64Segment
- / Different ways to encode a sequence of u64 values.
- Uuid
- UUID type. encoded as 16 bytes.
- Version
AuxData - Auxiliary Data attached to a version. Only load on-demand.
Enums§
- Shard
Status - Lifecycle status of a WAL shard. Drives drop-table two-phase commit: a SEALED shard refuses new writer claims (reversible) until the drop commits (the shard dir is deleted) or rolls back (status -> ACTIVE).