Module pb 

Protobuf definitions for encodings

These are the messages used for describing encoding in the 2.0 format

Modules

array_encoding

Nested message and enum types in ArrayEncoding.

buffer

Nested message and enum types in Buffer.

column_encoding

Nested message and enum types in ColumnEncoding.

full_zip_layout

Nested message and enum types in FullZipLayout.

nullable

Nested message and enum types in Nullable.

page_layout

Nested message and enum types in PageLayout.

Structs

AllNullLayout

/ A layout used for pages where all values are null / / There may be buffers of repetition and definition information / if required in order to interpret what kind of nulls are present

ArrayEncoding

Encodings that decode into an Arrow array

Binary

An array encoding for binary fields

Bitpacked

Items are bitpacked in a buffer

BitpackedForNonNeg

Items are bitpacked in a buffer

Blob

Marks a column as blob data. It will contain a packed struct with fields position and size (u64)

Block

Buffer

A pointer to a buffer in a Lance file

ByteStreamSplit

Byte Stream Split encoding for floating point values

ColumnEncoding

Encodings that describe a column of values

Compression

Constant

Compression algorithm where all values have a constant value

Dictionary

An array encoding for dictionary-encoded fields

FixedSizeBinary

FixedSizeList

An array encoding for fixed-size list fields

Flat

Fixed width items placed contiguously in a buffer

Fsst

FullZipLayout

/ A layout used for pages where the data is large / / In this case the cost of transposing the data is relatively small (compared to the cost of writing the data) / and so we just zip the buffers together

GeneralMiniBlock

General miniblock encoding - wraps another miniblock encoding with compression

InlineBitpacking

Opaque bitpacking variant where the bits per value are stored inline in the chunks themselves

List

An array encoding for variable-length list fields

MiniBlockLayout

/ A layout used for pages where the data is small / / In this case we can fit many values into a single disk sector and transposing buffers is / expensive. As a result, we do not transpose the buffers but compress the data into small / chunks (called mini blocks) which are roughly the size of a disk sector.

Nullable

An encoding that adds nullability to another array encoding

OutOfLineBitpacking

Transparent bitpacking variant where the number of bits per value is fixed through the whole buffer

PackedStruct

PackedStructFixedWidthMiniBlock

PageLayout

Rle

Run-Length Encoding for miniblock format

SimpleStruct

An array encoding for shredded structs that will never be null

Variable

ZoneIndex

Wraps a column with a zone map index that can be used to apply pushdown filters

Enums

RepDefLayer

/ Describes the meaning of each repdef layer in a mini-block layout