Expand description
Metadata for the unstable 2.3 planar page format.
Modules§
- compressive_
encoding - Nested message and enum types in
CompressiveEncoding. - sparse_
structural_ layer - Nested message and enum types in
SparseStructuralLayer. - sparse_
validity_ set - Nested message and enum types in
SparseValiditySet. - structural_
plane - Nested message and enum types in
StructuralPlane. - value_
plane - Nested message and enum types in
ValuePlane. - variable_
packed_ struct - Nested message and enum types in
VariablePackedStruct. - zipped_
values - Nested message and enum types in
ZippedValues.
Structs§
- Buffer
Compression - Compression applied to a single buffer of data
- Byte
Stream Split - A compression scheme where fixed-width values are transposed into a series of byte streams
- Chunked
Values - Compressive
Encoding - An encoding that compresses a data block into buffers
- Constant
- Compression algorithm where all values have a constant value (encoded in the description)
- Constant
Value - Delta
- An unstable Lance 2.3 non-decreasing unsigned fixed-width sequence represented by its first value and the differences between adjacent values.
- Dense
Chunk Levels - Dense
Structure - Dense
ZipLevels - Dictionary
- A compression scheme where common values are stored in a dictionary and the values are encoded as indices into the dictionary.
- Fixed
Size List - Converts a fixed-size-list of values into a flattened list of values
- Flat
- Fixed width items placed contiguously in a single buffer
- Fsst
- A compression scheme for variable-width data
- General
- A compression scheme that wraps the underlying data with general compression
- Indexed
Values - Sparse value slots reference independently addressable dictionary items. Buffers are: one encoded UInt32 index buffer (empty for a metadata-only codec), then the zipped item buffers. Index cardinality is the sparse leaf-slot domain; each index is less than items.num_items. Item values have no dense levels or nested dictionary.
- Inline
Bitpacking - Bitpacking variant where the bits per value are stored inline in the chunks themselves
- NoValues
- OutOf
Line Bitpacking - A compression scheme in which a single fixed-width block is “packed” into a smaller fixed-width block values where each value has fewer bits.
- Packed
Struct - Packs a struct containing only fixed-width children into a single fixed-width data block
- Planar
Layout - Unstable 2.3 page metadata. Structure maps rows to leaf slots; values map those slots to bytes. Neither plane may be absent. There is no compatibility contract with previous unstable revisions. All referenced messages and enums belong to this package; stable format schemas are not dependencies.
- Range
- A metadata-only arithmetic sequence for unstable Lance 2.3 only.
- Rle
- A compression scheme where runs of common values are encoded as a single value and a count
- Sparse
Fixed Size List Layer - Sparse
List Layer - Sparse
Structural Layer - A domain is a layer-local integer coordinate space [0, size). A slot is one element in that space. The outer-most domain is the page’s top-level rows; each layer’s child domain is the next layer’s parent domain, and the terminal child domain contains the leaf value slots. Parent sizes are derived from the page row count and preceding layers, not stored in each layer. The outer size is the page row count divided exactly by the product of all fixed-size-list dimensions, including those below lists.
- Sparse
Structure - Sparse
Validity Set - Structural
Plane - Value
Plane - Variable
- Variable width items have the values stored in one buffer and the offsets are output as a data block that may be further compressed.
- Variable
Packed Struct - Variable-width packed struct encoding
- Zipped
Values