Module pb21 

Protobuf definitions for encodings21

These are the messages used for describing encoding in the 2.1 format and any newer formats.

Modules

compressive_encoding

Nested message and enum types in CompressiveEncoding.

full_zip_layout

Nested message and enum types in FullZipLayout.

page_layout

Nested message and enum types in PageLayout.

Structs

AllNullLayout

A layout used for pages where all values are null

BufferCompression

Compression applied to a single buffer of data

ByteStreamSplit

A compression scheme where fixed-width values are transposed into a series of byte streams

CompressiveEncoding

An encoding that compresses a data block into buffers

Constant

Compression algorithm where all values have a constant value (encoded in the description)

Dictionary

A compression scheme where common values are stored in a dictionary and the values are encoded as indices into the dictionary.

FixedSizeList

Converts a fixed-size-list of values into a flattened list of values

Flat

Fixed width items placed contiguously in a single buffer

Fsst

A compression scheme for variable-width data

FullZipLayout

A layout used for pages where the data is large

General

A compression scheme that wraps the underlying data with general compression

InlineBitpacking

Bitpacking variant where the bits per value are stored inline in the chunks themselves

MiniBlockLayout

A layout used for pages where the data is small

OutOfLineBitpacking

A compression scheme in which a single fixed-width block is “packed” into a smaller fixed-width block values where each value has fewer bits.

PackedStruct

Converts a struct data block into either a fixed-width data block or a variable-width data block

PageLayout

Describes the structural encoding of a page

Rle

A compression scheme where runs of common values are encoded as a single value and a count

Variable

Variable width items have the values stored in one buffer and the offsets are output as a data block that may be further compressed.

Enums

CompressionScheme

RepDefLayer

Repetition and definition levels are described in more detail elsewhere. As we peel through the structure of an array we will encounter layers of struct and list. Each of these layers potentially adds a new level to the repetition and definition levels. This message describes the meaning of each layer.