Expand description
The top-level encoding module for Lance files.
Lance files are encoded using a FieldEncodingStrategy which choose
what encoder to use for each field.
Structural strategies build a tree of encoders for each field from the
version-free builders in structural. Struct and list encoders collect
validity and offsets; primitive leaf encoders accumulate values and emit
miniblock or full-zip pages.
Modules§
- structural
- Version-free structural field encoder builders.
Structs§
- Array
Field Encoding Strategy - Field-to-column composition for the
pb::ArrayEncodinggrammar. - Batch
Encoder - A batch encoder that encodes RecordBatch objects by delegating to field encoders for each top-level field in the batch.
- Column
Index Sequence - Keeps track of the current column index and makes a mapping from field id to column index
- Encoded
Array - An array encoded with the
pb::ArrayEncodinggrammar. - Encoded
Batch - An encoded batch of data and a page table describing it
- Encoded
Column - Encoded
Page - An encoded page of data
- Encoding
Options - Options that control the encoding process
- Field
Encoding Context - Context shared while one top-level field and all of its children are mapped to concrete field encoders.
- OutOf
Line Buffers - A tool to reserve space for buffers that are not in-line with the data
Constants§
- MIN_
PAGE_ BUFFER_ ALIGNMENT - The minimum alignment for a page buffer. Writers must respect this.
Traits§
- Array
Encoder - Encodes one data block and describes it with
pb::ArrayEncoding. - Array
Encoding Strategy - Selects an
ArrayEncoderfor one page. - Field
Encoder - Top level encoding trait to code any Arrow array type into one or more pages.
- Field
Encoding Strategy - A trait to pick which kind of field encoding to use for a field
Functions§
- encode_
batch - Helper method to encode a batch of data into memory
Type Aliases§
- Encode
Task - A task to create a page of data