Crate vortex_file

Expand description

Read and write Vortex layouts, a serialization of Vortex arrays.

A Vortex file stores a root Layout, the byte segments referenced by that layout, an optional file-level DType and statistics, and enough footer metadata to deserialize the tree. Layouts are recursive, so a file may organize data by row groups, columns, dictionaries, statistics, or other writer-chosen structures without changing the logical dtype seen by readers. The built-in layouts are:

FlatLayout: a single contiguously serialized array of buffers with a specific in-memory Alignment.
StructLayout: each column laid out at known offsets, permitting a subset of columns to be read in linear time with constant-time random access to any column.
ChunkedLayout: each chunk laid out at known offsets; locating the chunks covering a row range is an N log(N) search of the offsets.
DictLayout: a shared dictionary of values with a child layout holding indices.
ZonedLayout: a zone-map of statistics used for filter pruning.

A layout alone is not a standalone Vortex file: it is not self-describing, so the file pairs it with the dtype and footer metadata needed to deserialize it. This crate owns the file reader/writer APIs; the byte-level format is described under File Format below and specified in full in the docs: https://docs.vortex.dev/specs/file-format.html.

§Reading

Use OpenOptionsSessionExt::open_options to create VortexOpenOptions from a session. Opening reads or accepts a footer, builds a segment source, and returns VortexFile. Scans are configured from VortexFile::scan with projection/filter expressions, row ranges, Selection, split strategy, and concurrency settings from vortex-layout.

Supplying known metadata can reduce open-time IO:

VortexOpenOptions::with_file_size avoids a size request.
VortexOpenOptions::with_dtype is required for files written without an embedded dtype.
VortexOpenOptions::with_footer can open a file without reading footer bytes.
VortexOpenOptions::with_segment_cache reuses segment buffers across scans.

§Writing

Use WriteOptionsSessionExt::write_options or VortexWriteOptions::new to write an ArrayStream. The default WriteStrategyBuilder repartitions rows, builds statistics layouts, dictionary-encodes suitable columns, compresses chunks with the BtrBlocks-style compressor, and writes flat leaf layouts. Advanced users can replace the whole strategy or override individual fields.

§File Format

A Vortex file begins and ends with the 4-byte magic VTXF. Data segments are written first, in writer-chosen order, followed by the footer flatbuffers, a postscript, and an 8-byte end-of-file marker:

┌────────────────────────────┐
│     Magic bytes 'VTXF'     │  4 bytes
├────────────────────────────┤
│          Segments          │  serialized array chunks and per-column
│     (data & statistics)    │  statistics, in writer-chosen order
├────────────────────────────┤
│      DType flatbuffer      │  optional; omitted via `exclude_dtype`
├────────────────────────────┤
│      Layout flatbuffer     │  required; the root Layout tree
├────────────────────────────┤
│    Statistics flatbuffer   │  optional; file-level per-field statistics
├────────────────────────────┤
│      Footer flatbuffer     │  required; dictionary-encoded segment map
│                            │  and array/layout/compression/encryption specs
├────────────────────────────┤
│         Postscript         │  offsets of the four footer segments above;
│                            │  at most 65528 bytes
├────────────────────────────┤
│     8-byte End of File     │  u16 version, u16 postscript length,
│                            │  4 magic bytes 'VTXF'
└────────────────────────────┘

The postscript records the offset, length, and alignment of the dtype, layout, statistics, and footer segments, so a single read of the file tail (defaulting to 64KiB) is enough to locate and parse the footer. The byte-level format is specified in full at https://docs.vortex.dev/specs/file-format.html.

A Parquet-style file is realized by nesting a chunked layout of struct layouts of chunked layouts of flat layouts: the outer chunked layout models row groups and the inner one models pages. Layouts are adaptive, so the writer is free to build arbitrarily complex layouts to trade off locality and parallelism, or to elide statistics entirely when an external index supplies them.

FooterDeserializer supports incremental footer reads. It returns DeserializeStep values when it needs more bytes or a file size, and returns Footer once all required footer segments are available. VortexOpenOptions drives this state machine for ordinary file opens.

Modules§

multi: Builder for constructing a MultiLayoutDataSource from multiple Vortex files.
segments: Segment sources, caches, and sinks used by file readers and writers.
v2: Compatibility readers for newer file-statistics layout behavior.

Structs§

BlockingWrite: Blocking adapter for VortexWriteOptions.
BlockingWriter: A blocking adapter around a Writer, allowing incremental writing of arrays to a Vortex file.
FileStatistics: Contains statistical information about the data in a Vortex file.
Footer: Captures the layout information of a Vortex file.
FooterDeserializer: Deserialize a footer from the end of a Vortex file or created from a crate::footer::FooterSerializer.
FooterSerializer: Serializes a Footer into footer buffers and the trailing postscript/EOF marker.
SegmentSpec: The location of a segment within a Vortex file.
VortexFile: Represents a Vortex file, providing access to its metadata and content.
VortexOpenOptions: Open options for a Vortex file reader.
VortexWriteOptions: Configure a new writer, which can eventually be used to write an ArrayStream into a sink that implements VortexWrite.
WriteStrategyBuilder: Build a new writer strategy to compress and reorganize chunks of a Vortex file.
WriteSummary: Summary returned after a Vortex file is written.
Writer: An async API for writing Vortex files.

Enums§

DeserializeStep: Result of one FooterDeserializer::deserialize step.

Constants§

EOF_SIZE: The size of the EOF marker in bytes
MAGIC_BYTES: The magic bytes for a Vortex file
MAX_POSTSCRIPT_SIZE: The maximum length of a Vortex postscript in bytes
V1_FOOTER_FBS_SIZE: The size of the footer in bytes in Vortex version 1
VERSION: The current version of the Vortex file format
VORTEX_FILE_EXTENSION: The extension for Vortex files

Statics§

ALLOWED_ENCODINGS: Static registry of all allowed array encodings for file writing.

Traits§

OpenOptionsSessionExt: Extension trait for constructing VortexOpenOptions from a session.
WriteOptionsSessionExt: Extension trait for constructing VortexWriteOptions from a session.

Functions§

register_default_encodings: Register the default encodings use in Vortex files with the provided session.