Expand description
§🌪️ Vortex
📚 Documentation | 📊 Performance Benchmarks
§Overview
Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:
-
⚡️ Blazing Fast Performance
- 200x faster random access reads (vs. modern Apache Parquet)
- 2-10x faster scans
- 2-10x faster writes
- Similar compression ratios
- Efficient support for wide tables with zero-copy/zero-parse metadata
-
🔧 Extensible Architecture
- Modeled after Apache DataFusion’s extensible approach
- Pluggable encoding system, type system, compression strategy, & layout strategy
- Zero-copy compatibility with Apache Arrow
-
🗳️ Open Source, Neutral Governance
- A Linux Foundation (LF AI & Data) Project
- Apache-2.0 Licensed
-
↔️ Integrations
- Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
- Apache Iceberg (coming soon)
🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).
§Key Features
§Core Capabilities
- ✨ Logical Types - Clean separation between logical schema and physical layout
- 🔄 Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
- 🧩 Extensible Encodings - Pluggable physical layouts with built-in optimizations
- 📦 Cascading Compression - Support for nested encoding schemes
- 🚀 High-Performance Computing - Optimized compute kernels for encoded data
- 📊 Rich Statistics - Lazy-loaded summary statistics for optimization
§Technical Architecture
§Logical vs Physical Design
Vortex strictly separates logical and physical concerns:
- Logical Layer: Defines data types and schema
- Physical Layer: Handles encoding and storage implementation
- Built-in Encodings: Compatible with Apache Arrow’s memory format
- Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)
§Quick Start
§Installation
§Rust Crate
All features are exported through the main vortex
crate.
cargo add vortex
§Python Package
uv add vortex-data
§Command Line UI (vx)
For browsing the structure of Vortex files, you can use the vx
command-line tool.
# Install latest release
cargo install vortex-tui --locked
# Or build from source
cargo install --path vortex-tui --locked
# Usage
vx browse <file>
§Development Setup
§Prerequisites (macOS)
# Optional but recommended dependencies
brew install flatbuffers protobuf # For .fbs and .proto files
brew install duckdb # For benchmarks
# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup
# Initialize submodules
git submodule update --init --recursive
# Setup dependencies with uv
uv sync --all-packages
§Performance Optimization
For optimal performance, we suggest using MiMalloc:
#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;
§Project Information
§License
Licensed under the Apache License, Version 2.0.
§Governance
Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.
§Contributing
See CONTRIBUTING.md for guidelines.
§Reporting Vulnerabilities
If you discovery a security vulnerability, please email vuln-report@vortex.dev.
§Trademarks
Copyright © Vortex a Series of LF Projects, LLC. For terms of use, trademark policy, and other project policies please see https://lfprojects.org
§Acknowledgments 🏆
The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.
§Research in Vortex
- BtrBlocks - Efficient columnar compression
- FastLanes - High-performance integer compression
- FSST - Fast random access string compression
- ALP - Adaptive lossless floating-point compression
- Procella - YouTube’s unified data system
- Anyblob - High-performance access to object storage
- ClickHouse - Fast analytics for everyone
§Vortex in Research
- Anyblox - A Framework for Self-Decoding Datasets
§Open Source Inspiration
- Apache Arrow
- Apache DataFusion
- parquet2 by Jorge Leitao
- DuckDB
- Velox & Nimble
§Thanks to all contributors who have shared their knowledge and code with the community! 🚀
Re-exports§
pub use vortex_file as file;
pub use vortex_buffer as buffer;
pub use vortex_dtype as dtype;
pub use vortex_error as error;
pub use vortex_expr as expr;
pub use vortex_flatbuffers as flatbuffers;
pub use vortex_ipc as ipc;
pub use vortex_layout as layout;
pub use vortex_mask as mask;
pub use vortex_metrics as metrics;
pub use vortex_proto as proto;
pub use vortex_scalar as scalar;
pub use vortex_scan as scan;
pub use vortex_utils as utils;
Modules§
- accessor
- aliases
- Re-exports of third-party crates we use in macros exported from vortex-array.
- arrays
- All the built-in encoding schemes and arrays.
- arrow
- Utilities to work with
Arrow
data and types. - builders
- Builders for Vortex arrays.
- compress
- compressor
- compute
- Compute kernels on top of Vortex Arrays.
- data
- display
- Convert an array into a human-readable string representation.
- encodings
- iter
- Iterator over slices of an array, and related utilities.
- patches
- search_
sorted - serde
- session
- stats
- Traits and utilities to compute and access array statistics.
- stream
- validity
- Array validity and nullability behavior, used by arrays and compute functions.
- variants
- This module defines extension functionality specific to each Vortex DType.
- vtable
- This module contains the VTable definitions for a Vortex encoding.
Macros§
- register_
kernel - Register a kernel for a compute function. See each compute function for the correct type of kernel to register.
- vtable
Structs§
- Array
Adapter - Adapter struct used to lift the
VTable
trait into an object-safeArray
implementation. - Empty
Metadata - Empty array metadata
- Encoding
Adapter - Adapter struct used to lift the
VTable
trait into an object-safeEncoding
implementation. - Prost
Metadata - A utility wrapper for Prost metadata serialization.
- RawMetadata
- A utility wrapper for raw metadata serialization. This delegates the serialiation step to the arrays’ vtable.
- VTable
Context - A collection of encodings that can be addressed by a u16 positional index. This is used to map array encodings and layout encodings when reading from a file.
- VTable
Registry - A registry of encodings that can be used to construct a context for serde.
Enums§
- Canonical
- An enum capturing the default uncompressed encodings for each Vortex type.
Traits§
- Array
- The public API trait for all Vortex arrays.
- Array
Buffer Visitor - Array
Child Visitor - Array
Visitor - Array
Visitor Ext - Deserialize
Metadata - Trait for deserializing Vortex metadata from a vector of unaligned bytes.
- Encoding
- Marker trait for array encodings with their associated Array type.
- Into
Array - Trait for converting a type into a Vortex
ArrayRef
. - Serialize
Metadata - Trait for serializing Vortex metadata to a vector of unaligned bytes.
- ToCanonical
- Trait for types that can be converted from an owned type into an owned array variant.
Type Aliases§
- Array
Context - A collection of array encodings.
- Array
Ref - A reference counted pointer to a dynamic
Array
trait object. - Array
Registry - Encoding
Id - EncodingId is a globally unique name of the array’s encoding.
- Encoding
Ref