Crate icepick

Crate icepick 

Source
Expand description

Icepick: Production-ready cloud Iceberg catalogs

Specialized Rust library for Apache Iceberg catalog operations on cloud providers. Provides rock-solid implementations for AWS S3 Tables and Cloudflare R2 Data Catalog.

§Quick Start

§AWS S3 Tables (native platforms only)

use icepick::S3TablesCatalog;
use icepick::catalog::Catalog;
use icepick::spec::{TableIdent, NamespaceIdent};

let catalog = S3TablesCatalog::from_arn(
    "my-catalog",
    "arn:aws:s3tables:us-west-2:123456789012:bucket/my-bucket"
).await?;

let namespace = NamespaceIdent::new(vec!["namespace".to_string()]);
let table_id = TableIdent::new(namespace, "table".to_string());
let table = catalog.load_table(&table_id).await?;

§Cloudflare R2 (all platforms including WASM)

use icepick::R2Catalog;
use icepick::catalog::Catalog;
use icepick::spec::{TableIdent, NamespaceIdent};

let catalog = R2Catalog::new(
    "my-catalog",
    "account-id",
    "bucket-name",
    "api-token"
).await?;

let namespace = NamespaceIdent::new(vec!["namespace".to_string()]);
let table_id = TableIdent::new(namespace, "table".to_string());
let table = catalog.load_table(&table_id).await?;

Re-exports§

pub use error::Error;
pub use error::Result;
pub use io::FileIO;
pub use reader::DataFileEntry;
pub use scan::ArrowRecordBatchStream;
pub use scan::TableScan;
pub use scan::TableScanBuilder;
pub use spec::DataContentType;
pub use spec::DataFile;
pub use spec::NamespaceIdent;
pub use spec::NestedField;
pub use spec::PrimitiveType;
pub use spec::Schema;
pub use spec::Snapshot;
pub use spec::StructType;
pub use spec::Summary;
pub use spec::TableIdent;
pub use spec::TableMetadata;
pub use spec::Type;
pub use table::Table;
pub use transaction::Transaction;
pub use writer::arrow_to_parquet;
pub use writer::AppendOnlyTableWriter;
pub use writer::AppendResult;
pub use writer::PartitionFieldConfig;
pub use writer::PartitionTransform;
pub use writer::SchemaEvolutionPolicy;
pub use writer::TableWriterOptions;
pub use catalog::r2::R2Catalog;
pub use catalog::RestAuthProvider;
pub use catalog::RestCatalog;
pub use catalog::RestCatalogBuilder;
pub use catalog::s3_tables::S3TablesCatalog;

Modules§

arrow_convert
Convert between Iceberg and Arrow types
catalog
Iceberg REST catalog implementation with pluggable authentication
commit
Transaction commit orchestration
error
Error types for icepick catalog operations
io
I/O operations for Iceberg files WASM-compatible via OpenDAL
manifest
Iceberg manifest file handling
reader
Reading data from Iceberg tables
scan
Table scanning and reading
spec
Iceberg specification types
table
Iceberg table representation
transaction
Transaction API for writing to Iceberg tables
writer
Parquet writer and statistics collection