Lake Pulse

A Rust library for analyzing data lake table health — checking the pulse — across multiple formats (Delta Lake, Apache Iceberg, Apache Hudi, Lance) and storage providers (AWS S3, Azure Data Lake, GCS, HDFS, Local).

Supported Formats

Overview

Lake Pulse provides comprehensive health metrics for your data lake tables, including:

• File organization and compaction opportunities
• Metadata analysis and schema evolution
• Partition statistics
• Time travel/snapshot metrics
• Storage efficiency insights

Quick Start

Basic Example - Analyzing a Delta Lake table on AWS S3

use lake_pulse::{Analyzer, StorageConfig};

#[tokio::main]
async fn main() {
    let storage_config = StorageConfig::aws()
        .with_option("bucket", "my-bucket-1234")
        .with_option("region", "us-east-1")
        .with_option("access_key_id", "the_access_key_id")
        .with_option("secret_access_key", "the_secret_access_key")
        .with_option("session_token", "session_token_if_needed");
    
    let analyzer = Analyzer::builder(storage_config).build().await.unwrap();

    // Generate report
    let report = analyzer.analyze("my/table/path").await.unwrap();

    // Print pretty report
    println!("{}", report);
}

Supported Table Formats

• Delta Lake - Full support for transaction logs, deletion vectors, and Delta-specific metrics
• Apache Iceberg - Metadata analysis, snapshot management, and Iceberg-specific features
• Apache Hudi - Basic support for Hudi table structure analysis and metrics (requires hudi feature)
• Lance - Modern columnar format with vector search capabilities (requires lance feature)

Feature Flags

By default, Lake Pulse includes support for Delta Lake and Apache Iceberg. Additional table formats can be enabled via feature flags:

Usage

# In your Cargo.toml

# Default (Delta + Iceberg only)
lake-pulse = "0.2"

# With Hudi support
lake-pulse = { version = "0.2", features = ["hudi"] }

# With Lance support
lake-pulse = { version = "0.2", features = ["lance"] }

# With all table formats (Delta + Iceberg + Hudi + Lance)
lake-pulse = { version = "0.2", features = ["all"] }

Storage Configuration

Lake Pulse uses the object_store crate for cloud storage access. Configuration options are passed through to the underlying storage provider.

AWS S3 Configuration Options

Common options for S3 (see object_store AWS documentation):

• bucket - S3 bucket name
• region - AWS region (e.g., "us-east-1")
• access_key_id - AWS access key ID
• secret_access_key - AWS secret access key
• session_token - Optional session token for temporary credentials
• endpoint - Optional custom endpoint URL

Azure Configuration Options

Common options for Azure (see object_store Azure documentation):

• container - Azure container name
• account_name - Storage account name
• tenant_id - Azure tenant ID
• client_id - Service principal client ID
• client_secret - Service principal client secret

GCP Configuration Options

Common options for GCP (see object_store GCP documentation):

• bucket - GCS bucket name
• service_account_key - Path to service account JSON key file

HDFS Configuration Options

HDFS support is provided via the hdfs-native-object-store crate:

• url - HDFS namenode URL (e.g., "hdfs://namenode:8020")

let storage_config = StorageConfig::hdfs()
    .with_option("url", "hdfs://namenode:8020");
let analyzer = Analyzer::builder(storage_config).build().await.unwrap();
let report = analyzer.analyze("/path/to/table").await.unwrap();

Local Filesystem

let storage_config = StorageConfig::local();
let analyzer = Analyzer::builder(storage_config).build().await.unwrap();
let report = analyzer.analyze("/path/to/table").await.unwrap();

Examples

See the examples/ directory for more detailed usage examples:

• s3_store.rs - AWS S3 example
• adl_store.rs - Azure Data Lake example
• hdfs_store.rs - HDFS example
• local_store.rs - Local filesystem example
• local_store_iceberg.rs - Iceberg table example
• local_store_hudi.rs - Hudi table example (requires hudi feature)
• local_store_lance.rs - Lance table example (requires lance feature)

Run examples with:

cargo run --example s3_store

# For examples requiring feature flags:
cargo run --features hudi --example local_store_hudi
cargo run --features lance --example local_store_lance

Documentation

For detailed information on configuration options, refer to the object_store crate documentation:

• AWS S3 Configuration
• Azure Configuration
• GCP Configuration
• HDFS Configuration

Supported Storages

See LAKE_PULSE_SUPPORTED_STORAGES.md for a comparison of storage providers supported by Lake Pulse.

Minimum Supported Rust Version (MSRV)

This crate requires Rust 1.88 or later.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

See LICENSE files for details.