Skip to main content

Crate lance_hdfs_backend

Crate lance_hdfs_backend 

Source
Expand description

§lance-hdfs-backend

Read and write Lance datasets on HDFS with a pluggable Rust storage backend, powered by Apache OpenDAL.

§Features

  • Lance datasets on HDFS — create, read, and append to datasets using hdfs:// URIs, with access to committed dataset versions.
  • Atomic dataset commits — use HDFS atomic renames to publish new versions without overwriting an existing version during concurrent writes.
  • Session-based integration — register the HDFS provider in a Lance storage registry and share it through a session for reads and writes.
  • Storage operations — stream file contents, read byte ranges, write data, list objects, and delete files through Lance’s storage interface.
  • Hadoop configuration — configure the NameNode, HDFS user, Kerberos ticket cache, and atomic write directory through storage options and supported environment variables.

§Installation

Add the backend and the Lance crates it integrates with to your Cargo.toml:

[dependencies]
lance-hdfs-backend = "0.2.2"
lance = { version = "12", default-features = false }
lance-io = { version = "12", default-features = false }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

lance-io provides the storage registry the provider is registered in, lance provides the dataset API, and tokio runs the async calls. The default features of lance-hdfs-backend include the HDFS provider and the rename commit handler.

This release targets Lance 12. Keep lance and lance-io on the same major as the backend so both sides share one set of storage types; a different major resolves to two copies of lance-io and the provider will not type-check.

§Quickstart

§Prepare the environment

Before connecting to a cluster, make sure you have:

  • The Rust toolchain specified in rust-toolchain.toml
  • Java 11 or newer
  • Hadoop client libraries and configuration when connecting to HDFS

Configure the Java and Hadoop environment for your cluster:

JAVA_HOME
HADOOP_HOME
HADOOP_CONF_DIR
CLASSPATH
LD_LIBRARY_PATH

Register the provider, create a session, and read an existing HDFS dataset. Replace the URI with your NameNode and dataset path:

ⓘ
use std::sync::Arc;

use lance::dataset::{
    DEFAULT_INDEX_CACHE_SIZE, DEFAULT_METADATA_CACHE_SIZE, builder::DatasetBuilder,
};
use lance::session::Session;
use lance_hdfs_backend::{register, rename_commit_handler};
use lance_io::object_store::ObjectStoreRegistry;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let registry = Arc::new(ObjectStoreRegistry::default());
    register(&registry);

    let session = Arc::new(Session::new(
        DEFAULT_INDEX_CACHE_SIZE,
        DEFAULT_METADATA_CACHE_SIZE,
        registry,
    ));

    let dataset = DatasetBuilder::from_uri("hdfs://namenode:9000/user/data/dataset")
        .with_session(session)
        .with_commit_handler(rename_commit_handler())
        .load()
        .await?;

    println!("rows={}", dataset.count_rows(None).await?);
    Ok(())
}

To create or append to a dataset, pass the same session and rename_commit_handler() through WriteParams, and select WriteMode::Create or WriteMode::Append. See the dataset example for a complete write and read workflow, including Arrow batch construction.

§Configuration

Storage optionEnvironment variablePurpose
hdfs_name_nodeHDFS_NAME_NODENameNode address or nameservice URI
hdfs_userHADOOP_USER_NAME, then HDFS_USERHDFS user identity
hdfs_kerberos_ticket_cache_pathnoneKerberos ticket cache path
hdfs_atomic_write_dirnoneTemporary directory for atomic writes

For the NameNode, storage options take precedence over the environment, then the address in the hdfs:// URI. An explicit HDFS user takes precedence over the user environment variables.

§Notes

  • HDFS URIs: include a NameNode address, such as hdfs://namenode:8020/user/data/dataset, or a nameservice configured in your Hadoop client, such as hdfs://my-cluster/user/data/dataset.
  • Dataset commits: pass rename_commit_handler() explicitly for writes. It uses HDFS atomic renames to avoid overwriting an existing dataset version. Registering the storage provider alone does not configure the commit handler.
  • Authentication: use the storage options listed above for user identity and a Kerberos ticket cache. Keep the Hadoop client configuration consistent with the target cluster.
  • Configuration scope: the provider forwards the listed HDFS options and sets the storage root to /. Create-mode renames (dataset commits) request OpenDAL if-not-exists semantics, so a commit fails instead of replacing an already committed dataset version.

§Integration Tests

The HDFS integration tests are ignored by default:

HDFS_NAME_NODE=hdfs://localhost:9000 \
  cargo test --all-features --test hdfs_integration -- --ignored

The nameservice (HA) test activates only when HDFS_HA_NAMESERVICE is set. Point the Hadoop client configuration (HADOOP_CONF_DIR and CLASSPATH) at the target nameservice, export the variable, and run the same command. For a Kerberos-secured cluster, obtain a ticket with kinit first and pass hdfs_user and hdfs_kerberos_ticket_cache_path through storage options.

§Licenses

Licensed under the Apache License, Version 2.0. See the LICENSE file for the license text.

This project includes code originally developed by the Lance Authors. See the NOTICE file for attribution.

Structs§

HdfsStoreProvider
HDFS object store provider backed by OpenDAL.

Constants§

HDFS_SCHEME

Functions§

register
Registers the HDFS object store provider in a Lance registry.
rename_commit_handler
Returns the commit handler recommended for HDFS datasets.