Struct opendal::services::Hdfs

pub struct Hdfs { /* private fields */ }

Hadoop Distributed File System (HDFS™) support.

A distributed file system that provides high-throughput access to application data.

Capabilities

This service can be used to:

• read
• write
• list
• scan
• presign
• blocking

Differences with webhdfs

Webhdfs is powered by hdfs’s RESTful HTTP API.

Features

HDFS support needs to enable feature services-hdfs.

Configuration

• root: Set the work dir for backend.
• name_node: Set the name node for backend.

Refer to HdfsBuilder’s public API docs for more information.

Environment

HDFS needs some environment set correctly.

• JAVA_HOME: the path to java home, could be found via java -XshowSettings:properties -version
• HADOOP_HOME: the path to hadoop home, opendal relays on this env to discover hadoop jars and set CLASSPATH automatically.

Most of the time, setting JAVA_HOME and HADOOP_HOME is enough. But there are some edge cases:

• If meeting errors like the following:

error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory

Java’s lib are not including in pkg-config find path, please set LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}

The path of libjvm.so could be different, please keep an eye on it.

• If meeting errors like the following:

(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)

CLASSPATH is not set correctly or your hadoop installation is incorrect.

Example

Via Builder

use std::sync::Arc;

use anyhow::Result;
use opendal::services::Hdfs;
use opendal::Object;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Create fs backend builder.
    let mut builder = Hdfs::default();
    // Set the name node for hdfs.
    builder.name_node("hdfs://127.0.0.1:9000");
    // Set the root for hdfs, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root("/tmp");

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder)?.finish();

    // Create an object handle to start operation on object.
    let _: Object = op.object("test_file");

    Ok(())
}

Implementations

impl HdfsBuilder

pub fn root(&mut self, root: &str) -> &mut Self

Set root of this backend.

All operations will happen under this root.

pub fn name_node(&mut self, name_node: &str) -> &mut Self

Set name_node of this backend.

Valid format including:

• default: using the default setting based on hadoop config.
• hdfs://127.0.0.1:9000: connect to hdfs cluster.

Trait Implementations

impl Builder for HdfsBuilder

const SCHEME: Scheme = Scheme::Hdfs

Associated scheme for this builder.

type Accessor = HdfsBackend

The accessor that built by this builder.

fn from_map(map: HashMap<String, String>) -> Self

Construct a builder from given map.

fn build(&mut self) -> Result<Self::Accessor>

Consume the accessor builder to build a service.

fn from_iter(iter: impl Iterator<Item = (String, String)>) -> Selfwhere Self: Sized,

Construct a builder from given iterator.

fn from_env() -> Selfwhere Self: Sized,

Construct a builder from envs.

impl Debug for HdfsBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for HdfsBuilder

fn default() -> HdfsBuilder

Returns the “default value” for a type.