Expand description
Hadoop Distributed File System (HDFS™) support.
A distributed file system that provides high-throughput access to application data.
Features
HDFS support needs to enable feature services-hdfs
.
Configuration
root
: Set the work dir for backend.name_node
: Set the name node for backend.
Refer to Builder
’s public API docs for more information.
Environment
OPENDAL_HDFS_ROOT
OPENDAL_HDFS_NAME_NODE
HDFS also needs some environment set correctly.
JAVA_HOME
: the path to java home, could be found viajava -XshowSettings:properties -version
HADOOP_HOME
: the path to hadoop home, opendal relays on this env to discover hadoop jars and setCLASSPATH
automatically.
Most of the time, setting JAVA_HOME
and HADOOP_HOME
is enough. But there are some edge cases:
- If meeting errors like the following:
error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory
Java’s lib are not including in pkg-config find path, please set LD_LIBRARY_PATH
:
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
The path of libjvm.so
could be different, please keep an eye on it.
- If meeting errors like the following:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
CLASSPATH
is not set correctly or your hadoop installation is incorrect.
Example
Via Environment
Set environment correctly:
export OPENDAL_HDFS_ROOT=/path/to/dir/
export OPENDAL_HDFS_NAME_NODE=hdfs://127.0.0.1:9000
use std::sync::Arc;
use anyhow::Result;
use opendal::Object;
use opendal::Operator;
use opendal::Scheme;
#[tokio::main]
async fn main() -> Result<()> {
let op: Operator = Operator::from_env(Scheme::Hdfs)?;
// Create an object handle to start operation on object.
let _: Object = op.object("test_file");
Ok(())
}
Via Builder
use std::sync::Arc;
use anyhow::Result;
use opendal::services::hdfs;
use opendal::Accessor;
use opendal::Object;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
// Create fs backend builder.
let mut builder = hdfs::Builder::default();
// Set the name node for hdfs.
builder.name_node("hdfs://127.0.0.1:9000");
// Set the root for hdfs, all operations will happen under this root.
//
// NOTE: the root must be absolute path.
builder.root("/tmp");
// `Accessor` provides the low level APIs, we will use `Operator` normally.
let op: Operator = Operator::new(builder.build()?);
// Create an object handle to start operation on object.
let _: Object = op.object("test_file");
Ok(())
}
Structs
Builder for hdfs services