pub struct Webhdfs { /* private fields */ }
Expand description
WebHDFS’s REST API support.
There two implementations of WebHDFS REST API:
- Native via HDFS Namenode and Datanode, data are transferred between nodes directly.
- HttpFS is a gateway before hdfs nodes, data are proxied.
Capabilities
This service can be used to:
- read
- write
- list
-
scan -
presign - blocking
Differences with hdfs
Hdfs is powered by HDFS’s native java client. Users need to setup the hdfs services correctly. But webhdfs can access from HTTP API and no extra setup needed.
Configurations
root
: The root path of the WebHDFS service.endpoint
: The endpoint of the WebHDFS service.delegation
: The delegation token for WebHDFS.
Refer to Builder
’s public API docs for more information
Examples
Via Builder
use std::sync::Arc;
use anyhow::Result;
use opendal::services::Webhdfs;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
let mut builder = Webhdfs::default();
// set the root for WebHDFS, all operations will happen under this root
//
// Note:
// if the root is not exists, the builder will automatically create the
// root directory for you
// if the root exists and is a directory, the builder will continue working
// if the root exists and is a folder, the builder will fail on building backend
builder.root("/path/to/dir");
// set the endpoint of webhdfs namenode, controlled by dfs.namenode.http-address
// default is http://127.0.0.1:9870
builder.endpoint("http://127.0.0.1:9870");
// set the delegation_token for builder
builder.delegation("delegation_token");
let op: Operator = Operator::new(builder)?.finish();
Ok(())
}
Implementations§
source§impl WebhdfsBuilder
impl WebhdfsBuilder
sourcepub fn root(&mut self, root: &str) -> &mut Self
pub fn root(&mut self, root: &str) -> &mut Self
Set the working directory of this backend
All operations will happen under this root
Note
The root will be automatically created if not exists. If the root is occupied by a file, building of directory will fail
sourcepub fn endpoint(&mut self, endpoint: &str) -> &mut Self
pub fn endpoint(&mut self, endpoint: &str) -> &mut Self
Set the remote address of this backend
default to http://127.0.0.1:9870
Endpoints should be full uri, e.g.
https://webhdfs.example.com:9870
http://192.168.66.88:9870
If user inputs endpoint without scheme, we will
prepend http://
to it.
sourcepub fn delegation(&mut self, delegation: &str) -> &mut Self
pub fn delegation(&mut self, delegation: &str) -> &mut Self
Set the delegation token of this backend, used for authentication
Note
The builder prefers using delegation token over username. If both are set, delegation token will be used.
Trait Implementations§
source§impl Builder for WebhdfsBuilder
impl Builder for WebhdfsBuilder
source§fn build(&mut self) -> Result<Self::Accessor>
fn build(&mut self) -> Result<Self::Accessor>
build the backend
Note:
when building backend, the built backend will check if the root directory exits. if the directory does not exits, the directory will be automatically created if the root path is occupied by a file, a failure will be returned
source§impl Clone for WebhdfsBuilder
impl Clone for WebhdfsBuilder
source§fn clone(&self) -> WebhdfsBuilder
fn clone(&self) -> WebhdfsBuilder
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read more