Crate opendal

Expand description

OpenDAL is the Open Data Access Layer to freely, painlessly, and efficiently access data.

Services

Service represents a backend scheme that OpenDAL supported.

OpenDAL supports the following services now:

Services	Description
azblob	Azure Storage Blob services.
azdfs	Azure Data Lake Storage Gen2 services.
fs	POSIX alike file system.
ftp	FTP and FTPS support.
gcs	Google Cloud Storage service.
ghac	Github Action Cache service.
hdfs	Hadoop Distributed File System(HDFS).
http	HTTP read-only backend.
ipfs	IPFS HTTP Gateway support.
ipmfs	IPFS Mutable File System support.
memcached	Memcached serivce.
memory	In memory backend support.
moka	moka backend support.
obs	Huawei Cloud OBS service.
oss	Aliyun Object Storage Service (OSS).
redis	Redis service.
rocksdb	RocksDB service.
s3	AWS S3 alike services.
webdav	WebDAV services.

Different services capabilities could be found at services
More services support is tracked at opendal#5

Layers

Layer is the mechanism to intercept operations.

OpenDAL supports the following layers now:

Layers	Description
ConcurrentLimitLayer	Concurrent request limit.
ImmutableIndexLayer	Immutable in-memory index.
LoggingLayer	Logging for every operations.
MetricsLayer	Metrics for every operations.
RetryLayer	Retry for failed operations.
TracingLayer	Tracing for every operations.

Optional features

Layers

layers-all: Enable all layers support.
layers-metrics: Enable operator metrics support.
layers-tracing: Enable operator tracing support.

Services

services-ftp: Enable ftp service support.
services-hdfs: Enable hdfs service support.
services-moka: Enable moka service support.
services-ipfs: Enable ipfs service support.
services-redis: Enable redis service support.
services-rocksdb: Enable rocksdb service support.

Dependencies features

compress: Enable object decompress read support.
rustls: Enable TLS functionality provided by rustls, enabled by default
native-tls: Enable TLS functionality provided by native-tls
native-tls-vendored: Enable the vendored feature of native-tls

Examples

More examples could be found at https://opendal.databend.rs/examples/index.html

use anyhow::Result;
use backon::ExponentialBackoff;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::layers::LoggingLayer;
use opendal::layers::RetryLayer;
use opendal::services;
use opendal::Object;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init a fs operator
    let op = Operator::from_env::<services::Fs>()?
        // Init with logging layer enabled.
        .layer(LoggingLayer::default())
        // Init with retry layer enabled.
        .layer(RetryLayer::new(ExponentialBackoff::default()))
        .finish();

    // Create object handler.
    let o = op.object("test_file");

    // Write data
    o.write("Hello, World!").await?;

    // Read data
    let bs = o.read().await?;

    // Fetch metadata
    let name = o.name();
    let path = o.path();
    let meta = o.metadata().await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    o.delete().await?;

    // Readdir
    let o = op.object("test_dir/");
    let mut ds = o.list().await?;
    while let Some(entry) = ds.try_next().await? {
        let path = entry.path();
        let mode = entry.mode().await?;
    }

    Ok(())
}