Crate opendal

Expand description

OpenDAL is the Open Data Access Layer to freely, painlessly, and efficiently access data.

Services

Service represents a backend scheme that OpenDAL supported.

OpenDAL supports the following services now:

Services	Description
azblob	Azure Storage Blob services.
fs	POSIX alike file system.
ftp	FTP and FTPS support.
gcs	Google Cloud Storage service.
hdfs	Hadoop Distributed File System(HDFS).
http	HTTP read-only backend.
ipfs	IPFS HTTP Gateway support.
ipmfs	IPFS Mutable File System support.
memory	In memory backend support.
moka	moka backend support.
obs	Huawei Cloud OBS service.
oss	Aliyun Object Storage Service (OSS).
redis	Redis service.
rocksdb	RocksDB service.
s3	AWS S3 alike services.

More services support is tracked at opendal#5

Layers

Layer is the mechanism to intercept operations.

OpenDAL supports the following layers now:

Layers	Description
ConcurrentLimitLayer	Concurrent request limit.
ContentCacheLayer	Content cache.
ImmutableIndexLayer	Immutable in-memory index.
LoggingLayer	Logging for every operations.
MetadataCacheLayer	Metadata cache.
MetricsLayer	Metrics for every operations.
RetryLayer	Retry for failed operations.
SubdirLayer	Allow switching directory.
TracingLayer	Tracing for every operations.

Optional features

Layers

layers-all: Enable all layers support.
layers-metrics: Enable operator metrics support.
layers-tracing: Enable operator tracing support.

Services

services-ftp: Enable ftp service support.
services-hdfs: Enable hdfs service support.
services-moka: Enable moka service support.
services-ipfs: Enable ipfs service support.
services-redis: Enable redis service support.
services-rocksdb: Enable rocksdb service support.

Dependencies features

compress: Enable object decompress read support.

Examples

More examples could be found at https://opendal.databend.rs/examples/index.html

use anyhow::Result;
use backon::ExponentialBackoff;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::layers::LoggingLayer;
use opendal::layers::RetryLayer;
use opendal::Object;
use opendal::ObjectEntry;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::ObjectStreamer;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init a memory operator
    let op = Operator::from_env(Scheme::Memory)?
        // Init with logging layer enabled.
        .layer(LoggingLayer)
        // Init with retry layer enabled.
        .layer(RetryLayer::new(ExponentialBackoff::default()));

    // Create object handler.
    let o = op.object("test_file");

    // Write data
    o.write("Hello, World!").await?;

    // Read data
    let bs = o.read().await?;

    // Fetch metadata
    let name = o.name();
    let path = o.path();
    let meta = o.metadata().await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    o.delete().await?;

    // Readdir
    let o = op.object("test_dir/");
    let mut ds = o.list().await?;
    while let Some(entry) = ds.try_next().await? {
        let path = entry.path();
        let mode = entry.mode();
    }

    Ok(())
}