Crate opendal

source ·
Expand description

OpenDAL is the Open Data Access Layer to freely, painlessly, and efficiently access data.

Services

Service represents a backend scheme that OpenDAL supported.

OpenDAL supports the following services now:

ServicesDescription
azblobAzure Storage Blob services.
azdfsAzure Data Lake Storage Gen2 services.
fsPOSIX alike file system.
ftpFTP and FTPS support.
gcsGoogle Cloud Storage service.
ghacGithub Action Cache service.
hdfsHadoop Distributed File System(HDFS).
httpHTTP read-only backend.
ipfsIPFS HTTP Gateway support.
ipmfsIPFS Mutable File System support.
memcachedMemcached serivce.
memoryIn memory backend support.
mokamoka backend support.
obsHuawei Cloud OBS service.
ossAliyun Object Storage Service (OSS).
redisRedis service.
rocksdbRocksDB service.
s3AWS S3 alike services.
webdavWebDAV services.

More services support is tracked at opendal#5

Layers

Layer is the mechanism to intercept operations.

OpenDAL supports the following layers now:

LayersDescription
ConcurrentLimitLayerConcurrent request limit.
CacheLayerCache supports.
ImmutableIndexLayerImmutable in-memory index.
LoggingLayerLogging for every operations.
MetricsLayerMetrics for every operations.
RetryLayerRetry for failed operations.
SubdirLayerAllow switching directory.
TracingLayerTracing for every operations.

Optional features

Layers

  • layers-all: Enable all layers support.
  • layers-metrics: Enable operator metrics support.
  • layers-tracing: Enable operator tracing support.

Services

  • services-ftp: Enable ftp service support.
  • services-hdfs: Enable hdfs service support.
  • services-moka: Enable moka service support.
  • services-ipfs: Enable ipfs service support.
  • services-redis: Enable redis service support.
  • services-rocksdb: Enable rocksdb service support.

Dependencies features

  • compress: Enable object decompress read support.
  • rustls: Enable TLS functionality provided by rustls, enabled by default
  • native-tls: Enable TLS functionality provided by native-tls
  • native-tls-vendored: Enable the vendored feature of native-tls

Examples

More examples could be found at https://opendal.databend.rs/examples/index.html

use anyhow::Result;
use backon::ExponentialBackoff;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::layers::LoggingLayer;
use opendal::layers::RetryLayer;
use opendal::Object;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init a fs operator
    let op = Operator::from_env(Scheme::Fs)?
        // Init with logging layer enabled.
        .layer(LoggingLayer::default())
        // Init with retry layer enabled.
        .layer(RetryLayer::new(ExponentialBackoff::default()));

    // Create object handler.
    let o = op.object("test_file");

    // Write data
    o.write("Hello, World!").await?;

    // Read data
    let bs = o.read().await?;

    // Fetch metadata
    let name = o.name();
    let path = o.path();
    let meta = o.metadata().await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    o.delete().await?;

    // Readdir
    let o = op.object("test_dir/");
    let mut ds = o.list().await?;
    while let Some(entry) = ds.try_next().await? {
        let path = entry.path();
        let mode = entry.mode().await?;
    }

    Ok(())
}

Modules

Providing Layer trait and its implementations.
Raw modules provide raw APIs that used by underlying services
Providing specific services support.

Structs

BatchOperator is used to take batch operations like walk_dir and remove_all, should be constructed by Operator::batch().
Error is the error struct returned by all opendal functions.
Object is the handler for all object related operations.
ObjectLister is returned by Object::list to list objects.
Metadata carries all object metadata.
ObjectMultipart represent an ongoing multipart upload.
ObjectPart is generated by write_multipart operation, carries required information for complete_multipart.
ObjectReader is the public API for users.
Args for abort_multipart operation.
Args for complete_multipart operation.
Args for create operation.
Args for create_multipart operation.
Args for delete operation.
Args for list operation.
Args for presign operation.
Args for read operation.
Args for stat operation.
Args for write operation.
Args for write_multipart operation.
User-facing APIs for object and object streams.
Metadata for operator, users can use this metadata to get information of operator.

Enums

ErrorKind is all kinds of opendal’s Error.
ObjectMode represents the corresponding object’s mode.
Presign operation used for presign.
Services that OpenDAL supports

Traits

Layer is used to intercept the operations on the underlying storage.

Type Definitions

Result that is a wrapper of Reustl<T, opendal::Error>