Expand description
OpenDAL is the Open Data Access Layer to freely, painlessly, and efficiently access data.
Services
Service
represents a backend scheme that OpenDAL supported.
OpenDAL supports the following services now:
Services | Description |
---|---|
azblob | Azure Storage Blob services. |
fs | POSIX alike file system. |
ftp | FTP and FTPS support. |
gcs | Google Cloud Storage service. |
hdfs | Hadoop Distributed File System(HDFS). |
http | HTTP read-only backend. |
ipfs | IPFS HTTP Gateway support. |
ipmfs | IPFS Mutable File System support. |
memory | In memory backend support. |
moka | moka backend support. |
obs | Huawei Cloud OBS service. |
oss | Aliyun Object Storage Service (OSS). |
redis | Redis service. |
rocksdb | RocksDB service. |
s3 | AWS S3 alike services. |
More services support is tracked at opendal#5
Layers
Layer
is the mechanism to intercept operations.
OpenDAL supports the following layers now:
Layers | Description |
---|---|
ConcurrentLimitLayer | Concurrent request limit. |
ContentCacheLayer | Content cache. |
ImmutableIndexLayer | Immutable in-memory index. |
LoggingLayer | Logging for every operations. |
MetadataCacheLayer | Metadata cache. |
MetricsLayer | Metrics for every operations. |
RetryLayer | Retry for failed operations. |
SubdirLayer | Allow switching directory. |
TracingLayer | Tracing for every operations. |
Optional features
Layers
layers-all
: Enable all layers support.layers-metrics
: Enable operator metrics support.layers-tracing
: Enable operator tracing support.
Services
services-ftp
: Enable ftp service support.services-hdfs
: Enable hdfs service support.services-moka
: Enable moka service support.services-ipfs
: Enable ipfs service support.services-redis
: Enable redis service support.services-rocksdb
: Enable rocksdb service support.
Dependencies features
compress
: Enable object decompress read support.rustls
: Enable TLS functionality provided byrustls
, enabled by defaultnative-tls
: Enable TLS functionality provided bynative-tls
native-tls-vendored
: Enable thevendored
feature ofnative-tls
Examples
More examples could be found at https://opendal.databend.rs/examples/index.html
use anyhow::Result;
use backon::ExponentialBackoff;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::layers::LoggingLayer;
use opendal::layers::RetryLayer;
use opendal::Object;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::Operator;
use opendal::Scheme;
#[tokio::main]
async fn main() -> Result<()> {
// Init a fs operator
let op = Operator::from_env(Scheme::Fs)?
// Init with logging layer enabled.
.layer(LoggingLayer)
// Init with retry layer enabled.
.layer(RetryLayer::new(ExponentialBackoff::default()));
// Create object handler.
let o = op.object("test_file");
// Write data
o.write("Hello, World!").await?;
// Read data
let bs = o.read().await?;
// Fetch metadata
let name = o.name();
let path = o.path();
let meta = o.metadata().await?;
let mode = meta.mode();
let length = meta.content_length();
// Delete
o.delete().await?;
// Readdir
let o = op.object("test_dir/");
let mut ds = o.list().await?;
while let Some(entry) = ds.try_next().await? {
let path = entry.path();
let mode = entry.mode().await?;
}
Ok(())
}
Modules
Providing Layer trait and its implementations.
Raw modules provide raw APIs that used by underlying services
Providing specific services support.
Structs
BatchOperator is used to take batch operations like walk_dir and remove_all, should
be constructed by
Operator::batch()
.Error is the error struct returned by all opendal functions.
Object is the handler for all object related operations.
Metadata carries all object metadata.
ObjectMultipart represent an ongoing multipart upload.
ObjectPart is generated by
write_multipart
operation, carries
required information for complete_multipart
.ObjectReader is a bytes reader that carries it’s related metadata.
Users could fetch part of metadata that carried by read response.
Args for
abort_multipart
operation.Args for
complete_multipart
operation.Args for
create
operation.Args for
create_multipart
operation.Args for
delete
operation.Args for
list
operation.Args for
presign
operation.Args for
read
operation.Args for
stat
operation.Args for
write
operation.Args for
write_multipart
operation.User-facing APIs for object and object streams.
Metadata for operator, users can use this metadata to get information of operator.
Enums
ErrorKind is all kinds of opendal’s Error.
ObjectMode represents the corresponding object’s mode.
Presign operation used for presign.
Services that OpenDAL supports
Traits
Layer is used to intercept the operations on the underlying storage.
Type Definitions
Result that is a wrapper of
Reustl<T, opendal::Error>