Expand description
OpenDAL is the Open Data Access Layer to freely, painlessly, and efficiently access data.
Services
Service
represents a backend scheme that OpenDAL supported.
OpenDAL supports the following services now:
Services | Description |
---|---|
azblob | Azure Storage Blob services. |
fs | POSIX alike file system. |
ftp | FTP and FTPS support. |
gcs | Google Cloud Storage service. |
hdfs | Hadoop Distributed File System(HDFS). |
http | HTTP read-only backend. |
ipfs | IPFS HTTP Gateway support. |
ipmfs | IPFS Mutable File System support. |
memory | In memory backend support. |
moka | moka backend support. |
obs | Huawei Cloud OBS service. |
oss | Aliyun Object Storage Service (OSS). |
redis | Redis service. |
s3 | AWS S3 alike services. |
More services support is tracked at opendal#5
Layers
Layer
is the mechanism to intercept operations.
OpenDAL supports the following layers now:
Layers | Description |
---|---|
ConcurrentLimitLayer | Concurrent request limit. |
ContentCacheLayer | Content cache. |
ImmutableIndexLayer | Immutable in-memory index. |
LoggingLayer | Logging for every operations. |
MetadataCacheLayer | Metadata cache. |
MetricsLayer | Metrics for every operations. |
RetryLayer | Retry for failed operations. |
SubdirLayer | Allow switching directory. |
TracingLayer | Tracing for every operations. |
Optional features
Layers
layers-all
: Enable all layers support.layers-metrics
: Enable operator metrics support.layers-tracing
: Enable operator tracing support.
Services
services-ftp
: Enable ftp service support.services-hdfs
: Enable hdfs service support.services-moka
: Enable moka service support.services-ipfs
: Enable ipfs service support.services-redis
: Enable redis service support.
Dependencies features
compress
: Enable object decompress read support.
Examples
More examples could be found at https://opendal.databend.rs/examples/index.html
use anyhow::Result;
use backon::ExponentialBackoff;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::layers::LoggingLayer;
use opendal::layers::RetryLayer;
use opendal::Object;
use opendal::ObjectEntry;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::ObjectStreamer;
use opendal::Operator;
use opendal::Scheme;
#[tokio::main]
async fn main() -> Result<()> {
// Init a memory operator
let op = Operator::from_env(Scheme::Memory)?
// Init with logging layer enabled.
.layer(LoggingLayer)
// Init with retry layer enabled.
.layer(RetryLayer::new(ExponentialBackoff::default()));
// Create object handler.
let o = op.object("test_file");
// Write data
o.write("Hello, World!").await?;
// Read data
let bs = o.read().await?;
// Fetch metadata
let name = o.name();
let path = o.path();
let meta = o.metadata().await?;
let mode = meta.mode();
let length = meta.content_length();
// Delete
o.delete().await?;
// Readdir
let o = op.object("test_dir/");
let mut ds = o.list().await?;
while let Some(entry) = ds.try_next().await? {
let path = entry.path();
let mode = entry.mode();
}
Ok(())
}
Modules
Providing adapters and its implementations.
http_util contains the util types and functions that used across OpenDAL.
io_util Providing IO utils like
into_sink
, into_stream
.Providing Layer trait and its implementations.
Providing specific services support.
Structs
Metadata for accessor, users can use this metadata to get information of underlying backend.
BatchOperator is used to take batch operations like walk_dir and remove_all, should
be constructed by
Operator::batch()
.Handler for all object related operations.
ObjectEntry is returned by
ObjectStream
or ObjectIterate
during object list.Metadata carries all object metadata.
ObjectMultipart represent an ongoing multipart upload.
ObjectPart is generated by
write_multipart
operation, carries
required information for complete_multipart
.User-facing APIs for object and object streams.
Enums
AccessorCapability describes accessor’s advanced capability.
ObjectMode represents the corresponding object’s mode.
Services that OpenDAL supports
Traits
Underlying trait of all backends for implementors.
BlockingBytesRead represents a blocking reader of bytes.
BytesRead represents a reader of bytes.
BytesSink represents a sink of bytes.
BytesStream represents a stream of bytes.
BytesWrite represents a writer of bytes.
Layer is used to intercept the operations on the underlying storage.
Type Definitions
BlockingBytesReader is a boxed dyn
BlockingBytesRead
.BytesReader is a boxed dyn
BytesRead
.BytesWriter is a boxed dyn
BytesWrite
.ObjectIterator is a boxed dyn
ObjectIterate
ObjectStreamer is a boxed dyn
ObjectStream