Expand description
OpenDAL is the Open Data Access Layer that connect the whole world together.
Supported Services
Services | Description |
---|---|
azblob | Azure Storage Blob services. |
fs | POSIX alike file system. |
gcs | Google Cloud Storage service. |
hdfs | Hadoop Distributed File System(HDFS). |
http | HTTP read-only backend. |
memory | In memory backend support. |
s3 | AWS S3 alike services. |
Optional features
Layers
layers-retry
: Enable operator retry support.layers-metrics
: Enable operator metrics support.layers-tracing
: Enable operator tracing support.
Services
services-hdfs
: Enable hdfs service support.services-http
: Enable http service support.
Dependencies features
compress
: Enable object decompress read support.rustls
: Use rustls instead openssl for https connectionserde
: Implement serde::{Serialize,Deserialize} for ObjectMetadata.
Example
use anyhow::Result;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::DirEntry;
use opendal::DirStreamer;
use opendal::Object;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::Operator;
use opendal::Scheme;
#[tokio::main]
async fn main() -> Result<()> {
// Init Operator
let op = Operator::from_env(Scheme::Fs)?;
// Create object handler.
let o = op.object("test_file");
// Write data info object;
o.write("Hello, World!").await?;
// Read data from object;
let bs = o.read().await?;
// Read range from object;
let bs = o.range_read(1..=11).await?;
// Get object's path
let name = o.name();
let path = o.path();
// Fetch more meta about object.
let meta = o.metadata().await?;
let mode = meta.mode();
let length = meta.content_length();
let content_md5 = meta.content_md5();
let etag = meta.etag();
// Delete object.
o.delete().await?;
// List dir object.
let o = op.object("test_dir/");
let mut ds = o.list().await?;
while let Some(entry) = ds.try_next().await? {
let path = entry.path();
let mode = entry.mode();
}
Ok(())
}
Modules
Providing IO utils like into_sink
, into_stream
.
Providing Layer trait and its implementations.
Providing specific services support.
Structs
Metadata for accessor, users can use this metadata to get information of underlying backend.
BatchOperator is used to take batch operations like walk_dir and remove_all, should
be constructed by Operator::batch()
.
Handler for all object related operations.
Metadata carries all object metadata.
ObjectMultipart represent an ongoing multipart upload.
ObjectPart is generated by write_multipart
operation, carries
required information for complete_multipart
.
User-facing APIs for object and object streams.
Enums
ObjectMode represents the corresponding object’s mode.
Services that OpenDAL supports
Traits
Underlying trait of all backends for implementors.
BytesRead represents a reader of bytes.
BytesSink represents a sink of bytes.
BytesStream represents a stream of bytes.
BytesWrite represents a writer of bytes.
DirStream represents a stream of Dir.
Layer is used to intercept the operations on the underlying storage.
Type Definitions
BytesReader is a boxed dyn BytesRead
.
BytesWriter is a boxed dyn BytesWrite
.
DirStreamer is a boxed dyn DirStream