Struct opendal::Operator

source ·

pub struct Operator { /* private fields */ }

Expand description

Operator is the entry for all public async APIs.

Developer should manipulate the data from storage service through Operator only by right.

We will usually do some general checks and data transformations in this layer, like normalizing path from input, checking whether the path refers to one file or one directory, and so on. Read concepts for more about Operator.

Examples

Read more backend init examples in services

use opendal::services::Fs;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
    // Create fs backend builder.
    let mut builder = Fs::default();
    // Set the root for fs, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root("/tmp");

    // Build an `Operator` to start operating the storage.
    let _: Operator = Operator::new(builder)?.finish();

    Ok(())
}

Implementations§

source §

impl Operator

Operator basic API.

source

pub fn limit(&self) -> usize

Get current operator’s limit. Limit is usually the maximum size of data that operator will handle in one operation.

source

pub fn with_limit(&self, limit: usize) -> Self

Specify the batch limit.

Default: 1000

source

pub fn info(&self) -> OperatorInfo

Get information of underlying accessor.

Examples

use opendal::Operator;

let info = op.info();

source

pub fn blocking(&self) -> BlockingOperator

Create a new blocking operator.

This operation is nearly no cost.

source §

impl Operator

Operator async API.

source

pub async fn check(&self) -> Result<()>

Check if this operator can work correctly.

We will send a list request to path and return any errors we met.

use opendal::Operator;

op.check().await?;

source

pub async fn stat(&self, path: &str) -> Result<Metadata>

Get current path’s metadata without cache directly.

Notes

Use stat if you:

Want to detect the outside changes of path.
Don’t want to read from cached metadata.

You may want to use metadata if you are working with entries returned by Lister. It’s highly possible that metadata you want has already been cached.

Examples

use opendal::ErrorKind;
if let Err(e) = op.stat("test").await {
    if e.kind() == ErrorKind::NotFound {
        println!("file not exist")
    }
}

source

pub fn stat_with(&self, path: &str) -> FutureStat ⓘ

Get current path’s metadata without cache directly with extra options.

Notes

Use stat if you:

Want to detect the outside changes of path.
Don’t want to read from cached metadata.

You may want to use metadata if you are working with entries returned by Lister. It’s highly possible that metadata you want has already been cached.

Examples

use opendal::ErrorKind;
if let Err(e) = op.stat_with("test").if_match("<etag>").await {
    if e.kind() == ErrorKind::NotFound {
        println!("file not exist")
    }
}

source

pub async fn metadata( &self, entry: &Entry, flags: impl Into<FlagSet<Metakey>> ) -> Result<Metadata>

Get current metadata with cache.

metadata will check the given query with already cached metadata first. And query from storage if not found.

Notes

Use metadata if you are working with entries returned by Lister. It’s highly possible that metadata you want has already been cached.

You may want to use stat, if you:

Want to detect the outside changes of path.
Don’t want to read from cached metadata.

Behavior

Visiting not fetched metadata will lead to panic in debug build. It must be a bug, please fix it instead.

Examples

Query already cached metadata

By querying metadata with None, we can only query in-memory metadata cache. In this way, we can make sure that no API call will be sent.

use opendal::Entry;
let meta = op.metadata(&entry, None).await?;
// content length COULD be correct.
let _ = meta.content_length();
// etag COULD be correct.
let _ = meta.etag();

Query content length and content type

use opendal::Entry;
use opendal::Metakey;

let meta = op
    .metadata(&entry, Metakey::ContentLength | Metakey::ContentType)
    .await?;
// content length MUST be correct.
let _ = meta.content_length();
// etag COULD be correct.
let _ = meta.etag();

Query all metadata

By querying metadata with Complete, we can make sure that we have fetched all metadata of this entry.

use opendal::Entry;
use opendal::Metakey;

let meta = op.metadata(&entry, Metakey::Complete).await?;
// content length MUST be correct.
let _ = meta.content_length();
// etag MUST be correct.
let _ = meta.etag();

source

pub async fn is_exist(&self, path: &str) -> Result<bool>

Check if this path exists or not.

Example

use anyhow::Result;
use futures::io;
use opendal::Operator;

#[tokio::main]
async fn test(op: Operator) -> Result<()> {
    let _ = op.is_exist("test").await?;

    Ok(())
}

source

pub async fn create_dir(&self, path: &str) -> Result<()>

Create a dir at given path.

Notes

To indicate that a path is a directory, it is compulsory to include a trailing / in the path. Failure to do so may result in NotADirectory error being returned by OpenDAL.

Behavior

Create on existing dir will succeed.
Create dir is always recursive, works like mkdir -p

Examples

op.create_dir("path/to/dir/").await?;

source

pub async fn read(&self, path: &str) -> Result<Vec<u8>>

Read the whole path into a bytes.

This function will allocate a new bytes internally. For more precise memory control or reading data lazily, please use Operator::reader

Examples

let bs = op.read("path/to/file").await?;

source

pub fn read_with(&self, path: &str) -> FutureRead ⓘ

Read the whole path into a bytes with extra options.

This function will allocate a new bytes internally. For more precise memory control or reading data lazily, please use Operator::reader

Examples

let bs = op.read_with("path/to/file").await?;

source

pub async fn range_read( &self, path: &str, range: impl RangeBounds<u64> ) -> Result<Vec<u8>>

Read the specified range of path into a bytes.

This function will allocate a new bytes internally. For more precise memory control or reading data lazily, please use Operator::range_reader

Notes

The returning content’s length may be smaller than the range specified.

Examples

let bs = op.range_read("path/to/file", 1024..2048).await?;

source

pub async fn reader(&self, path: &str) -> Result<Reader>

Create a new reader which can read the whole path.

Examples

let r = op.reader("path/to/file").await?;

source

pub async fn range_reader( &self, path: &str, range: impl RangeBounds<u64> ) -> Result<Reader>

Create a new reader which can read the specified range.

Notes

The returning content’s length may be smaller than the range specified.

Examples

let r = op.range_reader("path/to/file", 1024..2048).await?;

source

pub fn reader_with(&self, path: &str) -> FutureReader ⓘ

Create a new reader with extra options

Examples

let r = op.reader_with("path/to/file").range((0..10)).await?;

source

pub async fn write(&self, path: &str, bs: impl Into<Bytes>) -> Result<()>

Write bytes into path.

Notes

Write will make sure all bytes has been written, or an error will be returned.

Examples

use bytes::Bytes;

op.write("path/to/file", vec![0; 4096]).await?;

source

pub async fn append(&self, path: &str, bs: impl Into<Bytes>) -> Result<()>

Append bytes into path.

Notes

Append will make sure all bytes has been written, or an error will be returned.
Append will create the file if it does not exist.
Append always write bytes to the end of the file.

Examples

use bytes::Bytes;

op.append("path/to/file", vec![0; 4096]).await?;

source

pub async fn copy(&self, from: &str, to: &str) -> Result<()>

Copy a file from from to to.

Notes

from and to must be a file.
to will be overwritten if it exists.
If from and to are the same, an IsSameFile error will occur.
copy is idempotent. For same from and to input, the result will be the same.

Examples


op.copy("path/to/file", "path/to/file2").await?;

source

pub async fn rename(&self, from: &str, to: &str) -> Result<()>

Rename a file from from to to.

Notes

from and to must be a file.
to will be overwritten if it exists.
If from and to are the same, an IsSameFile error will occur.

Examples


op.rename("path/to/file", "path/to/file2").await?;

source

pub async fn writer(&self, path: &str) -> Result<Writer>

Write multiple bytes into path.

Refer to Writer for more details.

Examples

use bytes::Bytes;

let mut w = op.writer("path/to/file").await?;
w.write(vec![0; 4096]).await?;
w.write(vec![1; 4096]).await?;
w.close().await?;

source

pub fn writer_with(&self, path: &str) -> FutureWriter ⓘ

Write multiple bytes into path with extra options.

Refer to Writer for more details.

Examples

use bytes::Bytes;

let mut w = op
    .writer_with("path/to/file")
    .content_type("application/octet-stream")
    .await?;
w.write(vec![0; 4096]).await?;
w.write(vec![1; 4096]).await?;
w.close().await?;

source

pub fn write_with(&self, path: &str, bs: impl Into<Bytes>) -> FutureWrite ⓘ

Write data with extra options.

Notes

Write will make sure all bytes has been written, or an error will be returned.

Examples

use bytes::Bytes;

let bs = b"hello, world!".to_vec();
let _ = op
    .write_with("path/to/file", bs)
    .content_type("text/plain")
    .await?;

source

pub async fn appender(&self, path: &str) -> Result<Appender>

Append multiple bytes into path.

Refer to Appender for more details.

Examples

use bytes::Bytes;

let mut a = op.appender("path/to/file").await?;
a.append(vec![0; 4096]).await?;
a.append(vec![1; 4096]).await?;
a.close().await?;

source

pub fn appender_with(&self, path: &str) -> FutureAppender ⓘ

Append multiple bytes into path with extra options.

Refer to Appender for more details.

Examples

use bytes::Bytes;

let mut a = op
    .appender_with("path/to/file")
    .content_type("application/octet-stream")
    .await?;
a.append(vec![0; 4096]).await?;
a.append(vec![1; 4096]).await?;
a.close().await?;

source

pub fn append_with(&self, path: &str, bs: impl Into<Bytes>) -> FutureAppend ⓘ

Append bytes with extra options.

Notes

Append will make sure all bytes has been written, or an error will be returned.
Append will create the file if it does not exist.
Append always write bytes to the end of the file.

Examples

use bytes::Bytes;

let bs = b"hello, world!".to_vec();
let _ = op
    .append_with("path/to/file", bs)
    .content_type("text/plain")
    .await?;

source

pub async fn delete(&self, path: &str) -> Result<()>

Delete the given path.

Notes

Deleting a file that does not exist won’t return errors.

Examples

op.delete("test").await?;

source

pub async fn remove(&self, paths: Vec<String>) -> Result<()>

Notes

If underlying services support delete in batch, we will use batch delete instead.

Examples

op.remove(vec!["abc".to_string(), "def".to_string()])
    .await?;

source

pub async fn remove_via( &self, input: impl Stream<Item = String> + Unpin ) -> Result<()>

remove will remove files via the given paths.

remove_via will remove files via the given stream.

We will delete by chunks with given batch limit on the stream.

Notes

If underlying services support delete in batch, we will use batch delete instead.

Examples

use futures::stream;
let stream = stream::iter(vec!["abc".to_string(), "def".to_string()]);
op.remove_via(stream).await?;

source

pub async fn remove_all(&self, path: &str) -> Result<()>

Remove the path and all nested dirs and files recursively.

Notes

If underlying services support delete in batch, we will use batch delete instead.

Examples

op.remove_all("path/to/dir").await?;

source

pub async fn list(&self, path: &str) -> Result<Lister>

List given path.

This function will create a new handle to list entries.

An error will be returned if given path doesn’t end with /.

Examples

use futures::TryStreamExt;
use opendal::EntryMode;
use opendal::Metakey;
use opendal::Operator;
let mut ds = op.list("path/to/dir/").await?;
while let Some(mut de) = ds.try_next().await? {
    let meta = op.metadata(&de, Metakey::Mode).await?;
    match meta.mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        EntryMode::Unknown => continue,
    }
}

source

pub fn list_with(&self, path: &str) -> FutureList ⓘ

List given path with OpList.

This function will create a new handle to list entries.

An error will be returned if given path doesn’t end with /.

Examples

List current dir

use futures::TryStreamExt;
use opendal::EntryMode;
use opendal::Metakey;
use opendal::Operator;
let mut ds = op
    .list_with("path/to/dir/")
    .limit(10)
    .start_after("start")
    .await?;
while let Some(mut de) = ds.try_next().await? {
    let meta = op.metadata(&de, Metakey::Mode).await?;
    match meta.mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        EntryMode::Unknown => continue,
    }
}

List all files recursively

We can use op.scan() as a shorter alias.

use futures::TryStreamExt;
use opendal::EntryMode;
use opendal::Metakey;
use opendal::Operator;
let mut ds = op.list_with("path/to/dir/").delimiter("").await?;
while let Some(mut de) = ds.try_next().await? {
    let meta = op.metadata(&de, Metakey::Mode).await?;
    match meta.mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        EntryMode::Unknown => continue,
    }
}

source

pub async fn scan(&self, path: &str) -> Result<Lister>

List dir in flat way.

Also, this function can be used to list a prefix.

An error will be returned if given path doesn’t end with /.

Notes

scan will not return the prefix itself.
scan is an alias of list_with(path).delimiter("")

Examples

use futures::TryStreamExt;
use opendal::EntryMode;
use opendal::Metakey;
use opendal::Operator;
let mut ds = op.scan("/path/to/dir/").await?;
while let Some(mut de) = ds.try_next().await? {
    let meta = op.metadata(&de, Metakey::Mode).await?;
    match meta.mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        EntryMode::Unknown => continue,
    }
}

source §

impl Operator

Operator presign API.

source

pub async fn presign_stat( &self, path: &str, expire: Duration ) -> Result<PresignedRequest>

Presign an operation for stat(head).

Example

use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

#[tokio::main]
async fn test(op: Operator) -> Result<()> {
    let signed_req = op.presign_stat("test",Duration::from_secs(3600)).await?;
    let req = http::Request::builder()
        .method(signed_req.method())
        .uri(signed_req.uri())
        .body(())?;

source

pub async fn presign_read( &self, path: &str, expire: Duration ) -> Result<PresignedRequest>

Presign an operation for read.

Example

use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

#[tokio::main]
async fn test(op: Operator) -> Result<()> {
    let signed_req = op.presign_read("test.txt", Duration::from_secs(3600)).await?;

signed_req.method(): GET
signed_req.uri(): https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
signed_req.headers(): { "host": "s3.amazonaws.com" }

We can download this file via curl or other tools without credentials:

curl "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>" -O /tmp/test.txt

source

pub fn presign_read_with( &self, path: &str, expire: Duration ) -> FuturePresignRead ⓘ

Presign an operation for read option described in OpenDAL rfc-1735.

You can pass OpRead to this method to specify the content disposition.

Example

use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

#[tokio::main]
async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_read_with("test.txt", Duration::from_secs(3600))
        .override_content_disposition("attachment; filename=\"othertext.txt\"")
        .await?;

source

pub async fn presign_write( &self, path: &str, expire: Duration ) -> Result<PresignedRequest>

Presign an operation for write.

Example

use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

#[tokio::main]
async fn test(op: Operator) -> Result<()> {
    let signed_req = op.presign_write("test.txt", Duration::from_secs(3600)).await?;

signed_req.method(): PUT
signed_req.uri(): https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
signed_req.headers(): { "host": "s3.amazonaws.com" }

We can upload file as this file via curl or other tools without credential:

curl -X PUT "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>" -d "Hello, World!"

source

pub fn presign_write_with( &self, path: &str, expire: Duration ) -> FuturePresignWrite ⓘ

Presign an operation for write with option described in OpenDAL rfc-0661

You can pass OpWrite to this method to specify the content length and content type.

Example

use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

#[tokio::main]
async fn test(op: Operator) -> Result<()> {
    let signed_req = op.presign_write_with("test", Duration::from_secs(3600))
                       .content_type("text/csv").await?;
    let req = http::Request::builder()
        .method(signed_req.method())
        .uri(signed_req.uri())
        .body(())?;

source §

impl Operator

Operator build API

Operator should be built via OperatorBuilder. We recommend to use Operator::new to get started:

use opendal::services::Fs;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
    // Create fs backend builder.
    let mut builder = Fs::default();
    // Set the root for fs, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root("/tmp");

    // Build an `Operator` to start operating the storage.
    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

source

pub fn new<B: Builder>(ab: B) -> Result<OperatorBuilder<impl Accessor>>

Create a new operator with input builder.

OpenDAL will call builder.build() internally, so we don’t need to import opendal::Builder trait.

Examples

Read more backend init examples in examples.

use opendal::services::Fs;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
    // Create fs backend builder.
    let mut builder = Fs::default();
    // Set the root for fs, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root("/tmp");

    // Build an `Operator` to start operating the storage.
    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

source

pub fn from_map<B: Builder>( map: HashMap<String, String> ) -> Result<OperatorBuilder<impl Accessor>>

Create a new operator from given map.

Notes

from_map is using static dispatch layers which is zero cost. via_map is using dynamic dispatch layers which has a bit runtime overhead with an extra vtable lookup and unable to inline. But from_map requires generic type parameter which is not always easy to be used.

Examples

use std::collections::HashMap;

use opendal::services::Fs;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
    let map = HashMap::from([
        // Set the root for fs, all operations will happen under this root.
        //
        // NOTE: the root must be absolute path.
        ("root".to_string(), "/tmp".to_string()),
    ]);

    // Build an `Operator` to start operating the storage.
    let op: Operator = Operator::from_map::<Fs>(map)?.finish();

    Ok(())
}

source

pub fn via_map(scheme: Scheme, map: HashMap<String, String>) -> Result<Operator>

Create a new operator from given scheme and map.

Notes

from_map is using static dispatch layers which is zero cost. via_map is using dynamic dispatch layers which has a bit runtime overhead with an extra vtable lookup and unable to inline. But from_map requires generic type parameter which is not always easy to be used.

Examples

use std::collections::HashMap;

use opendal::Operator;
use opendal::Scheme;
#[tokio::main]
async fn main() -> Result<()> {
    let map = HashMap::from([
        // Set the root for fs, all operations will happen under this root.
        //
        // NOTE: the root must be absolute path.
        ("root".to_string(), "/tmp".to_string()),
    ]);

    // Build an `Operator` to start operating the storage.
    let op: Operator = Operator::via_map(Scheme::Fs, map)?;

    Ok(())
}

source

pub fn layer<L: Layer<FusedAccessor>>(self, layer: L) -> Self

Create a new layer with dynamic dispatch.

Notes

OperatorBuilder::layer() is using static dispatch which is zero cost. Operator::layer() is using dynamic dispatch which has a bit runtime overhead with an extra vtable lookup and unable to inline.

It’s always recommended to use OperatorBuilder::layer() instead.

Examples

use opendal::layers::LoggingLayer;
use opendal::services::Fs;
use opendal::Operator;

let op = Operator::new(Fs::default())?.finish();
let op = op.layer(LoggingLayer::default());
// All operations will go through the new_layer
let _ = op.read("test_file").await?;