Module opendal::docs::comparisons::vs_object_store

OpenDAL vs object_store

NOTE: This document is written by OpenDAL’s maintainers and not reviewed by object_store’s maintainers. So it could not be very objective.

About object_store

object_store is

A focused, easy to use, idiomatic, high performance, async object store library interacting with object stores.

It was initially developed for InfluxDB IOx and later split out and donated to Apache Arrow.

Similarities

Language

Yes, of course. Both opendal and object_store are developed in Rust, a language empowering everyone to build reliable and efficient software.

License

Both opendal and object_store are licensed under Apache 2.0.

Domain

Both opendal and object_store can be used to access data stored on object storage services. The primary users of those projects are both cloud-native databases too:

• opendal is mainly used by databend: A modern Elasticity and Performance cloud data warehouse
• object_store is mainly used by:
  • datafusion: Apache Arrow DataFusion SQL Query Engine
  • Influxdb IOx: The new core of InfluxDB is written in Rust on top of Apache Arrow.

Differences

Owner

object_store is a part of Apache Arrow which means it’s hosted and maintained by Apache Software Foundation.

opendal is now hosted and maintained by Datafuse Labs, a new start-up focusing on building data cloud.

opendal has a plan to be donated to ASF. We are at the very early stage of preparing the donation. Welcome any suggestions and help!

Vision

opendal is Open Data Access Layer that accesses data freely, painlessly, and efficiently. object_store is more focused on async object store support.

You will see the different visions lead to very different routes.

Design

object_store exposed a trait called ObjectStore to users.

Users need to build a dyn ObjectStore and operate on it directly:

let object_store: Arc<dyn ObjectStore> = Arc::new(get_object_store());
let path: Path = "data/file01.parquet".try_into().unwrap();
let stream = object_store
    .get(&path)
    .await
    .unwrap()
    .into_stream();

opendal has a similar trait called Accessor

But opendal don’t expose this trait to end users directly. Instead, opendal expose a new struct called Operator and builds public API on it.

let op: Operator = Operator::from_env(Scheme::S3)?;
let r = op.object("data/file01.parquet").reader().await.unwrap();

Interception

Both object_store and opendal provide a mechanism to intercept operations.

object_store called Adapters:

let object_store = ThrottledStore::new(get_object_store(), ThrottleConfig::default())

opendal called Layer:

let op = op.layer(TracingLayer).layer(MetricsLayer);

At the time of writing:

object_store (v0.5.0) supports:

• ThrottleStore: Rate Throttling
• LimitStore: Concurrent Request Limit

opendal supports:

• ImmutableIndexLayer: immutable in-memory index.
• LoggingLayer: logging.
• MetadataCacheLayer: metadata cache.
• ContentCacheLayer: content data cache.
• MetricsLayer: metrics
• RetryLayer: retry
• SubdirLayer: Allow switch directory without changing original operator.
• TracingLayer: tracing

Services

opendal and object_store have different visions, so they have other services support:

service	opendal	object_store
azblob	Y	Y
fs	Y	Y
ftp	Y	N
gcs	Y	Y
hdfs	Y	Y (via datafusion-objectstore-hdfs)
http	Y (read only)	N
ipfs	Y (read only)	N
ipmfs	Y	N
memory	Y	Y
obs	Y	N
s3	Y	Y

opendal has an idea called AccessorCapability, so it’s services may have different capability sets. For example, opendal’s http and ipfs are read only.

Features

opendal and object_store have different visions, so they have different feature sets:

opendal	object_store	notes
metadata	-	get some metadata from underlying storage
create	put	-
read	get	-
read	get_range	-
-	get_ranges	opendal doesn’t support read multiple ranges
write	put	-
stat	head	-
delete	delete	-
-	list	opendal doesn’t support list with prefix
list	list_with_delimiter	-
-	copy	-
-	copy_if_not_exists	-
-	rename	-
-	rename_if_not_exists	-
presign	-	get a presign URL of object
multipart	multipart	both support, but API is different
blocking	-	opendal supports blocking API

Demo show

The most straightforward complete demo how to read a file from s3:

opendal

let mut builder = s3::Builder::default();
builder.bucket("example");
builder.access_key_id("access_key_id");
builder.secret_access_key("secret_access_key");

let store = Operator::new(builder.build()?);
let r = store.object("data.parquet").reader().await?;

object_store

let mut builder = AmazonS3Builder::new()
.with_bucket_name("example")
.with_access_key_id("access_key_id")
.with_secret_access_key("secret_access_key");

let store = Arc::new(builder.build()?);
let path: Path = "data.parquet".try_into().unwrap();
let stream = store.get(&path).await()?.into_stream();