OpenDAL
OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.
- Documentation: stable | main
- Release notes
Services
- ftp: FTP and FTPS
- http: HTTP read-only services
- sftp: SFTP services being worked on
- webdav: WebDAV Service
- azblob: Azure Storage Blob services
- cos: Tencent Cloud Object Storage services
- gcs: Google Cloud Storage Service
- obs: Huawei Cloud Object Storage Service (OBS)
- oss: Aliyun Object Storage Service (OSS)
- s3: AWS S3 alike services
- supabase: Supabase Storage Service being worked on
- wasabi: Wasabi Cloud Storage
- fs: POSIX alike file system
- azdls: Azure Data Lake Storage Gen2 services (As known as abfs)
- hdfs: Hadoop Distributed File System(HDFS)
- ipfs: InterPlanetary File System HTTP Gateway
- ipmfs: InterPlanetary File System MFS API being worked on
- webhdfs: WebHDFS Service
- gdrive: Google Drive being worked on
- onedrive: OneDrive being worked on
- cacache: cacache backend
- dashmap: dashmap backend
- memory: In memory backend
- persy: persy backend
- redis: Redis services
- rocksdb: RocksDB services
- sled: sled backend
- redb: redb backend
- tikv: tikv backend
- atomicserver: Atomicserver services
- ghac: GitHub Action Cache Service
- memcached: Memcached service
- mini_moka: Mini Moka backend
- moka: Moka backend
- vercel_artifacts: Vercel Remote Caching Service being worked on
Welcome to add any services that are not currently supported here.
Features
Access data freely
- Access different storage services in the same way
- Behavior tests for all services
Access data painlessly
- 100% documents covered
- Powerful
Layers
- Automatic retry support
- Full observability: logging, tracing, metrics.
- Native chaos testing
Access data efficiently
- Zero cost: Maps to API calls directly
- Best effort: Automatically selects best read/seek/next based on services
- Avoid extra calls: Reuses metadata when possible
Quickstart
use Result;
use LoggingLayer;
use services;
use Operator;
async
Examples
The examples are available at here.
Contributing
Check out the CONTRIBUTING guide for more details on getting started with contributing to this project.
License
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0