Hyperfile - aka Random Read Write Object Format

A file-like interface abstraction layer provides random read and write capabilities for S3 (object storage).

Overview

S3 (object storage) provides high availability and high durability, on-demand billing, low-cost, and elastic storage space for various data.

Immutability comes as S3's nature(despite S3 Express One Zone now support append), which is a key distinguishes between object storage and file storage, for most scenarios, immutability is acceptable.

However, in scenarios such as log collection, disk image file, backup, etc., the underlying storage system needs to provide append writes, random writes, sparse capability and other characteristics.

Hyperfile bridges the gap between file storage and object storage by providing a POSIX-like file API for access data direct on S3.

It enables applications to perform random read, write and truncate in aligned with block-boundary by implements a abstraction layer to reconstructing data index with Random Read Write Object Format.

With Hyperfile, application can perform any form of random read write directly on S3.

Hyperfile runs entirely in user space.

Terminology

• Staging - where data is stored in random read write format.
• Meta block - index of file address space organized in B+tree.
• Data block - block chunks of file content.
• Segment - container of all modified(dirty) data, including header, meta blocks and data blocks.

Concept

Hyperfile organized persistent data on S3 in random read write (log-structured) format by reconstructing data index using B+tree structure.

When flush happens, all modified(dirty) data blocks together with it's metadata blocks are collected and constructed into on-disk segment format.

Copy-on-Write (CoW)

Hyperfile optimize storage space by using copy-on-write space allocation strategy, deferring data space allocation until first modification in block level granularity.

Continuous Snapshot

Hyperfile never overwrite persisted data on S3, every new written segment will be assigned with a global sequentially increasing id identified as checkpoint.

Log-Structured Characteristics

Persistent data organized in Log-Structured style, which means, Hyperfile has the native capability to implement continous snapshot, time travel and point-in-time recovery for better data protection.

Features

• B+tree data structure based on btree-ondisk.
• Read/write data direct on S3 with file-like API.
• Block-level API to borrow a cached block for reading or in-place modification, for callers that use hyperfile as block storage rather than as a file — see Block API.
• Read-only queries for what reading a range would cost and where each block sits, for callers deciding whether a file is worth rearranging — see Seeing where blocks are.
• Support S3 and S3 Express One Zone as staging area.
• Optional: WAL(Write-Ahead-Log) support to perserve durability of unflushed data.
• Optional: data blocks cache and meta blocks cache on local disk to accelerate read path.

S3 permissions

On a general purpose bucket, reading a container at s3://bucket/P needs s3:GetObject on P/* and s3:ListBucket limited to the prefix P/*; writing also needs s3:PutObject. s3:ListBucket is not optional: without it S3 answers 403 instead of 404 for a missing object, and create and open fail. unlink and the WAL also need s3:DeleteObject. On a directory bucket, a ReadOnly or ReadWrite session from s3express:CreateSession. Every write is conditional, so a bucket policy enforcing conditional writes is compatible. See S3 permissions.

Consistency

Hyperfile guaranteed to be consistent after each successful flush.

Limitations

• Read/write path IO amplify.
• Extra storage space for metadata blocks.
• On-disk data layout changed (in contrast of a real S3 object).
• Segments prune/compaction effort.
• An unlink interrupted after the inode is deleted leaves objects that neither unlink nor the cleaner can reclaim — see Known gap: orphaned objects.

Examples

See examples for how to use.

Under Developement

:warning: This library is currently under developement and is NOT recommended for production.

Credits

In loving memory of my father, Mr. Dai Wenhua, Who bought me my first computer.

License

This project is licensed under the Apache-2.0 License.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.