Crate nydus_storage

Chunked blob storage service to support Rafs filesystem.

The Rafs filesystem is blob based filesystem with chunk deduplication. A Rafs filesystem is composed up of a metadata blob and zero or more data blobs. A blob is just a plain object storage containing data chunks. Data chunks may be compressed, encrypted and deduplicated by content digest value. When Rafs file is used for container images, Rafs metadata blob contains all filesystem metadatas, such as directory, file name, permission etc. Actually file contents are split into chunks and stored into data blobs. Rafs may built one data blob for each container image layer or build a single data blob for the whole image, according to building options.

The nydus-storage crate is used to manage and access chunked blobs for Rafs filesystem, which contains three layers:

• Backend: access raw blob objects on remote storage backends.
• Cache: cache remote blob contents onto local storage in forms optimized for performance.
• Device: public APIs for chunked blobs

There are several core abstractions provided by the public APIs:

• BlobInfo: provides information about blobs, which is typically constructed from the blob array in Rafs filesystem metadata.
• BlobDevice: provides access to all blobs of a Rafs filesystem, which is constructed from an array of BlobInfo objects.
• BlobChunkInfo: provides information about a data chunk, which is loaded from Rafs metadata.
• BlobIoDesc: a blob IO descriptor, containing information for a continuous IO range within a chunk.
• BlobIoVec: a scatter/gather list for blob IO operation, containing one or more blob IO descriptors

To read data from the Rafs filesystem, the Rafs filesystem driver will prepare a BlobIoVec object and submit it to the corresponding BlobDevice object to actually execute the IO operations.

Modules

• backend
  Storage backends to read blob data from Registry, OSS, disk, file system etc.
• cache
  A blob cache layer over storage backend to improve performance.
• device
  Blob Storage Public Service APIs
• factory
  Factory to create blob cache objects for blobs.
• meta
  Generate, manage and access blob meta information for RAFS v6 data blobs.
• utils
  Utility helpers to supprt the storage subsystem.

Enums

• StorageError
  Error codes related to storage subsystem.

Constants

• RAFS_BATCH_SIZE_TO_GAP_SHIFT
  Generate maximum gap between chunks from merging size.
• RAFS_DEFAULT_CHUNK_SIZE
  Default blob chunk size.
• RAFS_MAX_CHUNKS_PER_BLOB
  Maximum numbers of chunk per data blob
• RAFS_MAX_CHUNK_SIZE
  Maximum blob chunk size, 16MB.

Type Definitions

• StorageResult
  Specialized std::result::Result for storage subsystem.