Crate nydus_storage

Crate nydus_storage 

Source
Expand description

Chunked blob storage service to support Rafs filesystem.

The Rafs filesystem is blob based filesystem with chunk deduplication. A Rafs filesystem is composed up of a metadata blob and zero or more data blobs. A blob is just a plain object storage containing data chunks. Data chunks may be compressed, encrypted and deduplicated by content digest value. When Rafs file is used for container images, Rafs metadata blob contains all filesystem metadatas, such as directory, file name, permission etc. Actually file contents are split into chunks and stored into data blobs. Rafs may build one data blob for each container image layer or build a single data blob for the whole image, according to building options.

The nydus-storage crate is used to manage and access chunked blobs for Rafs filesystem, which contains three layers:

  • Backend: access raw blob objects on remote storage backends.
  • Cache: cache remote blob contents onto local storage in forms optimized for performance.
  • Device: public APIs for chunked blobs

There are several core abstractions provided by the public APIs:

  • BlobInfo: provides information about blobs, which is typically constructed from the blob array in Rafs filesystem metadata.
  • BlobDevice: provides access to all blobs of a Rafs filesystem, which is constructed from an array of BlobInfo objects.
  • BlobChunkInfo: provides information about a data chunk, which is loaded from Rafs metadata.
  • BlobIoDesc: a blob IO descriptor, containing information for a continuous IO range within a chunk.
  • BlobIoVec: a scatter/gather list for blob IO operation, containing one or more blob IO descriptors

To read data from the Rafs filesystem, the Rafs filesystem driver will prepare a BlobIoVec object and submit it to the corresponding BlobDevice object to actually execute the IO operations.

Modules§

backend
Storage backends to read blob data from Registry, OSS, disk, file system etc.
cache
A blob cache layer over storage backend to improve performance.
device
Blob Storage Public Service APIs
factory
Factory to create blob cache objects for blobs.
meta
Generate, manage and access blob meta information for RAFS v6 data blobs.
utils
Utility helpers to support the storage subsystem.

Enums§

StorageError
Error codes related to storage subsystem.

Constants§

RAFS_BATCH_SIZE_TO_GAP_SHIFT
Generate maximum gap between chunks from merging size.
RAFS_DEFAULT_CHUNK_SIZE
Default blob chunk size.
RAFS_MAX_CHUNKS_PER_BLOB
Maximum numbers of chunk per data blob
RAFS_MAX_CHUNK_SIZE
Maximum blob chunk size, 16MB.

Type Aliases§

StorageResult
Specialized std::result::Result for storage subsystem.