Crate mediagit_storage 

Storage abstraction layer for MediaGit

This crate provides a unified, asynchronous storage interface that supports multiple backends:

• Local filesystem (via mediagit-local-storage)
• AWS S3
• Azure Blob Storage
• Google Cloud Storage
• MinIO / S3-compatible
• Backblaze B2 / DigitalOcean Spaces

Architecture

The StorageBackend trait defines a minimal but complete interface for object storage operations, allowing implementations to handle various storage systems transparently.

Core Concepts

• Keys: Unique identifiers for stored objects (strings, typically hierarchical like file paths)
• Objects: Arbitrary binary data associated with a key
• Prefixes: String prefixes used for listing and organization (similar to S3 object prefixes)

Features

• Async-first: All operations are async using tokio for non-blocking I/O
• Thread-safe: All implementations must be Send + Sync for safe concurrent use
• Debuggable: All implementations must implement Debug
• Error handling: Uses anyhow::Result for ergonomic error management

Examples

Using the mock backend for testing:

use mediagit_storage::{StorageBackend, mock::MockBackend};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create an in-memory backend for testing
    let storage = MockBackend::new();

    // Store data
    storage.put("documents/resume.pdf", b"PDF content").await?;

    // Retrieve data
    let data = storage.get("documents/resume.pdf").await?;
    assert_eq!(data, b"PDF content");

    // Check existence
    if storage.exists("documents/resume.pdf").await? {
        println!("File exists");
    }

    // List objects with prefix
    let documents = storage.list_objects("documents/").await?;
    println!("Found {} documents", documents.len());

    // Delete object
    storage.delete("documents/resume.pdf").await?;

    Ok(())
}

Implementation Guide

When implementing StorageBackend:

1. Use #[async_trait] macro on your impl block
2. Return anyhow::Result<T> for all operations
3. Ensure your type implements Send + Sync + Debug
4. Handle empty keys gracefully (typically return an error)
5. List operations should return sorted results for consistency
6. Deleting non-existent objects should succeed (idempotent)

Error Handling

While the trait uses anyhow::Result, consider using the StorageError enum in error.rs for more structured error information:

use mediagit_storage::error::{StorageError, StorageResult};

fn validate_key(key: &str) -> StorageResult<()> {
    if key.is_empty() {
        Err(StorageError::invalid_key("key cannot be empty"))
    } else {
        Ok(())
    }
}

Re-exports

pub use b2_spaces::B2SpacesBackend;

pub use error::StorageError;

pub use error::StorageResult;

pub use local::LocalBackend;

pub use minio::MinIOBackend;

pub use s3::S3Backend;

Modules

b2_spaces

Backblaze B2 & DigitalOcean Spaces backend

cache

LRU cache implementation for object database

error

Storage error types and utilities

local

Local filesystem storage backend

minio

MinIO & S3-compatible storage backend

mock

In-memory mock storage backend for testing

s3

AWS S3 storage backend implementation

Traits

StorageBackend

Storage backend trait for object storage operations