zlayer-storage

S3-backed container layer persistence with crash-tolerant uploads and SQLite replication.

Features

• Layer Persistence - Persist container OverlayFS upper layers to S3
• Crash-Tolerant Uploads - Multipart S3 uploads with resume capability
• SQLite State Tracking - Track sync state in SQLite (via SQLx)
• SQLite S3 Replication - WAL-based SQLite replication to S3 with auto-restore
• Compression - Zstd compression for efficient storage

Installation

Add to your Cargo.toml:

[dependencies]
zlayer-storage = "0.8"

Usage

Layer Sync Manager

Persist container layers to S3:

use zlayer_storage::{LayerSyncManager, LayerStorageConfig};

let config = LayerStorageConfig {
    bucket: "my-bucket".to_string(),
    prefix: "layers/".to_string(),
    region: Some("us-west-2".to_string()),
    endpoint_url: None, // Or custom S3-compatible endpoint
    staging_dir: "/var/lib/zlayer/staging".into(),
    state_db_path: "/var/lib/zlayer/layer-state.sqlite".into(),
    ..Default::default()
};

let manager = LayerSyncManager::new(config).await?;

// Sync a container's upper layer to S3
let snapshot = manager.sync_layer(&container_id, "/path/to/upper").await?;

// Restore a layer from S3
let restored = manager.restore_layer(&container_id, "/path/to/restore").await?;

SQLite S3 Replicator

Replicate any SQLite database to S3 with WAL-based incremental backups:

use zlayer_storage::{SqliteReplicator, SqliteReplicatorConfig, LayerStorageConfig};

let replicator_config = SqliteReplicatorConfig {
    db_path: "/var/lib/zlayer/state.sqlite".into(),
    s3_bucket: "my-bucket".to_string(),
    s3_prefix: "sqlite-backups/".to_string(),
    cache_dir: "/var/lib/zlayer/replicator-cache".into(),
    max_cache_size: 100 * 1024 * 1024, // 100MB
    auto_restore: true,
    snapshot_interval_secs: 3600, // 1 hour
};

let s3_config = LayerStorageConfig { /* ... */ };
let replicator = SqliteReplicator::new(replicator_config, &s3_config).await?;

// Start background replication
replicator.start().await?;

// Check status
let status = replicator.status();
println!("Last sync: {:?}", status.last_sync);

// Force sync before shutdown
replicator.flush().await?;

Auto-Restore

If auto_restore: true and the local database is missing, it will be automatically restored from S3:

// Database will be restored from S3 if missing
let replicator = SqliteReplicator::new(config, &s3_config).await?;

// Or manually restore
replicator.restore().await?;

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      zlayer-storage                          │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌────────────────────┐    ┌────────────────────────────┐  │
│  │  LayerSyncManager  │    │    SqliteReplicator        │  │
│  │  - Multipart upload│    │    - WAL monitoring        │  │
│  │  - Resume support  │    │    - Local write cache     │  │
│  │  - Restore layers  │    │    - Async S3 shipping     │  │
│  └─────────┬──────────┘    └─────────────┬──────────────┘  │
│            │                              │                  │
│  ┌─────────▼──────────────────────────────▼──────────────┐ │
│  │                 SQLite (SQLx Pool)                     │ │
│  │                 - WAL mode                             │ │
│  │                 - State persistence                    │ │
│  └────────────────────────────────────────────────────────┘ │
│                              │                               │
└──────────────────────────────┼───────────────────────────────┘
                               │ async
               ┌───────────────▼───────────────┐
               │           S3 Bucket           │
               │  - Layer tarballs (.tar.zst)  │
               │  - SQLite snapshots           │
               │  - WAL segments               │
               │  - Metadata                   │
               └───────────────────────────────┘

S3 Key Structure

Layer Storage

s3://{bucket}/{prefix}/
├── {digest}.tar.zst      # Compressed layer tarball
└── {digest}.meta.json    # Layer metadata

SQLite Replication

s3://{bucket}/{prefix}/
├── snapshots/
│   └── {timestamp}.sqlite.zst   # Full database snapshots
├── wal/
│   └── {sequence:020}.wal.zst   # WAL segments
└── metadata.json                 # Replication state

Configuration

LayerStorageConfig

SqliteReplicatorConfig

S3-Compatible Services

Works with any S3-compatible storage:

// MinIO
let config = LayerStorageConfig {
    endpoint_url: Some("http://minio:9000".to_string()),
    ..Default::default()
};

// Cloudflare R2
let config = LayerStorageConfig {
    endpoint_url: Some("https://{account_id}.r2.cloudflarestorage.com".to_string()),
    ..Default::default()
};

// Backblaze B2
let config = LayerStorageConfig {
    endpoint_url: Some("https://s3.{region}.backblazeb2.com".to_string()),
    ..Default::default()
};

License

MIT - See LICENSE for details.