Module replicator 

SQLite WAL-based replication to S3

Provides automatic backup and restore of SQLite databases to S3, using WAL (Write-Ahead Logging) for incremental replication. This enables crash-tolerant persistence and cross-node database restoration.

Features

• WAL Monitoring: Detects database changes via WAL file modifications
• Network Tolerance: Local write cache buffers changes during network outages
• Automatic Snapshots: Periodic full database snapshots with configurable intervals
• S3 Backend: Stores snapshots and WAL segments in S3 with zstd compression
• Auto-Restore: Automatically restores from S3 on startup if local DB is missing

Architecture

SQLite DB (WAL mode)
       |
       v
WAL Monitor (notify) --> Write Cache --> S3 Backend
       |                     |               |
       v                     v               v
  Frame Detection      FIFO Queue    Upload/Download

Example

use zlayer_storage::replicator::{SqliteReplicator, SqliteReplicatorConfig};
use zlayer_storage::config::LayerStorageConfig;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let replicator_config = SqliteReplicatorConfig {
        db_path: "/var/lib/myapp/data.db".into(),
        s3_bucket: "my-bucket".to_string(),
        s3_prefix: "sqlite-backups/myapp/".to_string(),
        cache_dir: "/tmp/zlayer-replicator/cache".into(),
        max_cache_size: 100 * 1024 * 1024, // 100MB
        auto_restore: true,
        snapshot_interval_secs: 3600, // 1 hour
    };

    let s3_config = LayerStorageConfig::new("my-bucket");
    let replicator = SqliteReplicator::new(replicator_config, &s3_config).await?;

    // Optionally restore from S3 on startup
    replicator.restore().await?;

    // Start background replication
    replicator.start().await?;

    // ... run your application ...

    // Graceful shutdown
    replicator.flush().await?;
    Ok(())
}

Re-exports

pub use crate::config::LayerStorageConfig;

pub use crate::config::SqliteReplicatorConfig;

Structs

CacheEntry

A cached WAL segment entry

ReplicationMetadata

Replication metadata stored in S3

ReplicationStatus

Current replication status

SqliteReplicator

SQLite WAL-based replicator to S3

WalEvent

WAL file change event