Netabase Store

A type-safe, multi-backend key-value storage library for Rust with support for native (Sled, Redb) and WASM (IndexedDB) environments.

⚠️ Early Development: This crate is still in early development and will change frequently as it stabilizes. It is not advised to use this in a production environment until it stabilizes.

Installation

Add to your Cargo.toml:

[dependencies]
netabase_store = "0.0.3"

# Required dependencies for macros to work
bincode = { version = "2.0", features = ["serde"] }
serde = { version = "1.0", features = ["derive"] }
strum = { version = "0.27.2", features = ["derive"] }
derive_more = { version = "2.0.1", features = ["from", "try_into", "into"] }
anyhow = "1.0"  # Optional, for error handling

# For WASM support
[target.'cfg(target_arch = "wasm32")'.dependencies]
netabase_store = { version = "0.0.2", default-features = false, features = ["wasm"] }

Feature Flags

• native (default): Enable Sled and Redb backends
• sled: Enable Sled backend only
• redb: Enable Redb backend only
• redb-zerocopy: Enable zero-copy Redb backend (high-performance variant)
• wasm: Enable IndexedDB backend for WASM
• libp2p: Enable libp2p integration
• record-store: Enable RecordStore trait (requires libp2p)

Quick Start

1. Define Your Schema

use netabase_store::netabase_definition_module;
use netabase_store::traits::model::NetabaseModelTrait;

#[netabase_definition_module(BlogDefinition, BlogKeys)]
pub mod blog_schema {
    use netabase_store::{NetabaseModel, netabase};

    #[derive(NetabaseModel, bincode::Encode, bincode::Decode, Clone, Debug)]
    #[netabase(BlogDefinition)]
    pub struct User {
        #[primary_key]
        pub id: u64,
        pub username: String,
        #[secondary_key]
        pub email: String,
    }

    #[derive(NetabaseModel, bincode::Encode, bincode::Decode, Clone, Debug)]
    #[netabase(BlogDefinition)]
    pub struct Post {
        #[primary_key]
        pub id: u64,
        pub title: String,
        pub content: String,
        #[secondary_key]
        pub author_id: u64,
    }
}

use blog_schema::*;

2. Create a Store with Unified Configuration API (Recommended)

The unified configuration system provides consistent backend initialization:

use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;
use netabase_store::databases::sled_store::SledStore;

fn main() -> anyhow::Result<()> {
    // Create configuration - same for all file-based backends!
    let config = FileConfig::builder()
        .path("my_app.db".into())
        .cache_size_mb(512)
        .build();

    // Initialize backend with BackendStore trait
    let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;

    // Switch backends by changing only the type - same config works!
    // let store = <RedbStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;
    // let store = <RedbStoreZeroCopy<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;

    // Open a tree for users - works identically across all backends
    let user_tree = store.open_tree::<User>();

    // Insert a user
    let user = User {
        id: 1,
        username: "alice".to_string(),
        email: "alice@example.com".to_string(),
    };
    user_tree.put(user.clone())?;

    // Get by primary key
    let retrieved = user_tree.get(UserPrimaryKey(1))?.unwrap();
    assert_eq!(retrieved.username, "alice");

    // Query by secondary key
    let users_by_email = user_tree.get_by_secondary_key(
        UserSecondaryKeys::Email(UserEmailSecondaryKey("alice@example.com".to_string()))
    )?;
    assert_eq!(users_by_email.len(), 1);

    // Iterate over all users
    for result in user_tree.iter() {
        let (_key, user) = result?;
        println!("User: {} - {}", user.username, user.email);
    }

    // Access backend-specific features when needed
    store.flush()?; // Sled-specific method

    Ok(())
}

3. Alternative: Simple Configuration for Quick Setup

For simple use cases, use the convenience constructors:

use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;
use netabase_store::databases::sled_store::SledStore;

// Simple constructor with defaults
let config = FileConfig::new("my_app.db");
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::open(config)?;

// Or create a temporary database for testing
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::temp()?;

4. Use with IndexedDB (WASM)

use netabase_store::config::IndexedDBConfig;
use netabase_store::traits::backend_store::BackendStore;
use netabase_store::databases::indexeddb_store::IndexedDBStore;
use netabase_store::traits::tree::NetabaseTreeAsync;

#[cfg(target_arch = "wasm32")]
async fn wasm_example() -> Result<(), Box<dyn std::error::Error>> {
    // Create configuration
    let config = IndexedDBConfig::builder()
        .database_name("my_database".to_string())
        .version(1)
        .build();

    // Initialize IndexedDB store
    let store = <IndexedDBStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config).await?;

    // Note: WASM uses async API
    let user_tree = store.open_tree::<User>();

    let user = User {
        id: 1,
        username: "charlie".to_string(),
        email: "charlie@example.com".to_string(),
    };

    // All operations are async
    user_tree.put(user.clone()).await?;
    let retrieved = user_tree.get(user.primary_key()).await?;

    Ok(())
}

Advanced Usage

Configuration API

The new unified configuration system provides consistent backend initialization across all database types:

FileConfig - For File-Based Backends

use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;
use netabase_store::databases::sled_store::SledStore;

// Method 1: Builder pattern (recommended)
let config = FileConfig::builder()
    .path("app_data.db".into())
    .cache_size_mb(1024)
    .truncate(true)
    .build();

let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;

// Method 2: Simple constructor
let config = FileConfig::new("app_data.db");
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::open(config)?;

// Method 3: Temporary database
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::temp()?;

Switching Backends with Same Config

The power of the configuration API is that you can switch backends without changing your code:

use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;

let config = FileConfig::builder()
    .path("my_app.db".into())
    .cache_size_mb(512)
    .build();

// Try different backends - same config!
#[cfg(feature = "sled")]
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config.clone())?;

#[cfg(feature = "redb")]
let store = <RedbStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config.clone())?;

#[cfg(feature = "redb-zerocopy")]
let store = <RedbStoreZeroCopy<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;

// All have the same API from this point on!
let user_tree = store.open_tree::<User>();

Configuration Options Reference

FileConfig (for Sled, Redb, RedbZeroCopy):

• path: PathBuf - Database file/directory path
• cache_size_mb: usize - Cache size in megabytes (default: 256)
• create_if_missing: bool - Create if doesn't exist (default: true)
• truncate: bool - Delete existing data (default: false)
• read_only: bool - Open read-only (default: false)
• use_fsync: bool - Fsync for durability (default: true)

MemoryConfig (for in-memory backend):

• capacity: Option<usize> - Optional capacity hint

IndexedDBConfig (for WASM):

• database_name: String - IndexedDB database name
• version: u32 - Schema version (default: 1)

Batch Operations & Bulk Methods

For high-performance bulk operations, use the convenient bulk methods:

use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;
use netabase_store::databases::sled_store::SledStore;

// Create store with temporary database
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::temp()?;
let user_tree = store.open_tree::<User>();

// Bulk insert - 8-9x faster than loop!
let users: Vec<User> = (0..1000)
    .map(|i| User {
        id: i,
        username: format!("user{}", i),
        email: format!("user{}@example.com", i),
    })
    .collect();

user_tree.put_many(users)?;  // Single transaction

// Bulk read
let keys: Vec<UserPrimaryKey> = (0..100).map(UserPrimaryKey).collect();
let users: Vec<Option<User>> = user_tree.get_many(keys)?;

// Bulk secondary key queries
let email_keys = vec![
    UserSecondaryKeys::Email(UserEmailSecondaryKey("alice@example.com".to_string())),
    UserSecondaryKeys::Email(UserEmailSecondaryKey("bob@example.com".to_string())),
];
let results: Vec<Vec<User>> = user_tree.get_many_by_secondary_keys(email_keys)?;

Bulk Methods:

• put_many(Vec<M>) - Insert multiple models in one transaction
• get_many(Vec<M::Keys>) - Read multiple models in one transaction
• get_many_by_secondary_keys(Vec<SecondaryKey>) - Query multiple secondary keys in one transaction

Or use the batch API for more control:

use netabase_store::traits::batch::Batchable;

// Create a batch
let mut batch = user_tree.create_batch()?;

// Add many operations
for i in 0..1000 {
    batch.put(User { /* ... */ })?;
}

// Commit atomically - all or nothing
batch.commit()?;

3. Choose the Right API for Your Use Case

Serialization Overhead

The read-path overhead in Redb comes from type system limitations with Generic Associated Types (GATs). We prioritize safety over unsafe transmutes. For applications where this matters:

• Use bulk methods to amortize overhead
• Use explicit transactions for better performance
• Consider Sled backend for read-heavy workloads

See benchmark results and visualizations in docs/benchmarks/ for detailed performance analysis.

For 1.0.0

• Transaction support across multiple operations (COMPLETED)
• Zero-copy reads for redb backend (via redb-zerocopy feature) - Phase 1 Complete
• Allow modules to define more than one definition for flexible organization
• Migration utilities for schema changes
• Query builder for complex queries
• Range queries on ordered keys
• Compression support
• Encryption at rest
• Improved documentation and examples

Future Plans

• Distributed systems support with automatic sync
• CRDT-based conflict resolution
• WebRTC backend for peer-to-peer storage
• SQL-like query language
• GraphQL integration

Acknowledgments

Built with:

• Sled - Embedded database
• Redb - Embedded database
• IndexedDB - Browser storage
• Bincode - Binary serialization