Netabase Store

A type-safe, multi-backend key-value storage library for Rust with support for native (Sled, Redb) and WASM (IndexedDB) environments.

⚠️ Early Development: This crate is still in early development and will change frequently as it stabilizes. It is not advised to use this in a production environment until it stabilizes.

Features

✨ Core Features

• 🗄️ Multi-Backend Support:

  • Sled: High-performance embedded database for native platforms
  • Redb: Memory-efficient embedded database with ACID guarantees
  • RedbZeroCopy: Zero-copy variant for maximum performance (10-54x faster for bulk ops)
  • IndexedDB: Browser-based storage for WASM applications
  • In-Memory: Fast in-memory storage for testing and caching

• ⚙️ Unified Configuration API:

  • FileConfig, MemoryConfig, IndexedDBConfig with builder pattern
  • Consistent initialization across all backends
  • Switch backends by changing one line of code
  • Type-safe configuration with sensible defaults

• 🔒 Type-Safe Schema Definition:

  • Derive macros for automatic schema generation
  • Primary and secondary key support
  • Compile-time type checking for all database operations
  • Zero-cost abstractions with trait-based design

• 🌍 Cross-Platform:

  • Unified API across native and WASM targets
  • Feature flags for platform-specific backends
  • Seamless switching between backends with same configuration

• ⚡ High Performance:

  • Transaction API with type-state pattern (10-100x faster for bulk ops)
  • Batch operations for bulk inserts/updates
  • Efficient secondary key indexing
  • Minimal overhead (<5-10%) over raw backend operations
  • Zero-copy deserialization where possible

• 🔍 Secondary Key Indexing:

  • Fast lookups using secondary keys
  • Multiple secondary keys per model
  • Automatic index management

• 🔄 Iteration Support:

  • Efficient iteration over stored data
  • Type-safe iterators with proper error handling

• 🔗 libp2p Integration (Optional):

  • Record store implementation for distributed systems
  • Compatible with libp2p DHT
  • Enable via record-store feature

• 🧪 Testing Utilities:

  • Comprehensive test suite
  • Benchmarking tools included
  • WASM test support via wasm-pack

🔌 Extensibility

• Unified Trait-Based API:

  • NetabaseTreeSync for synchronous operations (native)
  • NetabaseTreeAsync for asynchronous operations (WASM)
  • Easy to implement custom backends
  • Full compatibility with existing code

• Batch Processing:

  • Batchable trait for atomic bulk operations
  • Significantly faster than individual operations
  • Backend-specific optimizations

Installation

Add to your Cargo.toml:

[dependencies]
netabase_store = "0.0.3"

# Required dependencies for macros to work
bincode = { version = "2.0", features = ["serde"] }
serde = { version = "1.0", features = ["derive"] }
strum = { version = "0.27.2", features = ["derive"] }
derive_more = { version = "2.0.1", features = ["from", "try_into", "into"] }
anyhow = "1.0"  # Optional, for error handling

# For WASM support
[target.'cfg(target_arch = "wasm32")'.dependencies]
netabase_store = { version = "0.0.2", default-features = false, features = ["wasm"] }

Feature Flags

• native (default): Enable Sled and Redb backends
• sled: Enable Sled backend only
• redb: Enable Redb backend only
• redb-zerocopy: Enable zero-copy Redb backend (high-performance variant)
• wasm: Enable IndexedDB backend for WASM
• libp2p: Enable libp2p integration
• record-store: Enable RecordStore trait (requires libp2p)

Quick Start

1. Define Your Schema

use netabase_store::netabase_definition_module;
use netabase_store::traits::model::NetabaseModelTrait;

#[netabase_definition_module(BlogDefinition, BlogKeys)]
pub mod blog_schema {
    use netabase_store::{NetabaseModel, netabase};

    #[derive(NetabaseModel, bincode::Encode, bincode::Decode, Clone, Debug)]
    #[netabase(BlogDefinition)]
    pub struct User {
        #[primary_key]
        pub id: u64,
        pub username: String,
        #[secondary_key]
        pub email: String,
    }

    #[derive(NetabaseModel, bincode::Encode, bincode::Decode, Clone, Debug)]
    #[netabase(BlogDefinition)]
    pub struct Post {
        #[primary_key]
        pub id: u64,
        pub title: String,
        pub content: String,
        #[secondary_key]
        pub author_id: u64,
    }
}

use blog_schema::*;

2. Use with NetabaseStore (Recommended)

The unified NetabaseStore provides a consistent API across all backends:

use netabase_store::NetabaseStore;

fn main() -> anyhow::Result<()> {
    // Create a store with any backend - easily switch by changing one line!

    // Option 1: Sled backend (high-performance)
    let store = NetabaseStore::<BlogDefinition, _>::sled("./my_db")?;

    // Option 2: Redb backend (memory-efficient, ACID)
    // let store = NetabaseStore::<BlogDefinition, _>::redb("./my_db.redb")?;

    // Option 3: Temporary store for testing
    // let store = NetabaseStore::<BlogDefinition, _>::temp()?;

    // Open a tree for users - works identically across all backends
    let user_tree = store.open_tree::<User>();

    // Insert a user
    let user = User {
        id: 1,
        username: "alice".to_string(),
        email: "alice@example.com".to_string(),
    };
    user_tree.put(user.clone())?;

    // Get by primary key
    let retrieved = user_tree.get(UserPrimaryKey(1))?.unwrap();
    assert_eq!(retrieved.username, "alice");

    // Query by secondary key
    let users_by_email = user_tree.get_by_secondary_key(
        UserSecondaryKeys::Email(UserEmailSecondaryKey("alice@example.com".to_string()))
    )?;
    assert_eq!(users_by_email.len(), 1);

    // Iterate over all users
    for result in user_tree.iter() {
        let (_key, user) = result?;
        println!("User: {} - {}", user.username, user.email);
    }

    // Access backend-specific features when needed
    store.flush()?; // Sled-specific method

    Ok(())
}

3. Direct Backend Usage (Advanced)

You can also use backends directly for backend-specific features:

use netabase_store::databases::sled_store::SledStore;
use netabase_store::databases::redb_store::RedbStore;

// Direct Sled usage
let sled_store = SledStore::<BlogDefinition>::temp()?;
let user_tree = sled_store.open_tree::<User>();

// Direct Redb usage
let redb_store = RedbStore::<BlogDefinition>::new("my_database.redb")?;
let user_tree = redb_store.open_tree::<User>();

// Both have identical APIs via NetabaseTreeSync trait

4. Use with IndexedDB (WASM)

use netabase_store::databases::indexeddb_store::IndexedDBStore;
use netabase_store::traits::tree::NetabaseTreeAsync;

#[cfg(target_arch = "wasm32")]
async fn wasm_example() -> Result<(), Box<dyn std::error::Error>> {
    // Create a store in the browser
    let store = IndexedDBStore::<BlogDefinition>::new("my_database").await?;

    // Note: WASM uses async API
    let user_tree = store.open_tree::<User>();

    let user = User {
        id: 1,
        username: "charlie".to_string(),
        email: "charlie@example.com".to_string(),
    };

    // All operations are async
    user_tree.put(user.clone()).await?;
    let retrieved = user_tree.get(user.primary_key()).await?;

    Ok(())
}

Advanced Usage

Configuration API

The new unified configuration system provides consistent backend initialization across all database types:

FileConfig - For File-Based Backends

use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;
use netabase_store::databases::sled_store::SledStore;

// Method 1: Builder pattern (recommended)
let config = FileConfig::builder()
    .path("app_data.db".into())
    .cache_size_mb(1024)
    .truncate(true)
    .build();

let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;

// Method 2: Simple constructor
let config = FileConfig::new("app_data.db");
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::open(config)?;

// Method 3: Temporary database
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::temp()?;

Switching Backends with Same Config

The power of the configuration API is that you can switch backends without changing your code:

use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;

let config = FileConfig::builder()
    .path("my_app.db".into())
    .cache_size_mb(512)
    .build();

// Try different backends - same config!
#[cfg(feature = "sled")]
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config.clone())?;

#[cfg(feature = "redb")]
let store = <RedbStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config.clone())?;

#[cfg(feature = "redb-zerocopy")]
let store = <RedbStoreZeroCopy<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;

// All have the same API from this point on!
let user_tree = store.open_tree::<User>();

Configuration Options Reference

FileConfig (for Sled, Redb, RedbZeroCopy):

• path: PathBuf - Database file/directory path
• cache_size_mb: usize - Cache size in megabytes (default: 256)
• create_if_missing: bool - Create if doesn't exist (default: true)
• truncate: bool - Delete existing data (default: false)
• read_only: bool - Open read-only (default: false)
• use_fsync: bool - Fsync for durability (default: true)

MemoryConfig (for in-memory backend):

• capacity: Option<usize> - Optional capacity hint

IndexedDBConfig (for WASM):

• database_name: String - IndexedDB database name
• version: u32 - Schema version (default: 1)

Batch Operations & Bulk Methods

For high-performance bulk operations, use the convenient bulk methods:

use netabase_store::NetabaseStore;

let store = NetabaseStore::<BlogDefinition, _>::temp()?;
let user_tree = store.open_tree::<User>();

// Bulk insert - 8-9x faster than loop!
let users: Vec<User> = (0..1000)
    .map(|i| User {
        id: i,
        username: format!("user{}", i),
        email: format!("user{}@example.com", i),
    })
    .collect();

user_tree.put_many(users)?;  // Single transaction

// Bulk read
let keys: Vec<UserPrimaryKey> = (0..100).map(UserPrimaryKey).collect();
let users: Vec<Option<User>> = user_tree.get_many(keys)?;

// Bulk secondary key queries
let email_keys = vec![
    UserSecondaryKeys::Email(UserEmailSecondaryKey("alice@example.com".to_string())),
    UserSecondaryKeys::Email(UserEmailSecondaryKey("bob@example.com".to_string())),
];
let results: Vec<Vec<User>> = user_tree.get_many_by_secondary_keys(email_keys)?;

Bulk Methods:

• put_many(Vec<M>) - Insert multiple models in one transaction
• get_many(Vec<M::Keys>) - Read multiple models in one transaction
• get_many_by_secondary_keys(Vec<SecondaryKey>) - Query multiple secondary keys in one transaction

Or use the batch API for more control:

use netabase_store::traits::batch::Batchable;

// Create a batch
let mut batch = user_tree.create_batch()?;

// Add many operations
for i in 0..1000 {
    batch.put(User { /* ... */ })?;
}

// Commit atomically - all or nothing
batch.commit()?;

Bulk operations are:

• ⚡ Faster: 8-10x faster than individual operations
• 🔒 Atomic: All succeed or all fail
• 📦 Efficient: Single transaction reduces overhead

Transactions (New!)

For maximum performance and atomicity, use the transaction API to reuse a single transaction across multiple operations:

use netabase_store::NetabaseStore;

let store = NetabaseStore::<BlogDefinition, _>::sled("./my_db")?;

// Read-only transaction - multiple concurrent reads allowed
let txn = store.read();
let user_tree = txn.open_tree::<User>();
let user = user_tree.get(UserPrimaryKey(1))?;
// Transaction auto-closes on drop

// Read-write transaction - exclusive access, atomic commit
let mut txn = store.write()?;
let mut user_tree = txn.open_tree::<User>();

// All operations share the same transaction
for i in 0..1000 {
    let user = User {
        id: i,
        username: format!("user{}", i),
        email: format!("user{}@example.com", i),
    };
    user_tree.put(user)?;
}

// Bulk helpers also work within transactions
user_tree.put_many(more_users)?;

// Commit all changes atomically
txn.commit()?;
// Or drop without committing to rollback

Transaction Benefits:

• 🚀 10-100x Faster: Single transaction for many operations (eliminates per-operation overhead)
• 🔒 Type-Safe: Compile-time enforcement of read-only vs read-write access
• ⚡ Zero-Cost: Phantom types compile away completely
• 🔄 ACID: Full atomicity for write transactions (Redb)

Compile-Time Safety:

let txn = store.read();  // ReadOnly transaction
let tree = txn.open_tree::<User>();
tree.put(user)?;  // ❌ Compile error: put() not available on ReadOnly!

Secondary Keys

Secondary keys enable efficient lookups on non-primary fields:

#[derive(NetabaseModel, Clone, bincode::Encode, bincode::Decode)]
#[netabase(BlogDefinition)]
pub struct Article {
    #[primary_key]
    pub id: u64,
    pub title: String,
    #[secondary_key]
    pub category: String,
    #[secondary_key]
    pub published: bool,
}

// Query by single secondary key
let tech_articles = article_tree
    .get_by_secondary_key(
        ArticleSecondaryKeys::Category(
            ArticleCategorySecondaryKey("tech".to_string())
        )
    )?;

// Bulk query multiple secondary keys (2-3x faster!)
let keys = vec![
    ArticleSecondaryKeys::Category(ArticleCategorySecondaryKey("tech".to_string())),
    ArticleSecondaryKeys::Category(ArticleCategorySecondaryKey("science".to_string())),
];
let results: Vec<Vec<Article>> = article_tree.get_many_by_secondary_keys(keys)?;
// results[0] = tech articles, results[1] = science articles

Multiple Models in One Store

use netabase_store::NetabaseStore;

let store = NetabaseStore::<BlogDefinition, _>::sled("blog_db")?;

// Different trees for different models
let user_tree = store.open_tree::<User>();
let post_tree = store.open_tree::<Post>();

// Each tree is independent but shares the same underlying database
user_tree.put(user)?;
post_tree.put(post)?;

Temporary Store for Testing

use netabase_store::NetabaseStore;

// Perfect for unit tests - no I/O, no cleanup needed
let store = NetabaseStore::<BlogDefinition, _>::temp()?;
let user_tree = store.open_tree::<User>();

user_tree.put(user)?;

Custom Backend Implementation

Netabase Store's trait-based design makes it easy to implement custom storage backends. Here's what you need to know:

Required Traits

To create a custom backend, implement one of these traits depending on your backend's characteristics:

1. NetabaseTreeSync - For Synchronous Backends

Use this for native, blocking I/O backends (like SQLite, file systems, etc.):

use netabase_store::traits::tree::NetabaseTreeSync;
use netabase_store::traits::model::NetabaseModelTrait;
use netabase_store::traits::definition::NetabaseDefinitionTrait;
use netabase_store::error::NetabaseError;

pub struct MyCustomBackend<D, M> {
    // Your backend state (connection, file handles, etc.)
}

impl<D, M> NetabaseTreeSync<D, M> for MyCustomBackend<D, M>
where
    D: NetabaseDefinitionTrait + TryFrom<M> + From<M>,
    M: NetabaseModelTrait<D> + TryFrom<D> + Clone,
    M::PrimaryKey: bincode::Encode + bincode::Decode<()> + Clone,
    M::SecondaryKeys: bincode::Encode + bincode::Decode<()>,
    // Add discriminant bounds...
{
    type PrimaryKey = M::PrimaryKey;
    type SecondaryKeys = M::SecondaryKeys;

    // Required: Insert or update a model
    fn put(&self, model: M) -> Result<(), NetabaseError> {
        // 1. Get primary key: model.primary_key()
        // 2. Get secondary keys: model.secondary_keys()
        // 3. Serialize model to bytes (use bincode or ToIVec trait)
        // 4. Store in your backend
        // 5. Create secondary key indexes
        todo!()
    }

    // Required: Retrieve by primary key
    fn get(&self, key: Self::PrimaryKey) -> Result<Option<M>, NetabaseError> {
        // 1. Serialize key to bytes
        // 2. Look up in your backend
        // 3. Deserialize bytes to model
        // 4. Return Some(model) or None
        todo!()
    }

    // Required: Delete by primary key
    fn remove(&self, key: Self::PrimaryKey) -> Result<Option<M>, NetabaseError> {
        // 1. Get the model (for cleanup)
        // 2. Delete primary key entry
        // 3. Delete secondary key indexes
        // 4. Return the deleted model
        todo!()
    }

    // Required: Query by secondary key
    fn get_by_secondary_key(
        &self,
        secondary_key: Self::SecondaryKeys
    ) -> Result<Vec<M>, NetabaseError> {
        // 1. Look up secondary key index
        // 2. Get all matching primary keys
        // 3. Retrieve all models with those keys
        // 4. Return vector of models
        todo!()
    }

    // Required: Check if empty
    fn is_empty(&self) -> Result<bool, NetabaseError> {
        Ok(self.len()? == 0)
    }

    // Required: Get count
    fn len(&self) -> Result<usize, NetabaseError> {
        todo!()
    }

    // Required: Delete all entries
    fn clear(&self) -> Result<(), NetabaseError> {
        todo!()
    }
}

2. NetabaseTreeAsync - For Asynchronous Backends

Use this for async backends (remote databases, web APIs, etc.):

use netabase_store::traits::tree::NetabaseTreeAsync;
use netabase_store::error::NetabaseError;
use std::future::Future;

pub struct MyAsyncBackend<D, M> {
    // Your async backend state
}

impl<D, M> NetabaseTreeAsync<D, M> for MyAsyncBackend<D, M>
where
    D: NetabaseDefinitionTrait + TryFrom<M> + From<M>,
    M: NetabaseModelTrait<D> + TryFrom<D> + Clone,
    // ... same bounds as sync version
{
    type PrimaryKey = M::PrimaryKey;
    type SecondaryKeys = M::SecondaryKeys;

    fn put(&self, model: M) -> impl Future<Output = Result<(), NetabaseError>> {
        async move {
            // Your async implementation
            todo!()
        }
    }

    fn get(
        &self,
        key: Self::PrimaryKey
    ) -> impl Future<Output = Result<Option<M>, NetabaseError>> {
        async move {
            todo!()
        }
    }

    // ... implement other methods with async
}

3. OpenTree - For Store-Level API

Implement this on your store type to allow opening trees:

use netabase_store::traits::store_ops::OpenTree;

pub struct MyStore<D> {
    // Store state
}

impl<D> OpenTree<D> for MyStore<D>
where
    D: NetabaseDefinitionTrait,
{
    type Tree<M> = MyCustomBackend<D, M>
    where
        M: NetabaseModelTrait<D>;

    fn open_tree<M>(&self) -> Self::Tree<M>
    where
        M: NetabaseModelTrait<D> + TryFrom<D> + Into<D>,
    {
        // Create and return a tree instance for model M
        MyCustomBackend {
            // Initialize with store's connection/state
        }
    }
}

4. Batchable (Optional) - For Batch Operations

If your backend supports atomic batching:

use netabase_store::traits::batch::{Batchable, BatchBuilder};

impl<D, M> Batchable<D, M> for MyCustomBackend<D, M>
where
    D: NetabaseDefinitionTrait + TryFrom<M> + From<M>,
    M: NetabaseModelTrait<D> + TryFrom<D> + Clone,
{
    type Batch = MyBatch<D, M>;

    fn batch(&self) -> Self::Batch {
        MyBatch::new(/* ... */)
    }
}

pub struct MyBatch<D, M> {
    // Accumulate operations
}

impl<D, M> BatchBuilder<D, M> for MyBatch<D, M> {
    fn put(&mut self, model: M) -> Result<(), NetabaseError> {
        // Queue the operation
        todo!()
    }

    fn remove(&mut self, key: M::PrimaryKey) -> Result<(), NetabaseError> {
        // Queue the operation
        todo!()
    }

    fn commit(self) -> Result<(), NetabaseError> {
        // Execute all queued operations atomically
        todo!()
    }
}

Implementation Tips

1. Serialization: Use bincode for efficient serialization:

   use bincode::{encode_to_vec, decode_from_slice, config::standard};
   
   let bytes = encode_to_vec(&model, standard())?;
   let (model, _) = decode_from_slice(&bytes, standard())?;
   

2. Secondary Key Indexing: Store composite keys:

   // Create composite key: secondary_key_bytes + primary_key_bytes
   let mut composite = secondary_key_bytes;
   composite.extend_from_slice(&primary_key_bytes);
   

3. Error Handling: Convert your backend errors to NetabaseError:

   use netabase_store::error::NetabaseError;
   
   my_backend_op().map_err(|e|
       NetabaseError::Storage(format!("Backend error: {}", e))
   )?;
   

4. Iterator Support: Implement iterators for efficient traversal:

   fn iter(&self) -> impl Iterator<Item = Result<(M::PrimaryKey, M), NetabaseError>> {
       // Return an iterator over all entries
   }
   

Complete Example

See the existing backends for reference:

• sled_store.rs: Example of sync backend with batch support
• redb_store.rs: Example of transactional backend
• indexeddb_store.rs: Example of async WASM backend
• memory_store.rs: Simple in-memory implementation

All existing code will work with your custom backend once you implement the traits!

Performance

Netabase Store is designed for high performance while maintaining type safety. The library provides multiple APIs optimized for different use cases, with comprehensive benchmarking and profiling support.

API Options for Performance

The library offers three APIs with different performance characteristics:

1. Standard Wrapper API: Simple, ergonomic API with auto-transaction per operation
2. Bulk Methods: put_many(), get_many(), get_many_by_secondary_keys() - single transaction for multiple items
3. ZeroCopy API: Explicit transaction management for maximum control

Benchmark Results

Comprehensive benchmarks comparing all implementations across multiple dataset sizes (10, 100, 500, 1000, 5000 items):

Insert Performance (1000 items)

Implementation	Time	vs Raw	Notes
Raw Redb (baseline)	1.42 ms	0%	Single transaction, manual index management
Wrapper Redb (bulk)	3.10 ms	+118%	put_many() - single transaction
Wrapper Redb (loop)	27.3 ms	+1,822%	Individual put() calls - creates N transactions
ZeroCopy (bulk)	3.51 ms	+147%	put_many() with explicit transaction
ZeroCopy (loop)	4.34 ms	+206%	Loop with single explicit transaction

Key Insights:

• Bulk methods provide 8-9x speedup over loop-based insertion (27.3ms → 3.10ms)
• Bulk wrapper API approaches raw performance (118% overhead vs 1,822% for loops)
• Transaction overhead dominates when creating N transactions vs 1 transaction

Read Performance (1000 items)

Implementation	Time	vs Raw	Notes
Raw Redb (baseline)	164 µs	0%	Single transaction
Wrapper Redb (bulk)	382 µs	+133%	get_many() - single transaction
Wrapper Redb (loop)	895 µs	+446%	Individual get() calls - creates N transactions
ZeroCopy (single txn)	692 µs	+322%	Explicit read transaction

Key Insights:

• Bulk get_many() provides 2.3x speedup over individual gets (895µs → 382µs)
• Transaction reuse is critical for read performance
• Even bulk methods have overhead due to transaction and deserialization costs

Secondary Key Queries (10 queries)

Implementation	Time	vs Raw	Notes
Raw Redb (baseline)	291 µs	0%	10 transactions, manual index traversal
Wrapper Redb (bulk)	470 µs	+61%	get_many_by_secondary_keys() - single transaction
Wrapper Redb (loop)	1.02 ms	+248%	10 separate get_by_secondary_key() calls
ZeroCopy (single txn)	5.41 µs	-98%	Single transaction, optimized index access

Key Insights:

• ZeroCopy API is 54x faster than raw redb for secondary queries (291µs → 5.4µs)
• Bulk secondary query method provides 2.2x speedup over loops
• Single transaction + efficient index access = dramatic performance gains

Performance Optimization Guide

1. Use Bulk Methods for Standard API (8-9x faster)

// ❌ Slow: Creates 1000 transactions
for user in users {
    tree.put(user)?;  // Each call = new transaction
}

// ✅ Fast: Single transaction
tree.put_many(users)?;  // 8-9x faster!

Available Bulk Methods:

• put_many(Vec<M>) - Bulk insert
• get_many(Vec<M::Keys>) - Bulk read
• get_many_by_secondary_keys(Vec<SecondaryKey>) - Bulk secondary queries

2. Use Explicit Transactions for Maximum Control

// For write-heavy workloads
let mut txn = store.write()?;
let mut tree = txn.open_tree::<User>();

for user in users {
    tree.put(user)?;  // All share same transaction
}

txn.commit()?;  // Single atomic commit

3. Choose the Right API for Your Use Case

Use Case	Recommended API	Reason
Simple CRUD, few operations	Standard wrapper	Simplest API, auto-commit
Bulk inserts/reads (100+ items)	Bulk methods	8-9x faster than loops
Complex transactions	Explicit transactions	Full control, atomic commits
Read-heavy queries	ZeroCopy API	Up to 54x faster for secondary queries

Profiling Support

The benchmarks include full profiling support via pprof and flamegraphs:

# Run benchmarks with profiling
cargo bench --bench cross_store_comparison --features native

# Analyze profiling data
./scripts/analyze_profiling.sh

# View flamegraphs (SVG files in target/criterion/)
firefox target/criterion/cross_store_insert/wrapper_redb_bulk/profile/flamegraph.svg

Flamegraphs show:

• Function call stacks and time distribution
• Serialization overhead (bincode operations)
• Transaction costs (redb internal operations)
• Memory allocation patterns
• Lock contention (if any)

Running Benchmarks

# Cross-store comparison (all backends, multiple sizes)
cargo bench --bench cross_store_comparison --features native

# Generate visualizations
uv run scripts/generate_benchmark_charts.py

# View results
open docs/benchmarks/insert_comparison_bars.png
open docs/benchmarks/overhead_percentages.png
open docs/benchmarks/bulk_api_speedup.png

Backend Comparison

Redb

• Best for: Write-heavy workloads, ACID guarantees
• Wrapper overhead: 118-133% for bulk operations
• Strengths: Excellent write performance, full ACID compliance, efficient storage
• Use when: Data integrity is critical, write performance matters

Sled

• Best for: Read-heavy workloads
• Wrapper overhead: ~20% for read operations
• Strengths: Very low read overhead, battle-tested
• Use when: Read performance is critical, workload is read-heavy

Technical Notes

Why Transaction Overhead Matters

Creating a new transaction has fixed costs:

• Lock acquisition
• MVCC snapshot creation
• Internal state setup

When you call put() in a loop, you pay these costs N times. Using put_many() or explicit transactions, you pay once.

Type Safety vs Performance

The wrapper APIs prioritize type safety and ergonomics. For applications where the overhead is significant:

1. Use bulk methods first - often solves the problem
2. Use explicit transactions - full control with same safety
3. Profile your workload - measure before optimizing
4. Consider ZeroCopy API - for specialized high-performance scenarios

Serialization Overhead

The read-path overhead in Redb comes from type system limitations with Generic Associated Types (GATs). We prioritize safety over unsafe transmutes. For applications where this matters:

• Use bulk methods to amortize overhead
• Use explicit transactions for better performance
• Consider Sled backend for read-heavy workloads

See benchmark results and visualizations in docs/benchmarks/ for detailed performance analysis.

Testing

# Run all tests
cargo test --all-features

# Run native tests only
cargo test --features native

# Run WASM tests (requires wasm-pack and Firefox)
wasm-pack test --headless --firefox --features wasm

Architecture

See ARCHITECTURE.md for a deep dive into the library's design.

High-Level Overview

┌───────────────────────────────────────────────────┐
│         Your Application Code                     │
│      (Type-safe models with macros)              │
└───────────────────────────────────────────────────┘
                      │
                      ↓
┌───────────────────────────────────────────────────┐
│         NetabaseStore<D, Backend>                 │
│    (Unified API layer - Recommended)             │
└───────────────────────────────────────────────────┘
                      │
        ┌─────────────┼─────────────┐
        ↓             ↓             ↓
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│  SledStore  │ │  RedbStore  │ │IndexedDBStore│
│   <D>       │ │   <D>       │ │    <D>       │
└─────────────┘ └─────────────┘ └─────────────┘
        │             │             │
        ↓             ↓             ↓
┌─────────────────────────────────────────────────┐
│         Trait Layer                             │
│  (NetabaseTreeSync, NetabaseTreeAsync)         │
│  (OpenTree, Batchable, StoreOps)               │
└─────────────────────────────────────────────────┘
        │             │             │
        ↓             ↓             ↓
    ┌─────┐      ┌──────┐     ┌─────────┐
    │ Sled│      │ Redb │     │IndexedDB│
    └─────┘      └──────┘     └─────────┘
    Native       Native        WASM

Roadmap

For 1.0.0

• Transaction support across multiple operations (COMPLETED)
• Zero-copy reads for redb backend (via redb-zerocopy feature) - Phase 1 Complete
• Allow modules to define more than one definition for flexible organization
• Migration utilities for schema changes
• Query builder for complex queries
• Range queries on ordered keys
• Compression support
• Encryption at rest
• Improved documentation and examples

Future Plans

• Distributed systems support with automatic sync
• CRDT-based conflict resolution
• WebRTC backend for peer-to-peer storage
• SQL-like query language
• GraphQL integration

Examples

See the test_netabase_store_usage crate for a complete working example.

Additional examples in the repository:

• examples/basic_store.rs - Basic CRUD operations
• examples/unified_api.rs - Working with multiple backends
• tests/wasm_tests.rs - WASM usage patterns

Why Netabase Store?

Problem

Working with different database backends in Rust typically means:

• Learning different APIs for each backend
• No type safety for keys and values
• Manual serialization/deserialization
• Difficulty switching backends
• Complex secondary indexing

Solution

Netabase Store provides:

• ✅ Single unified API across all backends
• ✅ Compile-time type safety for everything
• ✅ Automatic serialization with bincode
• ✅ Seamless backend switching
• ✅ Automatic secondary key management
• ✅ Cross-platform support (native + WASM)

Contributing

Contributions are welcome! Please:

1. Open an issue to discuss major changes
2. Follow the existing code style
3. Add tests for new features
4. Update documentation

License

This project is licensed under the GPL-3.0-only License - see the LICENSE file for details.

Links

• Documentation
• Crates.io
• Repository
• Issue Tracker

Acknowledgments

Built with:

• Sled - Embedded database
• Redb - Embedded database
• IndexedDB - Browser storage
• Bincode - Binary serialization