Crate netabase 

Netabase - Distributed Database System

Netabase is a distributed, peer-to-peer database system built on top of sled with libp2p integration. It provides a type-safe, macro-driven approach to defining database definitions and models with support for primary keys, secondary keys, and relational queries.

This crate is an attempt to provide a persistent Object mapped store for use in the libp2p implementation of the kademlia protocol

Key Features

• Type-Safe Models: Automatic code generation for database models using derive macros
• Primary & Secondary Keys: Efficient indexing and querying capabilities
• Distributed Architecture: Peer-to-peer networking with DHT-based record storage
• Network Transparency: Seamless data synchronization across network nodes

Quick Start

use netabase::Netabase;
use netabase_store::{ netabase_definition_module};
use netabase_store::traits::model::NetabaseModelTrait;
use netabase_store::{bincode, serde}; // Re-exported for convenience

// Define your data models
#[netabase_definition_module(BlogDefinition, BlogKeys)]
mod blog {
    use netabase_store::{NetabaseModel, netabase};

    #[derive(NetabaseModel, Clone, Debug, bincode::Encode, bincode::Decode, serde::Serialize, serde::Deserialize)]
    #[netabase(BlogDefinition)]
    pub struct User {
        #[primary_key]
        pub id: u64,
        pub name: String,
        #[secondary_key]
        pub email: String,
    }
}

use blog::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a distributed database instance
    let mut netabase = Netabase::<BlogDefinition>::new()?;
    netabase.start_swarm().await?;

    // Create and store a user
    let user = User {
        id: 1,
        name: "Alice".to_string(),
        email: "alice@example.com".to_string(),
    };

    // Store in the distributed hash table
    netabase.put_record(user).await?;

    // Retrieve from the network
    let user_key = UserKey::Primary(UserPrimaryKey(1));
    let result = netabase.get_record(user_key).await?;

    Ok(())
}

Architecture Overview

Netabase consists of three main layers:

1. Storage Layer (netabase_store)

• Embedded database operations using sled
• CRUD operations and indexing
• Secondary key and relational queries
• Local data persistence

2. Macro Layer (netabase_macros)

• Procedural macros for code generation
• Type-safe model and schema definitions
• Automatic key type generation
• Serialization trait implementations

3. Network Layer (netabase)

• Peer-to-peer networking with libp2p
• Distributed hash table (DHT) operations
• Record replication and discovery
• Event broadcasting and subscription

Local Database Usage

For local-only database operations without networking:

use netabase_store::databases::sled_store::SledStore;
use netabase_store::traits::model::NetabaseModelTrait;
use netabase_store::traits::tree::NetabaseTreeSync;
use netabase_store::netabase_definition_module;

// Define your data models
#[netabase_definition_module(BlogDefinition, BlogKeys)]
mod blog {
    use netabase_store::{NetabaseModel, netabase};

    #[derive(NetabaseModel, Clone, Debug, bincode::Encode, bincode::Decode, serde::Serialize, serde::Deserialize)]
    #[netabase(BlogDefinition)]
    pub struct User {
        #[primary_key]
        pub id: u64,
        pub name: String,
        #[secondary_key]
        pub email: String,
    }
}

use blog::*;

// Create local database
let db = SledStore::<BlogDefinition>::temp().unwrap();
let user_tree = db.open_tree::<User>();

let user = User { id: 1, name: "Some Name".to_string(), email: "some@email.com".to_string() };

// Standard CRUD operations
user_tree.put(user.clone()).unwrap();
let retrieved = user_tree.get(user.primary_key()).unwrap().unwrap();
assert_eq!(retrieved.name, "Some Name");

// Secondary key queries
let users_by_email = user_tree.get_by_secondary_key(
    UserSecondaryKeys::Email(EmailSecondaryKey("some@email.com".to_string()))
).unwrap();
assert_eq!(users_by_email.len(), 1);

Distributed Network Usage

For distributed operations across multiple nodes:

use netabase::Netabase;
use netabase_store::*;
use netabase_store::traits::model::NetabaseModelTrait;

/// Example definition module for testing netabase functionality
#[netabase_definition_module(TestDefinition, TestDefinitionKeys)]
pub mod test_definition {
    use super::*;
    use netabase_store::{NetabaseModel, netabase};

    /// Test user model
    #[derive(
        NetabaseModel,
        Clone,
        Debug,
        PartialEq,
        bincode::Encode,
        bincode::Decode,
        serde::Serialize,
        serde::Deserialize,
    )]
    #[netabase(TestDefinition)]
    pub struct TestUser {
        #[primary_key]
        pub id: u64,
        pub name: String,
    }
}
pub use test_definition::*;
#[tokio::main]
pub async fn main() {
    // Create distributed instance
    let mut netabase = Netabase::<test_definition::TestDefinition>::new().expect("Failed to initialise database for some reason");
    let user = test_definition::TestUser { id:1, name:"Some Name".to_string() };
    let key = user.primary_key();
    let start_swarm_result = netabase.start_swarm().await;

    // Network operations
    let bootstrap_result = netabase.bootstrap().await; // Join the network
    let put_record_result = netabase.put_record(user).await; // Store data
    let get_record_result = netabase.get_record::<TestUserKey>(key.clone().into()).await; // Retrieve data

    // Provider operations
    let provider_result = netabase.start_providing::<TestUserKey>(key.clone().into()).await;
    let providers_result = netabase.get_providers::<TestUserKey>(key.into()).await;

    // Subscribe to network events
    let mut receiver = netabase.subscribe_to_broadcasts();
    use tokio::time::{timeout, Duration};
     
    let duration = Duration::from_secs(5);
     
    loop {
        match timeout(duration, receiver.recv()).await {
            Ok(Ok(event)) => {
                println!("Network event: {:?}", event);
            }
            Ok(Err(_closed)) => {
                // channel closed
                break;
            }
            Err(_) => {
                // timeout elapsed
                println!("No event received for {}s — timing out", duration.as_secs());
                break; // or continue, or handle timeout
            }
        }
    }
}

Data Modeling Best Practices

Primary Keys

• Use simple, immutable types (u64, String, UUID)
• Ensure uniqueness across all records
• Consider using auto-incrementing integers or UUIDs

Secondary Keys

• Index frequently queried fields
• Balance query performance vs. storage/write overhead
• Use for fields with reasonable cardinality

TODO

Relations

[ ] Use foreign key fields to reference other models [ ] Enable efficient joins and referential integrity [ ] Consider using NetabaseRelationalQuery for complex relationships

Re-exports

pub use netabase_store;

Modules

errors

Re-export macro dependencies for user convenience. Users can access these through netabase::serde, netabase::bincode, etc. but the macros will work even without manual imports thanks to hygiene.

network

Structs

Netabase

Main Netabase instance that manages the distributed database.

NetabaseSwarmEvent

Enums

NetabaseBehaviourEvent

NetworkBehaviour::ToSwarm produced by NetabaseBehaviour.