Crate tonbo

tonbo is a structured embedded database, built on Apache Arrow & Parquet, designed to store, filter, and project structured data using LSM Tree. Its name comes from the Japanese word for dragonfly.

This database is fully thread-safe and all operations are atomic to ensure data consistency. most operations are asynchronous and non-blocking, providing efficient concurrent processing capabilities.

tonbo constructs an instance using the DB::new method to serve a specific Tonbo Record type.

Tonbo Record is automatically implemented by the macro [tonbo_record]. Support type

• String
• Boolean
• Int8
• Int16
• Int32
• Int64
• UInt8
• UInt16
• UInt32
• UInt64

ACID optimistic transactions for concurrent data reading and writing are supported with the DB::transaction method.

Examples

use std::ops::Bound;

use fusio::path::Path;
use futures_util::stream::StreamExt;
use tokio::fs;
use tonbo::{executor::tokio::TokioExecutor, DbOption, Projection, Record, DB};

// use macro to define schema of column family just like ORM
// it provides type safety read & write API
#[derive(Record, Debug)]
pub struct User {
    #[record(primary_key)]
    name: String,
    email: Option<String>,
    age: u8,
}

#[tokio::main]
async fn main() {
    // make sure the path exists
    let _ = fs::create_dir_all("./db_path/users").await;

    let options = DbOption::new(
        Path::from_filesystem_path("./db_path/users").unwrap(),
        &UserSchema,
    );
    // pluggable async runtime and I/O
    let db = DB::new(options, TokioExecutor::current(), UserSchema)
        .await
        .unwrap();
    // insert with owned value
    db.insert(User {
        name: "Alice".into(),
        email: Some("alice@gmail.com".into()),
        age: 22,
    })
    .await
    .unwrap();

    {
        // tonbo supports transaction
        let txn = db.transaction().await;

        // get from primary key
        let name = "Alice".into();

        // get the zero-copy reference of record without any allocations.
        let user = txn
            .get(
                &name,
                // tonbo supports pushing down projection
                Projection::All,
            )
            .await
            .unwrap();
        assert!(user.is_some());
        assert_eq!(user.unwrap().get().age, Some(22));

        {
            let upper = "Blob".into();
            // range scan of
            let mut scan = txn
                .scan((Bound::Included(&name), Bound::Excluded(&upper)))
                // tonbo supports pushing down projection
                .projection(&["email"])
                .take()
                .await
                .unwrap();
            while let Some(entry) = scan.next().await.transpose().unwrap() {
                assert_eq!(
                    entry.value(),
                    Some(UserRef {
                        name: "Alice",
                        email: Some("alice@gmail.com"),
                        age: Some(22),
                    })
                );
            }
        }

        // commit transaction
        txn.commit().await.unwrap();
    }
}

Re-exports

pub use arrow;

pub use once_cell;

pub use parquet;

pub use crate::option::*;

Modules

executor

fs

inmem

magic

option

record

snapshot

stream

timestamp

transaction

Macros

cast_arc_value

Cast the Arc<dyn Any> to the value of given type.

dyn_record

Creates a DynRecord from slice of values and primary key index, suitable for rapid testing and development.

dyn_schema

Creates a DynSchema from literal slice of values and primary key index, suitable for rapid testing and development.

implement_key

Structs

DB

Scan

scan configuration intermediate structure

Enums

DbError

Projection

Traits

Decode

Encode

SeqRead

Write

The core trait for writing data, it is similar to std::io::Write, but it takes the ownership of the buffer, because completion-based IO requires the buffer to be pinned and should be safe to cancellation.

Type Aliases

ParquetLru

Derive Macros

KeyAttributes

Record

used to define the structure of Record, will generate the implementation required in Tonbo, allowing derive expansion.