# Getting started
## Installation
### Prerequisite
To get started with Tonbo, ensure that Rust is installed on your system. If you haven't installed it yet, please follow the [installation instructions](https://www.rust-lang.org/tools/install).
### Installation
Tonbo supports various target platforms (native, AWS Lambda, browsers, etc.) and storage backends (memory, local disk, S3, etc.). Built on asynchronous Rust, Tonbo improves database operation efficiency, which means you must configure an async runtime for your target platform.
For native platforms, [Tokio](https://github.com/tokio-rs/tokio) is the most popular async runtime in Rust. To use Tonbo with Tokio, ensure the tokio feature is enabled in your `Cargo.toml` file (enabled by default):
```toml
tokio = { version = "1", features = ["full"] }
tonbo = { git = "https://github.com/tonbo-io/tonbo" }
```
For browser targets using OPFS as the storage backend, disable the `tokio` feature and enable the `wasm` feature because Tokio is incompatible with OPFS. Since `tokio` is enabled by default, you must disable default features. If you plan to use S3 as the backend, also enable the `wasm-http` feature:
```toml
tonbo = { git = "https://github.com/tonbo-io/tonbo", default-features = false, features = [
"wasm",
"wasm-http",
] }
```
## Using Tonbo
### Defining Schema
Tonbo offers an ORM-like macro that simplifies working with column families. Use the Record macro to define your column family's schema, and Tonbo will automatically generate all necessary code at compile time:
```rust
use tonbo::Record;
#[derive(Record, Debug)]
pub struct User {
#[record(primary_key)]
name: String,
email: Option<String>,
age: u8,
}
```
Further explanation of this example:
- `Record`: This attribute marks the struct as a Tonbo schema definition, meaning it represents the structure of a column family.
- `#[record(primary_key)]`: This attribute designates the corresponding field as the primary key. Note that Tonbo currently does not support compound primary keys, so the primary key must be unique.
- `Option`: When a field is wrapped in Option, it indicates that the field is nullable.
Tonbo supports the following data types:
- Number types: i8, i16, i32, i64, u8, u16, u32, u64
- Boolean type: bool
- String type: String
- Bytes type: bytes::Bytes
### Creating database
After defining your schema, you can create a `DB` instance using a customized `DbOption`.
```rust
use std::fs;
use fusio::path::Path;
use tonbo::{executor::tokio::TokioExecutor, DbOption, DB};
#[tokio::main]
async fn main() {
// make sure the path exists
fs::create_dir_all("./db_path/users").unwrap();
let options = DbOption::new(
Path::from_filesystem_path("./db_path/users").unwrap(),
&UserSchema,
);
let db = DB::<User, TokioExecutor>::new(options, TokioExecutor::current(), UserSchema)
.await
.unwrap();
}
```
Tonbo automatically generates the `UserSchema` struct at compile time, so you don’t need to handle it manually. However, ensure that the specified path exists before creating your DBOption.
When using Tonbo in a WASM environment, use `Path::from_opfs_path` instead of `Path::from_filesystem_path`.
### Operations on Database
After creating the DB, you can perform operations like `insert`, `remove`, and `get`. However, when you retrieve a record from Tonbo, you'll receive a `UserRef` instance—not a direct `User` instance. The `UserRef` struct, which implements the `RecordRef` trait, is automatically generated by Tonbo at compile time. It might look something like this:
```rust
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub struct UserRef<'r> {
pub name: &'r str,
pub email: Option<&'r str>,
pub age: Option<u8>,
}
impl RecordRef for UserRef<'_> {
// ......
}
```
### Insert
`DB::insert` takes a `Record` instance—specifically, an instance of the struct you've defined with `#[derive(Record)]`:
```rust
db.insert(User { /* ... */ }).await.unwrap();
```
### Remove
`DB::remove` accepts a Key, where the type of the key is defined by the field annotated with `#[record(primary_key)]`. This method removes the record associated with the provided key:
```rust
db.remove("Alice".into()).await.unwrap();
```
### Get
`DB::get` accepts a `Key` and processes the corresponding record using a closure that receives a `TransactionEntry`. Within the closure, you can call `TransactionEntry::get` to retrieve the record as a `RecordRef` instance:
```rust
let age = db.get(&"Alice".into(),
|entry| {
// entry.get() will get a `UserRef`
let user = entry.get();
println!("{:#?}", user);
user.age
})
.await
.unwrap();
```
### Scan
Similar to `DB::get`, `DB::scan` accepts a closure that processes a `TransactionEntry`. However, instead of a single key, `DB::scan` operates over a range of keys, applying the closure to every record that falls within that range:
```rust
let lower = "Alice".into();
let upper = "Bob".into();
let stream = db
.scan(
(Bound::Included(&lower), Bound::Excluded(&upper)),
|entry| {
let record_ref = entry.get();
record_ref.age
},
)
.await;
let mut stream = std::pin::pin!(stream);
while let Some(data) = stream.next().await.transpose().unwrap() {
// ...
}
```
#### Using transaction
Tonbo supports transaction. You can also push down filter, limit and projection operators in query.
```rust
// create transaction
let txn = db.transaction().await;
let name = "Alice".into();
txn.insert(User { /* ... */ });
let user = txn.get(&name, Projection::All).await.unwrap();
let upper = "Blob".into();
// range scan of user
let mut scan = txn
.scan((Bound::Included(&name), Bound::Excluded(&upper)))
.take()
.await
.unwrap();
while let Some(entry) = scan.next().await.transpose().unwrap() {
let data = entry.value(); // type of UserRef
// ......
}
```
### Persistence
Tonbo employs a Log-Structured Merge Tree (LSM) as its underlying data structure, meaning that some data may reside in memory. To persist this in-memory data, use the flush method.
When Write-Ahead Logging (WAL) is enabled, data is automatically written to disk. However, since Tonbo buffers WAL data by default, you should call the `flush_wal` method to ensure all data is recovered. If you prefer not to use WAL buffering, you can disable it by setting `wal_buffer_size` to 0:
```rust
let options = DbOption::new(
Path::from_filesystem_path("./db_path/users").unwrap(),
&UserSchema,
).wal_buffer_size(0);
```
If you don't want to use WAL, you can disable it by setting the `DbOption::disable_wal`.
```rust
let options = DbOption::new(
Path::from_filesystem_path("./db_path/users").unwrap(),
&UserSchema,
).disable_wal(true);
```
> **Note**: If you disable WAL, there is nothing to do with `flush_wal`. You need to call `flush` method to persist the memory data.
>
> Conversely, if WAL is enabled and `wal_buffer_size` is set to 0, WAL data is flushed to disk immediately, so calling `flush_wal` is unnecessary.
### Using with S3
If you want to use Tonbo with S3, you can configure `DbOption` to determine which portions of your data are stored in S3 and which remain on the local disk. The example below demonstrates how to set up this configuration:
```rust
let s3_option = FsOptions::S3 {
bucket: "bucket".to_string(),
credential: Some(AwsCredential {
key_id: "key_id".to_string(),
secret_key: "secret_key".to_string(),
token: None,
}),
endpoint: None,
sign_payload: None,
checksum: None,
region: Some("region".to_string()),
};
let options = DbOption::new(
Path::from_filesystem_path("./db_path/users").unwrap(),
&UserSchema,
).level_path(2, "l2", s3_option.clone())
).level_path(3, "l3", s3_option);
```
In this example, data for level 2 and level 3 will be stored in S3, while all other levels remain on the local disk. If there is data in level 2 and level 3, you can verify and access it in S3:
```bash
s3://bucket/l2/
├── xxx.parquet
├── ......
s3://bucket/l3/
├── xxx.parquet
├── ......
```
For more configuration options, please refer to the [Configuration](./usage/conf.md) section.
## What next?
- To learn more about tonbo in Rust or in WASM, you can refer to [Tonbo API](./usage/tonbo.md)
- To use tonbo in python, you can refer to [Python API](./usage/python.md)
- To learn more about tonbo in brower, you can refer to [WASM API](./usage/wasm.md)
- To learn more configuration about tonbo, you can refer to [Configuration](./usage/conf.md)
- There are some data structures for runtime schema, you can use them to [expole tonbo](./usage/advance.md). You can also refer to our [python](https://github.com/tonbo-io/tonbo/tree/main/bindings/python), [wasm](https://github.com/tonbo-io/tonbo/tree/main/bindings/js) bindings and [Tonbolite(a SQLite extension)](https://github.com/tonbo-io/tonbolite)
- To learn more about tonbo by examples, you can refer to [examples](https://github.com/tonbo-io/tonbo/tree/main/examples)