Getting started

Installation

Prerequisite

To get started with Tonbo, ensure that Rust is installed on your system. If you haven't installed it yet, please follow the installation instructions.

Installation

Tonbo supports various target platforms (native, AWS Lambda, browsers, etc.) and storage backends (memory, local disk, S3, etc.). Built on asynchronous Rust, Tonbo improves database operation efficiency, which means you must configure an async runtime for your target platform.

For native platforms, Tokio is the most popular async runtime in Rust. To use Tonbo with Tokio, ensure the tokio feature is enabled in your Cargo.toml file (enabled by default):

tokio = { version = "1", features = ["full"] }
tonbo = { git = "https://github.com/tonbo-io/tonbo" }

For browser targets using OPFS as the storage backend, disable the tokio feature and enable the wasm feature because Tokio is incompatible with OPFS. Since tokio is enabled by default, you must disable default features. If you plan to use S3 as the backend, also enable the wasm-http feature:

tonbo = { git = "https://github.com/tonbo-io/tonbo", default-features = false, features = [
    "wasm",
    "wasm-http",
] }

Using Tonbo

Defining Schema

Tonbo offers an ORM-like macro that simplifies working with column families. Use the Record macro to define your column family's schema, and Tonbo will automatically generate all necessary code at compile time:

use tonbo::Record;

#[derive(Record, Debug)]
pub struct User {
    #[record(primary_key)]
    name: String,
    email: Option<String>,
    age: u8,
}

Further explanation of this example:

  • Record: This attribute marks the struct as a Tonbo schema definition, meaning it represents the structure of a column family.
  • #[record(primary_key)]: This attribute designates the corresponding field as the primary key. Note that Tonbo currently does not support compound primary keys, so the primary key must be unique.
  • Option: When a field is wrapped in Option, it indicates that the field is nullable.

Tonbo supports the following data types:

  • Number types: i8, i16, i32, i64, u8, u16, u32, u64
  • Boolean type: bool
  • String type: String
  • Bytes type: bytes::Bytes

Creating database

After defining your schema, you can create a DB instance using a customized DbOption.

use std::fs;
use fusio::path::Path;
use tonbo::{executor::tokio::TokioExecutor, DbOption, DB};

#[tokio::main]
async fn main() {
    // make sure the path exists
    fs::create_dir_all("./db_path/users").unwrap();

    let options = DbOption::new(
        Path::from_filesystem_path("./db_path/users").unwrap(),
        &UserSchema,
    );
    let db = DB::<User, TokioExecutor>::new(options, TokioExecutor::current(), UserSchema)
        .await
        .unwrap();
}

Tonbo automatically generates the UserSchema struct at compile time, so you don’t need to handle it manually. However, ensure that the specified path exists before creating your DBOption.

When using Tonbo in a WASM environment, use Path::from_opfs_path instead of Path::from_filesystem_path.

Operations on Database

After creating the DB, you can perform operations like insert, remove, and get. However, when you retrieve a record from Tonbo, you'll receive a UserRef instance—not a direct User instance. The UserRef struct, which implements the RecordRef trait, is automatically generated by Tonbo at compile time. It might look something like this:

#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub struct UserRef<'r> {
    pub name: &'r str,
    pub email: Option<&'r str>,
    pub age: Option<u8>,
}
impl RecordRef for UserRef<'_> {
    // ......
}

Insert

DB::insert takes a Record instance—specifically, an instance of the struct you've defined with #[derive(Record)]:

db.insert(User { /* ... */ }).await.unwrap();

Remove

DB::remove accepts a Key, where the type of the key is defined by the field annotated with #[record(primary_key)]. This method removes the record associated with the provided key:

db.remove("Alice".into()).await.unwrap();

Get

DB::get accepts a Key and processes the corresponding record using a closure that receives a TransactionEntry. Within the closure, you can call TransactionEntry::get to retrieve the record as a RecordRef instance:

let age = db.get(&"Alice".into(),
    |entry| {
        // entry.get() will get a `UserRef`
        let user = entry.get();
        println!("{:#?}", user);
        user.age
    })
    .await
    .unwrap();

Scan

Similar to DB::get, DB::scan accepts a closure that processes a TransactionEntry. However, instead of a single key, DB::scan operates over a range of keys, applying the closure to every record that falls within that range:

let lower = "Alice".into();
let upper = "Bob".into();
let stream = db
    .scan(
        (Bound::Included(&lower), Bound::Excluded(&upper)),
        |entry| {
            let record_ref = entry.get();

            record_ref.age
        },
    )
    .await;
let mut stream = std::pin::pin!(stream);
while let Some(data) = stream.next().await.transpose().unwrap() {
    // ...
}

Using transaction

Tonbo supports transaction. You can also push down filter, limit and projection operators in query.

// create transaction
let txn = db.transaction().await;

let name = "Alice".into();

txn.insert(User { /* ... */ });
let user = txn.get(&name, Projection::All).await.unwrap();

let upper = "Blob".into();
// range scan of user
let mut scan = txn
    .scan((Bound::Included(&name), Bound::Excluded(&upper)))
    .take()
    .await
    .unwrap();

while let Some(entry) = scan.next().await.transpose().unwrap() {
    let data = entry.value(); // type of UserRef
    // ......
}

Persistence

Tonbo employs a Log-Structured Merge Tree (LSM) as its underlying data structure, meaning that some data may reside in memory. To persist this in-memory data, use the flush method.

When Write-Ahead Logging (WAL) is enabled, data is automatically written to disk. However, since Tonbo buffers WAL data by default, you should call the flush_wal method to ensure all data is recovered. If you prefer not to use WAL buffering, you can disable it by setting wal_buffer_size to 0:

let options = DbOption::new(
    Path::from_filesystem_path("./db_path/users").unwrap(),
    &UserSchema,
).wal_buffer_size(0);

If you don't want to use WAL, you can disable it by setting the DbOption::disable_wal.

let options = DbOption::new(
    Path::from_filesystem_path("./db_path/users").unwrap(),
    &UserSchema,
).disable_wal(true);

Note: If you disable WAL, there is nothing to do with flush_wal. You need to call flush method to persist the memory data.

Conversely, if WAL is enabled and wal_buffer_size is set to 0, WAL data is flushed to disk immediately, so calling flush_wal is unnecessary.

Using with S3

If you want to use Tonbo with S3, you can configure DbOption to determine which portions of your data are stored in S3 and which remain on the local disk. The example below demonstrates how to set up this configuration:

let s3_option = FsOptions::S3 {
    bucket: "bucket".to_string(),
    credential: Some(AwsCredential {
        key_id: "key_id".to_string(),
        secret_key: "secret_key".to_string(),
        token: None,
    }),
    endpoint: None,
    sign_payload: None,
    checksum: None,
    region: Some("region".to_string()),
};
let options = DbOption::new(
    Path::from_filesystem_path("./db_path/users").unwrap(),
    &UserSchema,
).level_path(2, "l2", s3_option.clone())
).level_path(3, "l3", s3_option);

In this example, data for level 2 and level 3 will be stored in S3, while all other levels remain on the local disk. If there is data in level 2 and level 3, you can verify and access it in S3:

s3://bucket/l2/
├── xxx.parquet
├── ......
s3://bucket/l3/
├── xxx.parquet
├── ......

For more configuration options, please refer to the Configuration section.

What next?