sonnerie 0.4.2

An easy timeseries database
sonnerie-0.4.2 is not a library.
Visit the last successful build: sonnerie-0.8.6

GitHub license Crates.io

Introduction

Sonnerie is a time-series database. Map a timestamp to a floating-point value. Store multiple of these series in a single database. Insert tens of millions of samples in minutes, on rotational media.

Sonnerie includes a Client API, which has its own API docs.

Features

  • A straight-forward protocol for reading and writing
  • Easy setup: insert data with "netcat" or "telnet" in 5 minutes
  • No query language
  • Transactional: a transaction is completely committed or not at all.
  • Isolated: A transaction doesn't see updates from other transactions or expose its changes until it has been committed.
  • Durable: committed data is resistant to loss from unexpected shutdown.
  • Nanosecond-resolution timestamps (64 bit), 1970-2554
  • No weird dependencies, no virtual machines, one single native binary
  • floating point and integer values, multiple columns per sample

Sonnerie runs on Unix-like systems and is developed on Linux.

Performance

Sonnerie is designed to accept millions of samples across disparate series quickly, and then later fetch ranges from individual series. Memory is used for write-combining, write-ahead-logs are used to keep commits fast while still durable.

Fundamentally, the database is append-only. Edits and insertions are costly.

Why

Sonnerie is for storing data that you can plot.

You intake a lot of samples related to different entities all at once, and then want to read a lot of data for a entity, the disk usage patterns become very different

Timestamp Entity 1 Entity 2 Entity 3
2000-01-01 00:00:00 50.0 23.0 95.3
2000-01-02 00:00:00 24.0
2000-01-03 00:00:00 51.5 25.0
2000-01-04 00:00:00 53.0 26.0 94.8

At each timestamp (row), you insert some samples (it can be millions).

Some time later on, you want to run an analysis on a single Entity, Sonnerie allows one to quickly access all its values (an entire column).

Quick Start

Install

Sonnerie is implemented in Rust, a systems programming language that runs blazingly fast. Installation from source therefor requires you to install the rust compiler, which is as simple as: curl https://sh.rustup.rs -sSf | sh.

Sonnerie can then be installed from Cargo: cargo install sonnerie.

Sonnerie consists of one executable, sonnerie (~/.cargo/bin/sonnerie)

Run Server

sonnerie start -d <database directory to use>.

Sonnerie is running in the background, listening on [::1]:5599 for connections.

Start the Sonnerie Client

sonnerie client

After you start the client, you enter a shell-like command line from within all of the below is run.

Insert data

Start a transaction:

begin --write

Create a series:

create fibonacci

Add a few values to the series

add fibonacci 2018-01-01T00:00:00 1
add fibonacci 2018-01-02T00:00:00 1
add fibonacci 2018-01-03T00:00:00 2
add fibonacci 2018-01-04T00:00:00 3
add fibonacci 2018-01-05T00:00:00 5
add fibonacci 2018-01-06T00:00:00 8

Read some of those values back:

read fibonacci -f 2018-01-03 -t 2018-01-06

Sonnerie replies with:

2018-01-03 00:00:00     2
2018-01-04 00:00:00     3
2018-01-05 00:00:00     5
2018-01-06 00:00:00     8

Commit the transaction:

commit

After commit completes, the data is definitely on disk.

Try help and read --help (or --help with any command) for more information.

Multiple columns

In the above example, we are storing one value with each timestamp. You can also store multiple values with each timestamp of differing types. This is useful for multi-dimensional data.

When you create a series, you must specify the format ("type") of the records. The format is specified as a bunch of single character codes, one for each value.

The character codes are:

  • f - a 32 bit float (f32)
  • F - a 64 bit float (f64)
  • u - a 32 bit unsigned integer (u32)
  • U - a 64 bit unsigned integer (u64)
  • i - a 32 bit signed integer (i32)
  • I - a 64 bit signed integer (i64)

We can create another series plotting the coordinates of a jet travelling over the Pacific Ocean in terms of its longitude and latitude (respectively):

create oceanic-airlines --format ff

And we can insert values for this flight as well:

add oceanic-airlines 2018-01-01T00:00:00 37.686751 -122.602227
add oceanic-airlines 2018-01-01T00:00:01 37.686810 -122.603713
add oceanic-airlines 2018-01-01T00:00:02 37.686873 -122.605997
add oceanic-airlines 2018-01-01T00:00:03 37.687022 -122.609997
add oceanic-airlines 2018-01-01T00:00:04 37.687364 -122.610945
add oceanic-airlines 2018-01-01T00:00:05 37.687503 -122.615211

Errata

The protocol

Telnet into Sonnerie (telnet ::1 5599) and type "help" to see what you can do. The protocol is text-based and very similar to the client frontend.

Commands use shell-like escaping, so spaces can be escaped with a backslash. Timestamps are nanoseconds since the Unix Epoch.

The protocol formats floats with enough precision such that they can represent themselves exactly.

Fast imports

In order to ensure durability, many fsyncs need to be called (a few per transaction). This can slow down imports. You should consider running sonnerie prefixed with eatmydata, which is a Debian package. It will temporarily suppress fsync. After your import is done, start Sonnerie again normally.

When doing your inputs, tweak the size of the transaction until you find the optimal size. This might be a megabyte or so of data.

Backup

Online incremental backups are possible (the file format is designed accordingly) but not yet implemented.

You can do a full online backup as such, maintaining the following order:

mkdir dst
sqlite3 src/meta .dump | sqlite3 dst/meta
cp src/blocks-*.wal dst/
cp src/blocks dst/blocks

(This method will no longer apply once compacting is implemented).

Note that this method works even if you use erasure or insertion.

Semver

The file format and protocol are subject to change between different 0.x versions. Once a 1.0 is released, no backwards incompatible changes will be permitted before Sonnerie 2.0.

Implementation details

Metadata is stored in an sqlite3 database. Metadata is things like block locations and the names of series. Actual timeseries data is stored in a very large file named "blocks". They're called "blocks" because each series is stored in pieces of chronological samples.

Sonnerie may not be well suited for SSDs. This is because the design expects that for each sample (on different series), one of those blocks will have to be modified, and SSDs don't like you to write to the same part of the drive many times. An SSD-friendly design would have us write in blocks that each store data from many disparate series and then optimizing to read simultanously from many of those blocks.

Nevertheless, if you find that you write many consecutive values to a single series in a single transaction, then Sonnerie will suit an SSD.

Changelog

Refer to Changelog.md.

Roadmap

  • Online incremental backups
  • Compacting (not useful without insertion and appending)
  • Compression
  • An HTTP-based protocol (won't be quite complete)
  • Store variable-sized data (a string or blob per timestamp)

Copyright

Sonnerie was implemented by Charles Samuels at e.ventures Management LLC.