Crate noria

This create contains client bindings for Noria.

What is Noria?

Noria is a new streaming data-flow system designed to act as a fast storage backend for read-heavy web applications based on this paper from OSDI’18. It acts like a databases, but pre-computes and caches relational query results so that reads are blazingly fast. Noria automatically keeps cached results up-to-date as the underlying data, stored in persistent base tables change. Noria uses partially-stateful data-flow to reduce memory overhead, and supports dynamic, runtime data-flow and query change.

Infrastructure

Like most databases, Noria follows a server-client model where many clients connect to a (potentially distributed) server. The server in this case is the noria-server binary, and must be started before clients can connect. See noria-server --help for details and the Noria repository README for details. Noria uses Apache ZooKeeper to announce the location of its servers, so ZooKeeper must also be running.

Quickstart example

If you just want to get up and running quickly, here’s some code to dig into. Note that this requires a nightly release of Rust to run for the time being.

#[tokio::main]
async fn main() {
    let zookeeper_addr = "127.0.0.1:2181";
    let mut db = ControllerHandle::from_zk(zookeeper_addr).await.unwrap();

    // if this is the first time we interact with Noria, we must give it the schema
    db.install_recipe("
        CREATE TABLE Article (aid int, title varchar(255), url text, PRIMARY KEY(aid));
        CREATE TABLE Vote (aid int, uid int);
    ").await.unwrap();

    // we can then get handles that let us insert into the new tables
    let mut article = db.table("Article").await.unwrap();
    let mut vote = db.table("Vote").await.unwrap();

    // let's make a new article
    let aid = 42;
    let title = "I love Soup";
    let url = "https://pdos.csail.mit.edu";
    article
        .insert(vec![aid.into(), title.into(), url.into()])
        .await
        .unwrap();

    // and then vote for it
    vote.insert(vec![aid.into(), 1.into()]).await.unwrap();

    // we can also declare views that we want want to query
    db.extend_recipe("
        VoteCount: \
          SELECT Vote.aid, COUNT(uid) AS votes \
          FROM Vote GROUP BY Vote.aid;
        QUERY ArticleWithVoteCount: \
          SELECT Article.aid, title, url, VoteCount.votes AS votes \
          FROM Article LEFT JOIN VoteCount ON (Article.aid = VoteCount.aid) \
          WHERE Article.aid = ?;").await.unwrap();

    // and then get handles that let us execute those queries to fetch their results
    let mut awvc = db.view("ArticleWithVoteCount").await.unwrap();
    // looking up article 42 should yield the article we inserted with a vote count of 1
    assert_eq!(
        awvc.lookup(&[aid.into()], true).await.unwrap(),
        vec![vec![DataType::from(aid), title.into(), url.into(), 1.into()]]
    );
}

Client model

Noria accepts a set of parameterized SQL queries (think prepared statements), and produces a data-flow program that maintains materialized views for the output of those queries. Reads now become fast lookups directly into these materialized views, as if the value had been directly read from a cache (like memcached). The views are automatically kept up-to-date by Noria through the data-flow.

Reads work quite differently in Noria compared to traditional relational databases. In particular, a query, or view, must be registered before it can be executed, much like SQL prepared statements. Use ControllerHandle::extend_recipe to register new base tables and views. Once a view has been registered, you can get a handle that lets you execute the corresponding query by passing the view’s name to ControllerHandle::view. The returned View can be used to query the view with different values for its declared parameters (values in place of ? in the query) through View::lookup and View::multi_lookup.

Writes are fairly similar to those in relational databases. To add a new table, you extend the recipe (using ControllerHandle::extend_recipe) with a CREATE TABLE statement, and then use ControllerHandle::table to get a handle to the new base table. Base tables support similar operations as SQL tables, such as Table::insert, Table::update, Table::delete, and also more esoteric operations like Table::insert_or_update.

Alternatives

Noria provides a MySQL adapter that implements the binary MySQL protocol, which provides a compatibility layer for applications that wish to continue to issue ad-hoc MySQL queries through existing MySQL client libraries.

Modules

debug

Types used when debugging Noria.

error

Noria errors.

prelude

The prelude contains most of the types needed in everyday operation.

results

Wrapper types for Noria query results.

Macros

row

Create a new row for insertion into a Table using column names.

update

Create an update for a given Table using column names.

Structs

ActivationResult

Represents the result of a recipe activation.

ControllerHandle

A handle to a Noria controller.

Table

A Table is used to perform writes, deletes, and other operations to data in base tables.

View

A View is used to query previously defined external views.

ZookeeperAuthority

Coordinator that shares connection information between workers and clients using ZooKeeper.

Enums

DataType

The main type used for user data throughout the codebase.

Modification

A modification to make to a column in an existing row.

Operation

A modification to make to an existing value.

TableOperation

An operation to apply to a base table.

Functions

trace_ops_in

The next Noria read or write issued from the current thread will be traced using tokio-trace.