Crate dbsp

The dbsp crate implements a computational engine for continuous analysis of changing data. With DBSP, a programmer writes code in terms of computations on a complete data set, but DBSP implements it incrementally, meaning that changes to the data set run in time proportional to the size of the change rather than the size of the data set. This is a major advantage for applications that work with large data sets that change frequently in small ways.

The tutorial is a good place to start for a guided tour. After that, if you want to look through the API on your own, the circuit module is a reasonable starting point. For complete examples, visit the examples directory in the DBSP repository.

Theory

DBSP is underpinned by a formal theory:

• Budiu, Chajed, McSherry, Ryzhyk, Tannen. DBSP: Automatic Incremental View Maintenance for Rich Query Languages, Conference on Very Large Databases, August 2023, Vancouver, Canada

• Here is a presentation about DBSP at the 2023 Apache Calcite Meetup.

The model provides two things:

1. Semantics. DBSP defines a formal language of streaming operators and queries built out of these operators, and precisely specifies how these queries must transform input streams to output streams.

2. Algorithm. DBSP also gives an algorithm that takes an arbitrary query and generates an incremental dataflow program that implements this query correctly (in accordance with its formal semantics) and efficiently. Efficiency here means, in a nutshell, that the cost of processing a set of input events is proportional to the size of the input rather than the entire state of the database.

Re-exports

• pub use crate::time::Timestamp;
• pub use algebra::IndexedZSet;
• pub use algebra::ZSet;
• pub use circuit::ChildCircuit;
• pub use circuit::Circuit;
• pub use circuit::CircuitHandle;
• pub use circuit::DBSPHandle;
• pub use circuit::RootCircuit;
• pub use circuit::Runtime;
• pub use circuit::RuntimeError;
• pub use circuit::SchedulerError;
• pub use circuit::Stream;
• pub use operator::CollectionHandle;
• pub use operator::InputHandle;
• pub use operator::OutputHandle;
• pub use operator::UpsertHandle;
• pub use trace::ord::OrdIndexedZSet;
• pub use trace::ord::OrdZSet;
• pub use trace::DBData;
• pub use trace::DBTimestamp;
• pub use trace::DBWeight;
• pub use trace::Rkyv;

Modules

• algebra
  This module contains declarations of abstract algebraic concepts: monoids, groups, rings, etc.
• circuit
  Synchronous circuits over streams.
• mimalloc
• monitor
  Trace monitor that validates event traces from a circuit. It is used to test both the tracing mechanism and the circuit engine itself.
• operator
  DBSP stream operators.
• profile
  Built-in profiling capabilities.
• time
  Logical time
• trace
  Traces
• tutorial
  Developer tutorial
• utils

Macros

• arcstr_format
• arcstr_literal
• circuit_cache_key
  Declare an anonymous struct type to be used as a key in the cache and associated value type.
• indexed_zset
  Create an indexed Z-set with specified elements.
• metadata
• zset
  Create a Z-set with specified elements.
• zset_set
  Create a Z-set with specified elements all with weight 1.

Enums

• Error

Traits

• DetailedError
• NumEntries
  Trait to report object size as the number of entries.
• RefPair
  Trait for types that can be converted into a pair of references

Functions

• default_hash
  Default hashing function used to shard records across workers.