Crate erdos

ERDOS is a platform for developing self-driving car and robotics applications. The system is built using techniques from streaming dataflow systems which is reflected by the API.

Applications are modeled as directed graphs, in which data flows through streams and is processed by operators. Because applications often resemble a sequence of connected operators, an ERDOS application may also be referred to as a pipeline.

Example

This example shows an ERDOS application which counts the number of objects detected from a stream of camera frames. The example consists of the driver part of the program, which is responsible for connecting operators via streams. For information on building operators, see § Operators.

// Capture arguments to set up an ERDOS node.
let args = erdos::new_app("ObjectCounter");
// Create an ERDOS node which runs the application.
let mut node = Node::new(Configuration::from_args(args));

// Stream of RGB images from a camera.
let camera_frames = erdos::connect_1_write!(
    CameraOperator,
    OperatorConfig::new().name("Camera")
);
// Stream of labeled bounding boxes for each RGB image.
let detected_objects = erdos::connect_1_write!(
    ObjectDetector,
    OperatorConfig::new().name("Detector"),
    camera_stream
);
// Stream of detected object count for each RGB image.
let num_detected = erdos::connect_1_write!(
    MapOperator,
    OperatorConfig::new()
        .name("Counter")
        .arg(|bboxes: &Vec<BBox>| -> usize { bbxes.len() }),
    detected_objects
);

// Run the application
node.run();

Driver

The driver section of the program connects operators together using streams to build an ERDOS application which may then be executed. The driver is typically the main function in main.rs.

The driver may also interact with a running ERDOS application. Using the IngestStream, the driver can send data to operators on a stream. The ExtractStream allows the driver to read data sent from an operator.

Streams

Data is broadcast to all receivers when sending on an ERDOS stream. Streams are typed on their data, and expose 2 classes of interfaces that access the underlying stream:

1. Read-interfaces expose methods to receive and process data. They allow pulling data by calling read() and try_read(). Often, they also support a push data model accessed by registering callbacks (e.g. add_callback and add_watermark_callback). Structures that implement read interfaces include:
   • ReadStream: used by operators to read data and register callbacks.
   • ExtractStream: used by the driver to read data.
2. Write-interfaces expose the send method to send data on a stream. Structures that implement write interfaces include:
   • WriteStream: used by operators to send data.
   • IngestStream: used by the driver to send data.

Some applications may want to introduce loops in their dataflow graphs which is possible using the LoopStream.

Operators

An ERDOS operator receives data on ReadStreams, and sends processed data on WriteStreams. We provide a standard library of operators for common dataflow patterns. While the standard operators are general and versatile, some applications may implement custom operators to better optimize performance and take fine-grained control over exection.

All operators must implement new and connect methods, in addition to the Operator trait.

• The new method takes an OperatorConfig, all ReadStreams from which the operator receives data, all WriteStreams on which the operator sends data, and returns Self. Within new, the state should be initialized, and callbacks may be registered across ReadStreams.
• The connect method takes references to the required ReadStreams and returns WriteStreams in the same order as in new.

For an example, see the implementation of the MapOperator.

While ERDOS manages the execution of callbacks, some operators require more finegrained control. Operators can take manual control over the thread of execution by overriding the run of the Operator trait and pulling data from ReadStreams. Callbacks are not invoked while run executes.

Performance

ERDOS is designed for low latency. Self-driving car pipelines require end-to-end deadlines on the order of hundreds of milliseconds for safe driving. Similarly, self-driving cars typically process gigabytes per second of data on small clusters. Therefore, ERDOS is optimized to send small amounts of data (gigabytes as opposed to terabytes) as quickly as possible.

Watermarks

Watermarks in ERDOS signal completion of computation. More concretely, sending a watermark with timestamp t on a stream asserts that all future messages sent on that stream will have timestamps t' > t. ERDOS also introduces a top watermark, which is a watermark with the maximum possible timestamp. Sending a top watermark closes the stream as there is no t' > t_top, so no more messages can be sent.

Determinism

ERDOS provides mechanisms to enable the building of deterministic applications. For instance, processing sets of messages separated by watermarks using watermark callbacks and time-versioned state turns ERDOS pipelines into Kahn process networks.

Re-exports

pub use dataflow::OperatorConfig;

Modules

dataflow	Functions and structures for building an ERDOS application.
node	Data structures for executing and ERDOS application.

Macros

add_watermark_callback	Adds a watermark callback across several read streams.
connect_0_write	Connects read streams to an operator that writes on 0 streams.
connect_1_write	Connects read streams to an operator that writes on 1 stream.
connect_2_write	Connects read streams to an operator that writes on 2 streams.
connect_3_write	Connects read streams to an operator that writes on 3 streams.

Structs

Configuration	Stores the configuration parameters of a node.
Uuid	Wrapper around uuid::Uuid that implements Abomonation for fast serialization.

Functions

generate_id	Produces a deterministic, unique ID.
get_terminal_logger	Returns a logger that prints messages to the console.
new_app	Defines command line arguments for running a multi-node ERDOS application.
reset	Resets seed and creates a new dataflow graph.

Type Definitions

OperatorId

A unique identifier for an operator.