logo

Crate erdos

source · []
Expand description

ERDOS is a platform for developing self-driving car and robotics applications. The system is built using techniques from streaming dataflow systems which is reflected by the API.

Applications are modeled as directed graphs, in which data flows through streams and is processed by operators. Because applications often resemble a sequence of connected operators, an ERDOS application may also be referred to as a pipeline.

Example

This example shows an ERDOS application which counts the number of objects detected from a stream of camera frames. The example consists of the driver part of the program, which is responsible for connecting operators via streams. For information on building operators, see § Operators.

// Capture arguments to set up an ERDOS node.
let args = erdos::new_app("ObjectCounter");
// Create an ERDOS node which runs the application.
let mut node = Node::new(Configuration::from_args(&args));

// Stream of RGB images from a camera.
let camera_frames = erdos::connect_source(
    CameraOperator::new,
    OperatorConfig::new().name("Camera")
);
// Stream of labeled bounding boxes for each RGB image.
let detected_objects = erdos::connect_one_in_one_out(
    ObjectDetector::new,
    || {},
    OperatorConfig::new().name("Detector"),
    &camera_frames
);
// Stream of detected object count for each RGB image.
let num_detected = erdos::connect_one_in_one_out(
    || { MapOperator::new(|bboxes: &Vec<BBox>| -> usize { bboxes.len() }) },
    || {},
    OperatorConfig::new().name("Counter"),
    &detected_objects
);

// Run the application
node.run();

Operators

ERDOS operators process received data, and use streams to broadcast Messages to downstream operators. ERDOS provides a standard library of operators for common dataflow patterns. While the standard operators are general and versatile, some applications may implement custom operators to better optimize performance and take fine-grained control over exection.

Implementing Operators

For an example, see the implementation of the FlatMapOperator.

Operators are structures which implement an operator trait reflecting their communication pattern. For example, the SplitOperator implements OneInTwoOut because it receives data on one input stream, and sends messages on two output streams.

Operators can support both push and pull-based models of execution by implementing methods defined in the operator traits. By implementing callbacks such as OneInOneOut::on_data, operators can process messages as they arrive. Moreover, operators can implement callbacks over watermarks (e.g. OneInOneOut::on_watermark) to ensure ordered processing over timestamps. ERDOS ensures lock-free, safe, and concurrent processing by ordering callbacks in an ERDOS-managed execution lattice, which serves as a run queue for the system’s multithreaded runtime.

While ERDOS manages the execution of callbacks, some operators require more finegrained control. Operators can use the pull-based model to take over the thread of execution by overriding the run method (e.g. OneInOneOut::run) of an operator trait, and pulling data from the ReadStreams. Callbacks are not invoked while run executes.

Performance

ERDOS is designed for low latency. Self-driving car pipelines require end-to-end deadlines on the order of hundreds of milliseconds for safe driving. Similarly, self-driving cars typically process gigabytes per second of data on small clusters. Therefore, ERDOS is optimized to send small amounts of data (gigabytes as opposed to terabytes) as quickly as possible.

Watermarks

Watermarks in ERDOS signal completion of computation. More concretely, sending a watermark with timestamp t on a stream asserts that all future messages sent on that stream will have timestamps t' > t. ERDOS also introduces a top watermark, which is a watermark with the maximum possible timestamp. Sending a top watermark closes the stream as there is no t' > t_top, so no more messages can be sent.

Determinism

ERDOS provides mechanisms to enable the building of deterministic applications. For instance, processing sets of messages separated by watermarks using watermark callbacks can turn ERDOS pipelines into Kahn process networks.

More Information

To read more about the ideas behind ERDOS, refer to our paper D3: A Dynamic Deadline-Driven Approach for Building Autonomous Vehicles. If you find ERDOS useful to your work, please cite our paper as follows:

@inproceedings{10.1145/3492321.3519576,
    author = {Gog, Ionel and Kalra, Sukrit and Schafhalter, Peter and Gonzalez, Joseph E. and Stoica, Ion},
    title = {D3: A Dynamic Deadline-Driven Approach for Building Autonomous Vehicles},
    year = {2022},
    isbn = {9781450391627},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3492321.3519576},
    doi = {10.1145/3492321.3519576},
    booktitle = {Proceedings of the Seventeenth European Conference on Computer Systems},
    pages = {453–471},
    numpages = {19},
    location = {Rennes, France},
    series = {EuroSys '22}
}

Re-exports

pub use dataflow::connect::*;
pub use dataflow::OperatorConfig;

Modules

Functions and structures for building an ERDOS application.

Data structures for executing and ERDOS application.

Structs

Stores the configuration parameters of a node.

Wrapper around uuid::Uuid that implements Abomonation for fast serialization.

Functions

Produces a deterministic, unique ID.

Defines command line arguments for running a multi-node ERDOS application.

Resets seed and creates a new dataflow graph.

Type Definitions

A unique identifier for an operator.