Expand description
ERDOS is a platform for developing self-driving car and robotics applications. The system is built using techniques from streaming dataflow systems which is reflected by the API.
Applications are modeled as directed graphs, in which data flows through streams and is processed by operators. Because applications often resemble a sequence of connected operators, an ERDOS application may also be referred to as a pipeline.
Example
This example shows an ERDOS application which counts the number of objects detected from a stream of camera frames. The example consists of the driver part of the program, which is responsible for connecting operators via streams. For information on building operators, see § Operators.
// Capture arguments to set up an ERDOS node.
let args = erdos::new_app("ObjectCounter");
// Create an ERDOS node which runs the application.
let mut node = Node::new(Configuration::from_args(&args));
// Stream of RGB images from a camera.
let camera_frames = erdos::connect_source(
CameraOperator::new,
OperatorConfig::new().name("Camera")
);
// Stream of labeled bounding boxes for each RGB image.
let detected_objects = erdos::connect_one_in_one_out(
ObjectDetector::new,
|| {},
OperatorConfig::new().name("Detector"),
&camera_frames
);
// Stream of detected object count for each RGB image.
let num_detected = erdos::connect_one_in_one_out(
|| { MapOperator::new(|bboxes: &Vec<BBox>| -> usize { bboxes.len() }) },
|| {},
OperatorConfig::new().name("Counter"),
&detected_objects
);
// Run the application
node.run();
Operators
ERDOS operators process received data, and use
streams to broadcast
Message
s to downstream operators.
ERDOS provides a standard library of operators
for common dataflow patterns.
While the standard operators are general and versatile, some applications
may implement custom operators to better optimize performance and take
fine-grained control over exection.
Implementing Operators
For an example, see the implementation of the
FlatMapOperator
.
Operators are structures which implement an
operator trait reflecting their
communication pattern.
For example, the SplitOperator
implements OneInTwoOut
because it receives data on one input stream, and sends messages on
two output streams.
Operators can support both push and pull-based models of execution
by implementing methods defined in the
operator traits.
By implementing callbacks such as
OneInOneOut::on_data
,
operators can process messages as they arrive.
Moreover, operators can implement callbacks over watermarks
(e.g. OneInOneOut::on_watermark
)
to ensure ordered processing over timestamps.
ERDOS ensures lock-free, safe, and concurrent processing by ordering
callbacks in an ERDOS-managed execution lattice, which serves as a run
queue for the system’s multithreaded runtime.
While ERDOS manages the execution of callbacks, some operators require
more finegrained control. Operators can use the pull-based model
to take over the thread of execution by overriding the run
method
(e.g. OneInOneOut::run
)
of an operator trait, and pulling data from
the ReadStream
s.
Callbacks are not invoked while run
executes.
Performance
ERDOS is designed for low latency. Self-driving car pipelines require end-to-end deadlines on the order of hundreds of milliseconds for safe driving. Similarly, self-driving cars typically process gigabytes per second of data on small clusters. Therefore, ERDOS is optimized to send small amounts of data (gigabytes as opposed to terabytes) as quickly as possible.
Watermarks
Watermarks in ERDOS signal completion of computation. More concretely,
sending a watermark with timestamp t
on a stream asserts that all future
messages sent on that stream will have timestamps t' > t
.
ERDOS also introduces a top watermark, which is a watermark with the
maximum possible timestamp. Sending a top watermark closes the stream as
there is no t' > t_top
, so no more messages can be sent.
Determinism
ERDOS provides mechanisms to enable the building of deterministic applications. For instance, processing sets of messages separated by watermarks using watermark callbacks can turn ERDOS pipelines into Kahn process networks.
More Information
To read more about the ideas behind ERDOS, refer to our paper D3: A Dynamic Deadline-Driven Approach for Building Autonomous Vehicles. If you find ERDOS useful to your work, please cite our paper as follows:
@inproceedings{10.1145/3492321.3519576,
author = {Gog, Ionel and Kalra, Sukrit and Schafhalter, Peter and Gonzalez, Joseph E. and Stoica, Ion},
title = {D3: A Dynamic Deadline-Driven Approach for Building Autonomous Vehicles},
year = {2022},
isbn = {9781450391627},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3492321.3519576},
doi = {10.1145/3492321.3519576},
booktitle = {Proceedings of the Seventeenth European Conference on Computer Systems},
pages = {453–471},
numpages = {19},
location = {Rennes, France},
series = {EuroSys '22}
}
Re-exports
pub use dataflow::connect::*;
pub use dataflow::OperatorConfig;
Modules
Functions and structures for building an ERDOS application.
Data structures for executing and ERDOS application.
Structs
Stores the configuration parameters of a node
.
Wrapper around uuid::Uuid
that implements Abomonation
for fast serialization.
Functions
Produces a deterministic, unique ID.
Defines command line arguments for running a multi-node ERDOS application.
Resets seed and creates a new dataflow graph.
Type Definitions
A unique identifier for an operator.