[−][src]Crate erdos
ERDOS is a platform for developing self-driving car and robotics applications. The system is built using techniques from streaming dataflow systems which is reflected by the API.
Applications are modeled as directed graphs, in which data flows through streams and is processed by operators. Because applications often resemble a sequence of connected operators, an ERDOS application may also be referred to as a pipeline.
Example
This example shows an ERDOS application which counts the number of objects detected from a stream of camera frames. The example consists of the driver part of the program, which is responsible for connecting operators via streams. For information on building operators, see § Operators.
// Capture arguments to set up an ERDOS node. let args = erdos::new_app("ObjectCounter"); // Create an ERDOS node which runs the application. let mut node = Node::new(Configuration::from_args(args)); // Stream of RGB images from a camera. let camera_frames = erdos::connect_1_write!( CameraOperator, OperatorConfig::new().name("Camera") ); // Stream of labeled bounding boxes for each RGB image. let detected_objects = erdos::connect_1_write!( ObjectDetector, OperatorConfig::new().name("Detector"), camera_stream ); // Stream of detected object count for each RGB image. let num_detected = erdos::connect_1_write!( MapOperator, OperatorConfig::new() .name("Counter") .arg(|bboxes: &Vec<BBox>| -> usize { bbxes.len() }), detected_objects ); // Run the application node.run();
Driver
The driver section of the program connects operators together using
streams to build an ERDOS application which may then be executed.
The driver is typically the main
function in main.rs
.
The driver may also interact with a running ERDOS application.
Using the IngestStream
,
the driver can send data to operators on a stream.
The ExtractStream
allows the driver to read data sent from an operator.
Streams
Data is broadcast to all receivers when sending on an ERDOS stream. Streams are typed on their data, and expose 2 classes of interfaces that access the underlying stream:
- Read-interfaces expose methods to receive and process data.
They allow pulling data by calling
read()
andtry_read()
. Often, they also support a push data model accessed by registering callbacks (e.g.add_callback
andadd_watermark_callback
). Structures that implement read interfaces include:ReadStream
: used by operators to read data and register callbacks.ExtractStream
: used by the driver to read data.
- Write-interfaces expose the
send
method to send data on a stream. Structures that implement write interfaces include:WriteStream
: used by operators to send data.IngestStream
: used by the driver to send data.
Some applications may want to introduce loops in their dataflow graphs
which is possible using the
LoopStream
.
Operators
An ERDOS operator receives data on
ReadStream
s,
and sends processed data on
WriteStream
s.
We provide a standard library of operators
for common dataflow patterns.
While the standard operators are general and versatile, some applications
may implement custom operators to better optimize performance and take
fine-grained control over exection.
All operators must implement new
and connect
methods, in addition to
the Operator
trait.
- The
new
method takes anOperatorConfig
, allReadStream
s from which the operator receives data, allWriteStream
s on which the operator sends data, and returnsSelf
. Withinnew
, the state should be initialized, and callbacks may be registered acrossReadStream
s. - The
connect
method takes references to the requiredReadStream
s and returnsWriteStream
s in the same order as innew
.
For an example, see the implementation of the
MapOperator
.
While ERDOS manages the execution of callbacks, some operators require
more finegrained control. Operators can take manual control over the
thread of execution by overriding the
run
of the
Operator
trait and pulling data from
ReadStream
s.
Callbacks are not invoked while run executes.
Performance
ERDOS is designed for low latency. Self-driving car pipelines require end-to-end deadlines on the order of hundreds of milliseconds for safe driving. Similarly, self-driving cars typically process gigabytes per second of data on small clusters. Therefore, ERDOS is optimized to send small amounts of data (gigabytes as opposed to terabytes) as quickly as possible.
Watermarks
Watermarks in ERDOS signal completion of computation. More concretely,
sending a watermark with timestamp t
on a stream asserts that all future
messages sent on that stream will have timestamps t' > t
.
ERDOS also introduces a top watermark, which is a watermark with the
maximum possible timestamp. Sending a top watermark closes the stream as
there is no t' > t_top
, so no more messages can be sent.
Determinism
ERDOS provides mechanisms to enable the building of deterministic applications. For instance, processing sets of messages separated by watermarks using watermark callbacks and time-versioned state turns ERDOS pipelines into Kahn process networks.
Re-exports
pub use dataflow::OperatorConfig; |
Modules
dataflow | Functions and structures for building an ERDOS application. |
node | Data structures for executing and ERDOS application. |
Macros
add_watermark_callback | Adds a watermark callback across several read streams. |
connect_0_write | Connects read streams to an operator that writes on 0 streams. |
connect_1_write | Connects read streams to an operator that writes on 1 stream. |
connect_2_write | Connects read streams to an operator that writes on 2 streams. |
connect_3_write | Connects read streams to an operator that writes on 3 streams. |
Structs
Configuration | Stores the configuration parameters of a |
Uuid | Wrapper around |
Functions
generate_id | Produces a deterministic, unique ID. |
get_terminal_logger | Returns a logger that prints messages to the console. |
new_app | Defines command line arguments for running a multi-node ERDOS application. |
reset | Resets seed and creates a new dataflow graph. |
Type Definitions
OperatorId | A unique identifier for an operator. |