timely/
lib.rs

1//! Timely dataflow is a framework for managing and executing data-parallel dataflow computations.
2//!
3//! The code is organized in crates and modules that are meant to depend as little as possible on each other.
4//!
5//! **Serialization**: The [`abomonation`] crate contains simple and highly unsafe
6//! serialization routines.
7//!
8//! **Communication**: The [`timely_communication`] crate defines several primitives for
9//! communicating between dataflow workers, and across machine boundaries.
10//!
11//! **Progress tracking**: The [`timely::progress`](progress) module defines core dataflow structures for
12//! tracking and reporting progress in a timely dataflow system, namely the number of outstanding
13//! dataflow messages and un-exercised message capabilities throughout the timely dataflow graph.
14//! It depends on `timely_communication` to exchange progress messages.
15//!
16//! **Dataflow construction**: The [`timely::dataflow`](dataflow) module defines an example dataflow system
17//! using `communication` and `progress` to both exchange data and progress information, in support
18//! of an actual data-parallel timely dataflow computation. It depends on `timely_communication` to
19//! move data, and `timely::progress` to provide correct operator notifications.
20//!
21//! # Examples
22//!
23//! The following is a hello-world dataflow program.
24//!
25//! ```
26//! use timely::*;
27//! use timely::dataflow::operators::{Input, Inspect};
28//!
29//! // construct and execute a timely dataflow
30//! timely::execute_from_args(std::env::args(), |worker| {
31//!
32//!     // add an input and base computation off of it
33//!     let mut input = worker.dataflow(|scope| {
34//!         let (input, stream) = scope.new_input();
35//!         stream.inspect(|x| println!("hello {:?}", x));
36//!         input
37//!     });
38//!
39//!     // introduce input, advance computation
40//!     for round in 0..10 {
41//!         input.send(round);
42//!         input.advance_to(round + 1);
43//!         worker.step();
44//!     }
45//! });
46//! ```
47//!
48//! The program uses `timely::execute_from_args` to spin up a computation based on command line arguments
49//! and a closure specifying what each worker should do, in terms of a handle to a timely dataflow
50//! `Scope` (in this case, `root`). A `Scope` allows you to define inputs, feedback
51//! cycles, and dataflow subgraphs, as part of building the dataflow graph of your dreams.
52//!
53//! In this example, we define a new scope of root using `scoped`, add an exogenous
54//! input using `new_input`, and add a dataflow `inspect` operator to print each observed record.
55//! We then introduce input at increasing rounds, indicate the advance to the system (promising
56//! that we will introduce no more input at prior rounds), and step the computation.
57
58#![forbid(missing_docs)]
59
60#[macro_use]
61extern crate abomonation_derive;
62extern crate abomonation;
63extern crate serde;
64#[macro_use]
65extern crate serde_derive;
66extern crate timely_communication;
67extern crate timely_bytes;
68extern crate timely_logging;
69
70pub use execute::{execute, execute_directly, example};
71#[cfg(feature = "getopts")]
72pub use execute::execute_from_args;
73pub use order::PartialOrder;
74
75pub use timely_communication::Config as CommunicationConfig;
76pub use worker::Config as WorkerConfig;
77pub use execute::Config as Config;
78
79pub use timely_container::Container;
80/// Re-export of the `timely_container` crate.
81pub mod container {
82    pub use timely_container::*;
83}
84
85/// Re-export of the `timely_communication` crate.
86pub mod communication {
87    pub use timely_communication::*;
88}
89
90/// Re-export of the `timely_bytes` crate.
91pub mod bytes {
92    pub use timely_bytes::*;
93}
94
95/// Re-export of the `timely_logging` crate.
96pub mod logging_core {
97    pub use timely_logging::*;
98}
99
100pub mod worker;
101pub mod progress;
102pub mod dataflow;
103pub mod synchronization;
104pub mod execute;
105pub mod order;
106
107pub mod logging;
108// pub mod log_events;
109
110pub mod scheduling;
111
112/// A composite trait for types usable as data in timely dataflow.
113///
114/// The `Data` trait is necessary for all types that go along timely dataflow channels.
115pub trait Data: Clone+'static { }
116impl<T: Clone+'static> Data for T { }
117
118/// A composite trait for types usable on exchange channels in timely dataflow.
119///
120/// The `ExchangeData` trait extends `Data` with any requirements imposed by the `timely_communication`
121/// `Data` trait, which describes requirements for communication along channels.
122pub trait ExchangeData: Data + communication::Data { }
123impl<T: Data + communication::Data> ExchangeData for T { }
timely/lib.rs

timely/
lib.rs
