Module timely::dataflow::operators::capture

Operators which capture and replay streams of records.

The capture_into and replay_into operators respectively capture what a unary operator sees as input (both data and progress information), and play this information back as a new input.

The capture_into method requires a P: EventPusher<T, D>, which is some type accepting Event<T, D> inputs. This module provides several examples, including the linked list EventLink<T, D>, and the binary EventWriter<T, D, W> wrapping any W: Write.

Examples

The type Rc<EventLink<T,D>> implements a typed linked list, and can be captured into and replayed from.

use std::rc::Rc;
use timely::dataflow::Scope;
use timely::dataflow::operators::{Capture, ToStream, Inspect};
use timely::dataflow::operators::capture::{EventLink, Replay};

timely::execute(timely::Configuration::Thread, |computation| {
    let handle1 = Rc::new(EventLink::new());
    let handle2 = handle1.clone();

    computation.scoped::<u64,_,_>(|scope1|
        (0..10).to_stream(scope1)
               .capture_into(handle1)
    );

    computation.scoped(|scope2| {
        handle2.replay_into(scope2)
               .inspect(|x| println!("replayed: {:?}", x));
    })
}).unwrap();

The types EventWriter<T, D, W> and EventReader<T, D, R> can be captured into and replayed from, respectively. The use binary writers and readers respectively, and can be backed by files, network sockets, etc.

use std::rc::Rc;
use std::net::{TcpListener, TcpStream};
use timely::dataflow::Scope;
use timely::dataflow::operators::{Capture, ToStream, Inspect};
use timely::dataflow::operators::capture::{EventReader, EventWriter, Replay};

timely::execute(timely::Configuration::Thread, |computation| {
    let list = TcpListener::bind("127.0.0.1:8000").unwrap();
    let send = TcpStream::connect("127.0.0.1:8000").unwrap();
    let recv = list.incoming().next().unwrap().unwrap();

    computation.scoped::<u64,_,_>(|scope1|
        (0..10u64)
            .to_stream(scope1)
            .capture_into(EventWriter::new(send))
    );

    computation.scoped::<u64,_,_>(|scope2| {
        EventReader::<_,u64,_>::new(recv)
            .replay_into(scope2)
            .inspect(|x| println!("replayed: {:?}", x));
    })
}).unwrap();

Structs

EventLink	A linked list of Event.
EventReader	A Wrapper for R: Read implementing EventIterator<T, D>.
EventWriter	A wrapper for W: Write implementing EventPusher<T, D>.

Enums

Event

Data and progres events of the captured stream.

Traits

Capture	Capture a stream of timestamped data for later replay.
EventIterator	Iterates over contained Event<T, D>.
EventPusher	Receives Event<T, D> events.
Replay	Replay a capture stream into a scope with the same timestamp.