Struct FileSource 

pub struct FileSource { /* private fields */ }

Source that reads a text file line-by-line.

The file is divided in chunks and is read concurrently by multiple replicas.

Implementations

impl FileSource

pub fn new<P>(path: P) -> Self

where P: Into<PathBuf>,

Create a new source that reads the lines from a text file.

The file is partitioned into as many chunks as replicas, each replica has to have the same file in the same path. It is guaranteed that each line of the file is emitted by exactly one replica.

Note: the file must be readable and its size must be available. This means that only regular files can be read.

Example

let source = FileSource::new("/datasets/huge.txt");
let s = env.stream(source);

Trait Implementations

impl Clone for FileSource

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FileSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for FileSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Operator for FileSource

type Out = String

fn setup(&mut self, metadata: &mut ExecutionMetadata<'_>)

Setup the operator chain. This is called before any call to next and it’s used to initialize the operator. When it’s called the operator has already been cloned and it will never be cloned again. Therefore it’s safe to store replica-specific metadata inside of it.

fn next(&mut self) -> StreamElement<String>

Take a value from the previous operator, process it and return it.

fn structure(&self) -> BlockStructure

A more refined representation of the operator and its predecessors.

impl Source for FileSource

fn replication(&self) -> Replication

The maximum parallelism offered by this operator.