Trait rlink::core::function::InputFormat

pub trait InputFormat where
    Self: InputSplitSource + NamedFunction + CheckpointFunction, {
    fn open(&mut self, input_split: InputSplit, context: &Context) -> Result<()>;
    fn record_iter(&mut self) -> Box<dyn Iterator<Item = Record> + Send>;
    fn close(&mut self) -> Result<()>;
    fn schema(&self, input_schema: FnSchema) -> FnSchema;
    fn parallelism(&self) -> u16;

    fn element_iter(&mut self) -> Box<dyn Iterator<Item = Element> + Send> { ... }
    fn daemon(&self) -> bool { ... }
}

The base interface for data sources that produces records.

Required methods

fn open(&mut self, input_split: InputSplit, context: &Context) -> Result<()>

Initialization of InputFormat, Each task will be called once when it starts.

fn record_iter(&mut self) -> Box<dyn Iterator<Item = Record> + Send>

return an Iterator of Record, if the next of Iterator is None, the task of InputFormat will be Terminated. the function is called by element_iter, a user-friendly function, usually you just need to implement it

fn close(&mut self) -> Result<()>

fn schema(&self, input_schema: FnSchema) -> FnSchema

fn parallelism(&self) -> u16

Provided methods

fn element_iter(&mut self) -> Box<dyn Iterator<Item = Element> + Send>

return an Iterator of Element, if the next of Iterator is None, the task of InputFormat will be Terminated. the function is called by runtime

fn daemon(&self) -> bool

mark the InputFormat is running in daemon mode, if true, this InputFormat is automatically terminated when any task instance ends

Implementors

impl<T> InputFormat for IteratorInputFormat<T> where
    T: FnOnce(InputSplit, Context) -> Box<dyn Iterator<Item = Record> + Send>,