Struct DataStreamWriter

pub struct DataStreamWriter { /* private fields */ }

DataStreamWriter provides the ability to output a StreamingQuery which can then be used to monitor the active stream

Implementations

impl DataStreamWriter

pub fn new(dataframe: DataFrame) -> Self

Create a new DataStreamWriter from a provided streaming DataFrame

Defaults

• format: None,
• output_mode: OutputMode,
• query_name: None,
• trigger: Trigger,
• partition_by: vec![],
• write_options: HashMap::new()

pub fn format(self, format: &str) -> Self

Target format to output the StreamingQuery

pub fn output_mode(self, output_mode: OutputMode) -> Self

Specifies the behavior when data or table already exists

Arguments:

• output_mode: OutputMode enum

pub fn partition_by<'a, I>(self, cols: I) -> Self

where I: IntoIterator<Item = &'a str>,

Partitions the output by the given columns on the file system

pub fn option(self, key: &str, value: &str) -> Self

Add an input option for the underlying data source

pub fn options<I, K, V>(self, options: I) -> Self

where I: IntoIterator<Item = (K, V)>, K: AsRef<str>, V: AsRef<str>,

Set many input options based on an iterator of (key/value pairs) for the underlying data source

pub fn query_name(self, name: &str) -> Self

pub fn trigger(self, trigger: Trigger) -> Self

Query trigger for data to be processed by

pub async fn start( self, path: Option<&str>, ) -> Result<StreamingQuery, SparkError>

Start a streaming job to save the contents of the StreamingQuery to a data source.

The data source is specified by the format and a set of options.

pub async fn to_table( self, table_name: &str, ) -> Result<StreamingQuery, SparkError>

Start a streaming job to save the contents of the StreamingQuery to a table.

Trait Implementations

impl Clone for DataStreamWriter

fn clone(&self) -> DataStreamWriter

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataStreamWriter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.