Struct DataFrameWriter

pub struct DataFrameWriter { /* private fields */ }

DataFrameWriter provides the ability to output a DataFrame to a specific file format supported by Spark

Implementations

impl DataFrameWriter

pub fn new(dataframe: DataFrame) -> Self

Create a new DataFrameWriter from a provided DataFrame

Defaults

• format: None,
• mode: SaveMode::Overwrite,
• bucket_by: None,
• partition_by: vec![],
• sort_by: vec![],
• write_options: HashMap::new()

pub fn format(self, format: &str) -> Self

Target format to output the DataFrame

pub fn mode(self, mode: SaveMode) -> Self

Specifies the behavior when data or table already exists

Arguments:

• mode: SaveMode enum from the protobuf

pub fn bucket_by<'a, I>(self, num_buckets: i32, buckets: I) -> Self

where I: IntoIterator<Item = &'a str>,

Buckets the output by the given columns. If specified, the output is laid out on the file system similar to Hive’s bucketing scheme.

pub fn sort_by<'a, I>(self, cols: I) -> Self

where I: IntoIterator<Item = &'a str>,

Sorts the output in each bucket by the given columns on the file system

pub fn partition_by<'a, I>(self, cols: I) -> Self

where I: IntoIterator<Item = &'a str>,

Partitions the output by the given columns on the file system

pub fn option(self, key: &str, value: &str) -> Self

Add an input option for the underlying data source

pub fn options<I, K, V>(self, options: I) -> Self

where I: IntoIterator<Item = (K, V)>, K: AsRef<str>, V: AsRef<str>,

Set many input options based on an iterator of (key/value pairs) for the underlying data source

pub async fn save(self, path: &str) -> Result<(), SparkError>

Save the contents of the DataFrame to a data source.

The data source is specified by the format and a set of options.

pub async fn save_as_table(self, table_name: &str) -> Result<(), SparkError>

Saves the context of the DataFrame as the specified table.

pub async fn insert_into(self, table_name: &str) -> Result<(), SparkError>

Inserts the content of the DataFrame to the specified table.

It requires that the schema of the DataFrame is the same as the schema of the target table.

Unlike saveAsTable(), this method ignores the column names and just uses position-based resolution

pub async fn csv<C: ConfigOpts>( self, path: &str, config: C, ) -> Result<(), SparkError>

Writes the DataFrame to a CSV file with the specified options.

pub async fn json<C: ConfigOpts>( self, path: &str, config: C, ) -> Result<(), SparkError>

Writes the DataFrame to a JSON file with the specified options.

pub async fn orc<C: ConfigOpts>( self, path: &str, config: C, ) -> Result<(), SparkError>

Writes the DataFrame to an ORC file with the specified options.

pub async fn parquet<C: ConfigOpts>( self, path: &str, config: C, ) -> Result<(), SparkError>

Writes the DataFrame to a Parquet file with the specified options.

pub async fn text<C: ConfigOpts>( self, path: &str, config: C, ) -> Result<(), SparkError>

Writes the DataFrame to a text file with the specified options.