Struct DataFrameReader

pub struct DataFrameReader { /* private fields */ }

DataFrameReader represents the entrypoint to create a DataFrame from a specific file format.

Implementations

impl DataFrameReader

pub fn new(spark_session: SparkSession) -> Self

Create a new DataFrameReader with a SparkSession

pub fn format(self, format: &str) -> Self

Specifies the input data source format

pub fn option(self, key: &str, value: &str) -> Self

Add an input option for the underlying data source

pub fn schema<T: ToSchema>(self, schema: T) -> Self

pub fn options<I, K, V>(self, options: I) -> Self

where I: IntoIterator<Item = (K, V)>, K: AsRef<str>, V: AsRef<str>,

Set many input options based on an iterator of (key/value pairs) for the underlying data source

pub fn load<'a, I>(self, paths: I) -> Result<DataFrame, SparkError>

where I: IntoIterator<Item = &'a str>,

Loads data from a data source and returns it as a DataFrame

Example:

let path = vec!["some/dir/path/on/the/remote/cluster/"];

// returns a DataFrame from a csv file with a header from a the specific path
let mut df = spark.read().format("csv").option("header", "true").load(path);

pub fn table( self, table_name: &str, options: Option<HashMap<String, String>>, ) -> Result<DataFrame, SparkError>

Returns the specific table as a DataFrame

Arguments:

• table_name: &str of the table name
• options: (optional Hashmap) contains additional read options for a table

Trait Implementations

impl Clone for DataFrameReader

fn clone(&self) -> DataFrameReader

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataFrameReader

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.