Struct spark_connect_rs::spark::read::DataSource

pub struct DataSource {
    pub format: Option<String>,
    pub schema: Option<String>,
    pub options: HashMap<String, String>,
    pub paths: Vec<String>,
    pub predicates: Vec<String>,
}

Fields

format: Option<String>

(Optional) Supported formats include: parquet, orc, text, json, parquet, csv, avro.

If not set, the value from SQL conf ‘spark.sql.sources.default’ will be used.

schema: Option<String>

(Optional) If not set, Spark will infer the schema.

This schema string should be either DDL-formatted or JSON-formatted.

options: HashMap<String, String>

Options for the data source. The context of this map varies based on the data source format. This options could be empty for valid data source format. The map key is case insensitive.

paths: Vec<String>

(Optional) A list of path for file-system backed data sources.

predicates: Vec<String>

(Optional) Condition in the where clause for each partition.

This is only supported by the JDBC data source.

Implementations

impl DataSource

pub fn format(&self) -> &str

Returns the value of format, or the default value if format is unset.

pub fn schema(&self) -> &str

Returns the value of schema, or the default value if schema is unset.

Trait Implementations

impl Clone for DataSource

fn clone(&self) -> DataSource

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DataSource

fn default() -> Self

Returns the “default value” for a type.

impl Message for DataSource

fn encoded_len(&self) -> usize

Returns the encoded length of the message without a length delimiter.

fn clear(&mut self)

Clears the message, resetting all fields to their default.

fn encode<B>(&self, buf: &mut B) -> Result<(), EncodeError>where B: BufMut, Self: Sized,

Encodes the message to a buffer.

fn encode_to_vec(&self) -> Vec<u8, Global>where Self: Sized,

Encodes the message to a newly allocated buffer.

fn encode_length_delimited<B>(&self, buf: &mut B) -> Result<(), EncodeError>where B: BufMut, Self: Sized,

Encodes the message with a length-delimiter to a buffer.

fn encode_length_delimited_to_vec(&self) -> Vec<u8, Global>where Self: Sized,

Encodes the message with a length-delimiter to a newly allocated buffer.

fn decode<B>(buf: B) -> Result<Self, DecodeError>where B: Buf, Self: Default,

Decodes an instance of the message from a buffer.

fn decode_length_delimited<B>(buf: B) -> Result<Self, DecodeError>where B: Buf, Self: Default,

Decodes a length-delimited instance of the message from the buffer.

fn merge<B>(&mut self, buf: B) -> Result<(), DecodeError>where B: Buf, Self: Sized,

Decodes an instance of the message from a buffer, and merges it into self.

fn merge_length_delimited<B>(&mut self, buf: B) -> Result<(), DecodeError>where B: Buf, Self: Sized,

Decodes a length-delimited instance of the message from buffer, and merges it into self.

impl PartialEq<DataSource> for DataSource

fn eq(&self, other: &DataSource) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for DataSource