Struct SparkOptions 

#[non_exhaustive]
pub struct SparkOptions {
    pub connection: String,
    pub runtime_version: String,
    pub container_image: String,
    pub properties: HashMap<String, String>,
    pub main_file_uri: String,
    pub py_file_uris: Vec<String>,
    pub jar_uris: Vec<String>,
    pub file_uris: Vec<String>,
    pub archive_uris: Vec<String>,
    pub main_class: String,
    /* private fields */
}

Options for a user-defined Spark routine.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

connection: String

Fully qualified name of the user-provided Spark connection object. Format: "projects/{project_id}/locations/{location_id}/connections/{connection_id}"

runtime_version: String

Runtime version. If not specified, the default runtime version is used.

container_image: String

Custom container image for the runtime environment.

properties: HashMap<String, String>

Configuration properties as a set of key/value pairs, which will be passed on to the Spark application. For more information, see Apache Spark and the procedure option list.

main_file_uri: String

The main file/jar URI of the Spark application. Exactly one of the definition_body field and the main_file_uri field must be set for Python. Exactly one of main_class and main_file_uri field should be set for Java/Scala language type.

py_file_uris: Vec<String>

Python files to be placed on the PYTHONPATH for PySpark application. Supported file types: .py, .egg, and .zip. For more information about Apache Spark, see Apache Spark.

jar_uris: Vec<String>

JARs to include on the driver and executor CLASSPATH. For more information about Apache Spark, see Apache Spark.

file_uris: Vec<String>

Files to be placed in the working directory of each executor. For more information about Apache Spark, see Apache Spark.

archive_uris: Vec<String>

Archive files to be extracted into the working directory of each executor. For more information about Apache Spark, see Apache Spark.

main_class: String

The fully qualified name of a class in jar_uris, for example, com.example.wordcount. Exactly one of main_class and main_jar_uri field should be set for Java/Scala language type.

Implementations

impl SparkOptions

pub fn new() -> Self

Creates a new default instance.

pub fn set_connection<T: Into<String>>(self, v: T) -> Self

Sets the value of connection.

Example

let x = SparkOptions::new().set_connection("example");

pub fn set_runtime_version<T: Into<String>>(self, v: T) -> Self

Sets the value of runtime_version.

Example

let x = SparkOptions::new().set_runtime_version("example");

pub fn set_container_image<T: Into<String>>(self, v: T) -> Self

Sets the value of container_image.

Example

let x = SparkOptions::new().set_container_image("example");

pub fn set_properties<T, K, V>(self, v: T) -> Self

where T: IntoIterator<Item = (K, V)>, K: Into<String>, V: Into<String>,

Sets the value of properties.

Example

let x = SparkOptions::new().set_properties([
    ("key0", "abc"),
    ("key1", "xyz"),
]);

pub fn set_main_file_uri<T: Into<String>>(self, v: T) -> Self

Sets the value of main_file_uri.

Example

let x = SparkOptions::new().set_main_file_uri("example");

pub fn set_py_file_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of py_file_uris.

Example

let x = SparkOptions::new().set_py_file_uris(["a", "b", "c"]);

pub fn set_jar_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of jar_uris.

Example

let x = SparkOptions::new().set_jar_uris(["a", "b", "c"]);

pub fn set_file_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of file_uris.

Example

let x = SparkOptions::new().set_file_uris(["a", "b", "c"]);

pub fn set_archive_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of archive_uris.

Example

let x = SparkOptions::new().set_archive_uris(["a", "b", "c"]);

pub fn set_main_class<T: Into<String>>(self, v: T) -> Self

Sets the value of main_class.

Example

let x = SparkOptions::new().set_main_class("example");

Trait Implementations

impl Clone for SparkOptions

fn clone(&self) -> SparkOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SparkOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SparkOptions

fn default() -> SparkOptions

Returns the “default value” for a type.

impl Message for SparkOptions

fn typename() -> &'static str

The typename of this message.

impl PartialEq for SparkOptions

fn eq(&self, other: &SparkOptions) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for SparkOptions