Struct SparkOptions

pub struct SparkOptions {
    pub archive_uris: Option<Vec<String>>,
    pub connection: Option<String>,
    pub container_image: Option<String>,
    pub file_uris: Option<Vec<String>>,
    pub jar_uris: Option<Vec<String>>,
    pub main_file_uri: Option<String>,
    pub properties: Option<HashMap<String, String>>,
    pub py_file_uris: Option<Vec<String>>,
    pub runtime_version: Option<String>,
}

Options for a user-defined Spark routine.

This type is not used in any activity, and only used as part of another schema.

Fields

archive_uris: Option<Vec<String>>

Archive files to be extracted into the working directory of each executor. For more information about Apache Spark, see Apache Spark.

connection: Option<String>

Fully qualified name of the user-provided Spark connection object. Format: "projects/{project_id}/locations/{location_id}/connections/{connection_id}"

container_image: Option<String>

Custom container image for the runtime environment.

file_uris: Option<Vec<String>>

Files to be placed in the working directory of each executor. For more information about Apache Spark, see Apache Spark.

jar_uris: Option<Vec<String>>

JARs to include on the driver and executor CLASSPATH. For more information about Apache Spark, see Apache Spark.

main_file_uri: Option<String>

The main file/jar URI of the Spark application. Exactly one of the definition_body field and the main_file_uri field must be set for Python. Exactly one of main_class and main_file_uri field should be set for Java/Scala language type.

properties: Option<HashMap<String, String>>

Configuration properties as a set of key/value pairs, which will be passed on to the Spark application. For more information, see Apache Spark.

py_file_uris: Option<Vec<String>>

Python files to be placed on the PYTHONPATH for PySpark application. Supported file types: .py, .egg, and .zip. For more information about Apache Spark, see Apache Spark.

runtime_version: Option<String>

Runtime version. If not specified, the default runtime version is used.

Trait Implementations

impl Clone for SparkOptions

fn clone(&self) -> SparkOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SparkOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SparkOptions

fn default() -> SparkOptions

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for SparkOptions

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for SparkOptions

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Part for SparkOptions