Struct PySparkBatch 

#[non_exhaustive]
pub struct PySparkBatch {
    pub main_python_file_uri: String,
    pub args: Vec<String>,
    pub python_file_uris: Vec<String>,
    pub jar_file_uris: Vec<String>,
    pub file_uris: Vec<String>,
    pub archive_uris: Vec<String>,
    /* private fields */
}

A configuration for running an Apache PySpark batch workload.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

main_python_file_uri: String

Required. The HCFS URI of the main Python file to use as the Spark driver. Must be a .py file.

args: Vec<String>

Optional. The arguments to pass to the driver. Do not include arguments that can be set as batch properties, such as --conf, since a collision can occur that causes an incorrect batch submission.

python_file_uris: Vec<String>

Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

jar_file_uris: Vec<String>

Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks.

file_uris: Vec<String>

Optional. HCFS URIs of files to be placed in the working directory of each executor.

archive_uris: Vec<String>

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Implementations

impl PySparkBatch

pub fn new() -> Self

pub fn set_main_python_file_uri<T: Into<String>>(self, v: T) -> Self

Sets the value of main_python_file_uri.

pub fn set_args<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of args.

pub fn set_python_file_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of python_file_uris.

pub fn set_jar_file_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of jar_file_uris.

pub fn set_file_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of file_uris.

pub fn set_archive_uris<T, V>(self, v: T) -> Self

where T: IntoIterator<Item = V>, V: Into<String>,

Sets the value of archive_uris.

Trait Implementations

impl Clone for PySparkBatch

fn clone(&self) -> PySparkBatch

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PySparkBatch

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PySparkBatch

fn default() -> PySparkBatch

Returns the “default value” for a type.

impl Message for PySparkBatch

fn typename() -> &'static str

The typename of this message.

impl PartialEq for PySparkBatch

fn eq(&self, other: &PySparkBatch) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for PySparkBatch