Struct SparkSql

#[non_exhaustive]
pub struct SparkSql {
    pub name: String,
    pub inputs: Vec<String>,
    pub sql_query: String,
    pub sql_aliases: Vec<SqlAlias>,
    pub output_schemas: Option<Vec<GlueSchema>>,
}

Specifies a transform where you enter a SQL query using Spark SQL syntax to transform the data. The output is a single DynamicFrame.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

name: String

The name of the transform node.

inputs: Vec<String>

The data inputs identified by their node names. You can associate a table name with each input node to use in the SQL query. The name you choose must meet the Spark SQL naming restrictions.

sql_query: String

A SQL query that must use Spark SQL syntax and return a single data set.

sql_aliases: Vec<SqlAlias>

A list of aliases. An alias allows you to specify what name to use in the SQL for a given input. For example, you have a datasource named "MyDataSource". If you specify From as MyDataSource, and Alias as SqlName, then in your SQL you can do:

select * from SqlName

and that gets data from MyDataSource.

output_schemas: Option<Vec<GlueSchema>>

Specifies the data schema for the SparkSQL transform.

Implementations

impl SparkSql

pub fn name(&self) -> &str

The name of the transform node.

pub fn inputs(&self) -> &[String]

The data inputs identified by their node names. You can associate a table name with each input node to use in the SQL query. The name you choose must meet the Spark SQL naming restrictions.

pub fn sql_query(&self) -> &str

A SQL query that must use Spark SQL syntax and return a single data set.

pub fn sql_aliases(&self) -> &[SqlAlias]

A list of aliases. An alias allows you to specify what name to use in the SQL for a given input. For example, you have a datasource named "MyDataSource". If you specify From as MyDataSource, and Alias as SqlName, then in your SQL you can do:

select * from SqlName

and that gets data from MyDataSource.

pub fn output_schemas(&self) -> &[GlueSchema]

Specifies the data schema for the SparkSQL transform.

If no value was sent for this field, a default will be set. If you want to determine if no value was sent, use .output_schemas.is_none().

impl SparkSql

pub fn builder() -> SparkSqlBuilder

Creates a new builder-style object to manufacture SparkSql.

Trait Implementations

impl Clone for SparkSql

fn clone(&self) -> SparkSql

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SparkSql

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for SparkSql

fn eq(&self, other: &SparkSql) -> bool

Tests for self and other values to be equal, and is used by ==.

const fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for SparkSql