Crate spark_connect_rs

source ·

Modules§

  • catalog
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Spark Catalog representation through which the user may create, drop, alter or query underlying databases, tables, functions, etc.
  • client
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Generic implementation of ChannelBuilder and SparkConnectClient
  • column
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Column represents a column in a DataFrame that holds a spark::Expression
  • dataframe
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core DataFrame representation for Spark Connection
  • errors
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Defines a SparkError for representing failures in various Spark operations. Most of these are wrappers for tonic or arrow error messages
  • expressions
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Traits for converting Rust Types to Spark Connect Expression Types
  • functions
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core A re-implementation of Spark functions
  • group
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core A DataFrame created with an aggregate statement
  • plan
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Logical Plan representation
  • readwriter
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core DataFrameReader & DataFrameWriter representations
  • session
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Spark Session containing the remote gRPC client
  • spark
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Spark Connect gRPC protobuf translated using tonic
  • storage
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Enum for handling Spark Storage representations
  • streaming
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Streaming implementation for the Spark Connect Client
  • types
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Rust Types to Spark Types
  • window
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core Utility structs for defining a window over a DataFrame

Structs§

  • DataFrame
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core DataFrame is composed of a SparkSession referencing a Spark Connect enabled cluster, and a LogicalPlanBuilder which represents the unresolved spark::Plan to be submitted to the cluster when an action is called.
  • DataFrameReader
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core DataFrameReader represents the entrypoint to create a DataFrame from a specific file format.
  • DataFrameWriter
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core DataFrameWriter provides the ability to output a DataFrame to a specific file format supported by Spark
  • SparkSession
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core The entry point to connecting to a Spark Cluster using the Spark Connection gRPC protocol.
  • SparkSessionBuilder
    The spark-connect-rs crate is currently just a meta-package shim for spark-connect-core SparkSessionBuilder creates a remote Spark Session a connection string.