Crate delta_sharing_server

Delta Sharing Server

Delta Sharing Server provides the building blocks to easily setup a server compatible with the Delta Sharing protocol.

Delta Sharing protocol

Delta Sharing is an open protocol for secure real-time exchange of large datasets, which enables organizations to share data in real time regardless of which computing platforms they use. It is a simple REST protocol that securely shares access to part of a cloud dataset and leverages modern cloud storage systems, such as S3, ADLS, or GCS, to reliably transfer data.

Design

In order to provide flexibility, this crate is build around three core abstractions.

• ShareReader: This trait is responsible for the communication between sharing server and the source system for shares, schemas and tables.
• TableReader: This trait is responsible for reading tables of a specified table format
• UrlSigner: This trait is responsible for signing the urls that will be used to access the data.

use delta_sharing_server::manager::dynamo::DynamoShareReader;
use delta_sharing_server::reader::delta::DeltaTableReader;
use delta_sharing_server::signer::s3::S3UrlSigner;
use delta_sharing_server::router::build_sharing_server_router;
use delta_sharing_server::state::SharingServerState;

#[tokio::main]
async fn main() {
    // configure table manager
    let config = aws_config::load_from_env().await;
    let ddb_client = aws_sdk_dynamodb::Client::new(&config);
    let table_manager = Arc::new(DynamoShareReader::new(ddb_client, "delta-sharing-table".to_owned(), "GSI1".to_owned()));
         
    // configure table readers
    let delta_table_reader = Arc::new(DeltaTableReader::new());

    // configure file url signers
    let s3_client = aws_sdk_s3::Client::new(&config);
    let s3_url_signer = Arc::new(S3UrlSigner::new(s3_client));

    // initialize server state
    let mut state = SharingServerState::new(table_manager);
    state.add_table_reader("DELTA", delta_table_reader);
    state.add_url_signer("S3", s3_url_signer);

    // start server
    let app = build_sharing_server_router(Arc::new(state));    
    axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
        .serve(app.into_make_service())
        .await
        .unwrap();
}

Modules

• error
• manager
  Types and traits for managing shared objects.
• protocol
  Types for implementing the Delta Sharing protocol.
• reader
  Types and traits for reading table data in object storage.
• router
  Router for the sharing server.
• signer
  Traits and types for creating pre-signed urls.
• state
  Delta Sharing server state.