Crate databricks_zerobus_ingest_sdk 

Databricks Zerobus Ingest SDK

A high-performance Rust client for streaming data ingestion into Databricks Delta tables.

Quick Start

use databricks_zerobus_ingest_sdk::{ZerobusSdk, JsonValue};

let sdk = ZerobusSdk::builder()
    .endpoint(zerobus_endpoint)
    .unity_catalog_url(uc_endpoint)
    .build()?;

let stream = sdk
    .stream_builder()
    .table("catalog.schema.table")
    .oauth(client_id, client_secret)
    .json()
    .build()
    .await?;

// Ingest a record and wait for acknowledgment
let offset = stream.ingest_record_offset(JsonValue(my_record)).await?;
stream.wait_for_offset(offset).await?;

stream.close().await?;

See the examples/ directory for complete working examples.

Modules

databricks

schema

Convert a Unity Catalog table schema into a protobuf DescriptorProto.

stream_options

Shared configuration options for stream creation and operation.

Structs

DefaultTokenFactory

Default OAuth 2.0 token factory for Unity Catalog authentication.

JsonString

Wrapper for pre-serialized JSON strings.

JsonValue

Wrapper for JSON-serializable objects with automatic serialization.

OAuthHeadersProvider

The default headers provider that uses OAuth 2.0 with Unity Catalog.

OffsetIdGenerator

Generates monotonically increasing offset IDs for ingested records.

ProtoBytes

Wrapper for pre-serialized protobuf bytes.

ProtoMessage

Wrapper for protobuf messages with automatic serialization.

ProxyConnector

A proxy connector for the gRPC channel.

SecureTlsConfig

Secure TLS configuration using system CA certificates.

StreamBuilder

A fluent builder for creating Zerobus ingestion streams.

StreamConfigurationOptions

Configuration options for stream creation, recovery of broken streams and flushing.

ZerobusSdk

The main interface for interacting with the Zerobus API.

ZerobusSdkBuilder

Builder for creating a ZerobusSdk instance with fluent configuration.

ZerobusStream

Represents an active ingestion stream to a Databricks Delta table.

Enums

EncodedBatch

EncodedBatchIter

EncodedRecord

StreamType

The type of the stream connection created with the server. Currently we only support ephemeral streams on the server side, so we support only that in the SDK as well.

ZerobusError

Represents all possible errors that can occur when using Zerobus.

Constants

DEFAULT_SDK_IDENTIFIER

Default identifier the SDK sends as the HTTP user-agent header on every request. Use ZerobusSdkBuilder::application_name to append an application suffix.

Traits

AckCallback

Callback trait for receiving acknowledgment notifications.

HeadersProvider

A trait for providing custom headers for gRPC requests.

TlsConfig

Trait for TLS configuration strategies.

Type Aliases

ConnectorFactory

Signature for caller-supplied proxy selection. Given the target host, return a configured connector or None for a direct connection.

JsonEncodedRecord

A type alias for a JSON-encoded record.

OffsetId

Offset ID type representing a logical (or physical) position in the stream.

ProtoEncodedRecord

A type alias for a protobuf-encoded record.

ZerobusResult