Crate databricks_zerobus_ingest_sdk 

Databricks Zerobus Ingest SDK

A high-performance Rust client for streaming data ingestion into Databricks Delta tables.

Quick Start

use databricks_zerobus_ingest_sdk::{ZerobusSdk, JsonValue};

let sdk = ZerobusSdk::builder()
    .endpoint(zerobus_endpoint)
    .unity_catalog_url(uc_endpoint)
    .build()?;

let stream = sdk
    .stream_builder()
    .table("catalog.schema.table")
    .oauth(client_id, client_secret)
    .json()
    .build()
    .await?;

// Idiomatic flow: ingest in a loop, then flush() once to confirm the batch.
for my_record in records {
    stream.ingest_record_offset(JsonValue(my_record)).await?;
}
stream.flush().await?; // Returns once every queued record is acknowledged.

stream.close().await?;

See the examples/ directory for complete working examples.

Modules

databricks

schema

Convert a Unity Catalog table schema into a protobuf DescriptorProto.

stream_options

Shared configuration options for stream creation and operation.

zeroparser

Zero-copy, single-pass protobuf parser driven by a prost_types::DescriptorProto.

Structs

ArrowSchema

Describes the meta-data of an ordered sequence of relative types.

ArrowStreamConfigurationOptions

Configuration options for Arrow Flight stream creation and operation.

CallbackHandlerHarness

Test-only harness driving the real callback handler task and teardown without a gRPC connection. Wires up the same spawn_callback_handler_task + channel as ZerobusStream::new_stream, and its teardown() runs the production ZerobusStream::shutdown_callback_task, so tests exercise the ack callback lifetime contract against real code.

DefaultTokenFactory

Default OAuth 2.0 token factory for Unity Catalog authentication.

Field

Describes a single column in a Schema.

JsonString

Wrapper for pre-serialized JSON strings.

JsonValue

Wrapper for JSON-serializable objects with automatic serialization.

MessageId

Opaque identifier returned by ingest methods on MultiplexedStream. Encodes the sub-stream index and sub-stream offset in a single i64.

MultiplexedStream

Distributes ingestion round-robin across a fixed set of ZerobusStreams.

NoAuthHeadersProvider

A headers provider that returns no headers.

NoTlsConfig

No-op TLS configuration for testing with plaintext http:// endpoints.

OAuthHeadersProvider

The default headers provider that uses OAuth 2.0 with Unity Catalog.

OffsetIdGenerator

Generates monotonically increasing offset IDs for ingested records.

ProtoBytes

Wrapper for pre-serialized protobuf bytes.

ProtoMessage

Wrapper for protobuf messages with automatic serialization.

ProxyConnector

A proxy connector for the gRPC channel.

RecordBatch

A two-dimensional batch of column-oriented data with a defined schema.

SecureTlsConfig

Secure TLS configuration using system CA certificates.

StreamBuilder

A fluent builder for creating Zerobus ingestion streams.

StreamConfigurationOptions

Configuration options for stream creation, recovery of broken streams and flushing.

ZerobusArrowStream

An Arrow Flight stream for ingesting Arrow RecordBatches into a Delta table.

ZerobusSdk

The main interface for interacting with the Zerobus API.

ZerobusSdkBuilder

Builder for creating a ZerobusSdk instance with fluent configuration.

ZerobusStream

Represents an active ingestion stream to a Databricks Delta table.

Enums

DataType

Datatypes supported by this implementation of Apache Arrow.

EncodedBatch

EncodedBatchIter

EncodedRecord

StreamType

The type of the stream connection created with the server. Currently we only support ephemeral streams on the server side, so we support only that in the SDK as well.

TimeUnit

An absolute length of time in seconds, milliseconds, microseconds or nanoseconds.

ZerobusError

Represents all possible errors that can occur when using Zerobus.

Constants

DEFAULT_SDK_IDENTIFIER

Default identifier the SDK sends as the HTTP user-agent header on every request. Use ZerobusSdkBuilder::application_name to append an application suffix.

Traits

AckCallback

Callback trait for receiving acknowledgment notifications.

HeadersProvider

A trait for providing custom headers for gRPC requests.

TlsConfig

Trait for TLS configuration strategies.

Type Aliases

ConnectorFactory

Signature for caller-supplied proxy selection. Given the target host, return a configured connector or None for a direct connection.

JsonEncodedRecord

A type alias for a JSON-encoded record.

OffsetId

Offset ID type representing a logical (or physical) position in the stream.

ProtoEncodedRecord

A type alias for a protobuf-encoded record.

ZerobusResult