Skip to main content

Module driver

faucet_core

Module driver 

Source
Expand description

Family driver traits (RFC 0014, #895).

A driver is a connector’s I/O and nothing else. The engine — this module for the family-neutral contract, faucet-common-sql and faucet-common-file for the SQL and object-store families — drives every lifecycle from the driver’s primitives: overwrite staging, the exactly-once commit, scoped cleanup, rollback, per-row outcomes and the row report. A connector built on a driver cannot get one of those contracts wrong by forgetting a hook, because it does not implement the hook.

LayerTraitAdapter to Sink / Source
any sinkSinkDriver (+ StagingDriver, CommitTokenDriver, CleanupDriver, SchemaDriver, RollbackDriver)DriverSink
any sourceSourceDriverDriverSource
messaging, produceMessageDriver (+ MessageTransactions)MessageSinkDriver → DriverSink
messaging, consumeMessageConsumerMessageSourceDriver → DriverSource
HTTPHttpDriverHttpSinkDriver → DriverSink
SQLfaucet_common_sql::SqlDriverfaucet_common_sql::SqlSinkDriver → DriverSink
object storagefaucet_common_file::ObjectStoreDriverfaucet_common_file::FileSink (a DriverSink inside)

The SQL and object-store traits live in their family crates because they are built on those crates’ engines (SqlSinkEngine / SqlDialect, FileWriter / StorageBackend), and faucet-core depends on no connector-family crate.

Values. Drivers exchange JSON Value records under the value-fidelity rules (#885); a destination that writes Arrow directly adds a ColumnarDriver (feature arrow) and receives the RecordBatch unconverted.

Calls. Every async primitive receives the run’s CallSlot; the adapter installs the run’s context into it. Make each backend request through it.

Migrating a connector. Keep the public type and config; hold the adapter and forward:

use faucet_core::driver::{DriverSink, PageWrite, RowPolicy, SinkDriver};
use faucet_core::{CallSlot, FaucetError, Sink, Value, async_trait, forward_sink};

/// The connector's I/O.
struct AcmeDriver;

#[async_trait]
impl SinkDriver for AcmeDriver {
    fn connector_name(&self) -> &'static str {
        "acme"
    }
    async fn write(
        &self,
        calls: &CallSlot,
        rows: &[Value],
        _policy: RowPolicy,
    ) -> Result<PageWrite, FaucetError> {
        let n = rows.len();
        calls
            .call_once("insert", async move { Ok::<_, FaucetError>(n) })
            .await?;
        Ok(PageWrite::accepted(n))
    }
}

/// The public connector type, unchanged for its users.
pub struct AcmeSink {
    inner: DriverSink<AcmeDriver>,
}

#[async_trait]
impl Sink for AcmeSink {
    forward_sink!(inner);
}

See RFC 0014 for the deprecated hooks these traits replace and the migration guide.

Structs§

AckToken
An opaque handle a consumer acknowledges a message by.
DriverSink
A SinkDriver as a Sink (RFC 0014).
DriverSource
A SourceDriver as a Source (RFC 0014): every fetch is the driver’s page stream, and the run’s call context reaches every read.
HttpSinkDriver
An HttpDriver as a SinkDriver.
InboundMessage
One received message.
MessageSinkDriver
A MessageDriver as a SinkDriver.
MessageSourceDriver
A MessageConsumer as a SourceDriver.
OutboundMessage
One message to publish.
PageWrite
What one SinkDriver::write did.

Enums§

HttpBatching
How a page becomes requests.
RowPolicy
How a write may report a row it could not write.

Traits§

CleanupDriver
The scoped-cleanup primitive of a destination (#478).
ColumnarDriverarrow
The columnar write primitive of a destination that writes Arrow directly.
CommitTokenDriver
The exactly-once primitives of a destination: rows and a commit token committed as one unit (#215).
HttpDriver
An HTTP destination’s primitives (RFC 0014). Wrap it in an HttpSinkDriver, then a DriverSink.
MessageConsumer
A consumer’s primitives (RFC 0014). Wrap it in a MessageSourceDriver, then a DriverSource.
MessageDriver
A producer’s primitives (RFC 0014). Wrap it in a MessageSinkDriver, then a DriverSink.
MessageTransactions
A producer’s transaction primitives. With them, MessageSinkDriver commits a page’s messages and its commit token in one transaction.
RollbackDriver
The run-rollback primitives of a destination (#706).
SchemaDriver
The schema primitives of a destination (#194).
SinkDriver
A destination’s I/O primitives (RFC 0014). Wrap it in a DriverSink to get a Sink.
SourceDriver
A dataset’s read primitives (RFC 0014). Wrap it in a DriverSource to get a Source.
StagingDriver
The overwrite (full refresh) primitives of a destination (#492).

Functions§

pages_of
Split a buffered read into pages of batch_size records (0: one page), the bookmark on the last page — the shape a source without a native paging primitive streams in. An empty read yields one empty page only when it carries a bookmark.

Type Aliases§

ColumnarStreamarrow
A stream of Arrow pages a SourceDriver reads.
NativeStream
A stream of raw byte batches a SourceDriver reads.
PageStream
A stream of pages a SourceDriver reads.