Trait SplitSource 

pub trait SplitSource {
    type Lane: SourceLane;

    // Required methods
    fn open_split(
        &mut self,
        opening: SplitOpening<'_>,
    ) -> Result<Self::Lane, SourceError>;
    fn encode_commit(
        &mut self,
        split: &SplitId,
        watermark: i64,
    ) -> Result<SplitProgress, SourceError>;
    fn sweep(
        &mut self,
        split: &SplitId,
    ) -> Result<Option<SplitProgress>, SourceError>;
    fn close_split(&mut self, split: &SplitId);

    // Provided methods
    fn validate_resume(
        &self,
        split: &SplitSpec,
        progress: &SplitProgress,
    ) -> Result<(), SourceError> { ... }
    fn take_finishing(&mut self) -> Vec<SplitId> { ... }
    fn begin_revoke(&mut self, split: &SplitId) -> bool { ... }
    fn drain_ready(
        &mut self,
        split: &SplitId,
    ) -> Result<Option<SplitProgress>, SourceError> { ... }
}

What the driver needs from the embedding source.

Implement it on the source’s lane-assembly context (the sub-struct that holds what lane construction needs), not on the source itself. The driver lives beside that context as a sibling field, so both can be borrowed disjointly.

Required Associated Types

type Lane: SourceLane

The data-plane lane type produced for gained splits.

Required Methods

fn open_split( &mut self, opening: SplitOpening<'_>, ) -> Result<Self::Lane, SourceError>

Materialize the lane for a gained (or re-assigned) split. Spawn fetchers here; never block on data.

fn encode_commit( &mut self, split: &SplitId, watermark: i64, ) -> Result<SplitProgress, SourceError>

Snapshot the split’s committable progress at an acked watermark. The snapshot carries the opaque resume state plus whether that watermark completes the split (fully delivered and fully acknowledged; the source owns its eof/emitted accounting).

fn sweep( &mut self, split: &SplitId, ) -> Result<Option<SplitProgress>, SourceError>

Completion sweep for an owned split with no new watermark this tick (empty splits; tails acked exactly at the previous commit). Returns Some(terminal progress) when complete, None while data is in flight.

fn close_split(&mut self, split: &SplitId)

The split’s lane is being retired (lost, fenced, completed, or shutdown). Detach its fetcher; never abort it, because the pipeline thread may still be draining the lane. Must not block.

This is the end of the tenancy. The driver never calls SplitSource::encode_commit or SplitSource::sweep for the split afterwards (its tenancy is retired first, and retired tenancies absorb late watermarks), so the source may drop the split’s state here.

Provided Methods

fn validate_resume( &self, split: &SplitSpec, progress: &SplitProgress, ) -> Result<(), SourceError>

Drift-check carried progress against this instance’s view of the split (etag pins, schema versions) before it is trusted; the default accepts everything.

A rejection is raised before the tenancy is recorded, so the split is never opened, and the driver reports it as poison: one delivery attempt is consumed, the split is handed back for another instance, and at the attempt cap it is quarantined. The error’s class then decides this run. ErrorClass::Fatal stops the pipeline; any other class is logged and the run continues with the split left to the coordinator.

fn take_finishing(&mut self) -> Vec<SplitId>

Splits whose lanes decided end-of-input since the last call (the edge, not the level). The driver surfaces them as SourceEvent::CommitReady so the runtime chases their final acks instead of waiting out the commit tick. The split then completes (and frees its working-set slot) within milliseconds of its last record becoming sink-durable. A latency hint only; the default reports none.

fn begin_revoke(&mut self, split: &SplitId) -> bool

Begin a cooperative revocation of an owned split. Stop its intake at a safe boundary while keeping its commit state, so the tail can still be chased to a final fenced commit. Unlike close_split (which ends the tenancy and lets the source drop the split’s state), the split stays commit- and sweep-adjacent here. The driver keeps committing its acked watermarks and then calls drain_ready until the drain finishes.

Return true to accept the revocation, false to decline it (the default). Contract: return false for any split this source has not opened or has already closed or completed. The driver also guards this (it declines tenancies without an open lane and feeds the decline back to the backend), but the source must not rely on that alone.

Declining is safe but not free. The split still leaves, because a revocation is the leader’s decision; the backend forces the release instead and this split’s uncommitted tail replays under its next owner. (The backend cancels a revocation the leader takes back before the decline lands, and then the split stays and keeps being read.) A source that can stop intake at a safe boundary should, because that is the difference between a replay-free move and a bounded-duplicate one.

While a split is draining, encode_commit must never report it completed. A drain cut can look terminal to the source (everything emitted is acked) while the split is half-read; the driver strips a completed flag it sees here and logs an error.

fn drain_ready( &mut self, split: &SplitId, ) -> Result<Option<SplitProgress>, SourceError>

Poll a draining split for its final progress. Returns Some(progress) with completed: false once every record it emitted is acked and that watermark is committed, so the resume point handed to the next owner covers everything this instance produced (a replay-free transfer); None while any of that tail is still in flight, to be retried on the next poll. Never reports completed; a revocation gives the split away rather than finishing it. The default reports None.

Level-triggered, unlike take_finishing: keep returning Some on every poll until the driver retires the split. The final store commit can defer on a store hiccup and is re-attempted from a fresh drain_ready answer, so an edge-triggered implementation would stall the drain until the backend forced it.

A source that accepts a revocation in begin_revoke must eventually answer Some here. The default None never finishes a drain; it pairs with the default begin_revoke, which declines.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors