Trait SplitSource 

pub trait SplitSource {
    type Lane: SourceLane;

    // Required methods
    fn open_split(
        &mut self,
        opening: SplitOpening<'_>,
    ) -> Result<Self::Lane, SourceError>;
    fn encode_commit(
        &mut self,
        split: &SplitId,
        watermark: i64,
    ) -> Result<SplitProgress, SourceError>;
    fn sweep(
        &mut self,
        split: &SplitId,
    ) -> Result<Option<SplitProgress>, SourceError>;
    fn close_split(&mut self, split: &SplitId);

    // Provided methods
    fn validate_resume(
        &self,
        split: &SplitSpec,
        progress: &SplitProgress,
    ) -> Result<(), SourceError> { ... }
    fn take_finishing(&mut self) -> Vec<SplitId> { ... }
    fn begin_revoke(&mut self, split: &SplitId) -> bool { ... }
    fn drain_ready(
        &mut self,
        split: &SplitId,
    ) -> Result<Option<SplitProgress>, SourceError> { ... }
}

What the driver needs from the embedding source.

Implement it on the source’s lane-assembly context (the sub-struct that holds what lane construction needs), not on the source itself — the driver lives beside that context as a sibling field, so both can be borrowed disjointly.

Required Associated Types

type Lane: SourceLane

The data-plane lane type produced for gained splits.

Required Methods

fn open_split( &mut self, opening: SplitOpening<'_>, ) -> Result<Self::Lane, SourceError>

Materialize the lane for a gained (or re-assigned) split. Spawn fetchers here; never block on data.

fn encode_commit( &mut self, split: &SplitId, watermark: i64, ) -> Result<SplitProgress, SourceError>

Snapshot the split’s committable progress at an acked watermark: the opaque resume state plus whether that watermark completes the split (fully delivered and fully acknowledged — the source owns its eof/emitted accounting).

fn sweep( &mut self, split: &SplitId, ) -> Result<Option<SplitProgress>, SourceError>

Completion sweep for an owned split with no new watermark this tick (empty splits; tails acked exactly at the previous commit): Some(terminal progress) when complete, None while data is in flight.

fn close_split(&mut self, split: &SplitId)

The split’s lane is being retired (lost, fenced, completed, or shutdown): detach its fetcher — never abort it, the pipeline thread may still be draining the lane. Must not block.

This is the end of the tenancy: the driver never calls SplitSource::encode_commit or SplitSource::sweep for the split afterwards (its tenancy is retired first, and retired tenancies absorb late watermarks), so the source may drop the split’s state here.

Provided Methods

fn validate_resume( &self, split: &SplitSpec, progress: &SplitProgress, ) -> Result<(), SourceError>

Drift-check carried progress against this instance’s view of the split (etag pins, schema versions) before it is trusted. Rejecting stops the pipeline — carried progress that no longer matches the input is unrecoverable divergence.

fn take_finishing(&mut self) -> Vec<SplitId>

Splits whose lanes decided end-of-input since the last call (the edge, not the level). The driver surfaces them as SourceEvent::CommitReady so the runtime chases their final acks instead of waiting out the commit tick — the split then completes (and frees its working-set slot) within milliseconds of its last record becoming sink-durable. Purely a latency hint; the default reports none.

fn begin_revoke(&mut self, split: &SplitId) -> bool

Begin a cooperative revocation of an owned split: stop its intake at a safe boundary while keeping its commit state, so the tail can still be chased to a final fenced commit. Unlike close_split (which ends the tenancy and lets the source drop the split’s state), the split stays commit- and sweep-adjacent here — the driver keeps committing its acked watermarks and then calls drain_ready until the drain finishes.

Return true to accept the revocation, false to decline it (the default). Contract: return false for any split this source has not opened or has already closed or completed — the driver also guards this (it declines tenancies without an open lane and feeds the decline back to the backend), but the source must not rely on that alone.

Declining is safe but not free. The split still leaves — a revocation is the leader’s decision, not a proposal — so the backend forces the release instead and this split’s uncommitted tail replays under its next owner. (The exception is the backend’s to decide, not the source’s: a revocation the leader takes back before the decline lands is cancelled, and then the split stays and keeps being read.) A source that can stop intake at a safe boundary should, because that is the difference between a replay-free move and a bounded-duplicate one.

While a split is handing off, encode_commit must never report it completed — a drain cut can look terminal to the source (everything emitted is acked) but the split is half-read; the driver strips a completed flag it sees here and logs an error, so a conforming source should never trigger it.

fn drain_ready( &mut self, split: &SplitId, ) -> Result<Option<SplitProgress>, SourceError>

Poll a handing-off split for its final progress: Some(progress) with completed: false once every record it emitted is acked and that watermark is committed, so the resume point handed to the next owner covers everything this instance produced (a replay-free transfer); None while any of that tail is still in flight, to be retried on the next poll. Never reports completed — a revocation gives the split away, it does not finish it. The default reports None.

Level-triggered, unlike take_finishing: keep returning Some on every poll until the driver retires the split — the final store commit can defer on a store hiccup and is re-attempted from a fresh drain_ready answer, so an edge-triggered implementation would stall the drain until the backend forced it.

A source that accepts a revocation in begin_revoke must eventually answer Some here. The default None never finishes a drain; it pairs with the default begin_revoke, which declines.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors