Sourcer

numaflow::source

Trait Sourcer 

Source 
pub trait Sourcer {
    // Required methods
    fn read<'life0, 'async_trait>(
        &'life0 self,
        request: SourceReadRequest,
        transmitter: Sender<Message>,
    ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn ack<'life0, 'async_trait>(
        &'life0 self,
        offset: Vec<Offset>,
    ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn nack<'life0, 'async_trait>(
        &'life0 self,
        offset: Vec<Offset>,
    ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn pending<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Option<usize>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn partitions<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Option<Vec<i32>>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
}
Expand description

Trait representing a user defined source.

§Example

Please refer to simple source for an example.

§NOTE

The standard convention for both Sourcer::read and Sourcer::ack is that they should be mutable, since they have to update some state. Unfortunately the SDK provides only a shared reference of self and thus makes it immutable. This is because gRPC tonic provides only a shared reference for its traits. This means, the implementer for trait will have to use SharedState pattern to mutate the values as recommended in issue-427. This might change in future as async traits evolves.

Required Methods§

Source

fn read<'life0, 'async_trait>( &'life0 self, request: SourceReadRequest, transmitter: Sender<Message>, ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Reads the messages from the source and sends them to the transmitter.

Source

fn ack<'life0, 'async_trait>( &'life0 self, offset: Vec<Offset>, ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Acknowledges the message that has been processed by the user-defined source.

Source

fn nack<'life0, 'async_trait>( &'life0 self, offset: Vec<Offset>, ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Negatively acknowledges the message that has been processed by the user-defined source.

Source

fn pending<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Option<usize>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Returns the number of messages that are yet to be processed by the user-defined source. The None value can be returned if source doesn’t support detecting the backlog.

Source

fn partitions<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Option<Vec<i32>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Returns the partitions associated with the source. This will be used by the platform to determine the partitions to which the watermark should be published. Some sources might not have the concept of partitions. Kafka is an example of source where a reader can read from multiple partitions. If None is returned, Numaflow replica-id will be returned as the partition.

Implementors§