numaflow 0.5.0

Rust SDK for Numaflow
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

//! A Rust SDK for [Numaflow]. The Rust SDK is experimental has only implemented the most important
//! features.
//! It will support all the core features eventually.
//! It supports [Map], [Reduce], [User Defined Sources], [User Defined Source Transformer] and [User Defined Sinks].
//!
//! Please note that the Rust SDK is experimental and will be refactored in the future to make it more
//! idiomatic.
//!
//! [Numaflow]: https://numaflow.numaproj.io/
//! [Map]: https://numaflow.numaproj.io/user-guide/user-defined-functions/map/map/
//! [Reduce]: https://numaflow.numaproj.io/user-guide/user-defined-functions/reduce/reduce/
//! [User Defined Sources]: https://numaflow.numaproj.io/user-guide/sources/user-defined-sources/
//! [User Defined Source Transformer]: https://numaflow.numaproj.io/user-guide/sources/transformer/
//! [User Defined Sinks]: https://numaflow.numaproj.io/user-guide/sinks/user-defined-sinks/
//! [Session reduce]: https://numaflow.numaproj.io/user-guide/user-defined-functions/reduce/windowing/session/

/// Shared utilities, traits, and common functionality
pub mod shared;

/// Generated protobuf modules
pub mod proto;

/// source is for building custom [user defined sources](https://numaflow.numaproj.io/user-guide/sources/overview/).
pub mod source;

/// Source transform functionality for writing [source data transformers](https://numaflow.numaproj.io/user-guide/sources/transformer/overview/).
pub mod sourcetransform;

/// Map functionality for writing the [map](https://numaflow.numaproj.io/user-guide/user-defined-functions/map/map/) handlers.
pub mod map;

/// Reduce functionality for writing the [reduce](https://numaflow.numaproj.io/user-guide/user-defined-functions/reduce/reduce/) handlers.
pub mod reduce;

/// Reduce stream functionality for writing streaming [reduce](https://numaflow.numaproj.io/user-guide/user-defined-functions/reduce/reduce/) handlers.
pub mod reducestream;

/// Sink functionality for writing [user defined sinks](https://numaflow.numaproj.io/user-guide/sinks/user-defined-sinks/).
pub mod sink;

/// Side input functionality for building [side input](https://numaflow.numaproj.io/user-guide/reference/side-inputs/) handlers.
pub mod sideinput;

/// batchmap is for writing the map in [batch mode](https://numaflow.numaproj.io/user-guide/user-defined-functions/map/map/#batch-map-mode) handlers.
pub mod batchmap;

/// mapstream is for writing the map in [stream mode](https://numaflow.numaproj.io/user-guide/user-defined-functions/map/map/#streaming-mode) handlers.
pub mod mapstream;

pub mod serving_store;

/// session_reduce is for implementing [session reduce](https://numaflow.numaproj.io/user-guide/user-defined-functions/reduce/windowing/session) handlers.
pub mod session_reduce;

/// accumulator is for implementing [accumulator](https://numaflow.numaproj.io/user-guide/user-defined-functions/reduce/windowing/accumulator) handlers.
pub mod accumulator;

// Error handling on Numaflow SDKs!
//
// Any non-recoverable error will cause the process to shutdown with a non-zero exit status. All errors are non-recoverable.
// If there are errors that are retryable, we (gRPC or Numaflow SDK) would have already retried it (hence not an error), that means,
// all errors raised by the SDK are non-recoverable.
//
// Task Ordering and error propagation.
//
//      level-1               level-2               level-3
//
//                   +---> (service_fn) ->
//                   |
//                   |
//                   |                     +---> (task)
//                   |                     |
//                   |                     |
// (gRPC Service) ---+---> (service_fn) ---+---> (task)
//      ^            |                     |
//      |            |                     |
//      |            |                     +---> (task)
//      |            |
//  (shutdown)       |
//      |            +---> (service_fn) ->
//      |
//      |
//   (user)
//
// If a task at level-3 has an error, then that error will be propagated to level-2 (service_fn) via a mpsc::channel using the response channel.
// The Response channel passes a Result type and by returning Err() in response channel, it notifies top service_fn that the task wants to abort itself.
// service_fn (level-2) will now use another mpsc::channel to tell the gRPC service to cancel all the service_fns. gRPC service will
// ask all the level-2 service_fns to abort using the CancellationToken. service_fn will call abort on all the tasks it created using internal
// mpsc::channel when CancellationToken has been dropped/cancelled.
//
// User can directly send shutdown request to the gRPC server which triggers the shutdown of the server by stop accepting new requests
// and draining the existing requests. Lastly we will cancel the cancellation token to make sure all the tasks are aborted.
//
// The above 3 level task ordering is only for complex cases like reduce, but for simpler endpoints like `map`, it only has 2 levels but
// the error propagation is handled the same way.

/// Error handling types and utilities
pub mod error;