1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
//! Waiting on several workflows at once: the first to finish, or all of them.
//!
//! A fan-out starts N workflows and then has to learn when they finish. Awaiting each handle in
//! turn already works and costs nothing in wall-clock — the caller takes about as long as the
//! slowest, which [`WorkflowRef::start_with`](crate::WorkflowRef::start_with) documents — but it
//! answers only in the order the caller happened to ask, and it holds N waits open where one would
//! do. These two calls are the other shape: **one query per interval whatever N is**, answering
//! either *which one finished first* or *tell me when they all have*.
//!
//! ```no_run
//! # async fn drain(dbos: &dbos::DBOS, handles: Vec<dbos::WorkflowHandle<u32>>) -> dbos::Result<()> {
//! let mut handles = handles;
//! while !handles.is_empty() {
//! // The borrow ends with the block, so the set can be narrowed below.
//! let id = {
//! let ids: Vec<&str> = handles.iter().map(|h| h.workflow_id()).collect();
//! dbos.select_workflow(&ids).await?
//! };
//! // Removed before it is awaited, so the next pass waits on the rest.
//! let at = handles.iter().position(|h| h.workflow_id() == id).expect("in the set");
//! let winner = handles.swap_remove(at);
//! println!("{id} finished: {:?}", winner.result().await);
//! }
//! # Ok(()) }
//! ```
//!
//! # What "finished" means here
//!
//! **Settled, not succeeded.** A workflow counts as finished once its status leaves `PENDING`,
//! `ENQUEUED` and `DELAYED` — so a cancelled or dead-lettered member ends the wait like any other.
//! The caller asked which race finished first, and a set that has to be re-waited because one
//! member was cancelled is a wait that can hang on a workflow nobody is running any more.
//!
//! Neither call returns an outcome, and that is not an omission. What a workflow *returned* comes
//! back through its own [`WorkflowHandle::result`](crate::WorkflowHandle::result), typed, and a
//! wait over a set has no one type to give. So these answer *when*, and the handle answers *what*.
//!
//! # Ids in, ids out
//!
//! **These return ids, not handles.** Handing back an owned
//! [`WorkflowHandle`](crate::WorkflowHandle) would mean taking the whole set by value and dropping
//! every loser, which is precisely the wrong thing for the loop the call exists for. So the answer
//! is the winner's **id** — the identity the handle carried anyway, and the same thing the
//! checkpoint stores, so nothing is projected on the way out and re-derived on replay.
//!
//! A position in the slice would be the only positional return in this crate (`cancel_all` and
//! its neighbours take ids and give back ids), is meaningful only against the exact slice it came
//! from, and would force a set with no repeats — all to skip a
//! [`position`](std::iter::Iterator::position) lookup in the drain loop above.
//!
//! `join_workflows` returns nothing: there is nothing to hand back that the caller did not already
//! have.
//!
//! **These take ids, not handles.** Every bulk call here that only names workflows takes
//! `&[&str]` — [`cancel_all`](crate::DBOS::cancel_all), [`delete_all`](crate::DBOS::delete_all),
//! [`resume_all`](crate::DBOS::resume_all) — and a wait is not different enough to spell its
//! argument a second way. It also makes the heterogeneous case expressible: a fan-out whose members
//! return different types has no common `WorkflowHandle<R, E>` to put in a slice.
//!
//! **Both waits are here.** A fan-out wants both questions — which member finished first, and
//! when have they all — and having one without the other would leave the natural pair half-built.
//!
//! **They are named for the concurrency primitives.** `select_workflow` and `join_workflows` name
//! the *shape* a Rust caller already knows it by, from `tokio::select!` and `tokio::join!` —
//! one branch of several wins, or every branch is joined. That is the name the macros over them
//! take, and a function and its macro that do the same thing under two different names is a seam to
//! learn for nothing.
//!
//! **The recorded step name follows the call**, `DBOS.selectWorkflow`: a step listing should
//! name the call the caller wrote. The all-wait records no step at all, so it needs no name.
//!
//! # Called from inside a workflow
//!
//! **`select_workflow` is checkpointed as a step of the calling workflow**, under
//! [`SELECT_WORKFLOW`](crate::sysdb::types::step_names::SELECT_WORKFLOW), because it records a
//! *choice*. A replay that raced again could see a different member finish first and take a
//! different branch, which would make the workflow nondeterministic in the one way a workflow may
//! never be. So the winner's id is the step's output, and a replay hands back the same id without
//! waiting.
//!
//! **`join_workflows` is checkpointed nowhere, on any of its three surfaces.** It decides nothing:
//! every member has settled by the time it returns, in whatever order, so there is no choice for a
//! step to pin. A step recording only that the wait happened is not worth a step id — for a reason
//! stronger than the row being cheap:
//!
//! - **Whatever it could pin is already pinned harder by what comes after it.** An all-wait is
//! written to be followed by [`result`](crate::WorkflowHandle::result) on each handle, and each
//! of *those* is a step recording the outcome the workflow branches on. A replay reads those
//! rows back and never looks at the children at all, so a recorded wait in front of them is a
//! write no replay path reads.
//! - **A wait whose results are never read has nothing worth pinning either.** The recorded fact
//! is "they had all settled", and the wait re-derives it by asking once — every member is
//! settled already, so the poll answers on its first pass.
//! - **The step id is the expensive part, not the row.** An id is a position in this workflow's
//! counter, and every call that takes one has to be written the same way on every execution or
//! each later step lands on a slot it did not record. A call that pins no decision should not be
//! spending one.
//!
//! **What that costs is one narrow case, and it is the honest reading of the call.** A member
//! *resumed* after it settled — a cancelled workflow put back on a queue — is waited for again by
//! a replay, where a recorded step would have returned at once. The wait answers a question about
//! the set *now*, and a `select_workflow` beside it still pins every choice made on the strength
//! of one.
//!
//! **A replay of a first-wait checks what its payload lets it check, and no more.** It can ask
//! whether the recorded winner is still in the set, because the winner is what it recorded anyway.
//! Step *inputs* are not checkpointed anywhere — the step row has no column for them — so a replay
//! whose arguments changed reads back the answer to the question it asked the first time, exactly
//! as a [`sleep`](crate::sleep()) whose duration changed keeps the deadline it recorded.
//!
//! Outside a workflow the first-wait is not checkpointed either, which is the operator's and the
//! client's case; inside a *step* it is plain too, by the leaf rule every id-allocating call in
//! this crate follows. [`StepPlacement`] owns those rules and the argument for each. The all-wait
//! is the same call in all of these places.
//!
//! # Three surfaces, split by where the caller stands
//!
//! Exactly as [`event`](crate::event) splits its reader, and for the same reason rather than for
//! symmetry. The free [`select_workflow`](fn@select_workflow) and
//! [`join_workflows`](fn@join_workflows) are what a **workflow body** calls: they take the executor
//! from the ambient context, so a workflow that waits needs no [`DBOS`] handle — and a registered
//! closure that captured one would be stored inside the very `Arc` it holds a strong reference to,
//! keeping the instance, its executor and its pool alive for the life of the process.
//! [`DBOS::select_workflow`] and [`DBOS::join_workflows`] are for code **outside** a workflow,
//! where there is nothing ambient to take an executor from;
//! [`Client::select_workflow`](crate::Client::select_workflow) and
//! [`Client::join_workflows`](crate::Client::join_workflows) are that same caller from outside the
//! application altogether.
//!
//! # Two macros over the same two calls
//!
//! [`select_workflow!`](macro@crate::select_workflow) and
//! [`join_workflows!`](macro@crate::join_workflows) are the same two waits taking **handles** and
//! giving back **typed results**, which is what the id-taking calls cannot do: a set of ids has no
//! one type, where a fixed list of handles has one per branch. **Each shares its name with the
//! function it wraps**, which costs nothing — macros live in their own namespace, and
//! `futures::join!` sits beside `futures::future::join` on the same terms — and says the true
//! thing: they are one operation in two spellings, taking ids or taking handles. Neither macro
//! records anything of its own; each expands to the wait beside it plus
//! [`result`](crate::WorkflowHandle::result) on the handles it needs, so the checkpoint that pins
//! the choice is the [`select_workflow`](fn@select_workflow) the macro already made.
//!
//! They are macros because a select's answer is a *sum* over branches that may differ in type, and
//! Rust has no anonymous sum type to return; a function would have to give back nested `Either`s,
//! or force one type on every branch. That is why `select!` is a macro everywhere it exists. Both
//! are `macro_rules!`, exported from this crate rather than a companion one, as `tokio::select!`
//! and `tokio::join!` are.
//!
//! Both take an optional instance before a semicolon — `select_workflow!(dbos; ..)` — which is the
//! three-surface split above spelled at the call site, because a macro has no ambient context of
//! its own to consult. The expansion names the *method* rather than a type, so a [`DBOS`] and a
//! [`Client`](crate::Client) are the same two lines.
//!
//! The id-taking calls are not superseded by any of this. A macro is fixed-arity and needs the
//! handles, so a set built in a loop, a set of ids read from somewhere else, and a
//! [`Client`](crate::Client) that never had handles are all still theirs.
//!
//! # Bounding the wait
//!
//! Neither takes a timeout, for the reason
//! [`await_workflow_result`](crate::sysdb::SystemDatabase::await_workflow_result) gives at length:
//! `tokio::time::timeout` around the future, or dropping it, ends the poll: the language already
//! supplies the bound, so the call needs no parameter for it.
use Arc;
use crate;
use crateConnection;
use crate;
use crateDBOS;
use crate;
use crate;
/// Waits until one of these workflows finishes, and reports **which**.
///
/// The waiter for a workflow body: it takes the executor from the ambient context, so a workflow
/// that waits on a fan-out needs no [`DBOS`] handle and its registered closure captures nothing.
/// That is what keeps the registry free of strong references back to the instance holding it, and
/// it is why [`get_event`](crate::get_event) is a free function too.
///
/// The wait is checkpointed as a step, so a replay returns the same winner instead of racing again.
/// **Its step id is taken here, at the call, not at the first poll** — see [`PendingStep`] — so a
/// wait built beside a step and driven with it by `tokio::join!` takes the same slot on every
/// execution, whatever it was passed: an empty set is placed like any other and its refusal
/// recorded as the step's own outcome, so a replay refuses again. The error is the *workflow's*
/// channel, like [`step`](crate::step())'s, so `?` needs no conversion. From inside a *step* it
/// waits plainly with no checkpoint, the step's own checkpoint standing for everything its body
/// did.
///
/// ```no_run
/// # async fn fan_out(child: dbos::WorkflowRef<u32, u32>) -> dbos::Result<u32> {
/// let mut handles = Vec::new();
/// for n in 0..3 {
/// handles.push(child.start(n).await?);
/// }
/// let id = {
/// let ids: Vec<&str> = handles.iter().map(|h| h.workflow_id()).collect();
/// dbos::select_workflow(&ids).await?
/// };
/// let at = handles.iter().position(|h| h.workflow_id() == id).expect("in the set");
/// handles.swap_remove(at).result().await
/// # }
/// ```
///
/// Outside a workflow there is no context to read, so this is [`Error::NotInWorkflow`]. That is
/// where [`DBOS::select_workflow`] is the call.
///
/// **This rather than a [`select_step!`](crate::select_step) over the handles' awaits**, wherever
/// the branches are all workflows. Both are durable — an await is a step like any other — but this
/// settles the whole set with one query per poll interval, where N raced awaits are N pollers
/// asking separately. `select_step!` is for a race whose branches are not all of one kind.
/// Waits until every one of these workflows has finished.
///
/// The all-form of the free [`select_workflow`](fn@select_workflow), and the waiter a fan-out that
/// needs every answer reaches for. Same context rules and same error channel — but **nothing is
/// checkpointed and no step id is taken**, here or on any other surface, for the reasons the
/// module doc gives: an all-wait decides nothing, and the
/// [`result`](crate::WorkflowHandle::result) calls it exists to precede record the outcomes a
/// replay actually reads.
///
/// So this is an ordinary `async fn` where the first-wait is a [`PendingStep`], and being inside a
/// workflow is what it needs only to find an executor to wait through — which it looks for when
/// the future is first polled, since it has nothing to claim before then. Outside a workflow that
/// is [`Error::NotInWorkflow`], and [`DBOS::join_workflows`] is the call.
///
/// **An empty set is a satisfied wait**, answered at once, where an empty
/// [`select_workflow`](fn@select_workflow) is refused. Answering early is safe only because this
/// takes no step id: a set computed from state could be empty on one execution and not the next,
/// and a call that took an id in one and none in the other would shift every later step.
///
/// **A replay asks the set again, rather than returning from anything it recorded.** That is the
/// point of taking no step id, and it is what makes the members' *current* state the answer. The
/// case where it shows is a member deleted between one execution and the next: it never settles, so
/// the replayed join keeps waiting, where [`result`](crate::WorkflowHandle::result) on the same
/// handle would have read the outcome back out of the parent's own row.
///
/// Neither is the defined behaviour of a deleted workflow. Removing one that a parent is still
/// replaying through is collecting what is not garbage — the parent's recorded outcomes make some
/// paths survive it, this one does not, and the difference is not a contract either way. A fan-out
/// that must tolerate it should await its handles rather than join them first.
///
/// ```no_run
/// # async fn fan_out(child: dbos::WorkflowRef<u32, u32>) -> dbos::Result<u32> {
/// let mut handles = Vec::new();
/// for n in 0..3 {
/// handles.push(child.start(n).await?);
/// }
/// let ids: Vec<&str> = handles.iter().map(|h| h.workflow_id()).collect();
/// dbos::join_workflows(&ids).await?;
/// // Every handle now resolves without waiting.
/// let mut total = 0;
/// for handle in handles {
/// total += handle.result().await?;
/// }
/// # Ok(total) }
/// ```
pub async
/// Races these workflow handles and runs the arm belonging to the one that finishes first.
///
/// The macro form of [`select_workflow`](fn@select_workflow). Where that answers with the winner's
/// **id**, this hands the winning arm the winner's **typed result** — so a fan-out over children
/// that return different types is one expression rather than an id, a lookup against the set, and a
/// cast.
///
/// ```no_run
/// # async fn race(
/// # counter: dbos::WorkflowRef<u32, u32>,
/// # namer: dbos::WorkflowRef<(), String>,
/// # ) -> dbos::Result<String> {
/// let count = counter.start(3).await?;
/// let name = namer.start(()).await?;
/// dbos::select_workflow! {
/// n = count => format!("the counter won with {}", n?),
/// s = name => format!("the namer won with {}", s?),
/// }
/// # }
/// ```
///
/// # Inside a workflow
///
/// **Nothing new is recorded.** The expansion is one [`select_workflow`](fn@select_workflow) over
/// the handles' ids — the checkpoint that already makes this choice survive a replay — and then
/// [`result`](crate::WorkflowHandle::result) on the winner alone. A replayed body reads the same
/// winner back out of that checkpoint and awaits the same handle, so the arm taken the second time
/// is the arm taken the first.
///
/// **Only the winner is awaited**, and that is what keeps the step ids stable rather than being an
/// optimisation: a loser's `result` is called on neither the run nor the replay, so no id is taken
/// on one and not the other. Dropping the losing handles does not stop those workflows — a
/// workflow runs on whether or not anything is watching it, which is the difference between racing
/// workflows and racing steps.
///
/// # Outside a workflow
///
/// An instance before a semicolon is the other form, and it is the [`DBOS::select_workflow`]
/// surface where the bare form is the free [`select_workflow`](fn@select_workflow) one — the same
/// split the calls themselves make, spelled at the call site because a macro has no ambient context
/// of its own to consult:
///
/// ```no_run
/// # async fn race(dbos: &dbos::DBOS, a: dbos::WorkflowHandle<u32>, b: dbos::WorkflowHandle<u32>)
/// # -> dbos::Result<u32> {
/// dbos::select_workflow! { dbos;
/// first = a => first?,
/// second = b => second?,
/// }
/// # }
/// ```
///
/// Anything with the two waits on it goes there, which today is a [`DBOS`] or a
/// [`Client`](crate::Client) — the macro names the method rather than a type, so a client's
/// undurable wait and an instance's checkpointed one are the same expansion. **Inside a workflow,
/// prefer the bare form**: it needs no handle, for the reason a registered closure that captured a
/// [`DBOS`] would keep the instance alive for the life of the process, and a client's wait inside a
/// workflow body is not checkpointed and so runs again on every replay.
///
/// One wrinkle belongs to this form alone: an instance answers in the engine's own channel, so the
/// expansion lifts that error into the caller's, and where nothing else says which channel that is
/// — a test, or any caller that never `?`s the result — it has to be annotated. Inside a workflow
/// the body's own return type settles it and the question does not arise.
///
/// # The shape of an arm
///
/// **A branch is a variable holding a handle, not an arbitrary expression.** The handle is named
/// twice, once for its id and once to consume it, and an expression would be evaluated twice —
/// which for `child.start(n).await?` would start the workflow twice. A handle a body means to race
/// has been bound to a name by the time it gets here in any case.
///
/// **Two branches naming the same workflow are answered by the first**, in the order the arms are
/// written. [`select_workflow`](fn@select_workflow) accepts a repeated id for the same reason: the
/// answer names one workflow however many entries pointed at it.
///
/// The value is a [`Result`], because the wait itself can fail. Each arm binds its own handle's
/// result, so an arm decides for itself whether to `?` it, match it, or report it.
/// Waits for all of these workflows and returns what each of them returned.
///
/// The macro form of [`join_workflows`](fn@join_workflows), and the reason to reach for it rather
/// than awaiting each handle in turn: one poll loop settles the whole set — **one query per
/// interval whatever N is** — and only then is each result read, so N handles cost one wait and N
/// row reads instead of N waits.
///
/// ```no_run
/// # async fn both(
/// # counter: dbos::WorkflowRef<u32, u32>,
/// # namer: dbos::WorkflowRef<(), String>,
/// # ) -> dbos::Result<String> {
/// let count = counter.start(3).await?;
/// let name = namer.start(()).await?;
/// let (count, name) = dbos::join_workflows!(count, name)?;
/// # Ok(format!("{name} counted to {count}")) }
/// ```
///
/// **The results are what get recorded, and the awaits stay sequential.** The expansion is one
/// [`join_workflows`](fn@join_workflows), which records nothing, and then
/// [`result`](crate::WorkflowHandle::result) on each handle in source order — so the whole of what
/// a replay reads back is the N `DBOS.getResult` rows. Sequential is not a concession here: the
/// set is already settled by the time the first result is read, so every one of them is a row
/// read that does not wait — and taking them in source order is what keeps each `DBOS.getResult`
/// on the step id its replay expects.
///
/// **The first failure ends it**, in source order, as `try_join!` does: the tuple holds values
/// rather than results, so a failed child is the value of the whole call. Where each child's own
/// failure matters, [`join_workflows`](fn@join_workflows) followed by a `result` per handle reports
/// all of them.
///
/// An instance before a semicolon is the outside-a-workflow form, exactly as it is for
/// [`select_workflow!`](macro@crate::select_workflow), which sets out when to reach for which:
///
/// ```no_run
/// # async fn both(dbos: &dbos::DBOS, a: dbos::WorkflowHandle<u32>, b: dbos::WorkflowHandle<u32>)
/// # -> dbos::Result<u32> {
/// let (a, b) = dbos::join_workflows!(dbos; a, b)?;
/// # Ok(a + b) }
/// ```
///
/// A branch is a variable holding a handle, for the reason
/// [`select_workflow!`](macro@crate::select_workflow) gives.
///
/// **A comma there is not caught.** `join_workflows!(dbos, a, b)` is a perfectly good list of three
/// idents, so it is read as three handles and fails inside the expansion, where the instance turns
/// out to have no `workflow_id`. That ambiguity is why the separator is a semicolon at all: unlike
/// [`select_workflow!`](macro@crate::select_workflow), whose arms cannot be mistaken for an
/// instance, nothing distinguishes the two shapes here.
/// The same two waits from outside the application.
///
/// Identical to [`DBOS`]'s, with the two differences a client always has: there is no launch
/// check, because a client is connected or it does not exist; and **nothing is checkpointed**,
/// because a client has no step counter of its own to agree with a workflow's. Called from inside
/// a workflow body, a client's wait runs again on replay — the line
/// [`WorkflowHandle::result`](crate::WorkflowHandle::result) already draws for a client's handle
/// awaited there.
///
/// Both waits are here because this is the surface an operator's tool wants most: a client is the
/// caller most likely to hold a set of ids it did not start.
/// Names an id set for an error message, without letting a fan-out of thousands *become* the
/// message.
///
/// The first few and a count: enough to see which set is meant, where the whole of a wide one
/// would be a wall of ids that says no more than its first line did.