1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
//! The SDK's observability contract: documented, stable `tracing` spans and
//! events describing the operation lifecycle.
//!
//! The SDK instruments itself through the [`tracing`] facade only: it never
//! installs a subscriber, adds no observability dependency, and works with
//! whatever subscriber the application provides. This module documents the
//! names and fields the SDK emits and exports them as constants, so a
//! subscriber, filter, or exporter can match on them without hard-coding
//! strings.
//!
//! # Relation to plugins
//!
//! The [`crate::plugin`] hooks, such as
//! [`Plugin::on_operation_start`](crate::plugin::Plugin::on_operation_start)
//! and [`Plugin::wrap_user_function`](crate::plugin::Plugin::wrap_user_function),
//! are independent of this module's spans and events: a plugin works without
//! a `tracing` subscriber, and a subscriber works without a plugin. The
//! plugin hooks receive structured operation info that the events do not
//! carry, such as the operations map
//! ([`InvocationInfo::operations`](crate::plugin::InvocationInfo::operations))
//! and the changed operations
//! ([`OperationChangeInfo::updated_operations`](crate::plugin::OperationChangeInfo::updated_operations)).
//!
//! # Stability
//!
//! Everything named in this module is a public, semver-meaningful contract:
//!
//! - Renaming or removing a span name, event name, or field name listed here
//! is a **breaking change**.
//! - Adding a new event, or adding a new field to an existing span or event,
//! is a **minor** (non-breaking) change. Write consumers to tolerate
//! unknown fields and unknown events.
//!
//! # Spans
//!
//! | Span name | Wraps | Fields |
//! |-----------|-------|--------|
//! | [`span_names::EXECUTION`] (`durable_execution`), root | one invocation of the handler | `executionArn`, `requestId`, `isReplay`, `tenantId` |
//! | [`span_names::EXECUTION`] (`durable_execution`), child | each child namespace (a `run_in_child_context` body, a map item, a parallel branch, a `wait_for_callback` body, recorded, virtual or flat), in its own span, a child of the span current when the namespace is created | the root's fields, plus `operationId` and `operationName` of the child operation |
//! | [`span_names::OPERATION`] (`durable_operation`) | each live step body and each `wait_for_condition` check | `executionArn`, `requestId`, `isReplay`, `tenantId`, `operationId`, `operationName`, `attempt` |
//!
//! A child span is a child of the span current when its namespace is
//! created, so a record inside a child body also descends from the spans
//! around it: the parent namespace's span, and a span a plugin entered
//! around `next()` in `wrap_invocation` or `wrap_user_function`. The child
//! span repeats the root's fields, so the innermost SDK span of a record
//! holds the active namespace's or operation's values; where a formatter
//! puts the fields of outer spans is the formatter's choice. A plugin's
//! user-function span is a child of the SDK span of the user function it
//! wraps. `tenantId` is omitted when the invocation has no tenant
//! ID, and `operationName` is omitted when the operation has no recorded
//! name. A map item or parallel branch always has one: the SDK records
//! `map-item-{index}` or `parallel-branch-{index}` when the caller gave none.
//!
//! Both spans are created at the `INFO` level. The `isReplay` field on the
//! `durable_execution` span is dynamic: the SDK re-records it after every
//! operation claim, so it flips to `false` the moment the invocation crosses
//! the replay high-water mark. [`crate::ReplayFilterLayer`] is built on
//! exactly this contract: a record without its own `isReplay` field is
//! suppressed when an SDK span between it and its nearest
//! `durable_execution` span, that one included, is replaying. Spans outside
//! that namespace, and spans the SDK did not create, decide nothing.
//! [`crate::init_default_subscriber`] installs a subscriber with it.
//!
//! # Events
//!
//! Lifecycle events are emitted at the `DEBUG` level with the target
//! [`TARGET`] (`aws_durable_execution_sdk::lifecycle`). The event's
//! `message` is its stable name.
//!
//! Operation events fire when the SDK records an operation transition
//! (`operation_started`, `operation_succeeded`, `operation_failed`,
//! `operation_retry_scheduled`) or short-circuits one from recorded state
//! (`operation_replayed`). The record-transition events cover **every**
//! operation type the SDK checkpoints (steps, waits, invokes, callbacks,
//! `wait_for_condition` polls, child contexts, and map/parallel batches and
//! items), and each one is emitted only **after** the checkpoint write that
//! persists the transition succeeds: a rejected checkpoint records nothing,
//! so it emits nothing. The converse holds too: every transition the
//! service records emits its event: the events fire the moment the service
//! accepts the write, before any fallible follow-up work (such as fetching
//! paginated execution state), so a failure *after* the write still leaves
//! the persisted transitions' events emitted. The code path that performs
//! the write owns the event, so under checkpoint buffering
//! ([`checkpoint_delay`](crate::OptionsBuilder::checkpoint_delay) /
//! [`checkpoint_batching`](crate::OptionsBuilder::checkpoint_batching)) a
//! caller cancelled while awaiting a batched write (a dropped `race` or
//! `select_ok` loser) does not suppress the events for updates the flush
//! still persists, and when a large batch is split into several requests,
//! each request's events are emitted as soon as that request succeeds:
//! even if a later request in the same batch fails.
//!
//! `operation_replayed` likewise covers every operation type: steps, waits,
//! invokes, callbacks, `wait_for_condition`, child contexts, map/parallel
//! batches, and the combinators (`try_join_all`, `join_all`, `select_ok`,
//! `race`). It is emitted only after the recorded outcome has actually been
//! reconstructed: a corrupt payload or a failing custom serdes surfaces as
//! an error **without** the event, because no recorded terminal outcome was
//! returned. A custom serdes error marked with
//! [`RetryableSerdesError`](crate::serdes::RetryableSerdesError) instead
//! ends the invocation, also without the event. One exception is deliberate: a child context or batch recorded
//! in *replay-children* mode (its result was too large to checkpoint)
//! re-executes its body rather than short-circuiting, so it emits no
//! `operation_replayed` of its own: the operations inside it emit theirs.
//!
//! | Event name | Emitted when | Fields |
//! |------------|--------------|--------|
//! | [`event_names::OPERATION_STARTED`] | the SDK records the start of a live operation | identity fields, `attempt`, `isReplay = false` |
//! | [`event_names::OPERATION_SUCCEEDED`] | the SDK records an operation's success | identity fields, `attempt`, `isReplay = false` |
//! | [`event_names::OPERATION_FAILED`] | the SDK records an operation's permanent failure | identity fields, `attempt`, `isReplay = false`, `error` |
//! | [`event_names::OPERATION_RETRY_SCHEDULED`] | a retry strategy schedules another attempt | identity fields, `attempt` (the attempt that failed), `isReplay = false`, `delaySeconds`, `error` |
//! | [`event_names::OPERATION_REPLAYED`] | a recorded terminal outcome is returned without re-running the operation (steps, waits, invokes, callbacks, `wait_for_condition`, child contexts, map/parallel batches, combinators) | identity fields, `attempt`, `isReplay = true` |
//! | [`event_names::EXECUTION_STARTED`] | an invocation begins with no recorded operations to replay | `executionArn`, `requestId` |
//! | [`event_names::EXECUTION_RESUMED`] | an invocation begins with recorded operations to replay | `executionArn`, `requestId` |
//! | [`event_names::EXECUTION_SUSPENDED`] | the invocation ends by suspending (reported `PENDING`) | `executionArn`, `requestId` |
//!
//! Exactly one of `execution_started` / `execution_resumed` is emitted per
//! invocation, and `execution_suspended` at most once per invocation. All
//! three are emitted while the handler's `durable_execution` span is
//! entered, so they are span events of that span: a subscriber that
//! groups events by span (including the OpenTelemetry bridge below) sees
//! them on the execution span rather than as orphans.
//!
//! # Fields
//!
//! | Field | Type | Meaning |
//! |-------|------|---------|
//! | [`field_names::EXECUTION_ARN`] (`executionArn`) | string | durable execution ARN |
//! | [`field_names::REQUEST_ID`] (`requestId`) | string | Lambda invocation request ID |
//! | [`field_names::OPERATION_ID`] (`operationId`) | string | wire operation ID (SHA-256 hex digest of the positional ID) |
//! | [`field_names::OPERATION_NAME`] (`operationName`) | string, omitted when the operation has no recorded name | the name recorded for the operation in the execution history: its `.name("...")`, or a map item's or parallel branch's generated name such as `map-item-0` |
//! | [`field_names::TENANT_ID`] (`tenantId`) | string, omitted when the invocation has none | Lambda tenant ID (spans only) |
//! | [`field_names::OPERATION_TYPE`] (`operationType`) | string | one of `Step`, `Wait`, `Context`, `Callback`, `ChainedInvoke` |
//! | [`field_names::OPERATION_SUB_TYPE`] (`operationSubType`) | string, omitted when absent | wire sub-type (e.g. `Step`, `Wait`, `Map`, `Parallel`, `RunInChildContext`, `WaitForCondition`, `WaitForCallback`) |
//! | [`field_names::ATTEMPT`] (`attempt`) | integer | 1-based attempt number the event describes |
//! | [`field_names::IS_REPLAY`] (`isReplay`) | boolean | whether the event describes replayed (already recorded) work |
//! | [`field_names::DELAY_SECONDS`] (`delaySeconds`) | integer | delay before the next attempt (`operation_retry_scheduled` only) |
//! | [`field_names::ERROR`] (`error`) | string | error message (`operation_failed` and `operation_retry_scheduled` only) |
//!
//! # Consuming the contract with `tracing-subscriber`
//!
//! Lifecycle events are `DEBUG`-level, so a default `INFO` subscriber does
//! not print them. Enable the lifecycle target explicitly:
//!
//! ```
//! use tracing_subscriber::EnvFilter;
//! use tracing_subscriber::layer::SubscriberExt as _;
//!
//! // INFO everywhere, plus every SDK lifecycle event.
//! let filter = EnvFilter::new(format!(
//! "info,{}=debug",
//! aws_durable_execution_sdk::observability::TARGET,
//! ));
//! let subscriber = tracing_subscriber::registry()
//! .with(tracing_subscriber::fmt::layer().json())
//! .with(filter);
//! # drop(subscriber);
//! ```
//!
//! # OpenTelemetry bridge
//!
//! Because the contract is plain `tracing` instrumentation, the standard
//! [`tracing-opentelemetry`](https://docs.rs/tracing-opentelemetry) bridge
//! exports it without SDK support: `durable_execution` and
//! `durable_operation` become OpenTelemetry spans carrying the fields above
//! as attributes, and lifecycle events become span events. The bridge crates
//! are the application's dependencies, not this SDK's: the SDK's dependency
//! allowlist stays closed. A typical Lambda `main` (versions current at the
//! time of writing; the example is illustrative, not compiled):
//!
//! ```ignore
//! // [dependencies]
//! // opentelemetry = "0.30"
//! // opentelemetry-otlp = "0.30"
//! // opentelemetry_sdk = "0.30"
//! // tracing-opentelemetry = "0.31"
//! // tracing-subscriber = { version = "0.3", features = ["env-filter"] }
//! use aws_durable_execution_sdk as durable;
//! use opentelemetry::trace::TracerProvider as _;
//! use tracing_subscriber::layer::SubscriberExt as _;
//! use tracing_subscriber::util::SubscriberInitExt as _;
//!
//! #[tokio::main]
//! async fn main() -> Result<(), lambda_runtime::Error> {
//! let exporter = opentelemetry_otlp::SpanExporter::builder()
//! .with_tonic()
//! .build()?;
//! let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder()
//! .with_batch_exporter(exporter)
//! .build();
//! let tracer = provider.tracer("my-durable-function");
//!
//! // Export the SDK's spans and lifecycle events to OTel; keep
//! // CloudWatch logs at INFO.
//! tracing_subscriber::registry()
//! .with(tracing_opentelemetry::layer().with_tracer(tracer))
//! .with(tracing_subscriber::fmt::layer().json())
//! .with(tracing_subscriber::EnvFilter::new(format!(
//! "info,{}=debug",
//! durable::observability::TARGET,
//! )))
//! .init();
//!
//! durable::run(handler).await
//! }
//! ```
//!
//! Replay awareness carries over: an exporter that must not re-emit
//! replayed work can drop events where `isReplay` is `true`.
//! [`crate::ReplayFilterLayer`] implements exactly that
//! rule for events that carry the field. It additionally suppresses an
//! application log line, which carries no field of its own, while an SDK
//! span between the line and its nearest `durable_execution` span, that one
//! included, has `isReplay = true`.
/// The `tracing` target of every lifecycle event in this contract.
///
/// Use it to enable or route lifecycle events independently of the rest of
/// the application's logs, e.g. `EnvFilter::new("info,{TARGET}=debug")`.
pub const TARGET: &str = "aws_durable_execution_sdk::lifecycle";
/// Names of the spans the SDK creates.
/// Stable names (the `message` field) of the lifecycle events.
/// Names of the structured fields carried by the spans and events.
///
/// The JSON record of [`crate::init_default_subscriber`] and
/// [`crate::init_default_subscriber_with_writer`] writes `requestId`,
/// `executionArn`, `tenantId`, `operationId`, `operationName` and `attempt`
/// as top-level keys, taken from the event or the nearest SDK span, so the
/// `CloudWatch` Logs Insights query
/// `filter coalesce(durableExecutionArn, executionArn) like "<arn>"`
/// matches it. That record omits `isReplay`. `lambda_runtime`'s own
/// subscriber and a stock `fmt::layer().json()` keep the event's fields
/// under `fields` and the span fields under `span` and `spans` instead.