wingfoil 6.0.4

graph based stream processing framework
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

# OTLP Adapter

Wingfoil adapter for OpenTelemetry export. Two independent pathways:

- **Metrics** ([`push.rs`](push.rs)): `OtlpPush::otlp_push` pushes stream
  values as OpenTelemetry gauge metrics to any OTLP-compatible backend.
- **Traces** ([`traces.rs`](traces.rs)): `OtlpSpans::otlp_spans` emits
  one OpenTelemetry span per hop on a `Stream<P: HasLatency>`, with
  caller-supplied attributes (session ID, request ID, etc.) attached to
  the parent span. Use this for high-cardinality per-request data that
  would explode Prometheus label cardinality.

## Module Structure

```
otlp/
  mod.rs               # Public API re-exports, module-level docs
  push.rs              # OtlpPush trait + push_consumer async fn (metrics)
  traces.rs            # OtlpSpans trait + spans_consumer async fn (traces)
  integration_tests.rs # Integration tests (testcontainers — no external setup needed)
  CLAUDE.md            # This file
```

## Key Design Decisions

- Uses the OpenTelemetry Rust SDK (`opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp`).
  Metrics are exported via HTTP/protobuf (`http-proto` feature) to the OTLP `/v1/metrics` endpoint.
  Traces are exported via the HTTP span exporter.
- A `SdkMeterProvider` with a 500 ms `PeriodicReader` is created per consumer invocation so that
  the final batch of metrics is flushed before the function returns.
- The metric name must be a `&'static str` (static string literal) to satisfy the OTel SDK's
  `f64_gauge` builder requirements.
- **Historical / backtesting mode**: the consumer checks `ctx.run_mode` via `RunParams` and
  drains the source stream without connecting to any external service. No OTel provider is
  built and no network calls are made.
- The adapter is push-based (contrast with `prometheus` which is pull-based).
- `OtlpPush` is implemented for `dyn Stream<T>` where `T: Display` so any numeric or string
  stream can be pushed without wrapping.
- `OtlpSpans` is Rust-only: trace export requires `P: Element + HasLatency`, which is not
  available for generic Python streams. Use `.otlp_spans()` from Rust; Python can use `.otlp_push()`
  for metrics.

## Feature Flags

- `otlp` — enables the adapter (pulls in `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp`,
  `reqwest`).
- `otlp-integration-test` — enables `otlp` + `testcontainers` for self-contained integration tests.

## Pre-Commit Requirements

```bash
# 1. Standard checks
cargo fmt --all
cargo clippy --workspace --all-targets --all-features

# 2. Unit / doc tests (no external dependencies)
cargo test --features otlp -p wingfoil -- adapters::otlp

# 3. Integration tests (testcontainers pulls the collector image automatically)
RUST_LOG=INFO cargo test --features otlp-integration-test -p wingfoil \
  -- --test-threads=1 --nocapture adapters::otlp::integration_tests
```

## Integration Tests

Tests use `testcontainers` with `SyncRunner` to start an `otel/opentelemetry-collector` container
automatically — no manual Docker setup required. The container is stopped when dropped.

## Gotchas

- The OTel SDK logs its own startup/shutdown messages to stderr even with `RUST_LOG` unset.
  This is expected and harmless in tests.
- `PeriodicReader` interval is set to 500 ms to ensure at least one export happens during
  short-running tests. Decrease if tests become flaky; increase if the backend is slow.
- Non-numeric stream values are parsed as f64 via `.to_string().parse()`. Values that fail
  parsing are recorded as `0.0` and a warning is logged.
- The collector image version is pinned (`0.116.0`) to avoid unexpected API changes.
  Update the pin deliberately and re-run integration tests.