pub enum HealthEvent {
Show 25 variants
SourceConnected {
feed_id: FeedId,
},
SourceDisconnected {
feed_id: FeedId,
reason: MediaError,
},
SourceReconnecting {
feed_id: FeedId,
attempt: u32,
},
InsecureRtspSource {
feed_id: FeedId,
redacted_url: String,
},
StageError {
feed_id: FeedId,
stage_id: StageId,
error: StageError,
},
StagePanic {
feed_id: FeedId,
stage_id: StageId,
},
FeedRestarting {
feed_id: FeedId,
restart_count: u32,
},
FeedStopped {
feed_id: FeedId,
reason: StopReason,
},
BackpressureDrop {
feed_id: FeedId,
frames_dropped: u64,
},
FrameLag {
feed_id: FeedId,
frame_age_ms: u64,
frames_lagged: u64,
},
ViewEpochChanged {
feed_id: FeedId,
epoch: u64,
},
ViewDegraded {
feed_id: FeedId,
stability_score: f32,
},
ViewCompensationApplied {
feed_id: FeedId,
epoch: u64,
},
OutputLagged {
messages_lost: u64,
},
SourceEos {
feed_id: FeedId,
},
DecodeDecision {
feed_id: FeedId,
outcome: DecodeOutcome,
preference: DecodePreference,
fallback_active: bool,
fallback_reason: Option<String>,
detail: String,
},
SinkPanic {
feed_id: FeedId,
},
SinkTimeout {
feed_id: FeedId,
},
SinkBackpressure {
feed_id: FeedId,
outputs_dropped: u64,
},
TrackAdmissionRejected {
feed_id: FeedId,
rejected_count: u32,
},
BatchError {
processor_id: StageId,
batch_size: u32,
error: StageError,
},
BatchSubmissionRejected {
feed_id: FeedId,
processor_id: StageId,
dropped_count: u64,
},
BatchTimeout {
feed_id: FeedId,
processor_id: StageId,
timed_out_count: u64,
},
BatchInFlightExceeded {
feed_id: FeedId,
processor_id: StageId,
rejected_count: u64,
},
ResidencyDowngrade {
feed_id: FeedId,
requested: String,
effective: String,
},
}Expand description
A health event emitted by the runtime.
Subscribe via Runtime::health_subscribe() (aggregate).
Per-feed filtering is the subscriber’s responsibility.
Variants§
SourceConnected
The video source connected successfully.
SourceDisconnected
The video source disconnected.
SourceReconnecting
The video source is attempting to reconnect.
InsecureRtspSource
An RTSP source is using insecure (non-TLS) transport.
Emitted once per session start when the effective RTSP URL uses
rtsp:// instead of rtsps://. This is informational — the feed
still operates, but the operator should consider migrating the
camera to TLS or a firewalled network segment.
Not emitted when RtspSecurityPolicy::RequireTls is set (that
policy rejects insecure sources at config time).
Fields
StageError
A stage returned an error for a single frame. The frame was dropped; the feed continues.
StagePanic
A stage panicked. The feed will restart per its restart policy.
FeedRestarting
The feed is restarting.
FeedStopped
The feed has stopped permanently.
BackpressureDrop
Frames were dropped due to backpressure.
FrameLag
Frames are being processed with significant wall-clock staleness.
Emitted when the age of a frame at processing time (wall-clock now minus the wall-clock timestamp assigned at media bridge) exceeds a threshold, indicating the consumer is falling behind the source — typically due to buffer-pool starvation, inference backlog, or TCP accumulation.
Events are coalesced: under sustained lag, the executor emits
one event per throttle window (1 second) with frames_lagged
reflecting the number of stale frames in that window.
Fields
ViewEpochChanged
The view epoch changed (camera discontinuity detected).
Fields
ViewDegraded
The view validity has degraded due to camera motion.
ViewCompensationApplied
A compensation transform was applied to active tracks.
OutputLagged
The output broadcast channel is saturated — the internal sentinel receiver observed ring-buffer wrap.
An internal sentinel receiver (the slowest possible consumer)
monitors the aggregate output channel. When production outpaces
the sentinel’s drain interval, the ring buffer wraps past the
sentinel’s read position and messages_lost reports how many
messages the sentinel missed.
What this means: the channel is under backpressure. The sentinel observes worst-case wrap — it does not prove that any specific external subscriber lost messages. A subscriber consuming faster than the sentinel will experience less (or no) loss. Treat this as a saturation / capacity warning, not a per-subscriber loss report.
Attribution: this is a runtime-global event. The output channel is shared across all feeds, so saturation is not attributable to any single feed.
messages_lost semantics: per-event delta — the number of
messages the sentinel missed since the previous OutputLagged
event (or since runtime start for the first event). This is
not a cumulative total and not a count of messages lost
by external subscribers.
Throttling: events are coalesced to prevent storms. The runtime emits one event on the initial saturation transition, then at most one per second during sustained saturation, each carrying the accumulated sentinel-observed delta for that interval.
Action: consider increasing output_capacity or improving
subscriber throughput.
Fields
SourceEos
The source reached end-of-stream (file sources).
For non-looping file sources this is terminal: the feed stops
with StopReason::EndOfStream rather than restarting.
DecodeDecision
A decode decision was made for a feed’s video stream.
Emitted once per session start (not per frame) when the backend identifies which decoder was selected for the stream. Provides visibility into hardware vs. software decode selection without exposing backend-specific element names in required fields.
detail is a backend-specific diagnostic string (e.g., the
GStreamer element name). It is intended for logging — do not
match on its contents.
Fields
outcome: DecodeOutcomeHardware, Software, or Unknown.
preference: DecodePreferenceThe user-requested decode preference that was in effect.
fallback_active: boolWhether the adaptive fallback cache overrode the requested preference for this session.
SinkPanic
The per-feed OutputSink panicked during emit().
The output is dropped but the feed continues processing.
The runtime wraps OutputSink::emit() in catch_unwind to
prevent a misbehaving sink from tearing down the worker thread.
SinkTimeout
The per-feed sink worker did not shut down within the timeout.
The sink thread is detached and a placeholder sink is installed.
This typically means OutputSink::emit() is blocked on
downstream I/O. Distinct from SinkPanic
to allow operators to route hung-sink alerts separately from
crash alerts.
SinkBackpressure
The per-feed sink queue is full — output was dropped.
The feed continues processing; only the output delivery is dropped to prevent slow downstream I/O from blocking the perception pipeline.
outputs_dropped is the number of outputs dropped since the
last SinkBackpressure event (per-event delta).
TrackAdmissionRejected
Tracks were rejected by the temporal store’s admission control because the hard cap was reached and no evictable candidates (Lost/Coasted/Tentative) were available.
The feed continues processing. This event indicates tracker
saturation — the scene has more confirmed objects than
max_concurrent_tracks allows.
Events are coalesced per frame: a single event is emitted with the total number of tracks rejected in that frame.
BatchError
The batch processor returned an error or panicked.
All frames in the affected batch are dropped. Each feed thread waiting on that batch receives the error and skips the frame.
BatchSubmissionRejected
A feed’s batch submission was rejected because the coordinator’s queue is full or the coordinator has shut down. The frame is dropped for this feed.
Events are coalesced: under sustained overload, the executor
emits one event per throttle window (1 second) with
dropped_count reflecting the number of frames rejected in
that window. On recovery (a subsequent successful submission),
any remaining accumulated count is flushed immediately.
Indicates the batch coordinator is overloaded — either the
processor is too slow for the combined frame rate, or
max_batch_size is too small.
Fields
BatchTimeout
A feed’s batch response timed out — the coordinator did not
return a result within max_latency + response_timeout.
Events are coalesced: under sustained timeout conditions, the
executor emits one event per throttle window (1 second) with
timed_out_count reflecting the number of timeouts in that
window. On recovery (a subsequent successful submission), any
remaining accumulated count is flushed immediately.
Indicates the batch processor is slower than the configured
safety margin. Consider increasing response_timeout or
reducing max_batch_size.
Fields
BatchInFlightExceeded
A feed’s batch submission was rejected because the feed already has the maximum number of in-flight items in the coordinator pipeline (default: 1).
This occurs when a prior submission timed out on the feed side but has not yet been processed (or drained) by the coordinator. The in-flight cap prevents one feed from accumulating orphaned items in the shared queue.
Events are coalesced with the same 1-second throttle window as other batch rejection events. On recovery, any remaining accumulated count is flushed immediately.
Fields
ResidencyDowngrade
The effective device residency is lower than what was requested.
This occurs when DeviceResidency::Cuda was requested but the
required GStreamer CUDA elements (cudaupload, cudaconvert)
are not available at runtime — the backend silently falls back to
host-memory decoding.
Emitted once per session start alongside DecodeDecision.
Action: install GStreamer >= 1.20 CUDA plugins, or switch to
a platform-specific provider (DeviceResidency::Provider) for
guaranteed GPU residency.
Trait Implementations§
Source§impl Clone for HealthEvent
impl Clone for HealthEvent
Source§fn clone(&self) -> HealthEvent
fn clone(&self) -> HealthEvent
1.0.0 · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more