mediadecode_ffmpeg/video/mod.rs
1//! `mediadecode::VideoStreamDecoder` impl with HW + SW fallback.
2//!
3//! [`FfmpegVideoStreamDecoder`] starts on the hardware path: an inner
4//! [`crate::VideoDecoder`] that auto-probes VideoToolbox / VAAPI /
5//! NVDEC / D3D11VA. When every HW backend fails — at `open` time
6//! (no backend opens) or mid-stream ([`crate::Error::AllBackendsFailed`]
7//! from `send_packet` / `receive_frame` / `send_eof`) — we transparently
8//! fall back to a **software** `ffmpeg::decoder::Video` opened from the
9//! same `Parameters`.
10//!
11//! Two HW-exhaustion shapes feed the same fallback, distinguished by an
12//! **explicit origin** the `AllBackendsFailed` carries
13//! ([`crate::error::FallbackOrigin`]) — *not* by whether its rescued
14//! `unconsumed_packets` is empty (both shapes can be empty: a probe-era
15//! failure on the first packet has no prior history, exactly like every
16//! post-commit failure):
17//!
18//! * **Probe-era** (pre-first-frame, [`crate::error::FallbackOrigin::Probe`]):
19//! the inner decoder buffered every packet it consumed and surfaces them in
20//! `unconsumed_packets`. We **replay exactly those** through the SW decoder
21//! (lossless — no frame was delivered yet), then route the still-unconsumed
22//! current packet (the one the inner decoder failed on / refused) to SW
23//! ourselves. This is the original pre-runtime-fallback behaviour and is
24//! unchanged.
25//! * **Post-commit** (after the first frame, the inner probe is gone,
26//! [`crate::error::FallbackOrigin::PostCommit`]): a runtime HW-decode failure
27//! — e.g. VideoToolbox choking on H.264 High 4:2:2 10-bit — is reclassified
28//! to `AllBackendsFailed` by the inner decoder with an **empty**
29//! `unconsumed_packets` (the probe buffer no longer exists). Here we
30//! **degrade and continue** rather than reconstruct: open the SW decoder with
31//! an empty replay set and let it **resync at the next keyframe**. Fed forward
32//! packets from the failure point, the SW decoder naturally produces nothing
33//! until that keyframe, then decodes normally from there. The bounded span
34//! from the failure point to the next keyframe is dropped — an accepted,
35//! **loudly logged** gap (a single `tracing::warn!`), not a silent one. The
36//! indexing pipeline this serves prefers a small logged gap over the
37//! error-prone mid-stream-reconstruction state machine a lossless replay
38//! would require (see findit-studio/mediadecode#12). The *bounded*-ness is
39//! **enforced, not assumed**: a post-commit fallback enters a degraded-resync
40//! mode that holds until a **keyframe-anchored** resync — the SW decoder
41//! delivering a frame *after* a keyframe was fed to it across the gap. (Gating
42//! on a keyframe, not on *any* frame, matters because a lenient codec will
43//! decode a lone P-frame from the dropped span into a concealed frame; that
44//! must not count as a resync, or the one-GOP bound isn't truly enforced.) If
45//! EOF is reached while the mode is still pending — no keyframe ever arrived
46//! across the gap and the whole tail was lost — `receive_frame` escalates with
47//! a distinct [`VideoDecodeError::PostCommitNeverResynced`] (and a
48//! `tracing::error!`) rather than surfacing a clean end-of-stream that would
49//! swallow the tail silently. So the gap is either bounded-and-logged (a real
50//! keyframe resync happened) or reported-at-EOF (it never did) — never
51//! silent-and-unbounded.
52//!
53//! The post-commit path retains and reconstructs **zero** frames: it opens SW
54//! cold, forwards only the failure arm's current packet (or EOF), and lets SW
55//! resync naturally. It never populates the replay-frame queue, so the
56//! replay/conversion machinery the probe-era path uses cannot touch it.
57//!
58//! The probe-era replay happens before the new packet (or the next
59//! `receive_frame` poll) is processed, so a probe-era HW exhaustion on a
60//! non-seekable input loses no compressed data. The post-commit path
61//! intentionally accepts the next-keyframe gap.
62//!
63//! After the transition the decoder stays on SW for the rest of its
64//! life — there's no probe-back-to-HW logic; once we've decided the
65//! stream isn't HW-decodable, that decision is sticky.
66//!
67//! Frames produced by either path are converted via
68//! [`crate::convert::av_frame_to_video_frame`] so the consumer sees
69//! the same `mediadecode::VideoFrame<PixelFormat, VideoFrameExtra,
70//! FfmpegBytes>` shape regardless of which backend produced it.
71
72use std::collections::VecDeque;
73
74/// Maximum number of frames the SW fallback replay path will buffer
75/// while draining the new SW decoder during packet/EOF replay.
76/// Replaying many compressed packets through SW can produce hundreds
77/// of decoded frames before the fallback commits; with no cap the
78/// resident memory grows unbounded (e.g. 4K frames at ~12 MB each ×
79/// 100s of frames). 64 frames is enough room to absorb every
80/// realistic codec's reorder/lookahead window without becoming a
81/// resource sink.
82const SW_REPLAY_FRAME_CAP: usize = 64;
83
84use derive_more::{IsVariant, TryUnwrap, Unwrap};
85use ffmpeg_next::{Packet, codec::Parameters, frame};
86use mediadecode::{
87 Received, Sent, Timebase,
88 decoder::{ScaledOutputCapability, VideoStreamDecoder},
89 frame::VideoFrame,
90 packet::VideoPacket,
91};
92
93use crate::{
94 Backend, DecoderLimits, Error, Ffmpeg, Frame, VideoDecoder, boundary,
95 convert::{self, ConvertError},
96 decoder::{build_codec_context, try_clone_parameters},
97 error::FallbackFailed,
98 extras::{VideoFrameExtra, VideoPacketExtra},
99 frame::alloc_av_video_frame,
100};
101
102/// Which decode path a video session takes — the choice
103/// [`CarrierVideoStreamDecoder::open_as`] is given.
104///
105/// # The arms differ in what they PERMIT, not only in where they start
106///
107/// [`Auto`](Self::Auto) is a preference: it starts on hardware and is
108/// free to end on software, at open or mid-stream. The other two are
109/// **pins**, and a pin that a mid-stream failure could quietly undo
110/// would not be one — so a session opened on either of them stays on
111/// the path it was opened on for its whole life, and a hardware failure
112/// that `Auto` would degrade through is reported instead.
113///
114/// That is the difference the two consumers of this door need. A
115/// determinism comparison decodes *one stream* both ways and compares
116/// the pixels; a run that silently swapped paths halfway would compare
117/// nothing and say it had. An operator turning hardware off for a lane
118/// over a driver that produces wrong pixels needs it to stay off.
119///
120/// # Observability is unchanged
121///
122/// [`is_hardware`](CarrierVideoStreamDecoder::is_hardware) and
123/// [`is_software`](CarrierVideoStreamDecoder::is_software) read where a
124/// session **is**, which stays a live reading — under
125/// [`Auto`](Self::Auto) it can still change once, and under the pins it
126/// answers what was pinned because nothing can move it.
127///
128/// This type deliberately grows **no** `is_*` predicates of its own,
129/// where most vocabularies in this crate do. They would spell the
130/// decoder's two questions a second time with a different meaning —
131/// `path.is_software()` is *what was asked for* and
132/// `decoder.is_software()` is *where it ended up*, and under
133/// [`Auto`](Self::Auto) those genuinely differ. A caller that needs to
134/// branch on the choice it made already holds the value and can
135/// `match` it.
136#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
137pub enum DecodePath {
138 /// Probe the platform's hardware backends in order and fall back to
139 /// software — at open, and again on a mid-stream hardware failure.
140 ///
141 /// What [`CarrierVideoStreamDecoder::open`] has always done, and what
142 /// it still does.
143 Auto,
144 /// **This hardware backend, or nothing.** No other backend is probed
145 /// and software is never opened.
146 ///
147 /// A backend that cannot be opened for the stream fails the
148 /// [`open_as`](CarrierVideoStreamDecoder::open_as) call. A backend
149 /// that opens and then fails to decode surfaces
150 /// [`Error::AllBackendsFailed`] from the send or receive road that
151 /// met it, carrying that backend and what it said — the same error
152 /// [`Auto`](Self::Auto) treats as its cue to degrade, reported here
153 /// because degrading is what this arm declines.
154 Hardware(Backend),
155 /// **Software, with no probe at all.**
156 ///
157 /// Opens `libavcodec`'s own decoder for the stream directly. There is
158 /// no hardware in this session to fail, so there is nothing for it to
159 /// fall back from — the terminal state [`Auto`](Self::Auto) reaches
160 /// by degrading, entered on purpose.
161 Software,
162}
163
164/// `mediadecode::VideoStreamDecoder` impl with transparent HW → SW
165/// fallback.
166pub struct CarrierVideoStreamDecoder<C: crate::FfmpegCarrier> {
167 state: DecodeState,
168 /// The path this session was opened on — see [`DecodePath`].
169 ///
170 /// Read for exactly one question, [`Self::may_open_software`]: whether
171 /// a hardware exhaustion is this session's cue to degrade or its cue
172 /// to report. Kept as the whole choice rather than reduced to that
173 /// bit so a session can say what it *is*, not only what it allows.
174 path: DecodePath,
175 /// Codec parameters retained so we can open a software
176 /// `ffmpeg::decoder::Video` if the HW probe exhausts.
177 parameters: Parameters,
178 /// HW-side scratch frame (filled by [`VideoDecoder::receive_frame`]).
179 hw_scratch: Frame,
180 /// SW-side scratch frame (filled by `ffmpeg::decoder::Video::receive_frame`).
181 sw_scratch: frame::Video,
182 /// Frames produced while draining the SW decoder during fallback
183 /// replay (see [`Self::fall_back_to_sw`]). The trait's
184 /// `receive_frame` delivers from this queue before pulling new
185 /// frames from the SW decoder. Empty in steady-state operation.
186 sw_replay_frames: VecDeque<frame::Video>,
187 /// Resource ceilings for the frames this decoder exports, and for the
188 /// `AVCodecContext`s it opens — HW candidates, the SW fallback, and
189 /// any decoder a later probe advance builds all get the same number.
190 limits: DecoderLimits,
191 /// `true` once `send_eof` has been called on the active decoder.
192 /// Used to propagate EOF to the SW decoder when fallback fires
193 /// during the drain phase — without this, codecs that hold tail
194 /// frames at EOF would hang waiting for an EOF they already saw on
195 /// the HW path.
196 eof_sent: bool,
197 /// `true` between a **post-commit** fallback firing and a *keyframe-anchored*
198 /// resync (the SW decoder delivering a frame **after** a keyframe was fed to
199 /// it across the gap). A post-commit fallback opens SW cold and drops the
200 /// bounded span up to the next keyframe; the promise is that the span is
201 /// *bounded* — SW resyncs at that keyframe. This flag makes the promise
202 /// enforced rather than assumed: while it is set we have no proof SW ever
203 /// recovered from a real keyframe. It is cleared only when SW delivers a frame
204 /// *and* [`Self::degraded_keyframe_seen`] is set (a lone concealed P-frame a
205 /// lenient codec emits from the gap does **not** clear it); if EOF is reached
206 /// while it is still set the loss is escalated (a distinct loud error) rather
207 /// than silently swallowing the whole tail. Probe-era fallbacks never set it —
208 /// they replay losslessly and produce frames immediately.
209 degraded_resync_pending: bool,
210 /// `true` once a **keyframe** packet has been successfully fed to the SW
211 /// decoder while [`Self::degraded_resync_pending`] is set — i.e. a real resync
212 /// anchor crossed the gap. The pending flag clears only on a delivered SW
213 /// frame *after* this is set, so a concealed non-keyframe frame (a lenient
214 /// codec decoding a lone P-frame from the dropped span) cannot masquerade as a
215 /// resync and prematurely clear the guard. Set alongside `enter`/cleared with
216 /// the pending flag.
217 degraded_keyframe_seen: bool,
218 /// Packets fed to the SW decoder since the post-commit fallback fired while
219 /// [`Self::degraded_resync_pending`] is set — i.e. across the unresolved
220 /// resync gap. Reported in the escalation message so the lost span is
221 /// quantified ("N packets, no keyframe found"). Reset whenever the flag
222 /// clears or on `flush`.
223 degraded_packets_since_fallback: u64,
224 /// Source-stream time base, used to label produced frames.
225 time_base: Timebase,
226 /// The lane this decoder captures into. A marker: the carrier
227 /// appears in the frames it produces, not in its own state.
228 /// `true` when the scratch frame holds a decoded frame whose
229 /// conversion has **not committed** — see
230 /// [`CarrierAudioStreamDecoder::scratch_pending`](crate::audio::CarrierAudioStreamDecoder)
231 /// for the reasoning, which is the same on both roads.
232 ///
233 /// **This decoder has two scratches and can change which one is
234 /// current, so the seat is enforced rather than merely recorded.**
235 /// While it is set, `send_packet` and `send_eof` answer
236 /// [`Sent::MustDrain`]: both are the roads that commit a
237 /// hardware-to-software fallback, and a fallback under a parked frame
238 /// would leave the retry reading the *other* scratch — delivering a
239 /// stale frame, or refusing permanently and stranding a decoded one.
240 /// Refusing makes the retry's state the state that parked it **by
241 /// construction**, which is a stronger guarantee than remembering
242 /// which road produced it.
243 ///
244 /// **The discipline is unchanged; only its spelling moved.** It was
245 /// `VideoDecodeError::FramePending`, and the escape was already
246 /// documented as "call `receive_frame`, or `flush` to abandon it" —
247 /// which is to say it was back pressure wearing an error's clothes.
248 /// Now it says so, and a caller can act on it without inspecting a
249 /// backend-specific error type. The subtitle decoder keeps the same
250 /// seat one road over, spelled the same way.
251 scratch_pending: bool,
252 _carrier: core::marker::PhantomData<C>,
253}
254
255/// Hardware-decode seam behind [`DecodeState::Hw`]. In production this is
256/// the real [`VideoDecoder`]; tests substitute a fake to drive the
257/// post-commit fallback path without a live GPU. Mirrors the subset of
258/// `VideoDecoder`'s surface the wrapper drives on the HW path.
259pub(crate) trait HwInner: Send {
260 /// See [`VideoDecoder::send_packet`].
261 fn send_packet(&mut self, packet: &Packet) -> Result<Sent, Error>;
262 /// See [`VideoDecoder::receive_frame`].
263 fn receive_frame(&mut self, frame: &mut Frame) -> Result<Received, Error>;
264 /// See [`VideoDecoder::send_eof`].
265 fn send_eof(&mut self) -> Result<Sent, Error>;
266 /// See [`VideoDecoder::flush`]. Returns `Result` for a uniform seam even
267 /// though the inherent method is infallible.
268 fn flush(&mut self) -> Result<(), Error>;
269 /// Downcast to the concrete [`VideoDecoder`] when this seam is the real
270 /// HW decoder, so [`FfmpegVideoStreamDecoder::hardware_inner`] can keep
271 /// exposing it. Returns `None` for a test fake.
272 fn as_video_decoder(&self) -> Option<&VideoDecoder>;
273
274 /// Whether a packet submitted **now** would be recorded for replay.
275 ///
276 /// The probe keeps a rescue history so that a decoder which exhausts
277 /// every backend can hand the caller everything FFmpeg consumed since
278 /// open. It records by `av_packet_ref`, and
279 /// [`AllBackendsFailed::into_unconsumed_packets`] hands those
280 /// recordings out as owned, **mutable** `Packet`s — which is why the
281 /// view lane must not share its carrier's storage into a submission
282 /// that could be recorded. See
283 /// [`CarrierVideoStreamDecoder::send_packet_impl`].
284 fn records_submissions(&self) -> bool;
285}
286
287impl HwInner for VideoDecoder {
288 #[inline]
289 fn records_submissions(&self) -> bool {
290 self.is_probing()
291 }
292
293 #[inline]
294 fn send_packet(&mut self, packet: &Packet) -> Result<Sent, Error> {
295 VideoDecoder::send_packet(self, packet)
296 }
297 #[inline]
298 fn receive_frame(&mut self, frame: &mut Frame) -> Result<Received, Error> {
299 VideoDecoder::receive_frame(self, frame)
300 }
301 #[inline]
302 fn send_eof(&mut self) -> Result<Sent, Error> {
303 VideoDecoder::send_eof(self)
304 }
305 #[inline]
306 fn flush(&mut self) -> Result<(), Error> {
307 VideoDecoder::flush(self);
308 Ok(())
309 }
310 #[inline]
311 fn as_video_decoder(&self) -> Option<&VideoDecoder> {
312 Some(self)
313 }
314}
315
316/// Internal: which backend is currently driving the decode.
317enum DecodeState {
318 /// Hardware-backed decoder (auto-probe). May transition to `Sw` on
319 /// `AllBackendsFailed`. Boxed behind [`HwInner`] so tests can inject a
320 /// fake HW decoder.
321 Hw(Box<dyn HwInner>),
322 /// Software decoder. Terminal state.
323 Sw(SwDecoder),
324}
325
326/// A software decoder and the callback state its codec context points
327/// at.
328///
329/// The state carries the allocator judge's byte budget and the
330/// `get_format` declination; it has to outlive the `AVCodecContext`
331/// that references it, which is why it is a field here rather than a
332/// value dropped at the end of `open_sw_decoder`.
333///
334/// `Deref` so that every call site keeps talking to the decoder and
335/// only the construction changed — this pairing is a lifetime fact, not
336/// a new abstraction.
337pub(crate) struct SwDecoder {
338 decoder: ffmpeg_next::decoder::Video,
339 /// Declared **after** the decoder: fields drop in declaration order,
340 /// so the codec context is freed before the state it points at.
341 _callback_state: Box<crate::ffi::CallbackState>,
342}
343
344impl SwDecoder {
345 /// The callback state this decoder's codec context points at.
346 ///
347 /// Handed out as a raw pointer so an error closure can consult it
348 /// while the decoder itself is mutably borrowed — every software send
349 /// / receive / EOF failure on this road goes through
350 /// [`crate::decoder::software_exit`] with it, so a frame the
351 /// allocator judge refused surfaces named instead of as the `EINVAL`
352 /// libavcodec also uses for corrupt input.
353 ///
354 /// `Deref` alone was not enough: it exposes the decoder and hides the
355 /// state, so every call site kept wrapping raw and the budget refusal
356 /// had no way out on the whole software road — including the replay
357 /// and cold-fallback helpers, which drop the state when they finish.
358 pub(crate) fn state(&self) -> *const crate::ffi::CallbackState {
359 &*self._callback_state
360 }
361}
362
363impl core::ops::Deref for SwDecoder {
364 type Target = ffmpeg_next::decoder::Video;
365 fn deref(&self) -> &Self::Target {
366 &self.decoder
367 }
368}
369
370impl core::ops::DerefMut for SwDecoder {
371 fn deref_mut(&mut self) -> &mut Self::Target {
372 &mut self.decoder
373 }
374}
375
376/// What the cold SW decoder is fed on a **post-commit** degrade transition,
377/// named by the failure arm so the three shapes stay mutually exclusive (a
378/// current packet and EOF are never forwarded together). The post-commit path
379/// retains no replay frames, so this is the *only* thing handed to the new SW
380/// decoder at fallback time. See [`FfmpegVideoStreamDecoder::degrade_to_sw`].
381enum PostCommitInput<'a> {
382 /// `send_packet` arm: forward this current packet — the one the HW decoder
383 /// refused (so it was never in any replay set). If it is a keyframe it is the
384 /// resync anchor.
385 Packet(&'a Packet),
386 /// `receive_frame` arm: a frame-time failure has no current packet to forward.
387 FrameTime,
388 /// `send_eof` arm: EOF was pending on the HW path; re-forward it to the cold
389 /// SW so tail-delaying codecs don't hang.
390 Eof,
391}
392
393impl<C: crate::FfmpegCarrier + crate::CarrierOps> CarrierVideoStreamDecoder<C> {
394 /// Opens a decoder for the given codec parameters with the default
395 /// HW backend probe order. If the HW probe can't open any backend,
396 /// falls back to a software `ffmpeg::decoder::Video` immediately —
397 /// `open` only returns `Err` when both paths fail.
398 ///
399 /// Subsequent mid-stream `AllBackendsFailed` from the HW path
400 /// triggers the same SW fallback (with rescued packets replayed).
401 ///
402 /// `limits` bounds what one decoded frame may cost. It is taken here
403 /// rather than through a builder because half of it —
404 /// [`DecoderLimits::max_pixels`] — is written into every
405 /// `AVCodecContext` this decoder opens, and a context's ceiling
406 /// cannot be moved after `avcodec_open2`. That includes the contexts
407 /// opened later, by a mid-stream fallback or a probe advance: the
408 /// limits are retained for exactly that reason.
409 pub(crate) fn open_impl(
410 parameters: Parameters,
411 time_base: Timebase,
412 limits: DecoderLimits,
413 ) -> Result<Self, Error> {
414 Self::open_as_impl(parameters, time_base, limits, DecodePath::Auto)
415 }
416
417 /// [`Self::open_impl`], with the decode path chosen rather than
418 /// probed. `DecodePath::Auto` is the constructor above, verbatim.
419 pub(crate) fn open_as_impl(
420 parameters: Parameters,
421 time_base: Timebase,
422 limits: DecoderLimits,
423 path: DecodePath,
424 ) -> Result<Self, Error> {
425 // ffmpeg-next's `Parameters` carries an optional `owner: Rc<dyn Any>`
426 // (when constructed from `stream.parameters()` it points back at
427 // the demuxer's `AVStream`). Upstream marks the type `Send`
428 // anyway, which is unsound the moment a non-`None` owner is in
429 // play — moving such a value across threads moves the `Rc`. We
430 // sidestep this by always storing a deep-cloned `Parameters`
431 // (`avcodec_parameters_copy` produces an owner-free copy), so
432 // the `FfmpegVideoStreamDecoder`'s `Send` reachability never
433 // depends on the caller's owner discipline.
434 //
435 // Use `try_clone_parameters` instead of `Parameters::clone` —
436 // ffmpeg-next's `clone` calls `Parameters::new()` which can
437 // return a `Parameters` whose inner pointer is null on OOM
438 // (`avcodec_parameters_alloc` returns null without indication);
439 // the subsequent `avcodec_parameters_copy` against that null
440 // destination is C UB. Our checked helper surfaces the OOM as
441 // an error instead.
442 let owned_parameters = try_clone_parameters(¶meters, limits.max_codec_parameter_bytes())?;
443 let hw_scratch = Frame::empty()?;
444 let sw_scratch = alloc_av_video_frame()?;
445 let state = match path {
446 DecodePath::Auto => match VideoDecoder::open_with_frame_limits(
447 try_clone_parameters(&owned_parameters, limits.max_codec_parameter_bytes())?,
448 limits,
449 ) {
450 Ok(hw) => DecodeState::Hw(Box::new(hw)),
451 Err(Error::AllBackendsFailed(_)) => {
452 // Open-time HW exhaustion: no rescued packets (open didn't
453 // see any). Just open SW directly from our owned copy.
454 let sw = open_sw_decoder(&owned_parameters, limits)?;
455 DecodeState::Sw(sw)
456 }
457 Err(other) => return Err(other),
458 },
459 // **The named backend, and no probe order at all.** Nothing is
460 // tried before it and nothing after it, which is what makes the
461 // arm a pin: an open that fails is the answer, where `Auto` would
462 // have read the same failure as a reason to look elsewhere.
463 DecodePath::Hardware(backend) => DecodeState::Hw(Box::new(VideoDecoder::open_with_limits(
464 try_clone_parameters(&owned_parameters, limits.max_codec_parameter_bytes())?,
465 backend,
466 limits,
467 )?)),
468 // The software decoder, opened on purpose rather than reached by
469 // degrading. `DecodeState::Sw` is terminal, so this session has
470 // nothing to keep it on its path but the shape of the state
471 // machine itself.
472 DecodePath::Software => DecodeState::Sw(open_sw_decoder(&owned_parameters, limits)?),
473 };
474 Ok(Self {
475 state,
476 path,
477 parameters: owned_parameters,
478 hw_scratch,
479 sw_scratch,
480 sw_replay_frames: VecDeque::new(),
481 eof_sent: false,
482 degraded_resync_pending: false,
483 degraded_keyframe_seen: false,
484 degraded_packets_since_fallback: 0,
485 time_base,
486 limits,
487 scratch_pending: false,
488 _carrier: core::marker::PhantomData,
489 })
490 }
491
492 /// Returns `true` when this decoder has fallen back to the software
493 /// path. `false` while still on the HW probe (the initial state).
494 #[cfg_attr(not(tarpaulin), inline(always))]
495 pub(crate) const fn is_software_impl(&self) -> bool {
496 matches!(self.state, DecodeState::Sw(_))
497 }
498
499 /// Returns `true` while the HW probe is still active.
500 #[cfg_attr(not(tarpaulin), inline(always))]
501 pub(crate) const fn is_hardware_impl(&self) -> bool {
502 matches!(self.state, DecodeState::Hw(_))
503 }
504
505 /// Whether this session can currently honor a
506 /// [`Self::request_scaled_output_impl`] request. See
507 /// [`ScaledOutputCapability`] for the determinism trade a caller
508 /// takes on by requesting one.
509 ///
510 /// **Always [`ScaledOutputCapability::Unsupported`] today, on every
511 /// path this crate opens — hardware included.** That is a census
512 /// finding, not an oversight this seam papers over:
513 ///
514 /// - **Hardware.** This crate's hardware backends (see [`Backend`])
515 /// open through FFmpeg's *generic* hwaccel negotiation —
516 /// `av_hwdevice_ctx_create` + a strict `get_format` callback — the
517 /// same road every `Backend` variant takes. That road hands the
518 /// destination size to libavcodec unconditionally
519 /// (`AVHWFramesContext.width/height` are set from the coded
520 /// dimensions during `avcodec_open2`, not from anything a caller
521 /// supplies), and the one FFmpeg API built for a caller-owned
522 /// VideoToolbox session with its own destination size —
523 /// `AVVideotoolboxContext` / `av_videotoolbox_default_init`,
524 /// `libavcodec/videotoolbox.h` — is not part of `ffmpeg-sys-next`'s
525 /// bound surface (only the generic `hwcontext_videotoolbox.h` is).
526 /// The download step confirms the same ceiling from the other
527 /// side: `av_hwframe_transfer_data`'s own contract requires "the
528 /// two frames must have matching allocated dimensions … since not
529 /// all device types support transferring a sub-rectangle" — so
530 /// even a downscale applied only at the CPU-transfer step is
531 /// outside what this crate's FFmpeg-mediated hardware path can do.
532 /// VideoToolbox is the one hardware backend this crate can prove
533 /// real and wired on the host that built it (see
534 /// [`Self::is_hardware_impl`]'s neighbours), and the finding above
535 /// is exactly as true for it as for [`Backend::Vaapi`] /
536 /// [`Backend::Cuda`] / [`Backend::D3d11va`], which this crate wires
537 /// in source (`Backend::av_hwdevice_type`, `probe_order`) but
538 /// cannot compile, run, or verify on a non-Linux, non-Windows host.
539 /// A real decompression-session-level (or `VTPixelTransferSession`-
540 /// level) implementation for VideoToolbox is
541 /// [mediadecode#55](https://github.com/findit-studio/mediadecode/issues/55);
542 /// the native scaling seam each of the other three backends has —
543 /// NVDEC/CUVID in-decode scaling
544 /// ([#56](https://github.com/findit-studio/mediadecode/issues/56)),
545 /// VAAPI VPP
546 /// ([#57](https://github.com/findit-studio/mediadecode/issues/57)),
547 /// the D3D11 Video Processor
548 /// ([#58](https://github.com/findit-studio/mediadecode/issues/58))
549 /// — is filed the same way, rather than fabricated here.
550 /// - **Software.** See [`Self::request_scaled_output_impl`] for the
551 /// software road's own, separate refusal.
552 #[cfg_attr(not(tarpaulin), inline(always))]
553 pub(crate) const fn scaled_output_capability_impl(&self) -> ScaledOutputCapability {
554 ScaledOutputCapability::Unsupported
555 }
556
557 /// Requests decode-time output scaling to `size`; always refuses.
558 /// See [`Self::scaled_output_capability_impl`] for the hardware-side
559 /// finding this shares.
560 ///
561 /// **The software road's refusal has its own, different shape**,
562 /// worth naming rather than folding into "no backend does this yet":
563 /// FFmpeg's software decoders have no *general* decode-time scaling
564 /// seam. The one option that comes close — `AVCodecContext.lowres`
565 /// (the CLI's `-lowres`) — falls short on three separate counts, any
566 /// one of which would disqualify it as this seam's software answer:
567 ///
568 /// 1. **Narrow codec coverage.** `lowres` is wired only into the
569 /// legacy MPEG-family decoders (MPEG-1/2/4 part 2, H.263) that
570 /// still carry the low-resolution IDCT machinery it depends on.
571 /// HEVC, AV1 and VP9 — the codecs a modern HDR pipeline actually
572 /// decodes — implement no `lowres` support at all.
573 /// 2. **The one codec that is wired is broken.** `lowres` on H.264
574 /// (also nominally covered) has been non-functional for years —
575 /// the decoder does not honor it correctly — so even the "old
576 /// family" half of the promise does not hold across the board.
577 /// 3. **It is not a resize, it is reduced reconstruction.** Where it
578 /// does work, `lowres` decodes at a coarser IDCT precision
579 /// (`1<<lowres`), skipping reconstruction detail rather than
580 /// decoding in full and scaling the result — later inter frames
581 /// drift from a reference the decoder itself degraded, which is a
582 /// different (and worse) contract than "the same picture, smaller".
583 ///
584 /// So the software road's answer is not "unimplemented" the way the
585 /// hardware road's is — it is "full-size decode, then the fused
586 /// conform walk downstream", by design, on every codec this crate
587 /// decodes in software.
588 #[cfg_attr(not(tarpaulin), inline(always))]
589 pub(crate) const fn request_scaled_output_impl(
590 &mut self,
591 size: (u32, u32),
592 ) -> ScaledOutputCapability {
593 let _ = size;
594 ScaledOutputCapability::Unsupported
595 }
596
597 /// Borrow the inner [`VideoDecoder`] when this decoder is still on the
598 /// real HW path. Returns `None` after the SW fallback has fired (or, in
599 /// tests, when the HW seam is a fake rather than a real decoder).
600 #[cfg_attr(not(tarpaulin), inline(always))]
601 pub(crate) fn hardware_inner_impl(&self) -> Option<&VideoDecoder> {
602 match &self.state {
603 DecodeState::Hw(hw) => hw.as_video_decoder(),
604 DecodeState::Sw(_) => None,
605 }
606 }
607
608 /// Returns the time base associated with the source stream.
609 #[cfg_attr(not(tarpaulin), inline(always))]
610 pub(crate) const fn time_base_impl(&self) -> Timebase {
611 self.time_base
612 }
613
614 /// Whether this session may open a software decoder in answer to a
615 /// hardware exhaustion.
616 ///
617 /// **The one place the pin is enforced**, consulted by all three
618 /// roads that can meet [`Error::AllBackendsFailed`] — the two send
619 /// arms and the receive arm. It is one predicate rather than three
620 /// conditions because the pin is one promise: a session opened on
621 /// [`DecodePath::Hardware`] ends on hardware or ends in an error, and
622 /// a road that forgot to ask would break that promise silently,
623 /// which is the failure mode a caller cannot see.
624 ///
625 /// [`DecodePath::Software`] answers `true` and it costs nothing:
626 /// `DecodeState::Sw` is terminal, so no hardware exhaustion can
627 /// reach a road that asks. Answering for it by state rather than by
628 /// pin would make the predicate say something it does not mean.
629 #[cfg_attr(not(tarpaulin), inline(always))]
630 const fn may_open_software(&self) -> bool {
631 !matches!(self.path, DecodePath::Hardware(_))
632 }
633
634 /// Internal: **probe-era** transition from HW to SW. Replays the rescued
635 /// packets (the inner decoder's buffered history, already accepted by the HW
636 /// probe but not yet decoded) through the new SW decoder so the stream resumes
637 /// seamlessly. No frame was delivered on the HW path yet, so replaying the
638 /// history is lossless.
639 ///
640 /// Only the probe-era branches drive this. The **post-commit** path does
641 /// *not* — it retains and reconstructs zero frames, opening SW cold via
642 /// [`Self::degrade_to_sw`] and resyncing at the next keyframe instead of
643 /// replaying. (That is why this method's replay/drain machinery — and the
644 /// finding that the in-transaction drain doesn't cover later frame
645 /// *conversion* — cannot affect the post-commit path: it never produces a
646 /// post-commit replay frame to convert.)
647 ///
648 /// **Transactional**: drained replay frames accumulate in a local
649 /// queue; we only commit them to `self.sw_replay_frames` and switch
650 /// `self.state` to `Sw` after the replay (and EOF re-forwarding, if
651 /// needed) succeed. On failure, the SW decoder, the local frame
652 /// queue, and (where reachable) any consumed packets are dropped —
653 /// `self` is left in its prior state.
654 ///
655 /// **EOF-aware**: when EOF was already accepted on the HW path
656 /// (`self.eof_sent`), the new SW decoder also receives `send_eof()`
657 /// after replay. Without this, codecs that delay tail frames hang
658 /// forever in the drain phase.
659 ///
660 /// **EAGAIN-aware**: if SW's `send_packet` returns EAGAIN during
661 /// replay, drain produced frames into the local queue and retry.
662 ///
663 /// `eof_pending` is passed as a **local** argument rather than read from
664 /// `self.eof_sent`: callers must not mutate `self.eof_sent` before this
665 /// transaction commits (see [`VideoStreamDecoder::send_eof`]), so the
666 /// in-transaction SW EOF re-forward keys off the local flag and `self`'s
667 /// EOF state is updated only after a clean commit.
668 fn fall_back_to_sw(
669 &mut self,
670 unconsumed_packets: std::vec::Vec<ffmpeg_next::Packet>,
671 eof_pending: bool,
672 ) -> Result<(), Error> {
673 tracing::info!(
674 packets_replayed = unconsumed_packets.len(),
675 eof_pending,
676 "mediadecode-ffmpeg: HW probe exhausted, falling back to software decode",
677 );
678 // Wrap the internal worker so any failure path returns the
679 // rescued packets to the caller via `Error::FallbackFailed`.
680 // Without this, non-seekable streams (live feeds, pipes) would
681 // lose every compressed byte the HW path had consumed when a
682 // fallback transition fails partway.
683 match self.fall_back_to_sw_inner(&unconsumed_packets, eof_pending) {
684 Ok(()) => Ok(()),
685 Err(source) => Err(Error::FallbackFailed(FallbackFailed::new(
686 Box::new(source),
687 unconsumed_packets,
688 ))),
689 }
690 }
691
692 /// Worker for [`Self::fall_back_to_sw`]. Returns the rescued packets
693 /// untouched on the borrowed slice; the wrapper takes ownership of
694 /// them and surfaces them in `FallbackFailed` if this returns Err.
695 fn fall_back_to_sw_inner(
696 &mut self,
697 unconsumed_packets: &[ffmpeg_next::Packet],
698 eof_pending: bool,
699 ) -> Result<(), Error> {
700 let mut sw = open_sw_decoder(&self.parameters, self.limits)?;
701 // Bound before the decoder is mutably borrowed, so the error
702 // closures below can still consult it.
703 let sw_state = sw.state();
704 let mut local_replay: VecDeque<frame::Video> = VecDeque::new();
705 // Helper: drain SW into the local replay queue, capped at
706 // `SW_REPLAY_FRAME_CAP`.
707 //
708 // Error discipline: stop the drain **only** on the transient
709 // backpressure signals EAGAIN / EOF (the decoder has no more output for
710 // now). Every other `ffmpeg_next::Error` — e.g. `InvalidData` from a
711 // corrupt replayed packet — is a real decode failure and is propagated,
712 // so a non-recoverable error surfaces as `FallbackFailed` (carrying the
713 // replay packets) instead of being silently swallowed and the fallback
714 // committed over corruption.
715 fn drain_into(
716 sw: &mut ffmpeg_next::decoder::Video,
717 state: *const crate::ffi::CallbackState,
718 local_replay: &mut VecDeque<frame::Video>,
719 ) -> std::result::Result<(), Error> {
720 loop {
721 let mut tmp = alloc_av_video_frame()?;
722 match sw.receive_frame(&mut tmp) {
723 Ok(()) => {
724 if local_replay.len() >= SW_REPLAY_FRAME_CAP {
725 tracing::error!(
726 cap = SW_REPLAY_FRAME_CAP,
727 "mediadecode-ffmpeg: SW fallback replay produced more frames than the \
728 replay cap allows; aborting fallback (no frames dropped — they're \
729 still in the SW decoder's internal queue and will be released when \
730 it drops)",
731 );
732 return Err(Error::Ffmpeg(ffmpeg_next::Error::Other {
733 errno: libc::ENOMEM,
734 }));
735 }
736 local_replay.push_back(tmp);
737 }
738 // EAGAIN / EOF: no more output for now — stop draining, success.
739 Err(ffmpeg_next::Error::Other { errno }) if errno == ffmpeg_next::error::EAGAIN => {
740 break;
741 }
742 Err(ffmpeg_next::Error::Eof) => break,
743 // Any other error is a genuine decode failure on a replayed
744 // packet — surface it so it is not masked as a clean fallback.
745 Err(other) => return Err(crate::decoder::software_exit(state, other)),
746 }
747 }
748 Ok(())
749 }
750
751 for pkt in unconsumed_packets {
752 let mut attempts: u32 = 0;
753 loop {
754 match sw.send_packet(pkt) {
755 Ok(()) => break,
756 Err(ffmpeg_next::Error::Other { errno }) if errno == ffmpeg_next::error::EAGAIN => {
757 drain_into(&mut sw, sw_state, &mut local_replay)?;
758 attempts += 1;
759 if attempts > 16 {
760 return Err(Error::Ffmpeg(ffmpeg_next::Error::Other {
761 errno: ffmpeg_next::error::EAGAIN,
762 }));
763 }
764 }
765 Err(other) => return Err(crate::decoder::software_exit(sw_state, other)),
766 }
767 }
768 }
769 // Re-forward EOF if the HW path already saw it. SW EOF can also
770 // return EAGAIN until prior output is drained — mirror the
771 // packet-replay loop.
772 if eof_pending {
773 let mut attempts: u32 = 0;
774 loop {
775 match sw.send_eof() {
776 Ok(()) => break,
777 Err(ffmpeg_next::Error::Other { errno }) if errno == ffmpeg_next::error::EAGAIN => {
778 drain_into(&mut sw, sw_state, &mut local_replay)?;
779 attempts += 1;
780 if attempts > 16 {
781 return Err(Error::Ffmpeg(ffmpeg_next::Error::Other {
782 errno: ffmpeg_next::error::EAGAIN,
783 }));
784 }
785 }
786 Err(other) => return Err(crate::decoder::software_exit(sw_state, other)),
787 }
788 }
789 }
790 // Final drain BEFORE commit — the transactional commit boundary. The
791 // EAGAIN-triggered drains above only fire when SW exerts backpressure mid
792 // replay; a SW decoder that ACCEPTS every replayed packet (and the EOF)
793 // without one then surfaces a non-transient error — `InvalidData` from a
794 // corrupt replayed packet, or any other decode failure — only on the *next*
795 // `receive_frame`. Without this drain that error would land after the
796 // commit (frames appended, `state` flipped to `Sw`, rescued packets
797 // dropped) and reach the caller as a plain decode failure, not
798 // `FallbackFailed` — breaking probe-era recovery on non-seekable input.
799 // Draining to EAGAIN/EOF here forces any such error to surface now, so it is
800 // wrapped as `FallbackFailed` (retaining the rescued packets) and the
801 // decoder stays on HW — nothing is committed. (Only the probe-era path
802 // reaches this; the post-commit path degrades via `degrade_to_sw` and never
803 // replays, so it has no drained frames to commit or convert.)
804 drain_into(&mut sw, sw_state, &mut local_replay)?;
805 // Commit: only after replay, any EOF forwarding, AND the final drain
806 // succeeded do we move the new SW decoder and queue into `self`.
807 self.sw_replay_frames.append(&mut local_replay);
808 self.state = DecodeState::Sw(sw);
809 Ok(())
810 }
811
812 /// **Post-commit** degrade-and-continue transition: open the SW decoder
813 /// **cold** and forward only the failure-arm's input, retaining and
814 /// reconstructing **zero** frames. This is the whole post-commit path: open
815 /// SW, forward the current packet (or EOF), degrade-track — nothing is drained
816 /// into `sw_replay_frames`, so there is no replayed frame to convert later and
817 /// no terminal-drain transaction to reason about. SW naturally produces no
818 /// frame until the next keyframe arrives across the gap, then decodes normally;
819 /// the failure-point→next-keyframe span is the accepted, logged drop.
820 ///
821 /// **Transactional (SW-open only)**: `self.state` flips to `Sw` *only after*
822 /// `open_sw_decoder` and the input forward succeed. On any failure the new SW
823 /// decoder is dropped and the decoder is left on its prior HW state, the error
824 /// surfaced as [`Error::FallbackFailed`] (with an empty rescue set — a
825 /// post-commit failure never carries unconsumed packets). With no replay-frame
826 /// retention there is nothing else to roll back.
827 ///
828 /// On a clean commit it enters degraded-resync mode (see
829 /// [`Self::enter_degraded_resync`]); if the forwarded current packet is itself
830 /// a keyframe, the resync anchor is recorded immediately
831 /// ([`Self::note_degraded_keyframe`]).
832 ///
833 /// # `eof_pending`
834 ///
835 /// Whether the session's end-of-stream has already been **committed**,
836 /// and so must be re-forwarded into the cold decoder. Carried as a
837 /// local argument for the same two reasons the probe-era road carries
838 /// it (see [`Self::fall_back_to_sw`]): it is read from `eof_sent`
839 /// before anything is mutated, so a fallback that fails leaves no
840 /// half-truth behind — and one question deserves one mechanism on
841 /// both fallback roads.
842 ///
843 /// It is **not** expressed by selecting [`PostCommitInput::Eof`],
844 /// even though that arm forwards the same call. That enum is named by
845 /// the *failure arm* — which road raised the exhaustion — and the
846 /// `warn!` each site emits says so; borrowing the EOF arm for a
847 /// frame-time failure would make it lie about where the failure came
848 /// from.
849 fn degrade_to_sw(&mut self, input: PostCommitInput<'_>, eof_pending: bool) -> Result<(), Error> {
850 match self.degrade_to_sw_inner(input, eof_pending) {
851 Ok(()) => Ok(()),
852 // **A budget refusal is not a fallback failure.** It travels
853 // unwrapped, and the spelling was chosen rather than inherited:
854 //
855 // * `FallbackFailed` means the fallback *machinery* could not
856 // complete, and its contract is to hand back the unconsumed
857 // packets so a caller can re-drive them. On this road that set
858 // is empty by construction — the probe buffer is gone and no
859 // replay frames are retained — so the envelope carries no
860 // recovery affordance at all, only a label.
861 // * And the label is the wrong one. Re-driving is the natural
862 // response to a fallback failure, and re-driving a budget
863 // refusal under the same limits refuses identically. Naming it
864 // a fallback failure invites an action that cannot succeed,
865 // while `FrameBudgetExceeded` names the one that can: raise
866 // the ceiling, or accept the refusal.
867 //
868 // So it keeps the same spelling here as on every other road. One
869 // fact, one name.
870 Err(budget @ Error::FrameBudgetExceeded(_)) => Err(budget),
871 // Everything else really is the machinery failing, and keeps the
872 // envelope — empty rescue set and all, which is what a
873 // post-commit failure has to hand back.
874 Err(source) => Err(Error::FallbackFailed(FallbackFailed::new(
875 Box::new(source),
876 std::vec::Vec::new(),
877 ))),
878 }
879 }
880
881 /// Worker for [`Self::degrade_to_sw`]. Opens SW cold, forwards the arm's input,
882 /// and on success commits + enters degraded-resync mode. Returns `Err` (and
883 /// commits nothing) if SW cannot open or the forward fails.
884 fn degrade_to_sw_inner(
885 &mut self,
886 input: PostCommitInput<'_>,
887 eof_pending: bool,
888 ) -> Result<(), Error> {
889 // The invariant [`PostCommitInput`] documents, stated where it can
890 // be checked: a current packet and an end-of-stream are never
891 // forwarded together. The send road cannot violate it — its own
892 // gate refuses every packet once `eof_sent` is committed — so this
893 // records the coupling rather than defending against it.
894 debug_assert!(
895 !(matches!(input, PostCommitInput::Packet(_)) && eof_pending),
896 "a current packet and a committed EOF must never be forwarded together",
897 );
898 let mut sw = open_sw_decoder(&self.parameters, self.limits)?;
899 // Captured before the decoder is borrowed for the forward, and
900 // before it can be dropped on the error road: this temporary
901 // decoder owns the callback state, so a `judge_buffer` refusal
902 // recorded during either forward below dies with it unless the
903 // reason is collected here. That was the last software road still
904 // wrapping libavcodec's `EINVAL` raw.
905 let state = sw.state();
906 let mut forwarded_keyframe = false;
907 let mut forwarded_packet = false;
908 match input {
909 PostCommitInput::Packet(pkt) => {
910 // The HW decoder REFUSED this packet, so it was never decoded; forward
911 // it to the cold SW. A failure here surfaces (it is not silently
912 // dropped) and rolls back to HW.
913 sw.send_packet(pkt)
914 .map_err(|e| crate::decoder::software_exit(state, e))?;
915 forwarded_keyframe = pkt.is_key();
916 forwarded_packet = true;
917 }
918 // Neither of these forwards a packet; the end-of-stream below is
919 // the only thing they can hand the cold decoder.
920 PostCommitInput::FrameTime | PostCommitInput::Eof => {}
921 }
922 // **The end of the stream is re-forwarded here, on every arm that
923 // has one, and that is the fix rather than an extra.**
924 //
925 // The cold decoder knows nothing: it was opened a moment ago, from
926 // codec parameters alone. If the session had already been told the
927 // stream ended and this new decoder is not, it answers `EAGAIN` to
928 // every drain — which reaches the caller as
929 // [`Received::NeedsInput`], an instruction to send another packet.
930 // On a session whose end is committed there is no legal way to obey
931 // that: both send gates refuse. The caller loops, or quietly
932 // accepts a truncated tail, until `flush`.
933 //
934 // It used to be reachable only through the `Eof` failure arm, so
935 // the frame-time road — a post-commit exhaustion raised *while
936 // draining*, after EOF was accepted — opened cold and stayed cold.
937 // A cold decoder has no buffered output, so this cannot answer
938 // `EAGAIN` itself.
939 if eof_pending {
940 sw.send_eof()
941 .map_err(|e| crate::decoder::software_exit(state, e))?;
942 }
943 // Commit: only after a clean open + forward.
944 self.state = DecodeState::Sw(sw);
945 self.enter_degraded_resync();
946 if forwarded_keyframe {
947 // The refused current packet was itself the resync anchor.
948 self.note_degraded_keyframe(true);
949 }
950 if forwarded_packet {
951 self.count_degraded_packet();
952 }
953 Ok(())
954 }
955
956 /// Enter post-commit degraded mode after a post-commit fallback commits: the
957 /// SW decoder opened cold and the span up to the next keyframe is being
958 /// dropped. We hold this mode until SW proves a *keyframe-anchored* resync
959 /// (a delivered frame after a keyframe was fed — see
960 /// [`Self::note_degraded_keyframe`] / [`Self::resync_on_frame`]) and the EOF
961 /// escalation in [`VideoStreamDecoder::receive_frame`]. Called only on the
962 /// post-commit path, only after a clean commit. Resets the keyframe-seen anchor
963 /// and the gap counter.
964 #[inline]
965 fn enter_degraded_resync(&mut self) {
966 self.degraded_resync_pending = true;
967 self.degraded_keyframe_seen = false;
968 self.degraded_packets_since_fallback = 0;
969 }
970
971 /// Record that a packet fed to the SW decoder across an unresolved post-commit
972 /// gap was a **keyframe** — the resync anchor. Only a frame delivered *after*
973 /// this clears the pending flag, so a lenient codec's concealed P-frame can't
974 /// masquerade as a resync. A no-op outside degraded mode, or for a
975 /// non-keyframe.
976 #[inline]
977 fn note_degraded_keyframe(&mut self, is_key: bool) {
978 if self.degraded_resync_pending && is_key {
979 self.degraded_keyframe_seen = true;
980 }
981 }
982
983 /// Count one packet fed to the SW decoder while a post-commit resync is still
984 /// unproven, so the EOF escalation can quantify the lost tail. A no-op once
985 /// SW has resynced (the flag is clear).
986 #[inline]
987 fn count_degraded_packet(&mut self) {
988 if self.degraded_resync_pending {
989 self.degraded_packets_since_fallback = self.degraded_packets_since_fallback.saturating_add(1);
990 }
991 }
992
993 /// A SW frame was delivered. Clear post-commit degraded mode **only if** a
994 /// keyframe was fed across the gap ([`Self::degraded_keyframe_seen`]) — that is
995 /// a real keyframe-anchored resync, so the dropped span is now the promised
996 /// *bounded* gap. A frame delivered with no keyframe yet (a concealed P-frame
997 /// from the dropped span) leaves the guard set, so the one-GOP bound stays
998 /// enforced and the EOF escalation still fires if no keyframe ever arrives.
999 /// Idempotent; a no-op outside degraded mode (steady state, probe-era replay).
1000 #[inline]
1001 fn resync_on_frame(&mut self) {
1002 if self.degraded_resync_pending && self.degraded_keyframe_seen {
1003 self.clear_degraded_resync();
1004 }
1005 }
1006
1007 /// Unconditionally reset post-commit degraded-mode state. Used where the gap
1008 /// is moot regardless of resync proof: a `flush` (seek/reset re-anchors the
1009 /// stream) and the cleanup after an EOF escalation has already fired (so a
1010 /// follow-up poll sees plain EOF, not a repeated escalation). The
1011 /// frame-delivery path uses the keyframe-gated [`Self::resync_on_frame`]
1012 /// instead.
1013 #[inline]
1014 fn clear_degraded_resync(&mut self) {
1015 self.degraded_resync_pending = false;
1016 self.degraded_keyframe_seen = false;
1017 self.degraded_packets_since_fallback = 0;
1018 }
1019
1020 /// The one place a delivered frame is committed.
1021 ///
1022 /// Every road that hands a frame to the caller passes through here —
1023 /// the hardware scratch, the software scratch, both replay-queue
1024 /// entries, and the retry of a parked frame — so the bookkeeping a
1025 /// delivery owes cannot be attached to some of them and forgotten on
1026 /// others. It was: a parked software frame delivered on the retry
1027 /// road skipped [`Self::resync_on_frame`], so the last recovered
1028 /// frame of a degraded stream could leave the resync guard standing
1029 /// and turn a clean EOF into a false
1030 /// [`PostCommitNeverResynced`].
1031 fn commit_delivery(
1032 &mut self,
1033 frame: VideoFrame<mediadecode::PixelFormat, VideoFrameExtra, C::Buffer>,
1034 dst: &mut VideoFrame<mediadecode::PixelFormat, VideoFrameExtra, C::Buffer>,
1035 ) {
1036 // The seat is free once a carrier exists for what it held.
1037 self.scratch_pending = false;
1038 // A delivered frame is what clears a keyframe-anchored resync. A
1039 // no-op on every road that never entered degraded mode, which is
1040 // why it can be unconditional here.
1041 self.resync_on_frame();
1042 *dst = frame;
1043 }
1044
1045 /// Where this session is. See
1046 /// [`SessionPhase`](crate::decoder::SessionPhase).
1047 ///
1048 /// The wrapper never sees a probe — that lives inside the hardware
1049 /// seam, which derives its own — so only the committed pair is
1050 /// reachable from here.
1051 const fn phase(&self) -> crate::decoder::SessionPhase {
1052 if self.eof_sent {
1053 crate::decoder::SessionPhase::Draining
1054 } else {
1055 crate::decoder::SessionPhase::Streaming
1056 }
1057 }
1058
1059 /// Reads a drain answer against the session's own committed end.
1060 ///
1061 /// Routes a settled end through the post-commit gap check.
1062 ///
1063 /// **The `NeedsInput`-past-the-end reading moved out of here.** It
1064 /// used to be this method's own comparison against `eof_sent` — one
1065 /// more road deriving the session's phase for itself, which is the
1066 /// habit [`SessionPhase`](crate::decoder::SessionPhase) ended. The
1067 /// classifier makes that reading now, for every road at once, and
1068 /// what is left here is the part that is genuinely this wrapper's:
1069 /// an end is not clean if a post-commit gap never closed.
1070 fn settle(&mut self, status: Received) -> Result<Received, VideoDecodeError> {
1071 match status {
1072 Received::Ended => self.ended(),
1073 other => Ok(other),
1074 }
1075 }
1076
1077 /// The end of the stream, read against a post-commit gap that never
1078 /// closed.
1079 ///
1080 /// One place, because there are now two spellings that reach it — the
1081 /// substrate's `AVERROR_EOF` and a settled [`Received::NeedsInput`]
1082 /// past a committed end — and a lost tail must escalate on both. The
1083 /// flag is cleared as it fires so a caller draining to the end sees
1084 /// the escalation once and the plain end afterwards.
1085 fn ended(&mut self) -> Result<Received, VideoDecodeError> {
1086 if !self.degraded_resync_pending {
1087 return Ok(Received::Ended);
1088 }
1089 let packets_lost = self.degraded_packets_since_fallback;
1090 tracing::error!(
1091 packets_lost,
1092 "mediadecode-ffmpeg: post-commit HW->SW fallback never resynced before EOF — \
1093 {packets_lost} packets fed to the software decoder produced no frame (no \
1094 keyframe found across the gap); the stream tail from the fallback point was \
1095 lost",
1096 );
1097 self.clear_degraded_resync();
1098 Err(VideoDecodeError::PostCommitNeverResynced(
1099 PostCommitNeverResynced::new(packets_lost),
1100 ))
1101 }
1102
1103 /// Internal: convert the active scratch frame into a
1104 /// `mediadecode::VideoFrame` and write into `dst`.
1105 fn deliver_frame(
1106 &mut self,
1107 dst: &mut VideoFrame<mediadecode::PixelFormat, VideoFrameExtra, C::Buffer>,
1108 ) -> Result<Received, VideoDecodeError> {
1109 let av_frame = match &mut self.state {
1110 DecodeState::Hw(_) => unsafe { self.hw_scratch.as_inner_mut().as_ptr() },
1111 DecodeState::Sw(_) => unsafe { self.sw_scratch.as_ptr() },
1112 };
1113 // SAFETY: the scratch frame is live — either just filled by the
1114 // inner decoder's `receive_frame`, or left holding a frame whose
1115 // conversion did not commit. Convert takes what it needs out of it,
1116 // so the scratch can be reused once this has committed.
1117 let converted = unsafe {
1118 convert::av_frame_to_video_frame_as::<C>(av_frame, self.time_base, self.limits.frame())
1119 };
1120 match converted {
1121 Ok(new_frame) => {
1122 self.commit_delivery(new_frame, dst);
1123 Ok(Received::Frame)
1124 }
1125 Err(e) => {
1126 // Park only what another attempt could survive.
1127 self.scratch_pending = e.parks_in_decode();
1128 Err(VideoDecodeError::Convert(e))
1129 }
1130 }
1131 }
1132}
1133
1134#[cfg(test)]
1135impl<C: crate::FfmpegCarrier + crate::CarrierOps> CarrierVideoStreamDecoder<C> {
1136 /// Build a decoder around an injected HW seam, bypassing the real probe.
1137 /// Lets tests drive the post-commit fallback path with a [`HwInner`] fake
1138 /// instead of a live GPU. The SW fallback still opens the **real**
1139 /// `ffmpeg::decoder::Video` from `parameters`, so a fallback in these tests
1140 /// genuinely decodes.
1141 pub(crate) fn from_hw_inner_for_test(
1142 hw: Box<dyn HwInner>,
1143 parameters: Parameters,
1144 time_base: Timebase,
1145 ) -> Result<Self, Error> {
1146 Self::from_hw_inner_for_test_as(hw, parameters, time_base, DecodePath::Auto)
1147 }
1148
1149 /// [`Self::from_hw_inner_for_test`], with the session's
1150 /// [`DecodePath`] named.
1151 ///
1152 /// The seam a **pinned** session's mid-stream behaviour is driven
1153 /// through: a pin's promise is about what happens when the hardware
1154 /// fails after opening, and the only way to reach that on a machine
1155 /// whose GPU works is to inject a seam that fails on demand. See
1156 /// `a_hardware_pin_reports_a_mid_stream_exhaustion_instead_of_degrading`.
1157 pub(crate) fn from_hw_inner_for_test_as(
1158 hw: Box<dyn HwInner>,
1159 parameters: Parameters,
1160 time_base: Timebase,
1161 path: DecodePath,
1162 ) -> Result<Self, Error> {
1163 let limits = DecoderLimits::default();
1164 let owned_parameters = try_clone_parameters(¶meters, limits.max_codec_parameter_bytes())?;
1165 Ok(Self {
1166 state: DecodeState::Hw(hw),
1167 path,
1168 parameters: owned_parameters,
1169 hw_scratch: Frame::empty()?,
1170 sw_scratch: alloc_av_video_frame()?,
1171 sw_replay_frames: VecDeque::new(),
1172 eof_sent: false,
1173 degraded_resync_pending: false,
1174 degraded_keyframe_seen: false,
1175 degraded_packets_since_fallback: 0,
1176 time_base,
1177 limits,
1178 scratch_pending: false,
1179 _carrier: core::marker::PhantomData,
1180 })
1181 }
1182
1183 /// Whether `send_eof` has been committed on the active decoder. Lets the
1184 /// rollback tests assert that a failed EOF fallback restores (never
1185 /// half-mutates) `eof_sent`.
1186 pub(crate) const fn eof_sent_for_test(&self) -> bool {
1187 self.eof_sent
1188 }
1189
1190 /// Whether a post-commit fallback is awaiting a keyframe-anchored resync.
1191 /// Lets the escalation tests observe the degraded-resync state machine.
1192 pub(crate) const fn degraded_resync_pending_for_test(&self) -> bool {
1193 self.degraded_resync_pending
1194 }
1195
1196 /// Whether a keyframe has been fed to the SW decoder across the unresolved
1197 /// post-commit gap (the resync anchor). Lets the keyframe-gating test confirm
1198 /// a concealed P-frame does not set it (so the resync clear stays blocked).
1199 pub(crate) const fn degraded_keyframe_seen_for_test(&self) -> bool {
1200 self.degraded_keyframe_seen
1201 }
1202
1203 /// Whether the post-commit path retained any replay frames — must always be
1204 /// empty for a post-commit fallback (it retains zero). Lets the finding-1
1205 /// dissolution test assert no replay frame was ever queued.
1206 pub(crate) fn sw_replay_frames_is_empty_for_test(&self) -> bool {
1207 self.sw_replay_frames.is_empty()
1208 }
1209
1210 /// Packets fed to SW across an unresolved post-commit resync gap. Lets the
1211 /// counter test confirm packets crossing the gap from the `send_packet` arm
1212 /// are tallied (and cleared on resync).
1213 pub(crate) const fn degraded_packets_since_fallback_for_test(&self) -> u64 {
1214 self.degraded_packets_since_fallback
1215 }
1216}
1217
1218impl<C: crate::FfmpegCarrier + crate::CarrierOps> CarrierVideoStreamDecoder<C> {
1219 /// The fault a submission after end-of-stream earns on this face.
1220 ///
1221 /// **Censused from the empty-seat road rather than invented.** With
1222 /// the seat free, a post-EOF `send_packet` or a repeated `send_eof`
1223 /// reaches libavcodec, which answers `AVERROR_EOF`, and all four
1224 /// roads through this wrapper — hardware and software, packet and
1225 /// EOF — surface it as exactly this value. The gates below short
1226 /// out to the same one so a parked seat cannot change *which* answer
1227 /// a caller gets, only how quickly. `the_post_eof_fault_is_the_one_the_substrate_gives`
1228 /// pins the two against each other.
1229 ///
1230 /// Deliberately **not** a new `VideoDecodeError` arm. The subtitle
1231 /// seam had to mint `AfterEof` because `avcodec_decode_subtitle2` has
1232 /// no state machine to refuse for it; this face already has an answer
1233 /// for the condition, and a second spelling for one fault on one
1234 /// surface is the disease this release is curing.
1235 fn after_eof() -> VideoDecodeError {
1236 VideoDecodeError::Decode(Error::Ffmpeg(ffmpeg_next::Error::Eof))
1237 }
1238
1239 pub(crate) fn send_packet_impl(
1240 &mut self,
1241 packet: &VideoPacket<VideoPacketExtra, C::Buffer>,
1242 ) -> Result<Sent, VideoDecodeError> {
1243 // **The end of the stream outranks the parked seat, and the order
1244 // is the whole point.**
1245 //
1246 // `Sent::MustDrain` is a promise: drain the output and this same
1247 // offer becomes acceptable. Past end-of-stream that promise is
1248 // false — draining empties the seat and the retry still faults,
1249 // until `flush`. Checking the seat first made the wrapper answer
1250 // `MustDrain` for a submission nothing could ever accept, which is
1251 // the same fault-under-back-pressure inversion the subtitle seam
1252 // carried: a caller obeying the contract loops, drains, re-offers,
1253 // and is refused anyway.
1254 //
1255 // It is reachable: `send_eof` is accepted and sets `eof_sent`, a
1256 // delayed tail frame comes out of the decoder, its carrier
1257 // allocation fails parkably, and the seat is taken on a session
1258 // that is already over.
1259 if !self.phase().accepts_input() {
1260 return Err(Self::after_eof());
1261 }
1262 // **Nothing is sent while a frame is parked.** Both send roads can
1263 // commit a hardware-to-software fallback, and a fallback under a
1264 // parked frame would leave the retry reading the other scratch. See
1265 // [`Self::scratch_pending`]. Nothing was consumed, so this is back
1266 // pressure and the packet is still the caller's to re-offer — which
1267 // is true precisely because the stream is not over, checked above.
1268 if self.scratch_pending {
1269 return Ok(Sent::MustDrain);
1270 }
1271 let phase = self.phase();
1272 // Scoped submission: the rebuilt `AVPacket` never leaves this call,
1273 // which is what lets the view lane share its buffer with libavcodec
1274 // rather than copy into it. See `boundary::with_ffmpeg_video_packet`.
1275 let limits = self.limits.packet_limits();
1276 // **The route depends on what this decoder does with what it is
1277 // sent.** While the hardware probe is open it `av_packet_ref`s
1278 // every accepted packet into a rescue history, and
1279 // `AllBackendsFailed::into_unconsumed_packets` hands those out as
1280 // owned, mutable `Packet`s — so a shared body would escape this
1281 // call as a live mutable alias of a carrier the caller may still be
1282 // reading. Inside that window the body is copied; once the probe
1283 // has committed, nothing is recorded and the send is zero-copy
1284 // again. The software road never records.
1285 let route = match &self.state {
1286 DecodeState::Hw(hw) if hw.records_submissions() => crate::carrier::BodyRoute::Copy,
1287 _ => crate::carrier::BodyRoute::Submission,
1288 };
1289 boundary::with_ffmpeg_video_packet::<C, _>(packet, limits, route, |av_pkt| {
1290 match &mut self.state {
1291 DecodeState::Hw(hw) => match hw.send_packet(av_pkt) {
1292 // The seam already classified libavcodec's back pressure, so
1293 // both states travel on unchanged.
1294 Ok(status) => Ok(status),
1295 Err(Error::AllBackendsFailed(p)) => {
1296 // **A pinned hardware session reports rather than degrades.**
1297 // See [`Self::may_open_software`]: this is the exhaustion
1298 // `DecodePath::Auto` reads as its cue to open software, and
1299 // the pin's whole content is that it is not that cue here.
1300 // Reported with the payload intact, so the caller keeps the
1301 // backend, its error, and any rescued packets.
1302 if !self.may_open_software() {
1303 return Err(VideoDecodeError::Decode(Error::AllBackendsFailed(p)));
1304 }
1305 // Route on the EXPLICIT origin, never on whether `rescued` is empty (a
1306 // probe-era first-packet cap trip is *also* empty).
1307 if p.origin().is_post_commit() {
1308 // Post-commit: DEGRADE AND CONTINUE. No lossless mid-stream
1309 // reconstruction — the SW decoder opens cold, retains zero replay
1310 // frames, and resyncs at the next keyframe. The current packet (the
1311 // one HW REFUSED) is forwarded to that cold SW: if it is the resync
1312 // keyframe SW decodes from it, otherwise SW drops it until a keyframe
1313 // arrives. The bounded span from here to that keyframe is dropped — a
1314 // loudly logged gap (see the `warn!`), not a silent one.
1315 tracing::warn!(
1316 backend = ?p.attempts().last().map(|(b, _)| *b),
1317 pts = ?av_pkt.pts(),
1318 "mediadecode-ffmpeg: HW decode failed post-commit; falling back to \
1319 software, resyncing at next keyframe — a bounded span of frames \
1320 may be dropped at this boundary",
1321 );
1322 // Transactional SW-open + current-packet forward; degrade-tracking
1323 // (incl. keyframe-anchor recording) happens inside on a clean commit.
1324 // A failure surfaces `FallbackFailed` and stays on HW.
1325 // A clean degrade forwarded this very packet into the
1326 // cold software decoder, so it was consumed.
1327 // `false`: this road is unreachable once the end is
1328 // committed — `send_packet_impl`'s first gate refuses
1329 // every packet past `eof_sent` — so there is no EOF to
1330 // re-forward, and forwarding one alongside a packet is
1331 // the pairing [`PostCommitInput`] forbids.
1332 return self
1333 .degrade_to_sw(PostCommitInput::Packet(av_pkt), false)
1334 .map(|()| Sent::Accepted)
1335 .map_err(VideoDecodeError::Decode);
1336 }
1337 // Probe-era: replay the inner decoder's buffered history (lossless —
1338 // no frame was delivered yet), then forward the still-unconsumed
1339 // current packet to SW.
1340 let rescued = p.into_unconsumed_packets();
1341 // `eof_pending` is the committed EOF state — never pre-mutated here.
1342 let eof_pending = self.eof_sent;
1343 self
1344 .fall_back_to_sw(rescued, eof_pending)
1345 .map_err(VideoDecodeError::Decode)?;
1346 // Forward the new (still-unconsumed) current packet to the
1347 // freshly-opened SW decoder — the HW decoder REFUSED it, so it was not
1348 // in the replay set. A failure here surfaces (it is not silently
1349 // dropped), and back pressure from the fresh decoder is reported as
1350 // such rather than mistaken for one: the fallback committed either
1351 // way, and the caller re-offers the packet.
1352 if let DecodeState::Sw(sw) = &mut self.state {
1353 let st = sw.state();
1354 if let Err(e) = sw.send_packet(av_pkt) {
1355 return crate::decoder::software_send(st, e, phase)
1356 .map_err(VideoDecodeError::Decode);
1357 }
1358 }
1359 Ok(Sent::Accepted)
1360 }
1361 Err(other) => Err(VideoDecodeError::Decode(other)),
1362 },
1363 DecodeState::Sw(sw) => {
1364 let st = sw.state();
1365 if let Err(e) = sw.send_packet(av_pkt) {
1366 // Funnel, then gate. **Nothing below runs on back pressure**,
1367 // which is the point of returning here rather than falling
1368 // through: a packet libavcodec did not take must not be
1369 // counted across the resync gap or recorded as a keyframe
1370 // anchor, or a caller's honest re-offer would double-count
1371 // it.
1372 return crate::decoder::software_send(st, e, phase).map_err(VideoDecodeError::Decode);
1373 }
1374 // A keyframe fed across an unresolved post-commit gap is the resync
1375 // anchor; record it so the next delivered frame can clear the guard.
1376 self.note_degraded_keyframe(av_pkt.is_key());
1377 // Count packets crossing an unresolved post-commit resync gap so the
1378 // escalation at EOF can report how much tail was lost.
1379 self.count_degraded_packet();
1380 Ok(Sent::Accepted)
1381 }
1382 }
1383 })
1384 .map_err(|e| VideoDecodeError::Decode(Error::PacketBuild(e)))?
1385 }
1386
1387 pub(crate) fn receive_frame_impl(
1388 &mut self,
1389 dst: &mut VideoFrame<mediadecode::PixelFormat, VideoFrameExtra, C::Buffer>,
1390 ) -> Result<Received, VideoDecodeError> {
1391 // Deliver any frames produced during SW fallback replay before
1392 // pulling new ones from the SW decoder. This is the queue
1393 // populated by `fall_back_to_sw` when SW returned EAGAIN during
1394 // packet replay — a **probe-era** path only (the post-commit path retains
1395 // no replay frames), so `resync_on_frame` here is a no-op (probe-era never
1396 // enters degraded mode).
1397 // **Peeked, not popped.** A replayed frame is the rescue history's
1398 // only copy: popping it before the conversion committed lost it to
1399 // any allocation failure, which is the one thing this queue exists
1400 // to prevent. It leaves the queue when a carrier exists for it.
1401 if let Some(replayed) = self.sw_replay_frames.front() {
1402 // SAFETY: `replayed` is a live AVFrame owned by this queue;
1403 // convert takes what it needs out of it.
1404 let converted = unsafe {
1405 convert::av_frame_to_video_frame_as::<C>(
1406 replayed.as_ptr(),
1407 self.time_base,
1408 self.limits.frame(),
1409 )
1410 };
1411 let new_frame = match converted {
1412 Ok(new_frame) => new_frame,
1413 Err(e) if e.parks_in_decode() => return Err(VideoDecodeError::Convert(e)),
1414 // A frame nothing can carry is dropped rather than re-offered
1415 // forever — the same rule the scratch seat follows.
1416 Err(e) => {
1417 self.sw_replay_frames.pop_front();
1418 return Err(VideoDecodeError::Convert(e));
1419 }
1420 };
1421 self.sw_replay_frames.pop_front();
1422 self.commit_delivery(new_frame, dst);
1423 return Ok(Received::Frame);
1424 }
1425 // A frame whose conversion did not commit is converted again before
1426 // the decoder is asked for another — see [`Self::scratch_pending`].
1427 // The scratch still holds it, and `deliver_frame` reads whichever
1428 // scratch the current state uses.
1429 if self.scratch_pending {
1430 return self.deliver_frame(dst);
1431 }
1432 let phase = self.phase();
1433 loop {
1434 match &mut self.state {
1435 DecodeState::Hw(hw) => match hw.receive_frame(&mut self.hw_scratch) {
1436 Ok(Received::Frame) => {
1437 // The frame is out of the decoder's queue from here; the
1438 // seat is what keeps it if the conversion cannot commit.
1439 self.scratch_pending = true;
1440 return self.deliver_frame(dst);
1441 }
1442 // The hardware seam already classified the two flow signals.
1443 // They still pass the session's own end: see [`Self::settle`].
1444 Ok(status) => return self.settle(status),
1445 Err(Error::AllBackendsFailed(p)) => {
1446 // The pin, on the receive road — see
1447 // [`Self::may_open_software`] and the identical gate on the
1448 // two send roads.
1449 if !self.may_open_software() {
1450 return Err(VideoDecodeError::Decode(Error::AllBackendsFailed(p)));
1451 }
1452 // HW exhausted at frame-time. There is no current packet here.
1453 // Route on the explicit origin.
1454 if p.origin().is_post_commit() {
1455 // Post-commit: DEGRADE AND CONTINUE — open SW cold (no current
1456 // packet to forward, no replay frames retained) and resync at the
1457 // next keyframe, dropping the bounded span up to it. Loud single
1458 // `warn!` marks that accepted gap. A clean commit enters degraded
1459 // mode; a SW-open failure surfaces `FallbackFailed` and stays HW.
1460 tracing::warn!(
1461 backend = ?p.attempts().last().map(|(b, _)| *b),
1462 "mediadecode-ffmpeg: HW decode failed post-commit at frame-time; \
1463 falling back to software, resyncing at next keyframe — a bounded \
1464 span of frames may be dropped at this boundary",
1465 );
1466 // **The committed end travels with the fallback.** Read
1467 // before anything mutates, exactly as the probe-era road
1468 // below reads it. Without it the cold decoder answers
1469 // `EAGAIN` forever on a session no send can feed.
1470 let eof_pending = self.eof_sent;
1471 self
1472 .degrade_to_sw(PostCommitInput::FrameTime, eof_pending)
1473 .map_err(VideoDecodeError::Decode)?;
1474 // Nothing to deliver yet — fall through to the loop; the next
1475 // iteration takes the Sw arm and pulls from the cold SW decoder.
1476 continue;
1477 }
1478 // Probe-era: replay the buffered history (lossless).
1479 let rescued = p.into_unconsumed_packets();
1480 // `eof_pending` is the committed EOF state — never pre-mutated here.
1481 let eof_pending = self.eof_sent;
1482 self
1483 .fall_back_to_sw(rescued, eof_pending)
1484 .map_err(VideoDecodeError::Decode)?;
1485 // If the replay produced any drained frames, return one
1486 // immediately — preserves stream order vs. whatever the
1487 // SW decoder will produce next.
1488 // **Peeked, not popped** — the second delivery path onto
1489 // this queue, and it owes the same discipline as the first
1490 // (see the head of `receive_frame_impl`). The replay queue
1491 // is the rescue history's only copy of these frames, so a
1492 // conversion that cannot commit must leave the head where
1493 // it is rather than advance past it.
1494 if let Some(replayed) = self.sw_replay_frames.front() {
1495 // SAFETY: `replayed` is a live AVFrame owned by this
1496 // queue; convert takes what it needs out of it.
1497 let converted = unsafe {
1498 convert::av_frame_to_video_frame_as::<C>(
1499 replayed.as_ptr(),
1500 self.time_base,
1501 self.limits.frame(),
1502 )
1503 };
1504 let new_frame = match converted {
1505 Ok(new_frame) => new_frame,
1506 Err(e) if e.parks_in_decode() => return Err(VideoDecodeError::Convert(e)),
1507 // A frame nothing can carry is dropped rather than
1508 // re-offered forever.
1509 Err(e) => {
1510 self.sw_replay_frames.pop_front();
1511 return Err(VideoDecodeError::Convert(e));
1512 }
1513 };
1514 self.sw_replay_frames.pop_front();
1515 self.commit_delivery(new_frame, dst);
1516 return Ok(Received::Frame);
1517 }
1518 // Fall through to the loop; next iteration takes the Sw arm.
1519 }
1520 Err(other) => return Err(VideoDecodeError::Decode(other)),
1521 },
1522 DecodeState::Sw(sw) => {
1523 // Convert inline (rather than via `deliver_frame`, which borrows all
1524 // of `self`) so only the disjoint fields `sw_scratch` / `time_base`
1525 // are touched alongside the `self.state` borrow `sw` holds.
1526 let st = sw.state();
1527 match sw.receive_frame(&mut self.sw_scratch) {
1528 Ok(()) => {
1529 // The frame is out of the decoder's queue from here; the
1530 // seat is what keeps it if the conversion cannot commit.
1531 self.scratch_pending = true;
1532 // SAFETY: the scratch frame is live (just filled by
1533 // `receive_frame`); convert takes what it needs out of
1534 // it, so the scratch can be reused once this commits.
1535 let converted = unsafe {
1536 convert::av_frame_to_video_frame_as::<C>(
1537 self.sw_scratch.as_ptr(),
1538 self.time_base,
1539 self.limits.frame(),
1540 )
1541 };
1542 let new_frame = match converted {
1543 Ok(new_frame) => new_frame,
1544 Err(e) => {
1545 self.scratch_pending = e.parks_in_decode();
1546 return Err(VideoDecodeError::Convert(e));
1547 }
1548 };
1549 // SW produced a frame. The commit point clears degraded mode only
1550 // if a keyframe was fed across the gap — a real keyframe-anchored
1551 // resync, so the dropped span is the promised bounded gap. A
1552 // concealed P-frame (no keyframe yet) does not clear it (see
1553 // `resync_on_frame`).
1554 self.commit_delivery(new_frame, dst);
1555 return Ok(Received::Frame);
1556 }
1557 // Funnel first — so a recorded budget refusal is named
1558 // rather than laundered — read as a status second (`EAGAIN`
1559 // is `NeedsInput`, `Eof` is `Ended`, and the errno stops
1560 // inside this crate either way), and settled against the
1561 // session's own end third.
1562 //
1563 // That last step is where a post-commit resync that never
1564 // closed becomes [`VideoDecodeError::PostCommitNeverResynced`]
1565 // instead of a clean end that would swallow the tail — and
1566 // it now catches the end however the codec spelled it. See
1567 // [`Self::settle`] and [`Self::ended`].
1568 Err(e) => {
1569 let status =
1570 crate::decoder::software_receive(st, e, phase).map_err(VideoDecodeError::Decode)?;
1571 return self.settle(status);
1572 }
1573 }
1574 }
1575 }
1576 }
1577 }
1578
1579 pub(crate) fn send_eof_impl(&mut self) -> Result<Sent, VideoDecodeError> {
1580 // The same two gates in the same order, for the same reason: a
1581 // repeated end-of-stream past a committed one is refused however
1582 // much is drained, so answering back pressure would be a promise
1583 // this face cannot keep. See [`Self::after_eof`].
1584 if !self.phase().accepts_input() {
1585 return Err(Self::after_eof());
1586 }
1587 // As `send_packet`: EOF can commit a fallback too, and the escalation
1588 // it may raise reads the resync standing a parked frame has not yet
1589 // had the chance to clear. Nothing was recorded, so drain and signal
1590 // again.
1591 if self.scratch_pending {
1592 return Ok(Sent::MustDrain);
1593 }
1594 let phase = self.phase();
1595 let outcome = match &mut self.state {
1596 DecodeState::Hw(hw) => match hw.send_eof() {
1597 // The seam classified libavcodec's back pressure already.
1598 Ok(status) => Ok(status),
1599 Err(Error::AllBackendsFailed(p)) => {
1600 // The pin, on the EOF road — see [`Self::may_open_software`].
1601 // Returned rather than folded into `outcome`: the commit below
1602 // fires only on `Ok(Sent::Accepted)`, so the two roads agree,
1603 // and leaving early keeps the fallback body at the nesting it
1604 // was written at.
1605 if !self.may_open_software() {
1606 return Err(VideoDecodeError::Decode(Error::AllBackendsFailed(p)));
1607 }
1608 // EOF is pending for this transaction, so the SW decoder must also
1609 // receive `send_eof` (codecs that delay tail frames hang otherwise).
1610 // We pass that intent locally rather than pre-setting `self.eof_sent`:
1611 // a fallback that fails returns `FallbackFailed` and stays on HW, and a
1612 // half-mutated `self.eof_sent = true` would then make a *later*
1613 // fallback inject an EOF into SW even though this `send_eof` errored.
1614 // `self.eof_sent` is committed only after the whole operation succeeds
1615 // (the `outcome` check below), keeping the fallback all-or-nothing.
1616 if p.origin().is_post_commit() {
1617 // Post-commit: DEGRADE AND CONTINUE — open SW cold, re-forward EOF
1618 // (no current packet, no replay frames). The cold SW produces no
1619 // frame from EOF alone, so the drain-to-EOF in `receive_frame`
1620 // escalates (`PostCommitNeverResynced`) unless a later keyframe-fed
1621 // poll resyncs first. A clean commit enters degraded mode; a SW-open
1622 // failure surfaces `FallbackFailed` and stays HW.
1623 tracing::warn!(
1624 backend = ?p.attempts().last().map(|(b, _)| *b),
1625 "mediadecode-ffmpeg: HW decode failed post-commit at EOF; falling \
1626 back to software — a bounded span of tail frames may be dropped",
1627 );
1628 // Both fallback roads forward the EOF inside their own
1629 // transaction, so a clean commit means it was recorded.
1630 // `true`: this *is* the end being sent. `eof_sent` is not
1631 // committed until the whole operation succeeds, so the
1632 // intent is passed locally rather than read back.
1633 self
1634 .degrade_to_sw(PostCommitInput::Eof, true)
1635 .map(|()| Sent::Accepted)
1636 .map_err(VideoDecodeError::Decode)
1637 } else {
1638 // Probe-era: replay the buffered history (lossless), re-forwarding
1639 // EOF inside the transaction.
1640 let rescued = p.into_unconsumed_packets();
1641 self
1642 .fall_back_to_sw(rescued, true)
1643 .map(|()| Sent::Accepted)
1644 .map_err(VideoDecodeError::Decode)
1645 }
1646 }
1647 Err(other) => Err(VideoDecodeError::Decode(other)),
1648 },
1649 DecodeState::Sw(sw) => {
1650 let st = sw.state();
1651 match sw.send_eof() {
1652 Ok(()) => Ok(Sent::Accepted),
1653 Err(e) => crate::decoder::software_send(st, e, phase).map_err(VideoDecodeError::Decode),
1654 }
1655 }
1656 };
1657 // Commit EOF state only when the EOF was actually **taken** — a failed
1658 // fallback left `self.eof_sent` untouched (restored-by-construction: we
1659 // never mutated it), so HW stays EOF-not-yet-sent and a retry behaves
1660 // correctly.
1661 //
1662 // **`is_ok()` is not the test any more, and that is not a stylistic
1663 // change.** `Ok(Sent::MustDrain)` means the decoder did not take the
1664 // end-of-stream; recording `eof_sent` there would make a later fallback
1665 // inject an EOF into the software decoder for a signal that was never
1666 // accepted — the exact half-mutation the local `eof_pending` argument
1667 // exists to prevent on the failure road.
1668 if matches!(outcome, Ok(Sent::Accepted)) {
1669 self.eof_sent = true;
1670 }
1671 outcome
1672 }
1673
1674 pub(crate) fn flush_impl(&mut self) -> Result<(), VideoDecodeError> {
1675 // Drop any frames buffered during SW fallback replay before
1676 // flushing the inner decoder — otherwise a seek/reset would
1677 // surface stale pre-flush frames on the next `receive_frame`.
1678 self.sw_replay_frames.clear();
1679 // And a parked frame belongs to the position being abandoned.
1680 self.scratch_pending = false;
1681 // Flush ends the drain phase; the decoder accepts new packets
1682 // after this, so reset EOF tracking.
1683 self.eof_sent = false;
1684 // A flush (seek/reset) re-anchors the stream — any in-flight post-commit
1685 // resync tracking from before the flush is moot. Clear it so the next EOF
1686 // doesn't escalate over a now-irrelevant pre-flush gap.
1687 self.clear_degraded_resync();
1688 match &mut self.state {
1689 // The HW seam's `flush` returns `Result` for a uniform trait; the
1690 // real `VideoDecoder::flush` is infallible (always `Ok`).
1691 DecodeState::Hw(hw) => hw.flush().map_err(VideoDecodeError::Decode)?,
1692 DecodeState::Sw(sw) => sw.flush(),
1693 }
1694 Ok(())
1695 }
1696}
1697
1698macro_rules! video_lane_face {
1699 ($($lane:ty),+ $(,)?) => { $(
1700 impl CarrierVideoStreamDecoder<$lane> {
1701 /// Opens a video decoder for `parameters`, probing hardware
1702 /// backends in order and falling back to software.
1703 ///
1704 /// [`open_as`](Self::open_as)`(.., DecodePath::Auto)`, which is
1705 /// what this has always done.
1706 pub fn open(
1707 parameters: Parameters,
1708 time_base: Timebase,
1709 limits: DecoderLimits,
1710 ) -> Result<Self, Error> {
1711 Self::open_impl(parameters, time_base, limits)
1712 }
1713
1714 /// Opens a video decoder on a **named decode path**.
1715 ///
1716 /// [`DecodePath::Auto`] is [`open`](Self::open) exactly; the
1717 /// other two arms pin the session to hardware or to software for
1718 /// its whole life. See [`DecodePath`] for what a pin promises and
1719 /// what it costs.
1720 ///
1721 /// Everything else about the session is unchanged — the same
1722 /// [`VideoStreamDecoder`] face, the same frames, the same
1723 /// [`is_hardware`](Self::is_hardware) / [`is_software`](Self::is_software)
1724 /// readings. The choice is *which decoder is behind them*, which
1725 /// is what a determinism comparison and a deployment policy each
1726 /// need and neither could reach.
1727 ///
1728 /// # Errors
1729 ///
1730 /// [`DecodePath::Hardware`] fails here when the named backend
1731 /// cannot be opened for the stream — where [`DecodePath::Auto`]
1732 /// would have gone on to software. [`DecodePath::Software`] fails
1733 /// only where libavcodec has no decoder for the stream, or the
1734 /// context cannot be built.
1735 ///
1736 /// # Examples
1737 ///
1738 /// ```no_run
1739 /// use mediadecode_ffmpeg::{DecodePath, DecoderLimits, FfmpegVideoStreamDecoder};
1740 /// # fn f(parameters: ffmpeg_next::codec::Parameters, time_base: mediadecode::Timebase)
1741 /// # -> Result<(), Box<dyn std::error::Error>> {
1742 /// // The same stream, decoded without a GPU anywhere in the story.
1743 /// let decoder = FfmpegVideoStreamDecoder::open_as(
1744 /// parameters,
1745 /// time_base,
1746 /// DecoderLimits::default(),
1747 /// DecodePath::Software,
1748 /// )?;
1749 /// assert!(decoder.is_software());
1750 /// # Ok(())
1751 /// # }
1752 /// ```
1753 pub fn open_as(
1754 parameters: Parameters,
1755 time_base: Timebase,
1756 limits: DecoderLimits,
1757 path: DecodePath,
1758 ) -> Result<Self, Error> {
1759 Self::open_as_impl(parameters, time_base, limits, path)
1760 }
1761
1762 /// Whether this decoder is currently running on software.
1763 pub const fn is_software(&self) -> bool {
1764 self.is_software_impl()
1765 }
1766
1767 /// Whether this decoder is currently running on hardware.
1768 pub const fn is_hardware(&self) -> bool {
1769 self.is_hardware_impl()
1770 }
1771
1772 /// Whether this session can currently emit pictures at a
1773 /// caller-requested output size. See
1774 /// [`ScaledOutputCapability`] and
1775 /// [`Self::request_scaled_output`].
1776 pub const fn scaled_output_capability(&self) -> ScaledOutputCapability {
1777 self.scaled_output_capability_impl()
1778 }
1779
1780 /// Requests decode-time output scaling to `size` (width, height)
1781 /// and reports whether it took effect. See
1782 /// [`VideoStreamDecoder::request_scaled_output`] for the full
1783 /// contract (never an error) and
1784 /// [`Self::scaled_output_capability`]'s documentation for why
1785 /// every path this crate opens answers
1786 /// [`ScaledOutputCapability::Unsupported`] today.
1787 pub const fn request_scaled_output(&mut self, size: (u32, u32)) -> ScaledOutputCapability {
1788 self.request_scaled_output_impl(size)
1789 }
1790
1791 /// The hardware wrapper, when one is in use.
1792 pub fn hardware_inner(&self) -> Option<&VideoDecoder> {
1793 self.hardware_inner_impl()
1794 }
1795
1796 /// The stream timebase every produced timestamp is stamped with.
1797 pub const fn time_base(&self) -> Timebase {
1798 self.time_base_impl()
1799 }
1800 }
1801
1802 impl VideoStreamDecoder for CarrierVideoStreamDecoder<$lane> {
1803 type Adapter = Ffmpeg;
1804 type Buffer = <$lane as crate::FfmpegCarrier>::Buffer;
1805 type Error = VideoDecodeError;
1806
1807 fn send_packet(
1808 &mut self,
1809 packet: &VideoPacket<VideoPacketExtra, Self::Buffer>,
1810 ) -> Result<Sent, Self::Error> {
1811 self.send_packet_impl(packet)
1812 }
1813
1814 fn receive_frame(
1815 &mut self,
1816 dst: &mut VideoFrame<mediadecode::PixelFormat, VideoFrameExtra, Self::Buffer>,
1817 ) -> Result<Received, Self::Error> {
1818 self.receive_frame_impl(dst)
1819 }
1820
1821 fn send_eof(&mut self) -> Result<Sent, Self::Error> {
1822 self.send_eof_impl()
1823 }
1824
1825 fn flush(&mut self) -> Result<(), Self::Error> {
1826 self.flush_impl()
1827 }
1828
1829 fn scaled_output_capability(&self) -> ScaledOutputCapability {
1830 self.scaled_output_capability_impl()
1831 }
1832
1833 fn request_scaled_output(&mut self, size: (u32, u32)) -> ScaledOutputCapability {
1834 self.request_scaled_output_impl(size)
1835 }
1836 }
1837 )+ };
1838}
1839
1840video_lane_face!(crate::View, crate::Owned);
1841
1842fn open_sw_decoder(parameters: &Parameters, limits: DecoderLimits) -> Result<SwDecoder, Error> {
1843 // Use the checked codec-context builder — ffmpeg-next's
1844 // `Context::from_parameters` calls `Context::new()` which doesn't
1845 // null-check `avcodec_alloc_context3`'s return value before
1846 // running `avcodec_parameters_to_context` against it. Under
1847 // memory pressure that's C-level UB; `build_codec_context`
1848 // surfaces the OOM as an error instead.
1849 let (ctx, callback_state) = build_codec_context(parameters, limits)?;
1850 // Opened without forming a bindgen enum from FFmpeg memory: the codec
1851 // is resolved off a raw `codec_id`, and the medium is proved off a raw
1852 // `codec_type`. See `crate::decoder::ensure_codec_type`.
1853 let codec = crate::decoder::find_decoder(parameters)?;
1854 let opened = ctx.decoder().open_as(codec).map_err(Error::Ffmpeg)?;
1855 crate::decoder::ensure_video_codec_type(&opened)?;
1856 Ok(SwDecoder {
1857 decoder: ffmpeg_next::decoder::Video(opened),
1858 _callback_state: callback_state,
1859 })
1860}
1861
1862/// Payload for [`VideoDecodeError::PostCommitNeverResynced`].
1863///
1864/// A **post-commit** HW->SW fallback degraded the stream (dropping the
1865/// bounded span up to the next keyframe) but the software decoder
1866/// reached EOF without ever producing a frame — it never resynced, so
1867/// the entire tail from the failure point was lost. The "bounded,
1868/// logged gap" the post-commit path promises did not materialise (no
1869/// keyframe arrived before EOF), so the loss is surfaced loudly here
1870/// instead of being silently swallowed as a clean end-of-stream.
1871#[derive(thiserror::Error, Debug)]
1872#[error(
1873 "post-commit HW->SW fallback never resynced before EOF: {packets_lost} packets fed to the \
1874 software decoder produced no frame (no keyframe found across the gap) — the stream tail \
1875 from the fallback point was lost"
1876)]
1877pub struct PostCommitNeverResynced {
1878 packets_lost: u64,
1879}
1880
1881impl PostCommitNeverResynced {
1882 /// Constructs a `PostCommitNeverResynced` payload.
1883 #[inline]
1884 pub const fn new(packets_lost: u64) -> Self {
1885 Self { packets_lost }
1886 }
1887 /// Packets fed to the software decoder across the unresolved resync
1888 /// gap.
1889 #[inline]
1890 pub const fn packets_lost(&self) -> u64 {
1891 self.packets_lost
1892 }
1893}
1894
1895/// Error type for [`FfmpegVideoStreamDecoder`] — **faults and the
1896/// send-side refusal**.
1897///
1898/// Every arm here is something that went wrong or something the push
1899/// face declined. The drain's *needs input* and *ended* are
1900/// [`Received`] states out of `receive_frame`; they used to arrive as
1901/// `Decode(Ffmpeg(Other { errno: EAGAIN }))` and `Decode(Ffmpeg(Eof))`,
1902/// which is to say they had no name at this tier at all.
1903/// [`Self::PostCommitNeverResynced`] is the deliberate exception on the
1904/// end-of-stream road: it is not "the stream ended", it is "the stream
1905/// ended and the tail was lost", which is a fault.
1906///
1907/// **Open fault taxonomy, so it is `#[non_exhaustive]`.** New ways to
1908/// fail are discovered — a backend, a ceiling, a corruption a codec
1909/// learns to report — and a consumer that meets one it has never heard
1910/// of should take its generic-fault path. That is exactly what the
1911/// wildcard arm this attribute forces is for. The two status
1912/// vocabularies opposite it,
1913/// [`Sent`](mediadecode::Sent) and [`Received`](mediadecode::Received),
1914/// are exhaustive for the mirror-image reason: their arms are the
1915/// substrate's fixed state set, and there the wildcard would be dead
1916/// weight hiding a state a consumer forgot.
1917#[derive(thiserror::Error, Debug, IsVariant, Unwrap, TryUnwrap)]
1918#[unwrap(ref, ref_mut)]
1919#[try_unwrap(ref, ref_mut)]
1920#[non_exhaustive]
1921pub enum VideoDecodeError {
1922 /// The wrapped decoder (HW or SW) reported an error.
1923 #[error(transparent)]
1924 Decode(#[from] Error),
1925 /// Frame conversion from FFmpeg's native types to mediadecode's
1926 /// types failed.
1927 #[error(transparent)]
1928 Convert(#[from] ConvertError),
1929 /// A **post-commit** HW->SW fallback degraded the stream but the
1930 /// software decoder reached EOF without ever producing a frame.
1931 #[error(transparent)]
1932 PostCommitNeverResynced(#[from] PostCommitNeverResynced),
1933}
1934
1935#[cfg(test)]
1936mod tests;