mediadecode-ffmpeg 0.19.0

FFmpeg adapter for the mediadecode abstraction layer, built on top of ffmpeg-next.

Implements mediadecode's VideoAdapter / AudioAdapter / SubtitleAdapter / ImageAdapter traits, the matching push-style *StreamDecoder traits, the one-shot ImageDecoder, and Demuxer.

Every byte a frame or packet carries is copied once, at the FFmpeg boundary, into an FfmpegBytes — mediadecode 0.9's carrier amputation contract. A delivered frame is owned, Send + Sync, and cheap to clone (a refcount bump); it holds nothing of libavcodec's open, so it can cross a channel, be read from several threads, and outlive the decoder that produced it. Through 0.8 the planes were refcounted views into AVBufferRef behind an FfmpegBuffer type, which meant every consumer inherited an FFmpeg lifetime it could not see. That type is gone.

FfmpegVideoStreamDecoder mirrors the send_packet / receive_frame shape of ffmpeg::decoder::Video, auto-probes the host's HW backends, and falls through to a software decoder when none takes the stream before its first picture. Audio and subtitles use parallel FfmpegAudioStreamDecoder / FfmpegSubtitleStreamDecoder types.

Backends

FfmpegVideoStreamDecoder::open walks this probe order, opening the first backend that accepts the stream:

Output frames are CPU-side, downloaded with av_hwframe_transfer_data (NV12 for 8-bit, P010/P012/P016/P210/P212/P216/P410/P412/P416 for 10/12/16-bit). Pixel-format conversion is intentionally out of scope — downstream colconv handles it.

The probe keeps every packet it consumes until the first picture comes out. When no backend takes the stream by then, open (that is, DecodePath::Auto) replays those packets into the software decoder, so nothing is lost; DecodePath::AnyHardware and a DecodePath::Hardware pin report VideoDecodeError::Decode(Error::AllBackendsFailed(p)) instead, carrying them (p.unconsumed_packets() / p.into_unconsumed_packets()) — so non-seekable callers (live streams, pipes, network sources) can replay them through a software decoder of their own without re-demuxing.

After the first picture nothing changes the road, on any path, and nothing is classified. A decoder failure, VideoDecodeError::Decode, is that picture's own error, reported as the decoder minted it, and nothing of it is remembered, so the next call reaches libavcodec. FFmpeg has no reliable signal that a hardware session is gone (AVERROR_EXTERNAL, for one, also answers a single picture), and whether a hardware session recovers is FFmpeg's, not this crate's. FFmpeg 9.0.1 does not always recover one. After a VideoToolbox restart that fails, every picture fails the same way until a new parameter set re-arms the restart, and flush does not change that: it drops the pictures and references the codec holds, and it rebuilds no hardware session. VideoDecoder's documentation cites the FFmpeg lines.

A VideoDecodeError::Convert is the wrapper's own failure, not the decoder's: FfmpegVideoStreamDecoder could not convert a decoded picture into a frame (a frame ceiling, a pixel format or plane layout it cannot carry, an allocation). One that failed on an allocation parks the picture: the next receive_frame converts it again, and until one delivers it, send_packet and send_eof answer Sent::MustDrain without reaching libavcodec. That is the wrapper's back pressure, not a failure.

So a caller that sees a hardware session's decoder failures persist rebuilds: it opens a session on DecodePath::Software from the same parameters and feeds it forward. What the new session can be fed depends on the road the failure came from:

• send_packet: the failure names the packet in hand, and the new session is given that packet.
• receive_frame: the failure may concern a packet accepted earlier. FFmpeg decouples input from output and may hold several pictures (libavcodec/avcodec.h 90–139 in FFmpeg 9.0.1), so no packet is named: the new session is given the next packet, and the caller accepts the gap.
• send_eof: the end has no packet to give a new session. What the hardware session still held is recoverable only from packets kept from before, or by a seek.

When to rebuild is the caller's policy, not the decoder's, because it is the caller that sees the failures on the three roads, the packets' key flags and what it has delivered. What a policy counts is the hardware session's decoder failures; a Convert error is reported, not counted. The usage example below carries the simplest one.

Usage

A file's video track, decoded under the simplest policy for a hardware session the caller stops trusting. It counts the hardware session's decoder failures on all three roads, and only a delivered picture ends the count. At FAILURES_BEFORE_SOFTWARE failures in a row it opens a session on DecodePath::Software from the same parameters and feeds it forward: the packet in hand when the failure came from send_packet, otherwise what comes next, the next packet or the end. It keeps no packets and replays nothing, so every picture the new session delivers is delivered as it is. A Convert error is reported, not counted: the example returns it, as it returns every error it does not handle.

use ffmpeg_next as ffmpeg;
use ffmpeg::{codec, format, media};
use mediadecode::{Received, Sent, Timebase, decoder::VideoStreamDecoder};
use mediadecode_ffmpeg::{
  DecodePath, DecoderLimits, Error, FfmpegVideoStreamDecoder, PacketLimits, VideoDecodeError,
  VideoFrame, VideoPacket, empty_video_frame, video_packet_from_ffmpeg_in,
};

type BoxError = Box<dyn std::error::Error>;

/// This caller's threshold, not the decoder's: how many decoder failures
/// in a row, with no picture delivered between them, it takes from a
/// hardware session before it rebuilds on software.
const FAILURES_BEFORE_SOFTWARE: u32 = 3;

fn main() -> Result<(), BoxError> {
  ffmpeg::init()?;

  let path = std::env::args().nth(1).expect("usage: <input-file>");
  let mut input = format::input(&path)?;
  let stream = input.streams().best(media::Type::Video).unwrap();
  let stream_index = stream.index();
  let time_base = Timebase::new(
    stream.time_base().numerator(),
    std::num::NonZeroI32::new(stream.time_base().denominator()).unwrap(),
  );
  // A copy of its own: a software session opened later opens from it.
  let parameters = stream.parameters().clone();

  let mut track = Track::open(parameters, time_base)?;
  for (s, av_packet) in input.packets() {
    if s.index() != stream_index { continue; }
    // `Ok(None)` is an empty packet; an `Err` is a payload that is
    // there and could not be referenced, which is never silently
    // skipped.
    // **By value.** The bare names are the view lane, where a packet's
    // payload is a window into libavformat's own buffer — so the source
    // is handed over rather than lent. (The borrowing doors,
    // `owned_*`, are the owned lane: they copy, so the packet stays
    // yours.)
    //
    // SAFETY: `packets()` reads each packet with `av_read_frame`, so
    // libavformat wrote its payload; it is consumed here.
    let Some(pkt) =
      (unsafe { video_packet_from_ffmpeg_in(av_packet, time_base, PacketLimits::default()) })?
    else { continue };
    track.push(&pkt)?;
  }
  track.finish()
}

/// A video track decoded under this caller's policy for a hardware
/// session it stops trusting.
///
/// The policy feeds forward and does nothing else: it keeps no packets,
/// replays nothing and matches no pictures. That costs the pictures from
/// a failure to the next keyframe: a new session holds no reference
/// pictures, so libavcodec drops or conceals what comes before one, and
/// what the hardware session still held goes with it. A caller that
/// cannot afford that gap keeps the packets since the last clean keyframe
/// and replays them with a picture identity of its own; that bookkeeping
/// is the caller's design and is not shown here, because a doc example
/// cannot carry it correctly.
struct Track {
  parameters: codec::Parameters,
  time_base: Timebase,
  decoder: FfmpegVideoStreamDecoder,
  frame: VideoFrame,
  /// The hardware session's decoder failures since the last picture
  /// delivered.
  failures_in_a_row: u32,
}

impl Track {
  fn open(parameters: codec::Parameters, time_base: Timebase) -> Result<Self, BoxError> {
    // Probes HW backends in order and falls back to software when none
    // takes the stream before its first picture — so an error here means
    // software could not open it either.
    let decoder =
      FfmpegVideoStreamDecoder::open(parameters.clone(), time_base, DecoderLimits::default())?;
    Ok(Self {
      parameters,
      time_base,
      decoder,
      frame: empty_video_frame(),
      failures_in_a_row: 0,
    })
  }

  /// The `send_packet` road. A failure here names this packet.
  fn push(&mut self, pkt: &VideoPacket) -> Result<(), BoxError> {
    loop {
      match self.decoder.send_packet(pkt) {
        Ok(Sent::Accepted) => return self.drain(),
        // Back pressure, not a failure: nothing was consumed, so drain
        // and offer this same packet again. The old idiom — submit
        // twice and treat the second failure as real — is what this
        // arm replaces.
        Ok(Sent::MustDrain) => self.drain()?,
        Err(VideoDecodeError::Decode(e)) => {
          // A software session opened for this failure is given this
          // packet; otherwise the packet is behind us.
          if self.failed(e)? {
            continue;
          }
          return self.drain();
        }
        // `VideoDecodeError` is `#[non_exhaustive]`: a fault this code
        // has never heard of takes the generic road, which is the right
        // handling for one.
        Err(e) => return Err(e.into()),
      }
    }
  }

  /// The `send_eof` road, then the tail. The end has no packet: a
  /// software session opened for a failure here is given the end and
  /// nothing else, so what the hardware session still held is lost.
  fn finish(&mut self) -> Result<(), BoxError> {
    loop {
      match self.decoder.send_eof() {
        Ok(Sent::Accepted) => return self.drain(),
        Ok(Sent::MustDrain) => self.drain()?,
        Err(VideoDecodeError::Decode(e)) => {
          if self.failed(e)? {
            continue;
          }
          // The end stays refused: what is ready is drained, and what
          // the session still holds is lost with it.
          return self.drain();
        }
        Err(e) => return Err(e.into()),
      }
    }
  }

  /// The `receive_frame` road: every picture the session has ready, up
  /// to `NeedsInput`, or to `Ended` once the end is taken. `Received` is
  /// a state, so the loop stops on one rather than on whatever the last
  /// error was.
  fn drain(&mut self) -> Result<(), BoxError> {
    loop {
      match self.decoder.receive_frame(&mut self.frame) {
        Ok(Received::Produced) => self.deliver(),
        Ok(Received::NeedsInput | Received::Ended) => return Ok(()),
        // A failure here may concern a packet accepted earlier: FFmpeg
        // decouples input from output and may hold several pictures. No
        // packet is named, so a software session opened for it is given
        // the next packet.
        Err(VideoDecodeError::Decode(e)) => {
          if self.failed(e)? {
            return Ok(());
          }
        }
        // Not counted: a `Convert` error is the wrapper's own, a decoded
        // picture it could not convert into a frame, and not a decoder
        // failure. It is reported, as is anything this code has never
        // heard of.
        Err(e) => return Err(e.into()),
      }
    }
  }

  /// A picture, delivered to the rest of the program as it is.
  fn deliver(&mut self) {
    // The count ends here, at a delivered picture, and nowhere else.
    self.failures_in_a_row = 0;
    // self.frame.pixel_format(), .width(), .height(), .planes() — view
    // carriers: read them here and drop. A frame held is a pool slot
    // held. Use the `Owned*` family when a frame has to outlive this.
  }

  /// A decoder failure on any of the three roads. On hardware it counts,
  /// and at `FAILURES_BEFORE_SOFTWARE` the session is replaced by one on
  /// `DecodePath::Software`, opened from the same parameters. Answers
  /// whether it was, so the road can feed the new session.
  fn failed(&mut self, e: Error) -> Result<bool, BoxError> {
    eprintln!("decode failure: {e}");
    // Software is where this policy ends: its failures are reported,
    // and the stream goes on.
    if !self.decoder.is_hardware() {
      return Ok(false);
    }
    self.failures_in_a_row += 1;
    if self.failures_in_a_row < FAILURES_BEFORE_SOFTWARE {
      return Ok(false);
    }
    self.decoder = FfmpegVideoStreamDecoder::open_as(
      self.parameters.clone(),
      self.time_base,
      DecoderLimits::default(),
      DecodePath::Software,
    )?;
    Ok(true)
  }
}

Audio and subtitle decoding share the shape — see examples/decode_via_trait.rs and tests/audio_subtitle_via_trait.rs for end-to-end demuxer-driven runs that cover all three streams.

Public surface map

• Decoders: FfmpegVideoStreamDecoder, FfmpegAudioStreamDecoder, FfmpegSubtitleStreamDecoder. Plus their error types: VideoDecodeError, AudioDecodeError, SubtitleDecodeError.
• Decode path: DecodePath and FfmpegVideoStreamDecoder::open_as — Auto (what open does: probe hardware, fall back to software before the first picture), AnyHardware (the same probe, never software), Software, or the pin Hardware(Backend). After the first picture no path changes its decoder, and a decoder failure is the picture's own error; when to stop trusting a hardware session is the caller's policy. is_hardware() / is_software() stay the live reading of where a session is.
• Demuxer: FfmpegDemuxer — mediadecode's Demuxer over libavformat, opened from a path (open) or from any Read + Seek byte source through a custom AVIOContext (open_reader). Plus DemuxError, and format() → ContainerFormat — libavformat's own identification of the bytes ("matroska,webm", "mov,mp4,m4a,3gp,3g2,mj2"), read from the container rather than guessed from a path.
• Codec identity: CodecId — the number FFmpeg keys on, with name() / long_name() reading libavcodec's own descriptor table for the word beside it ("h264", "aac"), so a stored track row can cross into a typed codec vocabulary. The number stays the identity; an id libavcodec has no descriptor for has no name, rather than a "unknown_codec" sentinel.
• Resampler (resample feature, on by default): FfmpegResampler — mediadecode's AudioResampler over swresample, built from two explicit ResampleSpecs (the source read off a track or off the opened decoder, the target the caller's). Plus ResampleError, which carries faults and the SourceChanged mid-stream refusal — "needs more input" and "the tail is finished" are mediadecode::Received states out of receive_frame, not error variants. Disabling the feature drops the type and the libswresample link along with it.
• FfmpegImageDecoder: the one-shot ImageDecoder — cover art in, ImageFrame out. Opened from an attachment track's codec parameters, which the demuxer's cover-art reclassification retains in full. The picture's EXIF orientation comes back typed on ImageFrameExtra, read off the display matrix libavcodec emits for it.
• Type aliases: VideoPacket, AudioPacket, SubtitlePacket, DataPacket, AttachmentPacket, DemuxedPacket, VideoFrame, AudioFrame, SubtitleFrame, ImageFrame, TrackInfo, TrackParams — the mediadecode generic types pre-parameterized with this crate's adapter / carrier / extras, so you don't have to spell them out.
• Carrier: FfmpegBytes — owned, Send + Sync, AsRef<[u8]>, cloning by refcount. Opaque over an Arc<[u8]>, because the storage gains a pooled strategy later (#35) and a consumer must not have to recompile for it. Construct one with FfmpegBytes::copy_from_slice / ::empty when feeding a packet back into a decoder.
• Boundary helpers: video_packet_from_ffmpeg, audio_packet_from_ffmpeg, subtitle_packet_from_ffmpeg — convert a borrowed ffmpeg::Packet into the matching mediadecode packet, copying the compressed payload out. Their *_in siblings (video_packet_from_ffmpeg_in, audio_packet_from_ffmpeg_in, subtitle_packet_from_ffmpeg_in, data_packet_from_ffmpeg_in) take the stream's timebase, so the produced Timestamp says what its ticks mean instead of carrying the 1/1 placeholder an AVPacket alone leaves you with. attachment_packet_from_ffmpeg wraps a cover-art packet, which has no timestamps to carry. Every one is unsafe: a borrow of an ffmpeg_next packet cannot vouch that its payload was ever written (Packet::new leaves it unwritten), so the caller does — a packet av_read_frame filled always was. The demuxer, FfmpegDemuxer, converts its own packets and needs no unsafe.
• Empty-frame builders: empty_video_frame, empty_audio_frame, empty_subtitle_frame — well-formed destinations for receive_frame.

Running tests and benches

The fixture-gated tests and the benchmark expect real media files, named by five environment variables. The unit tests run unconditionally; the gated ones are #[ignore]d, so --ignored is what opts into them.

HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo test --test hw_smoke -- --ignored
HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo bench

# The whole fixture-gated set at once.
HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_VIDEO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_AUDIO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_SUBTITLE=/path/to/subtitled.mkv \
MEDIADECODE_FX3_SAMPLE=/path/to/12_sony_fx3_xavc.mp4 \
  cargo test --all-features -- --ignored

A variable left unset is not always a quiet skip: the FX3 case prints a notice and returns, but the trait cases panic on a missing path once --ignored has opted into them.

Build requirements

• A system FFmpeg ≥ 5.1 linkable via pkg-config (we reference AV_PIX_FMT_P212LE / AV_PIX_FMT_P412LE, which were added in 5.1). Tested against 9.0. Verify with ffmpeg -hwaccels that your build has the backends you expect compiled in (e.g. videotoolbox on macOS, vaapi / cuda on Linux, d3d11va / cuda on Windows).
• Rust ≥ 1.95, edition 2024.

License

mediadecode-ffmpeg is under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2026 FinDIT Studio authors.