mediadecode-ffmpeg 0.9.0

FFmpeg adapter for the mediadecode abstraction layer, built on top of ffmpeg-next.

Implements mediadecode's VideoAdapter / AudioAdapter / SubtitleAdapter / ImageAdapter traits, the matching push-style *StreamDecoder traits, the one-shot ImageDecoder, and Demuxer.

Every byte a frame or packet carries is copied once, at the FFmpeg boundary, into an FfmpegBytes — mediadecode 0.9's D-seat amputation contract. A delivered frame is owned, Send + Sync, and cheap to clone (a refcount bump); it holds nothing of libavcodec's open, so it can cross a channel, be read from several threads, and outlive the decoder that produced it. Through 0.8 the planes were refcounted views into AVBufferRef behind an FfmpegBuffer type, which meant every consumer inherited an FFmpeg lifetime it could not see. That type is gone.

FfmpegVideoStreamDecoder mirrors the send_packet / receive_frame shape of ffmpeg::decoder::Video, auto-probes the host's HW backends, and falls through to a software decoder when none open. Audio and subtitles use parallel FfmpegAudioStreamDecoder / FfmpegSubtitleStreamDecoder types.

Backends

FfmpegVideoStreamDecoder::open walks this probe order, opening the first backend that accepts the stream:

Output frames are CPU-side, downloaded with av_hwframe_transfer_data (NV12 for 8-bit, P010/P012/P016/P210/P212/P216/P410/P412/P416 for 10/12/16-bit). Pixel-format conversion is intentionally out of scope — downstream colconv handles it.

If every HW backend opens but later fails at decode time and the software backend is also unavailable, the error surfaces as VideoDecodeError::Decode(Error::AllBackendsFailed(p)) carrying any packets the decoder had already accepted from the demuxer (accessible via p.unconsumed_packets() / p.into_unconsumed_packets()) — so non-seekable callers (live streams, pipes, network sources) can replay them through their own software decoder without re-demuxing.

Usage

use ffmpeg_next as ffmpeg;
use ffmpeg::{format, media};
use mediadecode::{Timebase, decoder::VideoStreamDecoder};
use mediadecode_ffmpeg::{
  DecoderLimits, Error as FfmpegError, FfmpegVideoStreamDecoder, PacketLimits,
  VideoDecodeError, empty_video_frame, video_packet_from_ffmpeg_in,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
  ffmpeg::init()?;

  let path = std::env::args().nth(1).expect("usage: <input-file>");
  let mut input = format::input(&path)?;
  let stream = input.streams().best(media::Type::Video).unwrap();
  let stream_index = stream.index();
  let time_base = Timebase::new(
    stream.time_base().numerator(),
    std::num::NonZeroI32::new(stream.time_base().denominator()).unwrap(),
  );

  // Probes HW backends in order, falls back to software.
  let mut decoder =
    match FfmpegVideoStreamDecoder::open(stream.parameters(), time_base, DecoderLimits::default())
    {
    Ok(d) => d,
    Err(FfmpegError::AllBackendsFailed(p)) => {
      // No backend at all could open this stream — including software.
      // `unconsumed_packets` is empty at open-time. Caller decides.
      let _unconsumed_packets = p.into_unconsumed_packets();
      return Ok(());
    }
    Err(e) => return Err(e.into()),
  };

  let mut frame = empty_video_frame();
  for (s, av_packet) in input.packets() {
    if s.index() != stream_index { continue; }
    // `Ok(None)` is an empty packet; an `Err` is a payload that is
    // there and could not be referenced, which is never silently
    // skipped.
    // **By value.** The bare names are the view lane, where a packet's
    // payload is a window into libavformat's own buffer — so the source
    // is handed over rather than lent. (The borrowing doors,
    // `owned_*`, are the owned lane: they copy, so the packet stays
    // yours.)
    let Some(pkt) =
      video_packet_from_ffmpeg_in(av_packet, time_base, PacketLimits::default())?
    else { continue };

    match decoder.send_packet(&pkt) {
      Ok(()) => {}
      Err(VideoDecodeError::Decode(FfmpegError::AllBackendsFailed(p))) => {
        // Runtime exhaustion: rescued packets are the bytes the decoder
        // already consumed from `input`. Replay them through your own
        // software decoder before the current packet so non-seekable
        // sources recover cleanly.
        let _unconsumed_packets = p.into_unconsumed_packets();
        return Ok(());
      }
      Err(e) => return Err(e.into()),
    }
    while decoder.receive_frame(&mut frame).is_ok() {
      // frame.pixel_format(), frame.width(), frame.height(),
      // frame.planes() — view carriers: read them here and drop. A
      // frame held is a pool slot held. Use the `Owned*` family when a
      // frame has to outlive the loop.
    }
  }
  decoder.send_eof()?;
  while decoder.receive_frame(&mut frame).is_ok() { /* drain */ }
  Ok(())
}

Audio and subtitle decoding share the shape — see examples/decode_via_trait.rs and tests/audio_subtitle_via_trait.rs for end-to-end demuxer-driven runs that cover all three streams.

Public surface map

• Decoders: FfmpegVideoStreamDecoder, FfmpegAudioStreamDecoder, FfmpegSubtitleStreamDecoder. Plus their error types: VideoDecodeError, AudioDecodeError, SubtitleDecodeError.
• Demuxer: FfmpegDemuxer — mediadecode's Demuxer over libavformat, opened from a path (open) or from any Read + Seek byte source through a custom AVIOContext (open_reader). Plus DemuxError.
• Resampler (resample feature, on by default): FfmpegResampler — mediadecode's AudioResampler over swresample, built from two explicit ResampleSpecs (the source read off a track or off the opened decoder, the target the caller's). Plus ResampleError, whose Again variant is the "needs more input" signal and whose SourceChanged variant is the mid-stream refusal. Disabling the feature drops the type and the libswresample link along with it.
• FfmpegImageDecoder: the one-shot ImageDecoder — cover art in, ImageFrame out. Opened from an attachment track's codec parameters, which the demuxer's cover-art reclassification retains in full. The picture's EXIF orientation comes back typed on ImageFrameExtra, read off the display matrix libavcodec emits for it.
• Type aliases: VideoPacket, AudioPacket, SubtitlePacket, DataPacket, AttachmentPacket, DemuxedPacket, VideoFrame, AudioFrame, SubtitleFrame, ImageFrame, TrackInfo, TrackParams — the mediadecode generic types pre-parameterized with this crate's adapter / carrier / extras, so you don't have to spell them out.
• Carrier: FfmpegBytes — owned, Send + Sync, AsRef<[u8]>, cloning by refcount. Opaque over an Arc<[u8]>, because the storage gains a pooled strategy later (#35) and a consumer must not have to recompile for it. Construct one with FfmpegBytes::copy_from_slice / ::empty when feeding a packet back into a decoder.
• Boundary helpers: video_packet_from_ffmpeg, audio_packet_from_ffmpeg, subtitle_packet_from_ffmpeg — convert a borrowed ffmpeg::Packet into the matching mediadecode packet, copying the compressed payload out. Their *_in siblings (video_packet_from_ffmpeg_in, audio_packet_from_ffmpeg_in, subtitle_packet_from_ffmpeg_in, data_packet_from_ffmpeg_in) take the stream's timebase, so the produced Timestamp says what its ticks mean instead of carrying the 1/1 placeholder an AVPacket alone leaves you with. attachment_packet_from_ffmpeg wraps a cover-art packet, which has no timestamps to carry.
• Empty-frame builders: empty_video_frame, empty_audio_frame, empty_subtitle_frame — well-formed destinations for receive_frame.

Running tests and benches

The fixture-gated tests and the benchmark expect real media files, named by five environment variables. The unit tests run unconditionally; the gated ones are #[ignore]d, so --ignored is what opts into them.

HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo test --test hw_smoke -- --ignored
HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo bench

# The whole fixture-gated set at once.
HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_VIDEO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_AUDIO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_SUBTITLE=/path/to/subtitled.mkv \
MEDIADECODE_FX3_SAMPLE=/path/to/12_sony_fx3_xavc.mp4 \
  cargo test --all-features -- --ignored

A variable left unset is not always a quiet skip: the FX3 case prints a notice and returns, but the trait cases panic on a missing path once --ignored has opted into them.

Build requirements

• A system FFmpeg ≥ 5.1 linkable via pkg-config (we reference AV_PIX_FMT_P212LE / AV_PIX_FMT_P412LE, which were added in 5.1). Tested against 9.0. Verify with ffmpeg -hwaccels that your build has the backends you expect compiled in (e.g. videotoolbox on macOS, vaapi / cuda on Linux, d3d11va / cuda on Windows).
• Rust ≥ 1.95, edition 2024.

License

mediadecode-ffmpeg is under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2026 FinDIT Studio authors.