Crate mediadecode_ffmpeg 

mediadecode-ffmpeg

FFmpeg adapter for the mediadecode abstraction layer, built on top of ffmpeg-next.

Implements mediadecode’s VideoAdapter / AudioAdapter / SubtitleAdapter / ImageAdapter traits, the matching push-style *StreamDecoder traits, the one-shot ImageDecoder, and Demuxer.

Every byte a frame or packet carries is copied once, at the FFmpeg boundary, into an FfmpegBytes — mediadecode 0.9’s D-seat amputation contract. A delivered frame is owned, Send + Sync, and cheap to clone (a refcount bump); it holds nothing of libavcodec’s open, so it can cross a channel, be read from several threads, and outlive the decoder that produced it. Through 0.8 the planes were refcounted views into AVBufferRef behind an FfmpegBuffer type, which meant every consumer inherited an FFmpeg lifetime it could not see. That type is gone.

FfmpegVideoStreamDecoder mirrors the send_packet / receive_frame shape of ffmpeg::decoder::Video, auto-probes the host’s HW backends, and falls through to a software decoder when none open. Audio and subtitles use parallel FfmpegAudioStreamDecoder / FfmpegSubtitleStreamDecoder types.

Backends

FfmpegVideoStreamDecoder::open walks this probe order, opening the first backend that accepts the stream:

Target	Probe order
macOS / iOS / tvOS	VideoToolbox → software
Linux	VAAPI → CUDA → software
Windows	D3D11VA → CUDA → software
other	software

Output frames are CPU-side, downloaded with av_hwframe_transfer_data (NV12 for 8-bit, P010/P012/P016/P210/P212/P216/P410/P412/P416 for 10/12/16-bit). Pixel-format conversion is intentionally out of scope — downstream colconv handles it.

If every HW backend opens but later fails at decode time and the software backend is also unavailable, the error surfaces as VideoDecodeError::Decode(Error::AllBackendsFailed(p)) carrying any packets the decoder had already accepted from the demuxer (accessible via p.unconsumed_packets() / p.into_unconsumed_packets()) — so non-seekable callers (live streams, pipes, network sources) can replay them through their own software decoder without re-demuxing.

Usage

use ffmpeg_next as ffmpeg;
use ffmpeg::{format, media};
use mediadecode::{Timebase, decoder::VideoStreamDecoder};
use mediadecode_ffmpeg::{
  DecoderLimits, Error as FfmpegError, FfmpegVideoStreamDecoder, PacketLimits,
  VideoDecodeError, empty_video_frame, video_packet_from_ffmpeg_in,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
  ffmpeg::init()?;

  let path = std::env::args().nth(1).expect("usage: <input-file>");
  let mut input = format::input(&path)?;
  let stream = input.streams().best(media::Type::Video).unwrap();
  let stream_index = stream.index();
  let time_base = Timebase::new(
    stream.time_base().numerator(),
    std::num::NonZeroI32::new(stream.time_base().denominator()).unwrap(),
  );

  // Probes HW backends in order, falls back to software.
  let mut decoder =
    match FfmpegVideoStreamDecoder::open(stream.parameters(), time_base, DecoderLimits::default())
    {
    Ok(d) => d,
    Err(FfmpegError::AllBackendsFailed(p)) => {
      // No backend at all could open this stream — including software.
      // `unconsumed_packets` is empty at open-time. Caller decides.
      let _unconsumed_packets = p.into_unconsumed_packets();
      return Ok(());
    }
    Err(e) => return Err(e.into()),
  };

  let mut frame = empty_video_frame();
  for (s, av_packet) in input.packets() {
    if s.index() != stream_index { continue; }
    // `Ok(None)` is an empty packet; an `Err` is a payload that is
    // there and could not be referenced, which is never silently
    // skipped.
    // **By value.** The bare names are the view lane, where a packet's
    // payload is a window into libavformat's own buffer — so the source
    // is handed over rather than lent. (The borrowing doors,
    // `owned_*`, are the owned lane: they copy, so the packet stays
    // yours.)
    let Some(pkt) =
      video_packet_from_ffmpeg_in(av_packet, time_base, PacketLimits::default())?
    else { continue };

    match decoder.send_packet(&pkt) {
      Ok(()) => {}
      Err(VideoDecodeError::Decode(FfmpegError::AllBackendsFailed(p))) => {
        // Runtime exhaustion: rescued packets are the bytes the decoder
        // already consumed from `input`. Replay them through your own
        // software decoder before the current packet so non-seekable
        // sources recover cleanly.
        let _unconsumed_packets = p.into_unconsumed_packets();
        return Ok(());
      }
      Err(e) => return Err(e.into()),
    }
    while decoder.receive_frame(&mut frame).is_ok() {
      // frame.pixel_format(), frame.width(), frame.height(),
      // frame.planes() — view carriers: read them here and drop. A
      // frame held is a pool slot held. Use the `Owned*` family when a
      // frame has to outlive the loop.
    }
  }
  decoder.send_eof()?;
  while decoder.receive_frame(&mut frame).is_ok() { /* drain */ }
  Ok(())
}

Audio and subtitle decoding share the shape — see examples/decode_via_trait.rs and tests/audio_subtitle_via_trait.rs for end-to-end demuxer-driven runs that cover all three streams.

Public surface map

• Decoders: FfmpegVideoStreamDecoder, FfmpegAudioStreamDecoder, FfmpegSubtitleStreamDecoder. Plus their error types: VideoDecodeError, AudioDecodeError, SubtitleDecodeError.
• Demuxer: FfmpegDemuxer — mediadecode’s Demuxer over libavformat, opened from a path (open) or from any Read + Seek byte source through a custom AVIOContext (open_reader). Plus DemuxError.
• Resampler (resample feature, on by default): FfmpegResampler — mediadecode’s AudioResampler over swresample, built from two explicit ResampleSpecs (the source read off a track or off the opened decoder, the target the caller’s). Plus ResampleError, whose Again variant is the “needs more input” signal and whose SourceChanged variant is the mid-stream refusal. Disabling the feature drops the type and the libswresample link along with it.
• FfmpegImageDecoder: the one-shot ImageDecoder — cover art in, ImageFrame out. Opened from an attachment track’s codec parameters, which the demuxer’s cover-art reclassification retains in full. The picture’s EXIF orientation comes back typed on ImageFrameExtra, read off the display matrix libavcodec emits for it.
• Type aliases: VideoPacket, AudioPacket, SubtitlePacket, DataPacket, AttachmentPacket, DemuxedPacket, VideoFrame, AudioFrame, SubtitleFrame, ImageFrame, TrackInfo, TrackParams — the mediadecode generic types pre-parameterized with this crate’s adapter / carrier / extras, so you don’t have to spell them out.
• Carrier: FfmpegBytes — owned, Send + Sync, AsRef<[u8]>, cloning by refcount. Opaque over an Arc<[u8]>, because the storage gains a pooled strategy later (#35) and a consumer must not have to recompile for it. Construct one with FfmpegBytes::copy_from_slice / ::empty when feeding a packet back into a decoder.
• Boundary helpers: video_packet_from_ffmpeg, audio_packet_from_ffmpeg, subtitle_packet_from_ffmpeg — convert a borrowed ffmpeg::Packet into the matching mediadecode packet, copying the compressed payload out. Their *_in siblings (video_packet_from_ffmpeg_in, audio_packet_from_ffmpeg_in, subtitle_packet_from_ffmpeg_in, data_packet_from_ffmpeg_in) take the stream’s timebase, so the produced Timestamp says what its ticks mean instead of carrying the 1/1 placeholder an AVPacket alone leaves you with. attachment_packet_from_ffmpeg wraps a cover-art packet, which has no timestamps to carry.
• Empty-frame builders: empty_video_frame, empty_audio_frame, empty_subtitle_frame — well-formed destinations for receive_frame.

Running tests and benches

The fixture-gated tests and the benchmark expect real media files, named by five environment variables. The unit tests run unconditionally; the gated ones are #[ignore]d, so --ignored is what opts into them.

Variable	Feeds
HWDECODE_SAMPLE_VIDEO	tests/decode.rs, tests/hw_smoke.rs, benches/decode.rs, and the three decoder::tests backend cases
MEDIADECODE_SAMPLE_VIDEO	tests/decode_via_trait.rs
MEDIADECODE_SAMPLE_AUDIO	the audio-through-trait case (any container with an audio track)
MEDIADECODE_SAMPLE_SUBTITLE	the subtitle-through-trait case (needs a container that really carries a subtitle track)
MEDIADECODE_FX3_SAMPLE	the Sony FX3 H.264 High 4:2:2 10-bit mid-stream HW→SW fallback case

HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo test --test hw_smoke -- --ignored
HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo bench

# The whole fixture-gated set at once.
HWDECODE_SAMPLE_VIDEO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_VIDEO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_AUDIO=/path/to/clip.mp4 \
MEDIADECODE_SAMPLE_SUBTITLE=/path/to/subtitled.mkv \
MEDIADECODE_FX3_SAMPLE=/path/to/12_sony_fx3_xavc.mp4 \
  cargo test --all-features -- --ignored

A variable left unset is not always a quiet skip: the FX3 case prints a notice and returns, but the trait cases panic on a missing path once --ignored has opted into them.

Build requirements

• A system FFmpeg ≥ 5.1 linkable via pkg-config (we reference AV_PIX_FMT_P212LE / AV_PIX_FMT_P412LE, which were added in 5.1). Tested against 9.0. Verify with ffmpeg -hwaccels that your build has the backends you expect compiled in (e.g. videotoolbox on macOS, vaapi / cuda on Linux, d3d11va / cuda on Windows).
• Rust ≥ 1.95, edition 2024.

License

mediadecode-ffmpeg is under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2026 FinDIT Studio authors.

Re-exports

pub use boundary::MediaKind;

pub use boundary::PacketBuildError;

pub use boundary::SendPayloadTooLarge;

pub use boundary::SendSideDataTooLarge;

pub use boundary::attachment_packet_from_ffmpeg;

pub use boundary::audio_packet_from_ffmpeg;

pub use boundary::audio_packet_from_ffmpeg_in;

pub use boundary::data_packet_from_ffmpeg_in;

pub use boundary::empty_audio_frame;

pub use boundary::empty_owned_audio_frame;

pub use boundary::empty_owned_subtitle_frame;

pub use boundary::empty_owned_video_frame;

pub use boundary::empty_subtitle_frame;

pub use boundary::empty_video_frame;

pub use boundary::ffmpeg_packet_from_audio_packet;

pub use boundary::ffmpeg_packet_from_owned_audio_packet;

pub use boundary::ffmpeg_packet_from_owned_subtitle_packet;

pub use boundary::ffmpeg_packet_from_owned_video_packet;

pub use boundary::ffmpeg_packet_from_subtitle_packet;

pub use boundary::ffmpeg_packet_from_video_packet;

pub use boundary::from_av_pixel_format;

pub use boundary::is_hardware_pix_fmt;

pub use boundary::owned_attachment_packet_from_ffmpeg;

pub use boundary::owned_audio_packet_from_ffmpeg_in;

pub use boundary::owned_data_packet_from_ffmpeg_in;

pub use boundary::owned_subtitle_packet_from_ffmpeg_in;

pub use boundary::owned_video_packet_from_ffmpeg_in;

pub use boundary::subtitle_packet_from_ffmpeg;

pub use boundary::subtitle_packet_from_ffmpeg_in;

pub use boundary::video_packet_from_ffmpeg;

pub use boundary::video_packet_from_ffmpeg_in;

pub use channel_layout::channel_layout_description_from_ffmpeg;

pub use channel_layout::channel_layout_from_ffmpeg;

pub use channel_layout::channel_order_from_ffmpeg;

pub use limits::DEFAULT_MAX_ATTACHMENT_BYTES;

pub use limits::DEFAULT_MAX_CODEC_PARAMETER_BYTES;

pub use limits::DEFAULT_MAX_FRAME_BYTES;

pub use limits::DEFAULT_MAX_IMAGE_INPUT_BYTES;

pub use limits::DEFAULT_MAX_IMAGE_SIDE_DATA_BYTES;

pub use limits::DEFAULT_MAX_PACKET_BYTES;

pub use limits::DEFAULT_MAX_PIXELS;

pub use limits::DEFAULT_MAX_PROBE_BYTES;

pub use limits::DEFAULT_MAX_STREAMS;

pub use limits::DEFAULT_MAX_TOTAL_ATTACHMENT_BYTES;

pub use limits::DEFAULT_MAX_TOTAL_CODEC_PARAMETER_BYTES;

pub use limits::DecoderLimits;

pub use limits::DemuxLimits;

pub use limits::FrameLimits;

pub use limits::PacketLimits;

Modules

boundary

Boundary conversions between FFmpeg’s bindgen integers and the unified mediadecode vocabulary.

channel_layout

Conversions from FFmpeg’s ffmpeg_next::ChannelLayout / ffmpeg_next::ffi::AVChannelOrder to the channel-layout vocabulary mediaframe owns (ChannelLayout, ChannelOrder, ChannelSpec, ChannelLayoutDescription).

convert

Conversion helpers from FFmpeg AVFrame / AVPacket to the mediadecode types parameterized by crate::Ffmpeg and FfmpegBytes.

extras

Backend-specific *Extra carriers used as the mediadecode::*Adapter::*Extra associated types.

limits

Resource ceilings — the finite budgets every copy across the FFmpeg boundary is checked against before it allocates.

Structs

CarrierAudioStreamDecoder

mediadecode::AudioStreamDecoder impl wrapping ffmpeg::decoder::Audio.

CarrierDemuxer

mediadecode::demuxer::Demuxer impl wrapping ffmpeg::format::context::Input.

CarrierImageDecoder

mediadecode::ImageDecoder impl wrapping ffmpeg::decoder::Video.

CarrierResamplerresample

mediadecode::resampler::AudioResampler impl wrapping swresample.

CarrierSubtitleStreamDecoder

mediadecode::SubtitleDecoder impl wrapping ffmpeg::decoder::Subtitle.

CarrierVideoStreamDecoder

mediadecode::VideoStreamDecoder impl with transparent HW → SW fallback.

CodecId

Codec identifier. Wraps the integer value of an AVCodecID enum variant; comparisons and storage work without ever transmuting back into the bindgen enum.

Corrupt

Payload for ImageDecodeError::Corrupt.

Ffmpeg

Zero-sized type carrying the FFmpeg adapter’s vocabulary.

FfmpegBuffer

Refcounted view onto a contiguous byte range inside an AVBufferRef.

FfmpegBytes

The bytes every packet and frame this crate produces are carried in.

Frame

CPU-side decoded video frame produced by crate::VideoDecoder.

FrameBudgetExceeded

Payload for Error::FrameBudgetExceeded.

HwSurfaceTooLarge

Payload for Error::HwSurfaceTooLarge.

HwTransferTooLarge

Payload for Error::HwTransferTooLarge.

InputTooLarge

Payload for ImageDecodeError::InputTooLarge.

OutputTooLargeresample

Payload for ResampleError::OutputTooLarge.

Owned

The owned lane: every byte copied once at the boundary.

ProbeBudgetExhausted

Payload for DemuxError::ProbeBudgetExhausted.

ResampleSpecresample

One end of a conversion: sample rate, sample format, channel layout.

SampleFormat

Audio sample format identifier.

TrustedPayload

Payload for PacketBufferError::TrustedPayload and crate::boundary::PacketBuildError::TrustedPayload.

UnsupportedSpecChannelCountresample

Payload for ResampleError::UnsupportedChannelCount.

VideoDecoder

Hardware-accelerated video decoder.

View

The view lane: refcounted zero-copy handles onto FFmpeg’s own allocations.

Enums

AudioDecodeError

Errors from [FfmpegAudioStreamDecoder].

Backend

Hardware decoding backend.

CorruptSource

Which side of the decode declared the corruption.

DemuxError

Errors from [FfmpegDemuxer].

Error

Errors returned from crate::VideoDecoder.

FrameMedium

Which kind of frame a FrameBudgetExceeded refers to.

ImageDecodeError

Errors from [FfmpegImageDecoder].

PacketBufferError

Why a packet could not be carried across the boundary — its payload, or the side data that comes with it.

ResampleErrorresample

Errors from [FfmpegResampler].

SpecEndresample

Which end of a conversion a refusal is about.

SubtitleDecodeError

Errors from [FfmpegSubtitleStreamDecoder].

VideoDecodeError

Error type for [FfmpegVideoStreamDecoder].

Traits

FfmpegCarrier

How a lane turns FFmpeg’s bytes into a carrier.

Type Aliases

AttachmentPacket

Attachment payload pre-parameterized with this crate’s extras and view carrier — a font, or the cover art FfmpegImageDecoder decodes.

AudioFrame

Decoded audio frame pre-parameterized with this crate’s sample format / channel layout / extras / view carrier.

AudioPacket

Compressed audio packet pre-parameterized with this crate’s extras and view carrier.

DataPacket

Timed opaque-data packet pre-parameterized with this crate’s extras and view carrier.

DemuxedPacket

The five-arm demux envelope FfmpegDemuxer delivers.

FfmpegAudioStreamDecoder

The audio decoder on the view lane.

FfmpegDemuxer

The demuxer, on the view lane — the ordinary road.

FfmpegImageDecoder

The still-image decoder on the view lane.

FfmpegOwnedAudioStreamDecoder

The audio decoder on the owned lane.

FfmpegOwnedDemuxer

The demuxer on the owned lane: every byte copied once at the boundary into memory Rust owns.

FfmpegOwnedImageDecoder

The still-image decoder on the owned lane.

FfmpegOwnedResamplerresample

The resampler, on the owned lane — every produced plane copied out of the output frame.

FfmpegOwnedSubtitleStreamDecoder

The subtitle decoder on the owned lane.

FfmpegOwnedVideoStreamDecoder

The video stream decoder on the owned lane.

FfmpegResamplerresample

The resampler, on the view lane — planes shared out of its own output frame.

FfmpegSubtitleStreamDecoder

The subtitle decoder on the view lane.

FfmpegVideoStreamDecoder

The video stream decoder on the view lane.

ImageFrame

Decoded still image pre-parameterized with this crate’s pixel format / extras / view carrier — what FfmpegImageDecoder produces.

OwnedAttachmentPacket

AttachmentPacket on the owned lane.

OwnedAudioFrame

AudioFrame on the owned lane.

OwnedAudioPacket

AudioPacket on the owned lane.

OwnedDataPacket

DataPacket on the owned lane.

OwnedDemuxedPacket

DemuxedPacket on the owned lane.

OwnedImageFrame

ImageFrame on the owned lane — what FfmpegOwnedImageDecoder produces.

OwnedSubtitleFrame

SubtitleFrame on the owned lane.

OwnedSubtitlePacket

SubtitlePacket on the owned lane.

OwnedVideoFrame

VideoFrame on the owned lane — planes copied out of the decoder’s buffer, so the frame outlives it and the pool slot goes straight back.

OwnedVideoPacket

VideoPacket on the owned lane.

Result

Crate result alias.

SubtitleFrame

Decoded subtitle frame pre-parameterized with this crate’s extras / view carrier.

SubtitlePacket

Compressed subtitle packet pre-parameterized with this crate’s extras and view carrier.

TrackInfo

One row of the track table FfmpegDemuxer::tracks returns.

TrackParams

A track’s per-kind codec parameters, as TrackInfo carries them.

VideoFrame

Decoded video frame pre-parameterized with this crate’s pixel format / extras / view carrier.

VideoPacket

Compressed video packet pre-parameterized with this crate’s extras and view carrier — the type FfmpegVideoStreamDecoder consumes via mediadecode::decoder::VideoStreamDecoder::send_packet.