Crate videotoolbox

Expand description

§videotoolbox

Safe Rust bindings for Apple’s VideoToolbox framework — hardware-accelerated encode/decode, pixel transfer/rotation, multipass helpers, HDR metadata, motion estimation, RAW processing, and VTFrameProcessor pipelines on macOS.

Status: experimental, but the crate now covers the main public VideoToolbox surfaces used by the doom-fish stack. Objective-C-only APIs use a small Swift bridge behind the frame_processor feature, and executor-agnostic encode/decode/RAW-processing async helpers live in videotoolbox::async_api behind the async feature.

§Features

Hardware-accelerated encoding + decoding — H.264, HEVC, and ProRes 422/4444
Pixel transfer / rotation / utilities — VTPixelTransferSession, VTPixelRotationSession, VTCreateCGImageFromCVPixelBuffer
Multipass + HDR helpers — VTFrameSilo, VTMultiPassStorage, VTHDRPerFrameMetadataGenerationSession
Advanced processing — VTFrameProcessor, VTMotionEstimationSession, VTRAWProcessingSession
Direct IOSurface input/output — zero-copy composition with apple-cf::iosurface
Builder pattern — fluent encoder configuration for bitrate, frame rate, keyframe interval, real-time mode, profile level, the encoder specification (hardware preferred/required/disabled, encoder ID, low-latency rate control, GPU registry ID) and source pixel buffer attributes, plus per-frame properties such as forced keyframes
Executor-agnostic async module — videotoolbox::async_api::{AsyncCompressionSession, AsyncDecompressionSession, AsyncRawProcessingSession} bridges one-shot frame callbacks to Futures and wraps RAW-parameter change notifications as a bounded async stream via doom-fish-utils
Mostly pure C bindings — optional Swift bridge only for Objective-C-only APIs
Minimal dependencies — apple-cf and doom-fish-utils, plus optional apple-metal for VTFrameProcessor command-buffer integration

§Async notes

AsyncRawProcessingSession::parameter_changes(...) exposes VTRAWProcessingSessionSetParameterChangedHandler as a bounded async stream.
One-shot async decompression accepts exactly one sample per CMSampleBuffer; multi-sample buffers return VTError::UnexpectedSampleCount before native submission.
VTDecompressionSessionSetMultiImageCallback remains sync-only for now: the audited C API requires a non-null callback and exposes no clear / unsubscribe hook for an RAII async stream wrapper.
encode_frame_async submits the frame when it is called. If the encoder is still holding the frame back (frame reordering is on by default) when the future is first polled, that poll forces it out with VTCompressionSessionCompleteFrames and blocks until the encoder emits it, so a sequential .await loop never hangs. Submit several frames before awaiting the first to keep reordering effective.

§Why not bindgen?

The full VideoToolbox SDK surface is large, but the useful set for real macOS media pipelines is still small enough to hand-audit. Hand-writing those declarations gives us:

No build-time dependency on clang
Type-safe Rust enums for codec types (instead of raw u32 four-character codes)
Builder APIs that map ergonomically to VT’s CFDictionary property bag

§Installation

[dependencies]
videotoolbox = "0.21"

§Requirements

macOS 13.0+
Apple Silicon or Intel Mac with hardware video encoder

APIs that need a newer macOS are looked up at run time, so binaries still load on macOS 13; on an older system they return VTError::Unsupported { api, minimum } (or false from the is_*_supported queries):

Needs	APIs
macOS 14.0	`CompressionSession::encode_multi_image`, `DecompressionSession::set_multi_image_callback`, `TaggedBufferGroup`, the MV-HEVC support queries
macOS 15.0	`DecompressionSession::decode_with_options` with frame options, `HdrMetadataSession`, the decoder and RAW-processor extension-property queries, `RawProcessingSession`
macOS 15.4 / 26.0	the `VTFrameProcessor` pipelines (each capability query reports what the running system supports)
macOS 26.0	`MotionEstimationSession`, `RawProcessingSession::metadata_for_sidecar_file`

The raw declarations in videotoolbox::ffi are ordinary imports. If you call one that is newer than your deployment target directly, link with -weak_framework VideoToolbox (and CoreMedia) and check availability first, or use the safe wrappers.

Sessions are Send but not Sync: VideoToolbox marks its session types non-Sendable and documents no concurrent use, so move a session to the thread that uses it or put it behind a Mutex.

Encoded H.264/HEVC frames (EncodedFrame::data) are AVCC-style length-prefixed NAL units without start codes; the parameter sets are in the sample buffer’s format description (CMFormatDescription::video_parameter_sets in apple-cf).

§Quick start

use videotoolbox::prelude::*;
use apple_cf::cm::CMTime;
use apple_cf::iosurface::{IOSurface, IOSurfaceLockOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Allocate a 1920×1080 BGRA IOSurface.
    let surface = IOSurface::create(1920, 1080, u32::from_be_bytes(*b"BGRA"), 4)
        .ok_or("failed to allocate")?;

    // Build a real-time H.264 encoder.
    let encoder = CompressionSession::builder(1920, 1080, Codec::H264)
        .with_real_time(true)
        .with_average_bit_rate(8_000_000)
        .with_expected_frame_rate(60.0)
        .with_max_keyframe_interval(120)
        .build()?;

    // Encode one frame and inspect the resulting CMSampleBuffer.
    let encoded = encoder.encode(&surface, CMTime::new(0, 60))?;
    println!("Got {} bytes of H.264", encoded.data.len());

    if let Some(sb) = encoded.cm_sample_buffer() {
        // Hand `sb` straight to avassetwriter::Writer::append_sample for
        // zero-copy muxing — no raw pointer hand-off needed.
        let _ = sb.is_valid();
    }

    Ok(())
}

§Composes with the rest of the doom-fish stack

screencapturekit-rs ──► IOSurface ──► videotoolbox-rs ──► H.264 bytes
                                              ↓
                                        avassetwriter-rs
                                              ↓
                                          .mp4 file

§Roadmap

VTCompressionSession (encoder)
VTDecompressionSession (decoder)
VTPixelTransferSession (pixel format / colour space conversion)
VTPixelRotationSession
VTMultiPassStorage + VTFrameSilo (two-pass encoding)
VTHDRPerFrameMetadataGenerationSession (Dolby Vision metadata)
VTFrameProcessor capability queries (super-resolution / optical flow detection)
VTFrameProcessor pipeline (super-resolution + motion blur + temporal noise + frame-rate conversion + optical flow + 2 low-latency variants)
VTMotionEstimationSession
VTRAWProcessingSession (with parameter introspection)
VTProfessionalVideoWorkflow decoder/encoder registration
VTCreateCGImageFromCVPixelBuffer
HEVC profile-level helpers
Executor-agnostic async encode/decode/RAW-processing module behind the async feature

§License

Licensed under either of Apache-2.0 or MIT at your option.

§API Documentation

Safe Rust bindings for Apple’s VideoToolbox framework — hardware-accelerated H.264, HEVC, and ProRes codecs on macOS.

Most of the crate uses direct extern "C" bindings against the system framework. Objective-C-only APIs (notably VTFrameProcessor, VTMotionEstimationSession, and VTRAWProcessingSession) use a small Swift bridge behind the frame_processor feature, while executor-agnostic encode/decode/RAW-processing async helpers live in async_api behind the async feature.

§Quick start

use videotoolbox::prelude::*;
use apple_cf::cm::CMTime;
use apple_cf::iosurface::IOSurface;

let surface = IOSurface::create(1920, 1080, u32::from_be_bytes(*b"BGRA"), 4)
    .ok_or("failed to allocate IOSurface")?;

let encoder = CompressionSession::builder(1920, 1080, Codec::H264)
    .with_real_time(true)
    .with_average_bit_rate(8_000_000)
    .with_expected_frame_rate(60.0)
    .build()?;

let encoded = encoder.encode(&surface, CMTime::new(0, 60))?;
println!("Got {} bytes of H.264", encoded.data.len());

Re-exports§

pub use decompression::DecodedFrame;
pub use decompression::DecodedMultiImageFrame;
pub use decompression::DecompressionSession;
pub use encoder_list::available_video_encoder_details;
pub use encoder_list::available_video_encoder_details_with_options;
pub use encoder_list::available_video_encoders;
pub use encoder_list::supported_property_dictionary_for_encoder;
pub use encoder_list::EncoderSupportedProperties;
pub use encoder_list::VideoEncoder;
pub use encoder_list::VideoEncoderDetails;
pub use encoder_list::VideoEncoderListOptions;
pub use error::VTError;
pub use hdr_metadata::HdrMetadataFormat;
pub use hdr_metadata::HdrMetadataSession;
pub use multipass::FrameSilo;
pub use multipass::MultiPassStorage;
pub use session::Codec;
pub use tagged_buffer_group::TaggedBufferGroup;
pub use transfer::DownsamplingMode;
pub use transfer::PixelRotationSession;
pub use transfer::PixelTransferSession;
pub use transfer::Rotation;
pub use transfer::ScalingMode;
pub use utilities::copy_raw_processor_extension_properties;
pub use utilities::copy_video_decoder_extension_properties;
pub use utilities::create_cg_image_from_pixel_buffer;
pub use utilities::is_hardware_decode_supported;
pub use utilities::register_professional_workflow_decoders;
pub use utilities::register_professional_workflow_encoders;
pub use utilities::register_supplemental_video_decoder_if_available;
pub use frame_processor::download_super_resolution_model;frame_processor
pub use frame_processor::download_super_resolution_model_for_configuration;frame_processor
pub use frame_processor::frame_processor_capabilities;frame_processor
pub use frame_processor::low_latency_super_resolution_supported_scale_factors;frame_processor
pub use frame_processor::super_resolution_model_percentage_available;frame_processor
pub use frame_processor::super_resolution_model_percentage_available_for_configuration;frame_processor
pub use frame_processor::super_resolution_model_status;frame_processor
pub use frame_processor::super_resolution_model_status_for_configuration;frame_processor
pub use frame_processor::super_resolution_supported_scale_factors;frame_processor
pub use frame_processor::FrameProcessor;frame_processor
pub use frame_processor::FrameProcessorCapabilities;frame_processor
pub use frame_processor::FrameProcessorFrame;frame_processor
pub use frame_processor::FrameProcessorOpticalFlow;frame_processor
pub use frame_processor::FrameProcessorSubmissionMode;frame_processor
pub use frame_processor::FrameRateConversionConfiguration;frame_processor
pub use frame_processor::FrameRateConversionSubmissionMode;frame_processor
pub use frame_processor::MotionBlurConfiguration;frame_processor
pub use frame_processor::OpticalFlowConfiguration;frame_processor
pub use frame_processor::SuperResolutionConfiguration;frame_processor
pub use frame_processor::SuperResolutionModelStatus;frame_processor
pub use frame_processor::VTFrameRateConversionConfigurationQualityPrioritization;frame_processor
pub use frame_processor::VTFrameRateConversionConfigurationRevision;frame_processor
pub use frame_processor::VTMotionBlurConfigurationQualityPrioritization;frame_processor
pub use frame_processor::VTMotionBlurConfigurationRevision;frame_processor
pub use frame_processor::VTOpticalFlowConfigurationQualityPrioritization;frame_processor
pub use frame_processor::VTOpticalFlowConfigurationRevision;frame_processor
pub use frame_processor::VTSuperResolutionScalerConfigurationInputType;frame_processor
pub use frame_processor::VTSuperResolutionScalerConfigurationQualityPrioritization;frame_processor
pub use frame_processor::VTSuperResolutionScalerConfigurationRevision;frame_processor
pub use motion_estimation::MotionEstimationResult;frame_processor
pub use motion_estimation::MotionEstimationSession;frame_processor
pub use motion_estimation::MotionEstimationSessionOptions;frame_processor
pub use raw_processing::RawProcessingParameter;frame_processor
pub use raw_processing::RawProcessingSession;frame_processor
pub use compression::CompressionSession;compression
pub use compression::CompressionSessionBuilder;compression
pub use compression::EncodedFrame;compression
pub use compression::EncoderGpu;compression
pub use compression::FrameProperties;compression
pub use compression::HardwareAcceleration;compression
pub use compression::ProfileLevel;compression

Modules§

async_apiasync: Async API for videotoolbox.
compressioncompression: CompressionSession — hardware H.264/HEVC/ProRes encoder.
decompression: DecompressionSession — hardware H.264/HEVC/ProRes decoder.
encoder_list: available_video_encoders / available_video_encoder_details — enumerate Apple’s installed video encoders and their selection metadata.
error: Errors produced by VideoToolbox APIs.
ffi: Raw extern "C" declarations against <VideoToolbox/VideoToolbox.h>, <CoreMedia/CoreMedia.h>, <CoreVideo/CVPixelBuffer.h>, and <CoreFoundation/CoreFoundation.h>.
frame_processorframe_processor: VTFrameProcessor capability queries plus retained wrappers for VTFrameProcessorFrame / VTFrameProcessorOpticalFlow and synchronous processing helpers for all public pipelines this crate exposes.
hdr_metadata: VTHDRPerFrameMetadataGenerationSession — generate Dolby Vision per-frame HDR metadata (macOS 15+).
motion_estimationframe_processor: VTMotionEstimationSession — between-frame motion-vector estimation backed by the Apple Neural Engine (macOS 26+).
multipass: VTFrameSilo + VTMultiPassStorage — multi-pass video encoding storage.
prelude: Common imports for users of this crate.
raw_processingframe_processor: VTRAWProcessingSession — ProRes RAW / CinemaDNG decoder with per-frame parameter controls (macOS 15+).
session: Video codecs supported by VideoToolbox.
tagged_buffer_group: Minimal wrapper around CoreMedia CMTaggedBufferGroupRef.
transfer: PixelTransferSession and PixelRotationSession — Apple’s zero-copy pixel-format conversion / scaling / rotation engines.
utilities: VTUtilities + VTProfessionalVideoWorkflow helpers.