video-interop 0.1.3

Video frame, DMA-BUF, synchronization, and ownership types for Rust and Elixir
Documentation

video-interop

The problem

A Rust producer and consumer in a zero-copy Linux GPU pipeline exchange DMA-BUF file descriptors instead of copying frame pixels through CPU memory. Each handoff also requires image layout, format, geometry, colorimetry, synchronization, file-descriptor ownership, and producer-buffer lifetime.

An integer file descriptor does not encode those properties and does not own the underlying allocation. Rust code must duplicate borrowed descriptors before retaining them beyond the current native call, then release each producer lease exactly once after GPU use finishes.

The solution

The video-interop crate provides the Rust types and lifecycle primitives for the VideoInterop contract. It describes owned binary frames, borrowed Linux DMA-BUF frames, sync-file acquire fences, frame geometry, colorimetry, and stream formats. Prepared and claimed frame types establish the ownership boundary between Elixir and native queues.

The default Rustler integration maps these types to the video_interop Hex package. This shared contract connects producer and consumer elements written in Rust to pipeline control written in Elixir. Optional EGL and Vulkan modules implement synchronization and import operations without taking ownership of the caller's graphics runtime.

The crate does not allocate producer buffers, create a renderer, or select a streaming framework.

Status

The core, Rustler, EGL, and Vulkan APIs are part of the supported 0.1 contract.

The crate supports Rust 1.91 and newer.

Installation

Rustler support is enabled by default:

[dependencies]
video-interop = "0.1"

Disable the default feature for a native program that only uses the frame types:

[dependencies]
video-interop = { version = "0.1", default-features = false }

Optional graphics helpers are separate features:

video-interop = { version = "0.1", features = ["egl"] }
video-interop = { version = "0.1", features = ["vulkan"] }

Binary storage

BinaryStorage owns its bytes and plane metadata. Cloning or decoding it copies the bytes, so it needs neither file-descriptor duplication nor a release lease. Binary formats include RGBA8888, RGB888, Gray8, Gray2, and BW1.

DMA-BUF ownership

Descriptor contains borrowed integer file descriptors. Validate it before use. Call Descriptor::duplicate_cloexec before keeping it after the current call. The returned OwnedDescriptor closes those duplicates when dropped.

dmabuf_allocation_size reads the complete allocation size exposed by a DMA-BUF fd. The reported size includes any exporter alignment and driver padding outside the bytes addressed by the image planes. Producers must store this size in Object::size. Vulkan imports compare the published size with the fd before using it.

A sync-file descriptor follows the same rule: it is borrowed until duplicated into an owned frame.

Rustler frames and leases

The default feature maps both storage kinds to the structs in the video_interop Hex package. It also provides prepared and claimed frames for moving either storage kind into native queues.

Frame::prepare_cloexec validates a frame. It copies owned binary bytes without acquiring a lease client and duplicates borrowed descriptors with CLOEXEC. If a prepared borrowed frame is dropped, the descriptors close but lease ownership stays with the Elixir caller.

PreparedVideoFrame::claim moves any lease release responsibility to native code. Owned binary claims carry no lease. Dropping a borrowed ClaimedVideoFrame or ClaimedLease sends the release through a ReleaseDispatcher.

The NIF lifecycle owner must call ReleaseDispatcher::close_and_join from a dirty I/O NIF after all claims have drained. Dispatcher and guard destructors do not join worker threads or send through OwnedEnv.

EGL

The egl feature loads native-fence function pointers supplied by the caller. It supports:

  • native sync-file fence creation and import;
  • fence duplication with FD_CLOEXEC;
  • server waits and bounded client waits;
  • a bounded poll fallback when EGL import is unavailable.

The caller must keep the EGL display, context, function pointers, and calling thread valid. The module has no link-time EGL or GL dependency.

Vulkan

The vulkan feature accepts a caller-owned ash device and provides DMA-BUF capability checks, memory import, sync-file waits, queue-family ownership transfer, and release-fence handling.

Imported images stay in a bounded source cache. With planar preference, non-linear NV12 is imported as a transfer-source image and copied plane-for-plane into ordinary renderer-owned Y and UV images; linear NV12 uses an imported transfer-source buffer. Packed RGBA and BGRA use a compute copy when the device does not support sampling the producer-linear image. Copy regions cover image bytes only; allocation padding remains mapped but is not copied.

The caller remains responsible for:

  • physical-device and queue selection;
  • serializing host access to the Vulkan queue;
  • choosing a supported import strategy;
  • renderer waits and release submission;
  • presentation and device-loss recovery.

Platform-specific qualification is tracked separately from the Vulkan API's supported status.

Features

Feature Default Provides
rustler yes Elixir schema codecs, prepared/claimed frames, release dispatcher
egl no Dynamically loaded EGL native-fence helpers
vulkan no DMA-BUF import and synchronization helpers
test-support no Rustler helpers used by schema tests

License

Apache-2.0. See the license.

Optional Vulkan GPU timing

Consumers can override VulkanDeviceContext::video_gpu_timing_enabled to omit query-pool allocation and sample_video_gpu_timing to select an entire staged acquire-to-release bracket. Defaults retain existing instrumentation. Keep the enable policy fixed for sync-owner lifetimes and make sampling independent of cached slot indices. report_video_gpu_timing receives resource status; timing is not a capability or completion authority. ImportedImageSync::has_gpu_timing_resources supports diagnostics.

Off/unsampled paths preserve copies, ownership transfers, waits, release submissions and fences, including release command buffers with no timestamp commands. Results remain optional and are collected only after release completion. Unsupported timestamps and query-pool OOM omit instrumentation; device loss is never optional success.

Existing consumers, including Emerge's Ganesh/headless renderer, do not need to implement the timing hooks: defaults enable timing for every supported staged frame and ignore status notifications. Direct imports remain untimed. This API has no Graphite dependency. Driver-free legacy-consumer tests cover these defaults, release-fence gating, and repeated sync-owner reuse.