video-interop
The problem
A Rust producer and consumer in a zero-copy Linux GPU pipeline exchange DMA-BUF file descriptors instead of copying frame pixels through CPU memory. Each handoff also requires image layout, format, geometry, colorimetry, synchronization, file-descriptor ownership, and producer-buffer lifetime.
An integer file descriptor does not encode those properties and does not own the underlying allocation. Rust code must duplicate borrowed descriptors before retaining them beyond the current native call, then release each producer lease exactly once after GPU use finishes.
The solution
The video-interop crate provides the Rust types and lifecycle primitives for
the VideoInterop contract. It describes owned binary frames, borrowed Linux
DMA-BUF frames, sync-file acquire fences, frame geometry, colorimetry, and stream
formats. Prepared and claimed frame types establish the ownership boundary
between Elixir and native queues.
The default Rustler integration maps these types to the video_interop Hex
package. This shared contract connects producer and consumer elements written in
Rust to pipeline control written in Elixir. Optional EGL and Vulkan modules
implement synchronization and import operations without taking ownership of the
caller's graphics runtime.
The crate does not allocate producer buffers, create a renderer, or select a streaming framework.
Status
The core, Rustler, EGL, and Vulkan APIs are part of the supported 0.1 contract.
The crate supports Rust 1.91 and newer.
Installation
Rustler support is enabled by default:
[]
= "0.1"
Disable the default feature for a native program that only uses the frame types:
[]
= { = "0.1", = false }
Optional graphics helpers are separate features:
= { = "0.1", = ["egl"] }
= { = "0.1", = ["vulkan"] }
Binary storage
BinaryStorage owns its bytes and plane metadata. Cloning or decoding it copies
the bytes, so it needs neither file-descriptor duplication nor a release lease.
Binary formats include RGBA8888, RGB888, Gray8, Gray2, and BW1.
DMA-BUF ownership
Descriptor contains borrowed integer file descriptors. Validate it before use.
Call Descriptor::duplicate_cloexec before keeping it after the current call.
The returned OwnedDescriptor closes those duplicates when dropped.
dmabuf_allocation_size reads the complete allocation size exposed by a
DMA-BUF fd. The reported size includes any exporter alignment and driver
padding outside the bytes addressed by the image planes. Producers must store
this size in Object::size. Vulkan imports compare the published size with the
fd before using it.
A sync-file descriptor follows the same rule: it is borrowed until duplicated into an owned frame.
Rustler frames and leases
The default feature maps both storage kinds to the structs in the
video_interop Hex package. It also provides prepared and claimed frames for
moving either storage kind into native queues.
Frame::prepare_cloexec validates a frame. It copies owned binary bytes without
acquiring a lease client and duplicates borrowed descriptors with CLOEXEC. If
a prepared borrowed frame is dropped, the descriptors close but lease ownership
stays with the Elixir caller.
PreparedVideoFrame::claim moves any lease release responsibility to native
code. Owned binary claims carry no lease. Dropping a borrowed
ClaimedVideoFrame or ClaimedLease sends the release through a
ReleaseDispatcher.
The NIF lifecycle owner must call ReleaseDispatcher::close_and_join from a
dirty I/O NIF after all claims have drained. Dispatcher and guard destructors do
not join worker threads or send through OwnedEnv.
EGL
The egl feature loads native-fence function pointers supplied by the caller.
It supports:
- native sync-file fence creation and import;
- fence duplication with
FD_CLOEXEC; - server waits and bounded client waits;
- a bounded
pollfallback when EGL import is unavailable.
The caller must keep the EGL display, context, function pointers, and calling thread valid. The module has no link-time EGL or GL dependency.
Vulkan
The vulkan feature accepts a caller-owned ash device and provides DMA-BUF
capability checks, memory import, sync-file waits, queue-family ownership
transfer, and release-fence handling.
Imported images stay in a bounded source cache. With planar preference, non-linear NV12 is imported as a transfer-source image and copied plane-for-plane into ordinary renderer-owned Y and UV images; linear NV12 uses an imported transfer-source buffer. Packed RGBA and BGRA use a compute copy when the device does not support sampling the producer-linear image. Copy regions cover image bytes only; allocation padding remains mapped but is not copied.
The caller remains responsible for:
- physical-device and queue selection;
- serializing host access to the Vulkan queue;
- choosing a supported import strategy;
- renderer waits and release submission;
- presentation and device-loss recovery.
Platform-specific qualification is tracked separately from the Vulkan API's supported status.
Features
| Feature | Default | Provides |
|---|---|---|
rustler |
yes | Elixir schema codecs, prepared/claimed frames, release dispatcher |
egl |
no | Dynamically loaded EGL native-fence helpers |
vulkan |
no | DMA-BUF import and synchronization helpers |
test-support |
no | Rustler helpers used by schema tests |
License
Apache-2.0. See the license.
Optional Vulkan GPU timing
Consumers can override VulkanDeviceContext::video_gpu_timing_enabled to omit
query-pool allocation and sample_video_gpu_timing to select an entire staged
acquire-to-release bracket. Defaults retain existing instrumentation. Keep the enable
policy fixed for sync-owner lifetimes and make sampling independent of cached slot
indices. report_video_gpu_timing receives resource status; timing is not a capability
or completion authority. ImportedImageSync::has_gpu_timing_resources supports diagnostics.
Off/unsampled paths preserve copies, ownership transfers, waits, release submissions and fences, including release command buffers with no timestamp commands. Results remain optional and are collected only after release completion. Unsupported timestamps and query-pool OOM omit instrumentation; device loss is never optional success.
Existing consumers, including Emerge's Ganesh/headless renderer, do not need to implement the timing hooks: defaults enable timing for every supported staged frame and ignore status notifications. Direct imports remain untimed. This API has no Graphite dependency. Driver-free legacy-consumer tests cover these defaults, release-fence gating, and repeated sync-owner reuse.