Skip to main content

Crate video_interop

Crate video_interop 

Source
Expand description

§video-interop

§The problem

A Rust producer and consumer in a zero-copy Linux GPU pipeline exchange DMA-BUF file descriptors instead of copying frame pixels through CPU memory. Each handoff also requires image layout, format, geometry, colorimetry, synchronization, file-descriptor ownership, and producer-buffer lifetime.

An integer file descriptor does not encode those properties and does not own the underlying allocation. Rust code must duplicate borrowed descriptors before retaining them beyond the current native call, then release each producer lease exactly once after GPU use finishes.

§The solution

The video-interop crate provides the Rust types and lifecycle primitives for the VideoInterop contract. It describes owned binary frames, borrowed Linux DMA-BUF frames, sync-file acquire fences, frame geometry, colorimetry, and stream formats. Prepared and claimed frame types establish the ownership boundary between Elixir and native queues.

The default Rustler integration maps these types to the video_interop Hex package. This shared contract connects producer and consumer elements written in Rust to pipeline control written in Elixir. Optional EGL and Vulkan modules implement synchronization and import operations without taking ownership of the caller’s graphics runtime.

The crate does not allocate producer buffers, create a renderer, or select a streaming framework.

§Status

The core, Rustler, EGL, and Vulkan APIs are part of the supported 0.1 contract.

The crate supports Rust 1.91 and newer.

§Installation

Rustler support is enabled by default:

[dependencies]
video-interop = "0.1"

Disable the default feature for a native program that only uses the frame types:

[dependencies]
video-interop = { version = "0.1", default-features = false }

Optional graphics helpers are separate features:

video-interop = { version = "0.1", features = ["egl"] }
video-interop = { version = "0.1", features = ["vulkan"] }

§Binary storage

BinaryStorage owns its bytes and plane metadata. Cloning or decoding it copies the bytes, so it needs neither file-descriptor duplication nor a release lease. Binary formats include RGBA8888, RGB888, Gray8, Gray2, and BW1.

§DMA-BUF ownership

Descriptor contains borrowed integer file descriptors. Validate it before use. Call Descriptor::duplicate_cloexec before keeping it after the current call. The returned OwnedDescriptor closes those duplicates when dropped.

dmabuf_allocation_size reads the complete allocation size exposed by a DMA-BUF fd. The reported size includes any exporter alignment and driver padding outside the bytes addressed by the image planes. Producers must store this size in Object::size. Vulkan imports compare the published size with the fd before using it.

A sync-file descriptor follows the same rule: it is borrowed until duplicated into an owned frame.

§Rustler frames and leases

The default feature maps both storage kinds to the structs in the video_interop Hex package. It also provides prepared and claimed frames for moving either storage kind into native queues.

Frame::prepare_cloexec validates a frame. It copies owned binary bytes without acquiring a lease client and duplicates borrowed descriptors with CLOEXEC. If a prepared borrowed frame is dropped, the descriptors close but lease ownership stays with the Elixir caller.

PreparedVideoFrame::claim moves any lease release responsibility to native code. Owned binary claims carry no lease. Dropping a borrowed ClaimedVideoFrame or ClaimedLease sends the release through a ReleaseDispatcher.

The NIF lifecycle owner must call ReleaseDispatcher::close_and_join from a dirty I/O NIF after all claims have drained. Dispatcher and guard destructors do not join worker threads or send through OwnedEnv.

§EGL

The egl feature loads native-fence function pointers supplied by the caller. It supports:

  • native sync-file fence creation and import;
  • fence duplication with FD_CLOEXEC;
  • server waits and bounded client waits;
  • a bounded poll fallback when EGL import is unavailable.

The caller must keep the EGL display, context, function pointers, and calling thread valid. The module has no link-time EGL or GL dependency.

§Vulkan

The vulkan feature accepts a caller-owned ash device and provides DMA-BUF capability checks, memory import, sync-file waits, queue-family ownership transfer, and release-fence handling.

Imported images stay in a bounded source cache. With planar preference, non-linear NV12 is imported as a transfer-source image and copied plane-for-plane into ordinary renderer-owned Y and UV images; linear NV12 uses an imported transfer-source buffer. Packed RGBA and BGRA use a compute copy when the device does not support sampling the producer-linear image. Copy regions cover image bytes only; allocation padding remains mapped but is not copied.

The caller remains responsible for:

  • physical-device and queue selection;
  • serializing host access to the Vulkan queue;
  • choosing a supported import strategy;
  • renderer waits and release submission;
  • presentation and device-loss recovery.

Platform-specific qualification is tracked separately from the Vulkan API’s supported status.

§Features

FeatureDefaultProvides
rustleryesElixir schema codecs, prepared/claimed frames, release dispatcher
eglnoDynamically loaded EGL native-fence helpers
vulkannoDMA-BUF import and synchronization helpers
test-supportnoRustler helpers used by schema tests

§License

Apache-2.0. See the license.

§Optional Vulkan GPU timing

Consumers can override VulkanDeviceContext::video_gpu_timing_enabled to omit query-pool allocation and sample_video_gpu_timing to select an entire staged acquire-to-release bracket. Defaults retain existing instrumentation. Keep the enable policy fixed for sync-owner lifetimes and make sampling independent of cached slot indices. report_video_gpu_timing receives resource status; timing is not a capability or completion authority. ImportedImageSync::has_gpu_timing_resources supports diagnostics.

Off/unsampled paths preserve copies, ownership transfers, waits, release submissions and fences, including release command buffers with no timestamp commands. Results remain optional and are collected only after release completion. Unsupported timestamps and query-pool OOM omit instrumentation; device loss is never optional success.

Existing consumers, including Emerge’s Ganesh/headless renderer, do not need to implement the timing hooks: defaults enable timing for every supported staged frame and ignore status notifications. Direct imports remain untimed. This API has no Graphite dependency. Driver-free legacy-consumer tests cover these defaults, release-fence gating, and repeated sync-owner reuse.

Modules§

egl
Dynamically loaded EGL native-fence synchronization helpers.
vulkan
Vulkan DMA-BUF import adapter.

Structs§

AbandonmentGuard
Fallback resource attached to one lease holder.
BinaryFormat
Rust representation of VideoInterop.Binary.Format.
BinaryPlane
BinaryStorage
ClaimedLease
ClaimedVideoFrame
Colorimetry
Rust representation of VideoInterop.Colorimetry.
Descriptor
DispatcherError
DispatcherProbe
DmaBufFormat
Rust representation of VideoInterop.DMABuf.Format.
Format
Rust representation of the VideoInterop.Format stream schema.
Frame
FrameDescriptor
Layer
Lease
Object
OwnedDescriptor
OwnedFrame
OwnedObject
Plane
PreparedLease
PreparedVideoFrame
Rect
ReleaseDispatcher
Lifecycle-owned dispatcher for normal and fallback lease messages.
SyncFile

Enums§

AcquireSync
AcquireSyncPolicy
Synchronization policy declared by VideoInterop.Format.
AlphaMode
BinaryPixelFormat
Bw1Polarity
ChromaLocation
ColorRange
DispatcherHealth
DmaBufAllocationSizeError
DuplicateError
FormatValidationError
Numeric validation errors for a stream format.
InterlaceMode
Matrix
Modifier
ModifierPolicy
DMA-BUF modifier policy declared by VideoInterop.DMABuf.Format.
OwnedAcquireSync
OwnedStorage
PrepareError
Primaries
Storage
StorageFormat
Transfer
ValidationError

Constants§

AV_DRM_MAX_ENTRIES

Functions§

dmabuf_allocation_size
Returns the complete allocation size exposed by a DMA-BUF fd.
is_abandonment_guard_resource
Creates the resource used by a thin producer-NIF guard constructor.
new_abandonment_guard

Type Aliases§

Rational
Positive rational represented as the Elixir {numerator, denominator} tuple.
ReleaseWorkerError
StreamAcquireSyncPolicy
Alias matching the stream-policy terminology used by native consumers.
StreamModifierPolicy
Alias matching the stream-policy terminology used by native consumers.