wgpu-external-frame 0.4.0

Zero-copy import of externally produced GPU frames into wgpu textures: Linux DMA-BUF, Android AHardwareBuffer, macOS and iOS IOSurface, and Windows shared handles
Documentation

wgpu-external-frame

Import an externally produced GPU frame into wgpu as a texture, without copying it through the CPU.

Browser engines, media decoders, screen capture, and compositors all publish their output as a platform handle to memory the GPU already holds. This crate turns each of those handles into a wgpu::Texture on a device you already own:

Platform Handle Import path
Linux DMA-BUF file descriptor Vulkan VK_EXT_external_memory_dma_buf, or EGL EGL_LINUX_DMA_BUF_EXT + glEGLImageTargetTexture2DOES
Android AHardwareBuffer Vulkan VK_ANDROID_external_memory_android_hardware_buffer; buffers of an external format are converted through a VkSamplerYcbcrConversion
macOS, iOS IOSurface MTLDevice newTextureWithDescriptor:iosurface:plane:, one texture per plane
Windows Shared texture HANDLE ID3D12Device::OpenSharedHandle

Each platform is its own module with its own frame types — DmaBufFrame, HardwareBufferFrame, PackedIoSurfaceFrame and Ycbcr420IoSurfaceFrame, SharedHandleFrame — because the handles have nothing in common beyond the goal. The Linux and Android sides additionally model the producer's lease on the buffer (DmaBufLease, HardwareBufferLease), since those buffers usually come from a pool the producer needs back, guarded by an explicit fence. An IOSurface frame takes the object its producer recycles, such as a pooled CVPixelBuffer, as its owner (with_owner), and drops it once the frame and every texture imported from it have been destroyed, which wgpu does only after every submission that used them has completed.

On Android an RGBA buffer imports as an Rgba8Unorm texture that aliases the buffer itself, so nothing is copied. A 4:2:0 YCbCr buffer imports as two plane views — luma R8Unorm, interleaved Cb/Cr Rg8Unorm — with the matrix and range the Vulkan driver reports for it:

  • when the driver maps the buffer to G8_B8R8_2PLANE_420_UNORM and the device has TEXTURE_FORMAT_NV12, the views are planes of one NV12 texture that aliases the buffer;
  • when the device lacks that feature, or the driver describes the buffer only with an implementation-defined external format — as some drivers do for every YCbCr buffer, camera frames included — sampling it needs a VkSamplerYcbcrConversion, which wgpu cannot express. The import then converts it on the GPU, in a small raw Vulkan pass on the importer's queue, into two textures wgpu owns. No pixel passes through the CPU.

The device must be opened with the extensions and the YCbCr conversion feature the import needs, which wgpu does not enable on its own; ahardware_buffer::request_device does that. The conversion needs nothing more, so it runs on every device that can import at all, the Android emulator included. It binds each buffer through a descriptor set from a pool of the device's maxCombinedImageSamplerDescriptorCount descriptors where it reports one (VK_KHR_maintenance6), and of 3, one per plane, otherwise; a driver that needs more is reported as HardwareBufferImportError::ConversionDescriptorPool. Building for Android compiles the conversion's GLSL shaders with the NDK's glslc, found through ANDROID_NDK_HOME or ANDROID_NDK_ROOT.

On Apple platforms, camera and video frames are biplanar 4:2:0 YCbCr surfaces, and each of their planes imports as its own texture at that plane's extent:

IOSurface pixel format Frame type Texture per plane Device feature
packed BGRA or RGBA PackedIoSurfaceFrame Bgra8Unorm or Rgba8Unorm —
420v, 420f (8-bit 4:2:0) Ycbcr420IoSurfaceFrame luma R8Unorm, chroma Rg8Unorm —
x420, xf20 (10-bit 4:2:0) Ycbcr420IoSurfaceFrame luma R16Unorm, chroma Rg16Unorm TEXTURE_FORMAT_16BIT_NORM

Every import reaches through wgpu to its hal layer, so a device must be on the backend its platform's import requires. Each entry point says which, and asserts it.

Status

This exists because wgpu has no portable external-memory import API yet. When the work in the lineage of wgpu#2320 lands, most of what is here becomes a thin adapter over it, and the hal reach-through goes away.