moq-video

Native video capture, encoding, decoding, and publishing for Media over QUIC.

The video counterpart to moq-audio. Everything is native per-platform code with no ffmpeg dependency: capture, color conversion, and the codec backends are all in-tree or thin wrappers over system frameworks / vendored static libs. The public API is codec-agnostic, so no signature, type, or error variant names a backend or a capture implementation; swapping or bumping a backend crate is not a breaking change.

Capture

Per-platform, picked at compile time:

• macOS: AVFoundation (camera) and ScreenCaptureKit (display), yielding zero-copy CVPixelBuffer surfaces straight to VideoToolbox.
• Linux: native V4L2 (camera; YUYV resampled, MJPEG via zune-jpeg) and xdg-desktop-portal + PipeWire (display; RGB -> CPU I420, behind the pipewire feature since it links libpipewire). Works on Wayland and X11; the desktop's picker dialog chooses the screen, and the portal's restore token is reused so demand-driven reopens don't re-prompt.
• Windows: native Media Foundation (camera; IMFSourceReader) and DXGI Desktop Duplication (display; BGRA -> CPU I420). Display capture is whole-monitor; select one with a bare index or display:{index}.

capture::cameras() lists AVFoundation, V4L2, or Media Foundation cameras with identifiers accepted by capture::Source::Camera. capture::displays() does the same for macOS and Windows displays. Linux display selection stays in the desktop portal picker, which does not expose a stable display identifier.

Encode

The codec is chosen via encode::Codec. Backends are tried in order (hardware first, then software) and the first that opens wins; encode::Kind narrows the choice (Auto / Hardware / Software / a named backend).

Every backend emits Annex-B with in-band parameter sets (SPS/PPS, plus VPS for H.265), so the matching moq_mux::codec importer handles framing and catalog registration directly. There is no software H.265 encoder (it's hardware-only).

Two public entry points:

• encode::publish_capture(...) captures a webcam, encodes it, and publishes on demand: the track and catalog are advertised up front, but the camera opens only while a subscriber is watching and is released when the last one leaves.
• encode::Producer publishes packets you encoded yourself, handling the catalog and framing.

The NVENC and VAAPI backends are Linux-only and gated behind their respective features. Both dlopen the vendor driver at runtime (and fall back to software where the driver is absent), so a feature-enabled binary still links on a GPU-less builder and still starts on a GPU-less machine.

Decode

decode::Consumer (the mirror of moq_audio::decode::Consumer) subscribes to an H.264, H.265, or AV1 track and returns raw frames. A hardware-decoded frame stays on the GPU: feeding it back to a compatible hardware encode::Encoder on the same device keeps it there (the transcode path), while into_i420() downloads it. An encoder that can't take that surface (openh264, or a different device) downloads it through I420 for you. Every frame carries a Surface, a #[non_exhaustive] enum naming where the pixels live (PixelBuffer on macOS, Texture on Windows, Cuda on Linux, or CPU I420). Match it to take a zero-copy path for a representation you recognize, and fall back to Surface::into_i420(), which always works. On macOS Surface::into_pixel_buffer() is the mirror: free for a hardware-decoded frame, an upload for a CPU one. Backends are tried hardware-first, like encode:

On macOS VideoToolbox decodes H.264 and H.265 on hardware, pulling the parameter sets (SPS/PPS, plus VPS for H.265) out of each keyframe to build the format description. On Windows the Microsoft decoder MFT runs synchronously with a Direct3D11 device bound to it, so the decode happens on the GPU through DXVA (NVDEC / Intel / AMD). H.264 falls back to openh264 on a GPU-less host; H.265 has no software decoder, so it needs the GPU path (on Windows, an HEVC decoder MFT: the inbox HEVC Video Extensions or a vendor one). On Linux, NVDEC decodes H.264, H.265, and 8-bit 4:2:0 AV1 to CUDA NV12 frames; AV1 is decode-only and is useful for AV1 source to H.264/H.265 transcode rungs. A non-H.264/H.265/AV1 rendition yields Error::UnsupportedCodec.