vk-video

A library for hardware video decoding and encoding using Vulkan Video, with wgpu integration.

Overview

The goal of this library is to provide easy access to hardware video coding. You can use it to decode and encode a video frame to/from Vec<u8> with pixel data, or wgpu::Texture. Currently, we only support H.264 (aka AVC or MPEG 4 Part 10) but we plan to support other codecs supported by Vulkan Video.

An advantage of using this library with wgpu is that decoded video frames never leave the GPU memory. There's no copying the frames to RAM and back to the GPU, so it should be quite fast if you want to use them for rendering.

This library was developed as a part of smelter, a tool for video composition.

Code samples

Decode video frame to `wgpu::Texture`

fn decode_video(
    window: &winit::window::Window,
    mut encoded_video_reader: impl std::io::Read,
) {
    let instance = vk_video::VulkanInstance::new().unwrap();
    let surface = instance.wgpu_instance().create_surface(window).unwrap();
    let adapter = instance.create_adapter(Some(&surface)).unwrap();
    let device = adapter
        .create_device(
            wgpu::Features::empty(),
            wgpu::ExperimentalFeatures::disabled(),
            wgpu::Limits::default(),
        )
        .unwrap();

    let mut decoder = device
        .create_wgpu_textures_decoder(
            vk_video::parameters::DecoderParameters::default()
        ).unwrap();

    let mut buffer = vec![0; 4096];

    while let Ok(n) = encoded_video_reader.read(&mut buffer) {
        if n == 0 {
            return;
        }

        let decoded_frames = decoder.decode(vk_video::EncodedInputChunk {
            data: &buffer[..n],
            pts: None
        }).unwrap();

        for frame in decoded_frames {
            // Each frame contains a wgpu::Texture you can sample for drawing.
            // device.wgpu_device() will give you a wgpu::Device and device.wgpu_queue()
            // a wgpu::Queue. You can use these for interacting with the frames.
        }
    }
}

Encode video frame from `wgpu::Texture`

fn encode_video(
    window: &winit::window::Window,
    frame_receiver: std::sync::mpsc::Receiver<wgpu::Texture>,
) {
    use std::num::NonZeroU32;

    let instance = vk_video::VulkanInstance::new().unwrap();
    let surface = instance.wgpu_instance().create_surface(window).unwrap();
    let adapter = instance.create_adapter(Some(&surface)).unwrap();
    let device = adapter
        .create_device(
            wgpu::Features::empty(),
            wgpu::ExperimentalFeatures::disabled(),
            wgpu::Limits::default(),
        )
        .unwrap();

    let mut encoder = device
        .create_wgpu_textures_encoder(
            device
                .encoder_parameters_high_quality(
                    vk_video::parameters::VideoParameters {
                        width: NonZeroU32::new(1920).unwrap(),
                        height: NonZeroU32::new(1080).unwrap(),
                        target_framerate: 30.into(),
                    },
                    vk_video::parameters::RateControl::VariableBitrate {
                        average_bitrate: 500_000,
                        max_bitrate: 2_000_000,
                        virtual_buffer_size: std::time::Duration::from_secs(2),
                    },
                )
                .unwrap(),
        )
        .unwrap();

    for frame in frame_receiver.iter() {
        // Encodes NV12 texture and returns encoded frame bytes
        let encoded_frame = unsafe {
            encoder
                .encode(
                    vk_video::Frame {
                        data: frame,
                        pts: None,
                    },
                    false,
                )
                .unwrap()
        };
    }
}

Be sure to check out our examples, especially the player example, which is a simple video player built using this library and wgpu. Because the player is very simple, you need to extract the raw h264 data from a container before usage. Here's an example on how to extract the h264 bytestream out of an mp4 file using ffmpeg:

ffmpeg -i input.mp4 -c:v copy -bsf:v h264_mp4toannexb -an output.h264

Then you can run the example with:

git clone https://github.com/software-mansion/smelter.git
cd smelter/vk-video
cargo run --example player -- output.h264 FRAMERATE

Compatibility

On Linux, the library should work on NVIDIA and AMD GPUs out of the box with recent Mesa drivers. For AMD GPUs with a bit older Mesa drivers, you may need to set the RADV_PERFTEST=video_decode,video_encode environment variable:

RADV_PERFTEST=video_decode,video_encode cargo run

It should work on Windows with recent drivers out of the box. Be sure to submit an issue if it doesn't.

vk-video is created by Software Mansion

Since 2012 Software Mansion is a software agency with experience in building web and mobile apps as well as complex multimedia solutions. We are Core React Native Contributors and experts in live streaming and broadcasting technologies. We can help you build your next dream product – Hire us.

vk-video 0.2.1