Crate vk_video 

👎Deprecated since 0.3.1:

This crate has been renamed. Please migrate to gpu-video.

⚠️ DEPRECATED ⚠️

This crate has been renamed to gpu-video.

Please update your Cargo.toml to use the new crate. This crate is no longer maintained.

vk-video

A library for hardware video decoding and encoding using Vulkan Video, with wgpu integration.

Overview

The goal of this library is to provide easy access to hardware video coding. You can use it to decode and encode a video frame to/from Vec<u8> with pixel data, or wgpu::Texture. Currently, we only support H.264 (aka AVC or MPEG 4 Part 10) but we plan to support other codecs supported by Vulkan Video.

An advantage of using this library with wgpu is that decoded video frames never leave the GPU memory. There’s no copying the frames to RAM and back to the GPU, so it should be quite fast if you want to use them for rendering.

This library was developed as a part of smelter, a tool for video composition.

Code samples

Decode video frame to wgpu::Texture

fn decode_video(
    window: &winit::window::Window,
    mut encoded_video_reader: impl std::io::Read,
) {
    let instance = vk_video::VulkanInstance::new().unwrap();
    let surface = instance.wgpu_instance().create_surface(window).unwrap();
    let adapter = instance.create_adapter(&vk_video::parameters::VulkanAdapterDescriptor {
        compatible_surface: Some(&surface),
        ..Default::default()
    }).unwrap();
    let device = adapter
        .create_device(&vk_video::parameters::VulkanDeviceDescriptor::default())
        .unwrap();

    let mut decoder = device
        .create_wgpu_textures_decoder(
            vk_video::parameters::DecoderParameters::default()
        ).unwrap();

    let mut buffer = vec![0; 4096];

    while let Ok(n) = encoded_video_reader.read(&mut buffer) {
        if n == 0 {
            return;
        }

        let decoded_frames = decoder.decode(vk_video::EncodedInputChunk {
            data: &buffer[..n],
            pts: None
        }).unwrap();

        for frame in decoded_frames {
            // Each frame contains a wgpu::Texture you can sample for drawing.
            // device.wgpu_device() will give you a wgpu::Device and device.wgpu_queue()
            // a wgpu::Queue. You can use these for interacting with the frames.
        }
    }
}

Encode video frame from wgpu::Texture

fn encode_video(
    window: &winit::window::Window,
    frame_receiver: std::sync::mpsc::Receiver<wgpu::Texture>,
) {
    use std::num::NonZeroU32;

    let instance = vk_video::VulkanInstance::new().unwrap();
    let surface = instance.wgpu_instance().create_surface(window).unwrap();
    let adapter = instance
        .create_adapter(&vk_video::parameters::VulkanAdapterDescriptor {
            compatible_surface: Some(&surface),
            ..Default::default()
        })
        .unwrap();
    let device = adapter
        .create_device(&vk_video::parameters::VulkanDeviceDescriptor::default())
        .unwrap();

    let mut encoder = device
        .create_wgpu_textures_encoder(
            vk_video::parameters::EncoderParameters {
                output_parameters: device
                    .encoder_output_parameters_high_quality(
                        vk_video::parameters::RateControl::VariableBitrate {
                            average_bitrate: 500_000,
                            max_bitrate: 2_000_000,
                            virtual_buffer_size: std::time::Duration::from_secs(2),
                        },
                    )
                    .unwrap(),
                input_parameters: vk_video::parameters::VideoParameters {
                    width: NonZeroU32::new(1920).unwrap(),
                    height: NonZeroU32::new(1080).unwrap(),
                    target_framerate: 30.into(),
                },
            }
        )
        .unwrap();

    for frame in frame_receiver.iter() {
        // Encodes NV12 texture and returns encoded frame bytes
        let encoded_frame = encoder
            .encode(
                vk_video::InputFrame {
                    data: frame,
                    pts: None,
                },
                false,
            )
            .unwrap();
    }
}

Be sure to check out our examples, especially the player example, which is a simple video player built using this library and wgpu. Because the player is very simple, you need to extract the raw h264 data from a container before usage. Here’s an example on how to extract the h264 bytestream out of an mp4 file using ffmpeg:

ffmpeg -i input.mp4 -c:v copy -bsf:v h264_mp4toannexb -an output.h264

Then you can run the example with:

git clone https://github.com/software-mansion/smelter.git
cd smelter/vk-video
cargo run --example player -- output.h264 FRAMERATE

Compatibility

On Linux, the library should work on NVIDIA and AMD GPUs out of the box with recent Mesa drivers. For AMD GPUs with a bit older Mesa drivers, you may need to set the RADV_PERFTEST=video_decode,video_encode environment variable:

RADV_PERFTEST=video_decode,video_encode cargo run

It should work on Windows with recent drivers out of the box. Be sure to submit an issue if it doesn’t.

vk-video is created by Software Mansion

Since 2012 Software Mansion is a software agency with experience in building web and mobile apps as well as complex multimedia solutions. We are Core React Native Contributors and experts in live streaming and broadcasting technologies. We can help you build your next dream product – Hire us.

Modules

capabilitiesDeprecated

parametersDeprecated

Structs

BytesDecoderDeprecated

A decoder that outputs frames stored as Vec<u8> with the raw pixel data.

BytesEncoderDeprecated

An encoder that takes input frames as Vec<u8> with raw pixel data (in NV12)

EncodedInputChunkDeprecated

Represents a chunk of encoded video data used for decoding.

EncodedOutputChunkDeprecated

Represents a chunk of encoded video data returned by the encoder.

FrameMetadataDeprecated

Additional information about the decoded frame.

InputFrameDeprecated

Represents a frame to be encoded.

OutputFrameDeprecated

Represents a single decoded frame.

RawFrameDataDeprecated

VulkanAdapterDeprecated

Represents a handle to a physical device. Can be used to create VulkanDevice.

VulkanDeviceDeprecated

Open connection to a coding-capable device. Also contains a wgpu::Device, a wgpu::Queue and a wgpu::Adapter.

VulkanInstanceDeprecated

Context for all encoders and decoders. Also contains a wgpu::Instance.

WgpuNv12ToRgbaConverterDeprecated

Helper that lets you convert OutputFrame<wgpu::Texture> into RGBA wgpu::Texture. Use WgpuNv12ToRgbaConverter::create_input_bind_group to create wgpu::BindGroup which represents NV12 bind group acceptable by the converter.

WgpuRgbaToNv12ConverterDeprecated

Helper that lets you convert RGBA wgpu::Texture into NV12 wgpu::Texture. Use WgpuRgbaToNv12Converter::create_input_bind_group to create wgpu::BindGroup which represents RGBA bind group acceptable by the converter.

WgpuTexturesDecoderDeprecated

A decoder that outputs frames stored as wgpu::Textures

WgpuTexturesEncoderDeprecated

An encoder that takes input frames as wgpu::Textures (in wgpu::TextureFormat::NV12)

Enums

DecoderErrorDeprecated

DecoderEventDeprecated

Represents all events that can be sent to the decoder

H264ParserErrorDeprecated

ReferenceManagementErrorDeprecated

VulkanCommonErrorDeprecated

VulkanDecoderErrorDeprecated

VulkanEncoderErrorDeprecated

VulkanInitErrorDeprecated

WgpuConverterInitErrorDeprecated

WgpuInitErrorDeprecated