zvidlib 0.4.1

Frame-accurate video and audio I/O for native and WebAssembly applications
Documentation

zvidlib

zvidlib is a Rust library for frame-accurate video and synchronized audio I/O on native and WebAssembly targets. Its primary jobs are reading an MP4 into a GL/WebGL canvas and writing canvas frames plus an audio stream into an MP4, behind a small API centered on indexed get and put operations.

Project status: zvidlib is pre-1.0 and its API may still change in a future minor release. The portable foundation now includes checked timeline arithmetic, validated media buffers, asynchronous byte I/O, CPU/GL/WebGL transfer contracts, bounded ordinary/fragmented MP4 sample indexing, normalized codec factories, bounded exact-frame video decoding, exact AAC, Opus and Vorbis sample reads, audio-clock playback control and adapter contracts, encoder contracts, strict indexed output, seekable MP4 muxing, and the browser WebAssembly boundary. The generated JavaScript package includes BigInt-safe values, stable errors, Blob/stream input, Blob output, session/stream/playback handles, real HEVC/AV1/VP9 video decode through the browser's native WebCodecs VideoDecoder (falling back to the pure-Rust HEVC Main/Main 10, AV1 and VP9 profile 0 decoders when WebCodecs cannot decode a track), and AAC packet/config exports for browser AudioDecoder playback. Native builds include HEVC Main and AV1 Main decoders and encoders (native_hevc_video_decoder_factory, native_hevc_video_encoder_factory, native_av1_video_decoder_factory, native_av1_video_encoder_factory), VP8 and VP9 profile 0 decoders and encoders (native_vp8_video_decoder_factory, native_vp8_video_encoder_factory, native_vp9_video_decoder_factory, native_vp9_video_encoder_factory), plus AAC-LC decode and default-device PCM output for synchronized playback; HEVC decode uses NVIDIA NVDEC on supported 64-bit Windows/Linux systems, a D3D11-aware Media Foundation decoder on Windows, or VideoToolbox on macOS, and otherwise retains the dependency-free pure-Rust fallback; HEVC encode can use a hardware Media Foundation encoder (NVENC, Quick Sync, or AMF) on Windows or VideoToolbox's hardware encoder on macOS. Color AV1 encoding beyond the monochrome profile and fully portable audio-device abstractions remain planned; a pure-Rust AAC encoder is not -- see the writer-core notes under Planned API examples for where each audio encoder comes from, and Implemented browser boundary for the browser's own WebCodecs-backed AAC-LC and Opus encode bridge. The complete workflow interfaces below remain intentionally aspirational.

Documentation

Full API documentation is published at lsegal.github.io/zvidlib and rebuilt from main on every push.

Goals

  • Return exactly the requested video frame, including across codecs that use bidirectionally predicted frames.
  • Keep sequential access fast by retaining decoded frames and codec state between requests.
  • Keep audio aligned to the video timeline for indexed reads, writes, seeking, and playback.
  • Move frames through CPU buffers or GL/WebGL textures and framebuffers, using zero-copy paths when the platform permits.
  • Read and write ISO Base Media File Format/MP4, initially targeting HEVC/H.265, AV1 and VP9 video (VP9 written as vp09) with AAC or Opus audio, and WebM with Opus or Vorbis audio alongside its video.
  • Present equivalent Rust and JavaScript concepts on native and wasm32 targets.
  • Make containers, codecs, storage, and graphics backends replaceable without exposing backend details in the common API.

The project is a library, not a media CLI or an FFmpeg binding. FFmpeg is a feature-set reference only; zvidlib will not copy or incorporate FFmpeg source.

Implemented browser boundary

The web feature exposes MediaInput, MediaOutput, Playback, VideoStream, AudioStream, OpenOptions, CreateOptions, PlaybackOptions, PreviewIndex, PreviewOptions, Preview, FrameIndex, Timestamp, Rational, SampleRange, VideoFrame, and AudioBuffer through wasm-bindgen.

MediaInput.open accepts a Blob, ReadableStream<Uint8Array>, ArrayBuffer, or typed-array view. It consumes streams, always releases its reader lock, and supports cancellation through OpenOptions.signal. Input bytes are copied into owned WebAssembly storage; bytes() returns a fresh JavaScript snapshot rather than a view into growable WebAssembly memory.

import init, { FrameIndex, MediaInput, OpenOptions, errorCode } from "./pkg/zvidlib.js";

await init();

const controller = new AbortController();
const options = new OpenOptions(64n * 1024n * 1024n);
options.signal = controller.signal;

const response = await fetch("./clip.mp4");
const input = await MediaInput.open(await response.blob(), options);
console.log(input.byteLength); // BigInt
console.log(new FrameIndex(18_446_744_073_709_551_615n).value);

try {
  // Decodes the real frame via the browser's native WebCodecs `VideoDecoder`
  // for HEVC/AV1/VP9 input tracks, or via zvidlib's own software decoder when
  // WebCodecs cannot decode the track.
  const frame = await input.video(0).get(0n);
  console.log(frame.width, frame.height, frame.pixels.length);
} catch (error) {
  // "UNSUPPORTED" if neither WebCodecs nor the software decoder can decode
  // the track, rather than a fake or nearest frame.
  console.log(errorCode(error));
}

input.close();

An input AudioStream reads exact sample ranges: getRange(start, end) resolves to an AudioBuffer holding exactly the half-open range [start, end) of the track's presentation samples, after its priming, end padding and edit list, and sampleCount() is one past the last of them. AAC and Opus tracks decode through the browser's WebCodecs AudioDecoder where it supports them, and Opus falls back to zvidlib's own decoder where it does not or where the browser's output does not line up with the track. decoderConfig() returns a track's WebCodecs AudioDecoderConfig for a caller driving its own decoder with packet().

MediaOutput.finish() returns a browser-owned Blob with the configured MIME type. writeEncodedChunk() remains the raw byte-sink boundary for a caller supplying its own already-muxed container bytes. Indexed put on any video track encodes through a built-in WebCodecs bridge (AV1 Main, HEVC Main, VP8 or VP9 profile 0; CreateOptions.videoCodec selects "av1" (the default), "hevc", "vp8" or "vp9") and mixes every track's samples into a real multi-track MP4 at finish(); call videoEncodeSupport(hardware, codec) first to check whether the current browser can encode the selected codec. A VP9 track is written as a vp09 sample entry whose vpcC takes its colour description from the encoder's first key frame. Input frames accept VideoFrame.rgba(), VideoFrame.bgra8(), or VideoFrame.yuv420p8(). Track 0 of an audio stream likewise encodes through a WebCodecs AudioEncoder bridge from interleaved f32 PCM: AAC-LC by default, or Opus with CreateOptions.audioCodec = "opus", which takes 48 kHz input, the rate Opus decodes at, and writes the encoder's pre-skip and end padding into the MP4's edit list so the track reads back with exactly the samples that were put. Call audioEncodeSupport(hardware, codec) first; it reports false for "vorbis", which no browser encodes and MP4 cannot carry. finish() muxes whichever tracks were used into one MP4. Other audio track indices and other codecs/profiles remain unsupported until their backends land. setCoverArt(bytes, "image/jpeg" | "image/png"), called any time before finish(), embeds an already-encoded picture as iTunes-style covr metadata, which Windows Explorer, Finder and most players show as the file's thumbnail without decoding the video. Without one, finish() generates the cover itself from video frame 4 of the first video track, shrunk to at most 512 pixels on its longest edge and encoded as PNG, or from the last frame of a shorter stream; set CreateOptions.coverFrame to another zero-based index, or to null for no generated cover.

import { CreateOptions, MediaOutput } from "./pkg/zvidlib.js";

const createOptions = new CreateOptions("mp4");
createOptions.maxOutputBytes = 512n * 1024n * 1024n;
const output = await MediaOutput.create(createOptions);

// A future MP4 muxer supplies already-encoded container chunks here.
output.writeEncodedChunk(new Uint8Array([0, 0, 0, 8, 0x66, 0x72, 0x65, 0x65]));
const blob = await output.finish();
console.log(blob.type); // "video/mp4"

WebM

WebM is read and written alongside MP4 through the same indexed get(n)/put(n) API. MediaInput.open tells the two apart by the bytes themselves, never by a file name or MIME type: an EBML header whose DocType is webm or matroska is WebM, and input.container reports "webm", "mp4", or null. A WebM input's AV1 video tracks then answer get(n), frameDuration(n), randomAccessPoints() and previews() exactly as an MP4's do. new CreateOptions("webm") makes MediaOutput.finish() mux the encoded video into a seekable WebM and return a video/webm Blob; supportedContainers() lists both names.

WebM permits only VP8, VP9 or AV1 video and Vorbis or Opus audio, so WebM output is AV1, VP8 or VP9 video with Opus or Vorbis audio: CreateOptions.audioCodec defaults to "opus" on a WebM output and refuses "aac", videoCodec = "vp8" encodes VP8 through the browser's WebCodecs VideoEncoder (videoEncodeSupport(hardware, "vp8") reports whether it can) and is refused with UNSUPPORTED on an "mp4" output, since MP4 has no widely supported VP8 mapping, and setCoverArt() and videoCodec = "hevc" reject with UNSUPPORTED. On input, Opus and Vorbis tracks read through AudioStream.getRange() with their CodecDelay and DiscardPadding trims applied, and other audio tracks are skipped rather than refused, so a browser MediaRecorder capture still opens for its video; VP8 video decodes through the browser's WebCodecs vp8 decoder, or zvidlib's own VP8 decoder where WebCodecs cannot, and VP9 profile 0 video through WebCodecs with a vp09 codec string, or zvidlib's own VP9 decoder where WebCodecs cannot. A VP8 frame the encoder hid (an alternate reference stored as a block of its own) is currently indexed as a frame, and asking for that frame fails. A VP9 track is written as V_VP9, with the profile, level, bit depth and chroma subsampling of its vpcC as the CodecPrivate features, and reads back with an equivalent vpcC.

const options = new CreateOptions("webm");
options.setTimeline(30, 1, 48_000);
const output = await MediaOutput.create(options);
for (let frameIndex = 0n; frameIndex < 90n; frameIndex++) {
  await output.video(0).put(frameIndex, VideoFrame.rgba(width, height, pixels));
}
const webm = await output.finish(); // Blob { type: "video/webm" }
const input = await MediaInput.open(webm);
console.log(input.container); // "webm"
const frame = await input.video(0).get(45n);

Natively, WebmDemuxer::open builds the same Mp4Track sample index Mp4Demuxer::open does, so to_encoded_video_samples and ExactFrameReader read a WebM track unchanged. It handles unknown-size Segments and Clusters (as live recorders write them), SimpleBlocks and BlockGroups, Xiph, EBML and fixed-size lacing, and any TimestampScale; WebmDemuxer::seek_point starts a decode from the file's Cues when it has them and from the scanned keyframes when it does not. WebmMuxer takes the same Mp4TrackConfig declarations Mp4Muxer does, for AV1, VP8 and VP9 video (a V_VP8 track carries no CodecPrivate, so its decoder_config is empty) and Opus and Vorbis audio, writes payload bytes as samples arrive, and at finish fills in the Segment and Cluster sizes, the Duration, a SeekHead, and Cues with one cue per keyframe, which is what lets Chrome and Firefox seek the file. Samples are written in one presentation-time order across tracks, because a WebM Cluster interleaves them. An Opus or Vorbis track is indexed into the same Mp4Track an MP4 audio track is - an Opus OpusHead rewritten as the dOps box opus_config reads, a Vorbis CodecPrivate as stored for vorbis_config - and WebmDemuxer::audio_timing turns its CodecDelay and last DiscardPadding into the priming and padding AudioSampleReader trims. WebmMuxer writes an Opus track's pre-skip as CodecDelay with an 80 ms SeekPreRoll, and the padding set_audio_gapless declares as the track's last block's DiscardPadding, holding each audio track's newest block back until it knows whether that block is the last; FFmpeg decodes zvidlib's Opus WebM to exactly the encoded length. An Opus or Vorbis track is indexed into the same Mp4Track an MP4 audio track is - an Opus OpusHead rewritten as the dOps box opus_config reads, a Vorbis CodecPrivate as stored for vorbis_config - and WebmDemuxer::audio_timing turns its CodecDelay and last DiscardPadding into the priming and padding AudioSampleReader trims. WebmMuxer writes an Opus track's pre-skip as CodecDelay with an 80 ms SeekPreRoll, and the padding set_audio_gapless declares as the track's last block's DiscardPadding, holding each audio track's newest block back until it knows whether that block is the last; FFmpeg decodes zvidlib's Opus WebM to exactly the encoded length. probe_container detects either container from a ByteSource, and container_capabilities reports both.

All exported 64-bit frame, sample, and timestamp values return JavaScript BigInt. Inputs accept BigInt across the full Rust range or validated Number values only within JavaScript's safe-integer range. Rust and boundary failures reject with native Error instances named ZvidError; their stable code values can be read directly or with errorCode(error).

Input VideoStream handles also expose frameDuration(index), which resolves to that presentation frame's MP4 duration in milliseconds. This timing query is independent of codec availability, so browser applications can pace fallback rendering even when get(index) reports UNSUPPORTED.

video.previews(options) builds the seek preview tier ARCHITECTURE.md section 3.2 requires a seek to be answered from: one downscaled picture every stride frames, on a decode session of its own. The pass has no thread to run on in a browser, so the caller advances it a preview at a time with await index.step(signal) from requestIdleCallback or a requestAnimationFrame slice, and index.nearest(frame) answers from whatever it has reached so far. That lookup is synchronous and never decodes - it returns a Preview carrying the picture and the frame it is actually of, or null - which is what lets it stay inside seekLatencyBudgetMs() however far a drag jumped.

Planned API examples

The examples in this section show the intended complete workflows. Rust examples are deliberately marked ignore, and browser examples depend on media backends that are not fully registered yet: video decoding works (see Implemented browser boundary), but Playback, audio decode/encode, and video encode still reject with UNSUPPORTED until their implementations land. The boundary classes exist for all of these. Names may change before the first release, while the ownership, synchronization, and cleanup steps are part of the design.

Read, play audio, and render with native OpenGL

The application owns the window, current GL context, and audio device. zvidlib owns the demuxer and decoder state, uploads exact frames into the caller's texture, and uses the audio device clock as the playback clock.

use std::time::Duration;
use zvidlib::{AudioOutput, GlContext, OpenOptions, Playback};

#[tokio::main]
async fn main() -> zvidlib::Result<()> {
    // Application-specific setup (glutin/winit, SDL, etc.). The GL context must
    // stay current on this thread for every zvidlib and draw call below.
    let (window, gl): (_, GlContext) = create_window_and_current_gl_context()?;
    let audio_device: AudioOutput = open_default_audio_device()?;
    let texture = create_rgba_texture(&gl, window.drawable_size())?;

    let input = zvidlib::open("clip.mp4", OpenOptions::default()).await?;
    let video = input.video(0)?;
    let audio = input.audio(0)?;
    let mut playback = Playback::builder(video, audio)
        .audio_output(audio_device)
        .video_destination(gl.texture_2d(texture))
        .build()
        .await?;

    playback.play().await?;
    while !window.should_close() && !playback.is_finished() {
        window.poll_events();

        // `present` schedules audio ahead, asks `get(n)` for the exact frame
        // selected by the audio clock, and uploads that frame to `texture`.
        if playback.present().await? {
            draw_fullscreen_texture(&gl, texture)?;
            window.swap_buffers()?;
        }

        tokio::time::sleep(Duration::from_millis(1)).await;
    }

    playback.stop().await?;
    input.close().await?;
    Ok(())
}

The helpers that create the native window, GL objects, shaders, and audio device depend on the application's windowing/audio crates and are intentionally left to the host. present never substitutes a nearby keyframe: it resolves the current presentation index and calls the same exact-frame path as video.get(n).

Read, play audio, and render with WebGL

Build the package as described in Building and using WebAssembly, then serve this page and clip.mp4 from the same directory. Browser security rules require HTTP(S); opening the HTML as a file: URL will not work reliably.

<!doctype html>
<html lang="en">
  <meta charset="utf-8">
  <title>zvidlib WebGL playback</title>
  <canvas id="video" width="1280" height="720"></canvas>
  <button id="play" type="button">Play</button>
  <script type="module">
    import init, { MediaInput, Playback } from "./pkg/zvidlib.js";

    await init();

    const canvas = document.querySelector("#video");
    const playButton = document.querySelector("#play");
    const gl = canvas.getContext("webgl2", { alpha: false });
    if (!gl) throw new Error("WebGL 2 is unavailable");

    const response = await fetch("./clip.mp4");
    if (!response.ok) throw new Error(`clip.mp4: ${response.status}`);

    const input = await MediaInput.open(await response.blob());
    const video = input.video(0);
    const audio = input.audio(0);
    const audioContext = new AudioContext();
    const playback = await Playback.create({
      video,
      audio,
      audioContext,
      webgl: { context: gl, canvas },
    });

    let animationFrame = 0;
    async function render() {
      // Schedules audio and renders the exact `video.get(n)` result selected by
      // the AudioContext clock into the canvas's default framebuffer.
      const { finished } = await playback.present();
      if (!finished) animationFrame = requestAnimationFrame(render);
    }

    playButton.addEventListener("click", async () => {
      playButton.disabled = true;
      await audioContext.resume(); // Must follow a user gesture.
      await playback.play();
      animationFrame = requestAnimationFrame(render);
    }, { once: true });

    window.addEventListener("pagehide", () => {
      cancelAnimationFrame(animationFrame);
      playback.close();
      input.close();
      audioContext.close();
    }, { once: true });
  </script>
</html>

Write synchronized native GL and audio input

Writing uses the same zero-based frame index. The application captures audio for the exact half-open interval represented by each video frame and submits both values before advancing the index.

use zvidlib::{CreateOptions, GlContext, VideoCodec};

#[tokio::main]
async fn main() -> zvidlib::Result<()> {
    let (window, gl): (_, GlContext) = create_window_and_current_gl_context()?;
    let microphone = open_default_audio_input()?;
    let framebuffer = create_recording_framebuffer(&gl, 1920, 1080)?;

    let mut output = zvidlib::create(
        "recording.mp4",
        CreateOptions::mp4()
            .video(VideoCodec::Av1, 1920, 1080, 30, 1)
            .aac_audio(microphone.sample_rate(), microphone.channels()),
    ).await?;
    let video = output.video(0)?;
    let audio = output.audio(0)?;

    for frame_index in 0_u64..300 {
        render_scene_into(&gl, framebuffer, frame_index)?;
        let samples = microphone.capture(audio.interval_for_frame(frame_index)?).await?;

        video.put(frame_index, gl.framebuffer(framebuffer)).await?;
        audio.put(frame_index, samples).await?;
    }

    // `finish` drains delayed video/audio packets and writes the MP4 indexes.
    output.finish().await?;
    window.close();
    Ok(())
}

Write a WebGL canvas and Web Audio stream

This browser example records ten seconds. A real application can replace drawScene and microphone with any WebGL renderer and timestamped Web Audio source.

import init, { MediaOutput } from "./pkg/zvidlib.js";

await init();

const canvas = document.querySelector("#video");
const gl = canvas.getContext("webgl2", { preserveDrawingBuffer: true });
const audioContext = new AudioContext();
const microphone = await navigator.mediaDevices.getUserMedia({ audio: true });
const output = await MediaOutput.create({
  container: "mp4",
  video: { codec: "av1", width: canvas.width, height: canvas.height, fps: 30 },
  audio: { codec: "aac", sampleRate: audioContext.sampleRate, channels: 2 },
});
const video = output.video(0);
const audio = output.audio(0);
const capture = await audio.captureFromMediaStream(audioContext, microphone);

for (let frameIndex = 0n; frameIndex < 300n; frameIndex++) {
  drawScene(gl, frameIndex);
  const samples = await capture.get(audio.intervalForFrame(frameIndex));
  await video.put(frameIndex, { webgl: gl, framebuffer: null });
  await audio.put(frameIndex, samples);
}

capture.close();
microphone.getTracks().forEach((track) => track.stop());
const mp4 = await output.finish();
const link = Object.assign(document.createElement("a"), {
  href: URL.createObjectURL(mp4),
  download: "recording.mp4",
  textContent: "Download recording",
});
document.body.append(link);

JavaScript frame indices are BigInt so the wrapper can preserve the Rust u64 range. Both writers reject skipped or repeated indices by default, and finish() is required to drain codecs and finalize the MP4.

zvidlib writes no audio codec of its own design. Every AudioEncoder it ships is either the platform's or a faithful port of the codec's reference encoder, because a mediocre encoder under this crate's name is harder for callers to route around than no encoder at all, and with a port the quality to defend is the reference's. native_aac_audio_encoder_factory() delegates to AudioToolbox on macOS and to Media Foundation on Windows, and reports HardwareUnavailable elsewhere rather than falling back to an encoder of its own: a pure-Rust AAC-LC encoder means owning a filter bank, a psychoacoustic model, and rate control with no open reference to port. native_opus_audio_encoder_factory() runs opus-pure, a pure-Rust port of libopus, and native_vorbis_audio_encoder_factory() runs zvidlib's own port of the libvorbis 1.3.7 encoder, whose packets are bit-identical to libvorbis's; both run on every target. MediaOutput muxes an audio track given any AudioEncoder, so a caller with an encoder of its own still plugs it in.

The implemented portable writer core exposes VideoEncoder and AudioEncoder contracts plus MediaOutput. An encoder backend declares its MP4 codec configuration and exact output timescale, then returns EncodedSample values with DTS, PTS, duration, sync, and dependency metadata. MediaOutput::put_video accepts the shared CPU/GL/WebGL FrameSource; it and put_audio enforce zero-based consecutive indices and exact frame-aligned audio ranges. finish drains both encoders, records audio priming and padding in an edit list, finalizes sample indexes, and flushes the seekable ByteSink. MediaOutput::set_cover_art (or Mp4Muxer::set_cover_art) embeds a caller-encoded JPEG or PNG CoverArt in moov/udta/meta/ilst/covr any time before finish, so a recorder can pick a frame after capture; operating-system file browsers show it as the thumbnail without an HEVC or AV1 decoder, output without cover art is unchanged, and Mp4Demuxer::cover_art reads it back. When no cover is set, MediaOutput generates one: OutputOptions::cover_source defaults to CoverSource::Frame(4), and the writer shrinks that presentation-order frame to at most 512 pixels on its longest edge as frames are written, then encodes it as a PNG at finish with no extra dependency. A stream with fewer frames uses its last frame, a frame supplied as a GPU resource gives no cover, and CoverSource::None turns the generated cover off. The capture holds one thumbnail, at most 768 KiB, and costs nothing after the chosen frame. CoverArt::from_video_frame makes the same thumbnail from any CPU frame. Mp4Muxer never generates a cover, so its output without cover art is unchanged.

native_av1_video_encoder_factory() supplies a dependency-free software backend for 8-bit monochrome AV1 Main profile. It accepts Gray8 CPU frames, emits one independently decodable keyframe access unit per input frame, and generates the matching standardized av1C box. VideoEncoderConfig::configuration selects the quantization profile: empty encodes losslessly with the 4x4 WHT (the default, and the output verified against an independent ffmpeg decode), and a single byte carrying a nonzero base_q_idx encodes non-lossless through the forward DCT with a per-block transform-size, transform-type, and partition decision. Non-lossless streams are interchange-grade: they round-trip through native_av1_video_decoder_factory() and decode through an independent ffmpeg 7.1 (dav1d) within an asserted distortion bound across several quantizers, frame sizes, and content patterns. VideoEncoderConfig::timescale and frame_duration define the exact constant-rate clock used for emitted DTS, PTS, and duration values. Unsupported color formats, hardware requirements, malformed private configuration, out-of-level rates/dimensions, and resource-limit violations are rejected during capability discovery or creation.

native_vp8_video_encoder_factory() supplies a dependency-free software VP8 encoder. It accepts limited-range Rgba8 or Bgra8 CPU frames of any size up to 16383 pixels a side, top-down or bottom-up, converts them to 4:2:0 YCbCr with the BT.601 matrix VP8 defines, and emits one VP8 frame per input frame for WebmMuxer; VP8 carries no configuration record, so EncoderConfig::decoder_config is empty. It codes key frames and inter frames predicted from the previous frame, choosing per macroblock between the four whole-macroblock and ten subblock intra modes and ZEROMV, NEARESTMV, NEARMV or NEWMV with a quarter-sample motion search, skipping a residual whose bits buy too little, adapting the token probabilities to each frame, and running the loop filter. Its reconstruction is the decoder's own, so native_vp8_video_decoder_factory() and libvpx decode exactly the pictures it predicts from, which a test checks byte for byte against libvpx through ffmpeg. VideoEncoderConfig::configuration takes the HEVC encoder's forms with a VP8 quantizer: empty is a fixed quantizer index of 32, one byte a fixed quantizer index from 0 to 127, and four big-endian bytes a target bit rate in bits a second, optionally followed by four more giving the keyframe interval in frames; without one, a key frame starts every second. Under a target bit rate the quantizer follows each frame's size towards the budget, carrying any surplus or deficit forward, so over two seconds of the tests' moving pattern the average lands within 2 percent of the target. Encoding runs on the calling thread at about 400 ms a 1080p frame on one desktop core, so it suits short clips and small frames rather than real-time HD capture. There is no hardware VP8 encoder behind it, so Require reports HardwareUnavailable.

native_vp9_video_encoder_factory() supplies a dependency-free software backend for VP9 profile 0 (8-bit 4:2:0), written to MP4 as a vp09 sample entry with a vpcC box naming the level the frame size and rate need, or to WebM as a V_VP9 track. It accepts Yuv420p8, Rgba8, Bgra8, or Gray8 CPU frames of any size up to 4096 pixels wide, top-down or bottom-up; RGB is converted to 4:2:0 with the BT.601 matrix in the configured range, and the bitstream signals BT.601. It emits key frames and inter frames: a key frame starts every group of pictures, and each inter frame predicts from the frame before it with motion vectors from a whole-pixel diamond search refined to quarter pixels, or eighth pixels below quantizer index 200, through VP9's regular 8-tap filters, splitting each 64x64 superblock into blocks from 64x64 down to 8x8 and choosing per block between motion compensation and DC, V, H, or TM intra prediction, with transforms from 4x4 up to 32x32, all by rate and distortion. Key frames are sync samples marked independent and inter frames are marked dependent, so ExactFrameReader restarts a seek at the nearest key frame. Frames are deblocked by the VP9 loop filter at the level that brings them closest to the source, and adapt their entropy coding probabilities from frame to frame as the decoder does, so the coding follows the content; each key frame starts again from VP9's default probabilities. VideoEncoderConfig::configuration is empty (quantizer index 80 and a key frame every 60 frames), a single nonzero base_q_idx byte (higher is smaller and softer), that byte followed by a nonzero big-endian u16 key frame interval in frames, or those three bytes followed by a flags byte. Flag bit 0 (1) codes every frame error resilient, with the default probabilities and without the previous frame's motion vectors, so each frame decodes from its reference picture alone at the cost of a larger stream; the other bits must be zero. The output decodes identically, pixel for pixel, through ffmpeg's VP9 decoder and libvpx, and plays through the browser's own WebCodecs VP9 decoder. It also round-trips frame-accurately through native_vp9_video_decoder_factory(), sequentially and under seeks across key frames.

The same factory selects a hardware VP9 encoder when VideoEncoderConfig::hardware asks for one. On Windows, Prefer and Require route to the GPU vendor's hardware Media Foundation VP9 encoder where one is registered, such as Intel Quick Sync's; on macOS, to VideoToolbox's hardware VP9 encoder on Macs whose media engine has one. Either takes limited-range input at even dimensions: Rgba8, Bgra8, or Yuv420p8 on Windows, and Rgba8 or Bgra8 on macOS. The hardware encoder takes the same configuration as the software one: it is rate controlled by quality, which base_q_idx maps onto, with a key frame at least every configured interval. It cannot code error-resilient frames, so a configuration with the error-resilient flag always encodes in software. Its stream is muxed exactly as the software encoder's is, as vp09/vpcC in MP4 and V_VP9 in WebM, with a vpcC built from the colour space and range its first key frame signals, and a sample is a sync sample exactly when it opens on a key frame. capability() reports Supported { implementation: Hardware } only when such an encoder is usable for the configuration; otherwise Prefer falls back to the software encoder and Require reports CodecSupport::HardwareUnavailable, or InvalidConfiguration for a configuration no hardware encoder takes. Avoid always encodes in software, and Linux has no hardware VP9 encoder yet.

native_hevc_video_encoder_factory() supplies the matching dependency-free software backend for 8-bit 4:2:0 HEVC Main profile. It accepts limited-range Rgba8 CPU frames whose dimensions are multiples of 16, emits one IDR access unit per input frame, and generates the matching standardized hvcC box. VideoEncoderConfig::configuration selects the operating point the same way AV1's does: empty codes every coding unit as a pcm_flag == 1 PCM block, so the coded picture is exactly the source, a single byte carrying SliceQpY in 0..=51 codes ITU-T H.265 §7.3.8.11 quantized residual at that QP, and four big-endian bytes give a nonzero target bitrate in bits a second, optionally followed by four more giving a keyframe interval in frames (zero meaning one second), which the native writer meets trivially because every picture it writes is an IDR. Lossless PCM stays the default, so callers that configure nothing get byte-identical output to earlier releases; unlike AV1's base_q_idx, a 0 byte is not a lossless request, because HEVC QP 0 is simply the finest quantizer step. The two lossy forms differ in what they optimize as well as in who picks the quantizer: a fixed QP takes the closest picture at that QP, because a caller naming a QP is asking for a picture, while a bitrate target picks SliceQpY per picture against what the previous picture actually cost and charges every intra candidate for the residual bits it would code, because with a rate to hit the bits a decision saves buy quality elsewhere. Lossy streams decode through native_hevc_video_decoder_factory() with distortion that tracks the requested QP, and a bitrate-targeted stream settles within 25 percent of its per-picture budget — about as tight as an integer QP chosen once a picture allows. The writer searches the intra prediction mode per coding unit and runs both in-loop filters -- §8.7.2 deblocking, and §8.7.3 SAO wherever the pass earns the sao( ) syntax it costs against the picture's own rate-distortion curve. VideoEncoderConfig::timescale and frame_duration define the exact constant-rate clock used for emitted DTS, PTS, and duration values.

The same factory selects a platform encoder when VideoEncoderConfig::hardware asks for one. On Windows, Prefer and Require route a target-bitrate configuration to Media Foundation: the GPU vendor's asynchronous hardware HEVC encoder (NVENC, Quick Sync, or AMF) first, and under Prefer Microsoft's software HEVC encoder from the HEVC Video Extensions next and the native encoder last; Require reports CodecSupport::HardwareUnavailable instead of falling back, and a lossless or fixed-QP configuration, which has no hardware form, is an invalid configuration for it. The Media Foundation backend encodes HEVC Main at a constant frame rate with the requested bitrate and keyframe interval, no B-frames (so every sample's DTS equals its PTS), and low latency. It accepts limited-range Rgba8, Bgra8, or Yuv420p8 CPU frames with even dimensions: RGB goes to a hardware encoder that takes ARGB32 as-is, so the colour conversion runs on the GPU, and otherwise converts to NV12 on the CPU, in both cases with the BT.601 studio-swing matrix the crate's decoder uses. Its Annex B output is rewritten as length-prefixed hvc1 samples, and the hvcC is built from the parameter sets the encoder actually wrote, so the stream muxes through Mp4Muxer and decodes through native_hevc_video_decoder_factory() and ffmpeg alike. finish() drains the encoder; dropping a pending encode() or finish() future cancels the stream, after which every call reports ErrorKind::Cancelled; and a lost GPU device is reported as ErrorKind::Graphics, after which a new encoder must be created. VideoEncoder::implementation() and VideoEncoder::backend_name() say which encoder a factory actually created -- the latter names the Media Foundation encoder, for example NVIDIA HEVC Encoder MFT. Linux has no platform HEVC encoder yet, so there Prefer selects the native encoder and Require is unavailable.

On macOS, Prefer and Require route a target-bitrate configuration to VideoToolbox's hardware HEVC encoder, through a VTCompressionSession created with hardware required; Prefer falls back to the native encoder where the host has none, and Require reports CodecSupport::HardwareUnavailable. It encodes HEVC Main in real-time mode at a constant frame rate with the requested bitrate and keyframe interval and no B-frames, so every sample's DTS equals its PTS. It accepts limited-range Rgba8 or Bgra8 CPU frames with even dimensions, top-down or bottom-up: the pixels are copied into BGRA pixel buffers - as they are for Bgra8, with the channels swapped for Rgba8 - and VideoToolbox converts them to YCbCr on the media engine with the BT.601 matrix, so no CPU colour conversion runs. Because config() has to declare the hvcC before the first frame and VideoToolbox only reveals its parameter sets once it has encoded something, creating the encoder encodes one black priming frame first and discards its sample. VideoToolbox's samples are already length-prefixed; the parameter sets are moved out of them into the hvcC, and sync flags come from the encoder. In real time VideoToolbox may drop a frame it cannot keep up with, and the sample before a dropped frame is lengthened to cover it, so decode timestamps stay contiguous. finish() drains the encoder, and dropping it unfinished cancels it. backend_name() reports VideoToolbox HEVC. On macos-latest CI it encodes 1080p at about 180 fps, six times real time.

Frame indices are zero-based. A video get(n) returns exactly frame n in presentation order, not merely the nearest keyframe. The matching audio get(n) returns the half-open sample interval covered by video frame n; sample-accurate APIs will also be available for audio-only use.

Runnable versions of the native GL and browser WebGL workflows above, driven against a real Big Buck Bunny MP4 sample, live in examples/.

Data paths

Video frames will support two families of destination/source:

  • CPU-backed planes with explicit pixel format, dimensions, stride, color space, and transfer metadata.
  • GPU-backed images associated with a GL texture or framebuffer on native targets and a WebGL texture or framebuffer in browsers.

GPU interop is capability-based. A backend may share an image without a copy, copy on the GPU, or fall back through CPU memory. Callers can require a particular behavior and receive an unsupported-capability error instead of an implicit expensive fallback.

Audio buffers carry their sample format, channel layout, sample rate, exact timeline interval, and any priming or padding information needed for gapless synchronization.

Platform expectations

Capability Native WebAssembly/browser
Input/output Files, memory, caller storage Blob, streams, memory, File System Access handles when supplied
Video acceleration Pluggable software or platform backend WebCodecs when available, otherwise the pure-Rust software decoder
Graphics OpenGL-family context supplied by caller WebGL context supplied by caller
Audio Raw buffers and pluggable device integration Web Audio buffers/nodes supplied by caller
Concurrency Worker threads where safe Async tasks; workers/threads only when browser isolation permits

Codec and hardware availability varies by browser and operating system. Opening media will report capabilities explicitly; container support does not imply that every platform can encode or decode every advertised codec.

The native HEVC Main backends each factory can select, in the order it tries them. Avoid always takes the pure-Rust backend; Require takes only the hardware ones.

HEVC Main Windows macOS Linux
Decode (native_hevc_video_decoder_factory) NVDEC (64-bit), Media Foundation (D3D11), then pure Rust VideoToolbox, then pure Rust NVDEC (64-bit), then pure Rust
Encode (native_hevc_video_encoder_factory) Media Foundation hardware (NVENC, Quick Sync, AMF), then Media Foundation software, then pure Rust VideoToolbox hardware, then pure Rust pure Rust

The decoder also takes HEVC Main 10 (CodecProfile::HevcMain10, 8- to 10-bit 4:2:0) on every platform, always in pure Rust: none of the hardware backends is wired for more than 8 bits per sample, so Require reports a Main 10 track unavailable.

The VP8 and VP9 decoders choose the same way:

Decode Windows macOS Linux
VP8 (native_vp8_video_decoder_factory) NVDEC (64-bit), Media Foundation (D3D11), then pure Rust pure Rust NVDEC (64-bit), then pure Rust
VP9 profile 0 (native_vp9_video_decoder_factory) NVDEC (64-bit), Media Foundation (D3D11), then pure Rust VideoToolbox, then pure Rust NVDEC (64-bit), then pure Rust

native_vp8_video_decoder_factory selects its backends the same way: NVDEC on 64-bit Windows and Linux hosts with an NVIDIA adapter that decodes VP8, then Media Foundation on Windows hosts whose adapter exposes the D3D11 VP8 decoder profile (Intel and some AMD drivers do; NVIDIA drivers do not) and that have a D3D11-aware VP8 decoder transform installed, such as the one in Microsoft's VP9 Video Extensions, then pure Rust, and pure Rust on macOS, where VideoToolbox has no VP8 decoder. The hardware backends' frames are cropped and converted to RGBA by the same code as the pure-Rust decoder's, so every backend returns the same pixels.

native_vp9_video_decoder_factory() decodes VP9 profile 0 (CodecProfile::Vp9Profile0, 8-bit 4:2:0) to Rgba8. Prefer and Require select a hardware backend: NVDEC on 64-bit Windows and Linux hosts with an NVIDIA adapter that decodes VP9, then Media Foundation on Windows hosts whose adapter exposes the D3D11 VP9 profile 0 decoder profile and that have a D3D11-aware VP9 decoder transform installed, such as the one in Microsoft's VP9 Video Extensions, then VideoToolbox on Macs whose media engine decodes VP9. Prefer falls back to pure Rust where none is available, Require reports HardwareUnavailable there, and Avoid always decodes in pure Rust. A chunk's headers are read before it reaches the hardware decoder, so every backend refuses the samples the pure-Rust decoder refuses and returns exactly one frame per sample, and the decoded picture is cropped to the size its header names and converted by the pure-Rust decoder's own conversion, so every backend returns the same pixels. NVDEC's parser never displays a show_existing_frame, so that backend reads the surface the named reference slot holds back itself. VideoToolbox never outputs a hidden frame, so that backend shows a hidden frame again by replaying the samples since the last key frame through the pure-Rust decoder, which encoders make it do rarely. The hardware backends decode a stream at its configured size only. It reads VP9 from MP4 vp09 sample entries, whose vpcC box is the track's decoder_config, and from WebM V_VP9 tracks, whose CodecPrivate becomes the same box (Vp9CodecConfig parses both), and takes each sample as one VP9 chunk: a frame, or a superframe of hidden frames followed by the one it shows. Every coding tool of the profile is decoded, including show_existing_frame, intra-only frames, reference frames of another size, tiles, segmentation and lossless coding, and the output matches libvpx bit for bit on every profile 0 test vector of libvpx's own test suite. A stream that changes resolution produces frames of each size, which ExactFrameReader refuses because it holds every frame to the track's configured size. Like the other software decoders, it decodes on the calling thread and suits thumbnails, previews and seeks rather than real-time playback of large frames.

Every native encoder, by operating system, and what does the encoding. A hardware encoder runs on the GPU or media engine. A platform encoder is the operating system's own codec; it reports CodecImplementation::Hardware because zvidlib runs none of the codec itself. A software encoder is zvidlib's own pure-Rust code and reports CodecImplementation::Software. VideoEncoder::implementation() and backend_name() report which one a created video encoder is.

Encoder Windows macOS Linux
HEVC Main (native_hevc_video_encoder_factory) Hardware: Media Foundation (NVENC, Quick Sync, AMF) for Prefer/Require target-bitrate configurations, with Prefer falling back to Media Foundation's software HEVC encoder. Software: pure Rust for Avoid, lossless, fixed QP, and the last Prefer fallback Hardware: VideoToolbox for Prefer/Require target-bitrate configurations (Rgba8/Bgra8, even dimensions). Software: pure Rust for Avoid, lossless, fixed QP, and the Prefer fallback Software: pure Rust; Require reports HardwareUnavailable
AV1 Main, 8-bit monochrome (native_av1_video_encoder_factory) Software: pure Rust Software: pure Rust Software: pure Rust
VP8 (native_vp8_video_encoder_factory) Software: pure Rust; Require reports HardwareUnavailable Software: pure Rust; Require reports HardwareUnavailable Software: pure Rust; Require reports HardwareUnavailable
VP9 profile 0 (native_vp9_video_encoder_factory) Hardware: Media Foundation (for example Quick Sync) for Prefer/Require where the adapter has a VP9 encoder. Software: pure Rust for Avoid, and for Prefer without one Hardware: VideoToolbox for Prefer/Require on Macs with a VP9 encoder. Software: pure Rust for Avoid, and for Prefer without one Software: pure Rust; Require reports HardwareUnavailable
AAC-LC (native_aac_audio_encoder_factory) Platform: Media Foundation AAC encoder MFT (44.1 or 48 kHz, mono or stereo) Platform: AudioToolbox None: reports HardwareUnavailable
Opus (native_opus_audio_encoder_factory) Software: opus-pure, a pure-Rust port of libopus (48 kHz, mono or stereo) Software: opus-pure Software: opus-pure
Vorbis (native_vorbis_audio_encoder_factory) Software: zvidlib's port of the libvorbis 1.3.7 encoder (mono or stereo, every rate libvorbis supports) Software: the libvorbis port Software: the libvorbis port
Audio decoder Windows macOS Linux
AAC-LC (NativeAacDecoder) Platform: Media Foundation AAC decoder MFT (mono or stereo) Platform: AudioToolbox (mono or stereo) Software: Symphonia's AAC decoder (mono or stereo)
Opus (NativeOpusDecoder) Software: opus-pure Software: opus-pure Software: opus-pure
Vorbis (NativeVorbisDecoder) Software: vendored Symphonia Vorbis decoder Software: vendored Symphonia Vorbis decoder Software: vendored Symphonia Vorbis decoder

The native audio decoders are pure Rust except AAC's on macOS and Windows: NativeAacDecoder runs the platform's AAC-LC decoder there, AudioToolbox on macOS and Media Foundation's AAC decoder MFT on Windows, and Symphonia's AAC decoder on Linux and the other native targets, which have no platform AAC decoder; it takes mono and stereo streams. NativeVorbisDecoder runs a copy of Symphonia's Vorbis decoder vendored in crates/zvidlib-vorbis-decoder/src/vorbis_decoder/ with its surround decoding fixed, for one to eight channels in the Vorbis channel order, and NativeOpusDecoder runs opus-pure, which passes all twelve RFC 8251 decoder conformance vectors in stereo and in mono. Opus is read from and written to MP4 (Opus sample entries with dOps) and WebM; Vorbis has no MP4 mapping, so it is read from and written to WebM, VorbisConfig parses and writes the Xiph-laced CodecPrivate WebM carries, and Mp4Muxer refuses a Vorbis track.

The browser build encodes through WebCodecs instead: AV1 Main, HEVC Main or VP9 profile 0 video and AAC-LC or Opus audio, wherever the browser provides those encoders (see Implemented browser boundary).

Repository layout

The repository contains a dependency-free portable core that validates native and WASM build configuration and implements the foundational timeline, media, I/O, frame-transfer, bounded MP4 reading, codec factory, exact-frame decoding, encoder, indexed-output, and seekable MP4 writing layers. Planned modules and dependency boundaries are described in ARCHITECTURE.md. Runtime dependencies will be added only with a documented portability, maintenance, size, and licensing rationale.

Using a release

zvidlib is published to crates.io, with API documentation on docs.rs. For a native build, depend on the zvidlib crate:

[dependencies]
zvidlib = { version = "<version>" }

with the latest version from crates.io in place of <version>, or run cargo add zvidlib. The workspace's zvidlib-* crates are published alongside zvidlib at the same version, but they are implementation crates that zvidlib re-exports; depend on zvidlib only. The native API requires Rust 1.85 or later, as recorded by rust-version in Cargo.toml; platform codec adapters retain their documented platform capability checks.

To pin a release tag, or to use a commit that is not yet on crates.io, use a Git dependency instead, with the release version in place of X.Y.Z:

[dependencies]
zvidlib = { git = "https://github.com/lsegal/zvidlib.git", tag = "vX.Y.Z" }

Each GitHub release also attaches the .crate archives it published, such as zvidlib-X.Y.Z.crate, for inspection or offline packaging.

Choosing codecs

Every native codec is a Cargo feature, one per codec and direction, and the default features enable all of them along with the platform hardware backends. An application that needs only some codecs turns the defaults off and names the ones it uses, and builds none of the others:

[dependencies]
# Decode VP9 only, in software.
zvidlib = { version = "<version>", default-features = false, features = ["vp9-decoder"] }
# Both AV1 directions and the HEVC encoder, with hardware acceleration.
zvidlib = { version = "<version>", default-features = false, features = ["av1", "hevc-encoder", "hardware"] }
Codec Decoder Encoder Both
AV1 av1-decoder av1-encoder av1
VP8 vp8-decoder vp8-encoder vp8
VP9 vp9-decoder vp9-encoder vp9
HEVC hevc-decoder hevc-encoder hevc
Opus opus-decoder opus-encoder opus
Vorbis vorbis-decoder vorbis-encoder vorbis
AAC aac-decoder aac-encoder aac
  • all enables every codec feature in the table. The default features are all and hardware.
  • hardware builds the platform backends (NVDEC, Media Foundation, VideoToolbox) for whichever enabled codecs have one. Without it the enabled codecs are pure Rust: HardwarePreference::Require reports HardwareUnavailable, and Prefer falls back to software, as it does on a host with no backend.
  • Features are additive and mix freely. The containers, the codec traits and the configuration records the containers read and write (OpusHead, VorbisConfig, the AV1, VP9 and HEVC syntax) are built in every configuration, so MP4 and WebM files with any track open regardless; a track whose codec is not enabled has no native decoder or encoder.
  • The AAC decoder runs on the operating system's decoder on macOS and Windows and on Symphonia's elsewhere; the AAC encoder needs macOS or Windows.
  • In a web build, a codec whose feature is off has no software fallback behind WebCodecs.
  • native gates no library code. It only keeps the native-only examples out of wasm32 builds, so applications do not need it for the native codecs.

The browser package is not on crates.io or npm: it is a GitHub release asset. For a browser build, download the zvidlib-web-vX.Y.Z.tgz asset from the latest GitHub release and install it:

npm install ./zvidlib-web-vX.Y.Z.tgz

The package contains zvidlib.js, zvidlib.d.ts, and zvidlib_bg.wasm generated by wasm-pack --target web; import it as an ES module and initialize it once as shown below. Browser and Cargo consumers must use the same release version when they exchange media or API contracts.

zvidlib follows pre-1.0 semantic versioning: patch releases preserve documented public API and behavior, while a minor release may make breaking API changes. Every release publishes its Rust crates to crates.io and attaches its browser package to the GitHub release; release notes call out any platform capability change.

Building the library

Install stable Rust with the wasm32-unknown-unknown target, then run:

cargo check --workspace --features native
cargo check --target wasm32-unknown-unknown --features web
cargo fmt --all -- --check
cargo clippy --workspace --all-targets --features native -- -D warnings

These commands validate the portable core on both targets. The repository is a Cargo workspace: zvidlib at the root, which is the only crate to depend on, and the codec, container and shared-core crates it is built from under crates/ (see ARCHITECTURE.md). Without --workspace or -p <crate>, a cargo command at the root covers the root package only; cargo test -p zvidlib-vp8 runs one crate's tests, cargo test --workspace --features native every package's. Native builds include accelerated HEVC Main decode through NVDEC on supported 64-bit Windows/Linux systems, Media Foundation on Windows, and VideoToolbox on macOS, with a dependency-free pure-Rust fallback. They also include the pure-Rust AV1 Main decoder (8-bit 4:2:0 colour and monochrome streams with every Main-profile coding tool and in-loop filter, verified frame-for-frame against an independent libdav1d decode), the pure-Rust VP9 profile 0 decoder (verified against libvpx's per-frame digests of its own test vectors), a dependency-free VP9 profile 0 encoder with inter frames (verified pixel for pixel against ffmpeg's VP9 decoder and libvpx), a VP8 decoder (native_vp8_video_decoder_factory, every bitstream version, bit-exact with libvpx on its VP8 test vectors, including hidden alternate-reference frames) that uses NVDEC or Media Foundation where available and pure Rust otherwise, dependency-free HEVC Main and monochrome AV1 Main-profile encoders (lossless, plus a non-lossless path verified against both the crate's own decoder and an independent ffmpeg decode), AAC-LC, Opus and Vorbis decode, pure-Rust Opus and Vorbis encoders, and default-device PCM output described above. Native builds also encode HEVC Main in hardware through VideoToolbox on macOS and Media Foundation on Windows, and AAC-LC through AudioToolbox on macOS and Media Foundation on Windows. The encoder table under Platform expectations lists every encoder by operating system. The browser build reads exact AAC, Opus and Vorbis sample ranges through WebCodecs AudioDecoder with software Opus and Vorbis fallbacks, and encodes AAC-LC, Opus or Vorbis through its own bridge (see Implemented browser boundary). Color AV1 encoding beyond the monochrome profile and fully portable audio-device abstractions remain planned.

Native compressed-codec backends use the public VideoDecoderConformanceVector and VideoEncoderConformanceVector runners before registration. Decoder vectors pin canonical SHA-256 fingerprints for every presentation frame and are tested under sequential, reverse, and seek-heavy access. Encoder vectors validate standard configuration and packet timing, then decode through an independently conforming backend and enforce an explicit PSNR floor. This gives HEVC and AV1 work finite, reusable acceptance targets without delegating codec behavior to an external library.

The VP9 decoder is additionally checked against every VP9 profile 0 test vector libvpx's own test suite decodes, which CI downloads from the WebM project. To run that check locally, download the vectors named in crates/zvidlib-vp9-decoder/tests/fixtures/libvpx_vp9_test_vectors.txt, each with its .md5 file, from https://storage.googleapis.com/downloads.webmproject.org/test_data/libvpx/ into one directory, then:

ZVIDLIB_VP9_VECTORS=/path/to/vectors cargo test -p zvidlib-vp9-decoder --lib vp9_dec::tests::libvpx_test_vectors -- --ignored

Building, testing, and using WebAssembly

The web feature excludes native-only integrations. During scaffold development, install the Rust target and verify it directly:

rustup target add wasm32-unknown-unknown
cargo check --target wasm32-unknown-unknown --features web

Install wasm-pack and create a browser-ready ES module package:

cargo install wasm-pack
wasm-pack build --target web --out-dir pkg --features web
python -m http.server 8000

The build creates pkg/zvidlib.js, pkg/zvidlib.d.ts, pkg/zvidlib_bg.wasm, and package metadata. Open http://localhost:8000/ and import the generated module with import init, { ... } from "./pkg/zvidlib.js", as in the browser examples. Call await init() exactly once before using an exported class. Deploy the generated JavaScript and WASM files together, with the server returning application/wasm for the .wasm file.

For the browser example specifically, examples/web_canvas/package.json wraps the wasm-pack build and serving steps above into one pnpm dev command; see examples/README.md.

Run the browser integration suite in an installed Chrome browser with:

wasm-pack test --headless --chrome --features web

The suite verifies Blob and stream input, cancellation, reader-lock cleanup, BigInt range handling, typed-array copy lifetimes, browser-object ownership, stable errors, Blob output, decoding the bundled HEVC sample and a VP9 track through WebCodecs, and the software fallback for browsers whose WebCodecs cannot decode a track. The base build does not require WASM threads or cross-origin isolation. Future optional threaded builds will document their additional headers and browser requirements separately.

The web feature's video decoder uses web-sys's WebCodecs bindings, which that crate gates behind --cfg=web_sys_unstable_apis because the spec is still evolving. .cargo/config.toml sets that flag for the wasm32-unknown-unknown target automatically, so the cargo/wasm-pack commands above need no extra flags.

Roadmap

  1. Specify and test timeline arithmetic, frame/audio value types, errors, and capability discovery.
  2. Build incremental MP4 parsing and writing with deterministic fixture tests.
  3. Add decoder/encoder traits and one end-to-end decode path before expanding codec coverage.
  4. Add native GL and WebGL transfer backends, then synchronized playback and recording adapters.
  5. Stabilize Rust and JavaScript APIs after cross-platform conformance testing.

See ARCHITECTURE.md for component boundaries, seeking and caching behavior, MP4 responsibilities, and portability constraints.