alhazen-core

A native media engine for Rust. Give it a file or a URL; it hands you video frames on time and plays the sound. Pure-Rust decoders everywhere, the operating system's GPU decoders where available, and no GStreamer, bundled ffmpeg or browser engine.

alhazen-core has no UI dependency. Use it to build a video player in any toolkit: GPUI (see alhazen-gpui), egui, iced, Slint, a game engine, or your own renderer.

Named after Ibn al-Haytham (Alhazen, c. 965–1040), whose Book of Optics first explained how images form, the science behind every camera and screen.

use alhazen_core::{Player, PlayerConfig, Source};

let player = Player::open(Source::parse("https://example.com/movie.mkv")?, PlayerConfig::default())?;
player.play();
// In your render loop:
if let Some(frame) = player.current_frame() { /* draw it */ }

Features

• Plays out of the box, on every platform: AV1, VP9 (multi-threaded), VP8, ProRes, Opus, Vorbis, FLAC, AAC (LC, HE-AAC v1/v2), MP3 and PCM in pure Rust, under permissive licences.
• Music files too: .mp3, .aac, .flac, .wav, .ogg and .opus play natively, with their tags and cover art (Player::metadata()).
• The OS's own decoders on Windows: H.264, HEVC, VP9 and AV1 on the GPU, plus AAC, MP3, AC-3, E-AC-3 and ALAC, through Media Foundation. Nothing to install; Windows covers the codec licences.
• NVIDIA GPUs on Linux: H.264, HEVC, VP8, VP9 and AV1 (8- and 10-bit) decode on the GPU through NVDEC, scaled to your display size on the GPU. Loaded from the driver at runtime: nothing to install or build.
• Everything else through the user's ffmpeg, if they have one: found at runtime and run as a separate program, so nothing is linked and ffmpeg's licence never touches your app.
• Files and the web: local files, file://, and HTTP(S) streaming with Range seeking and automatic reconnects.
• HLS streaming, VOD and live: .m3u8 playlists with MPEG-TS, fragmented MP4 or packed-audio segments, audio in a separate playlist (even on another host), adaptive bitrate with manual override, and AES-128. Demuxed natively: no ffmpeg needed.
• A/V sync done right: audio drives the clock; video follows, drops late frames and catches up after stalls. The encoder's start-up padding (MP4 edit lists, Matroska CodecDelay) is trimmed, so sound lines up with the picture, and MP3/Opus/Vorbis files play gapless (LAME delay and padding, Ogg granules).
• Smooth on big files: frames are scaled to the size you display them at before colour conversion, and a decoder that can't keep up hands over to a faster one automatically.

Contents

• Getting started
• Integrating with a UI toolkit
• Supported formats
• How decoding is chosen
• Cargo features
• Configuration
• Events and state
• Environment variables
• Platform requirements
• Performance
• Development
• License

Getting started

[dependencies]
alhazen-core = "0.5"

# Decoding unoptimized is many times slower than real time: optimize the decoders even in
# debug builds.
[profile.dev.package.rav1d]
opt-level = 3
[profile.dev.package.vp9-mt]
opt-level = 3

use std::time::Duration;
use alhazen_core::{Player, PlayerConfig, PlayerEvent, Source, VideoFrame};

fn main() -> alhazen_core::Result<()> {
    let player = Player::open(Source::parse("movie.mkv")?, PlayerConfig::default())?;
    let events = player.events(); // crossbeam Receiver<PlayerEvent>
    player.play();

    loop {
        // The frame due now, by the audio clock. Call it every time you draw.
        if let Some(VideoFrame::Cpu { width, height, bgra, pts }) = player.current_frame() {
            // bgra: tightly packed BGRA, width * 4 bytes per row. Upload it and draw.
        }
        for event in events.try_iter() {
            match event {
                PlayerEvent::Warning(w) => eprintln!("warning: {w}"),
                PlayerEvent::Ended => return Ok(()),
                _ => {}
            }
        }
        std::thread::sleep(Duration::from_millis(4));
    }
}

Player is Send + Sync; keep it in an Arc and call it from any thread.

Integrating with a UI toolkit

A player in any toolkit comes down to three things:

1. Draw current_frame() every frame while playing. It returns the frame due at the current audio position. Cache the texture and re-upload only when pts changes.
2. Tell the engine how big you draw: player.set_max_output_size(Some((w, h))) with the view's size in physical pixels. 4K frames shown in a small view are then scaled down before colour conversion, which saves most of the CPU time.
3. Listen to events() to repaint when paused (FrameReady), and to show Warnings and errors.

alhazen-gpui is a complete example: about 450 lines for the player entity and the element.

Supported formats

Sources

Containers

Containers are recognised by their content, not their extension. Raw video streams (.h264, .hevc) need a container for now.

HLS

Source::parse("https://host/master.m3u8") plays HTTP Live Streaming through the same Player:

• Playlists: master playlists (variants, EXT-X-MEDIA audio renditions) and media playlists; VOD seeks like a file; live streams start three target durations behind the edge, follow the playlist, and end when it stops updating (live streams are not seekable).
• Segments: MPEG-TS, fragmented MP4 / CMAF (EXT-X-MAP), and packed audio (ADTS/MP3 with an ID3 timestamp), with byte ranges and discontinuities (a new resolution or codec mid-stream reopens the decoder).
• Separate audio: variants whose audio lives in its own playlist (AUDIO="group"), possibly on another host, play in sync with the video; Player::audio_renditions() and set_audio_rendition(i) switch between languages.
• Quality: adaptive bitrate by default (throughput weighted by download time, buffer-aware, one step up at a time, straight down when the link drops); Player::variants(), current_variant() and set_variant(Variant::Index(i)) / set_variant(Variant::Auto); PlayerEvent::VariantChanged(i) reports switches.
• Encryption: METHOD=AES-128. DRM (SAMPLE-AES, FairPlay, Widevine, PlayReady) is refused with Error::Unsupported.
• Robustness: segment downloads retry with backoff; a missing segment fails VOD with an error naming it and is skipped on live streams; dropping the player never waits on a stalled download.

Not yet: subtitles (WebVTT renditions), low-latency HLS, DASH.

Tags and cover art

Player::metadata() returns the title, artist, album, album artist, track, year, genre and the embedded cover (still encoded, with its MIME type) from ID3v2/ID3v1, Vorbis comments, FLAC pictures, MP4 ilst, Matroska tags and attachments, and RIFF INFO.

# use alhazen_core::{Player, PlayerConfig, Source};
let player = Player::open(Source::parse("song.mp3")?, PlayerConfig::default())?;
if let Some(m) = player.metadata() {
    println!("{} — {}", m.artist.as_deref().unwrap_or("?"), m.title.as_deref().unwrap_or("?"));
}
# Ok::<(), alhazen_core::Error>(())

Video

¹ Used when the GPU decodes it and prefer_hardware is on (the default); otherwise the pure-Rust decoder is used. ² Needs the HEVC Video Extensions from the Microsoft Store (preinstalled on many PCs).

8- and 10-bit video is supported. Frames are delivered as 8-bit BGRA, converted with the stream's BT.601/BT.709 matrix and range.

Audio

Audio is resampled to the output device's rate and mixed to its channel layout.

If a file's audio can't be decoded, the video still plays and a Warning explains why. If its video can't be decoded, Player::open fails with Error::UnsupportedCodec, listing the backends it tried.

How decoding is chosen

Each stream is offered to the backends in priority order; the first that claims it decodes it.

• PlayerConfig::prefer_hardware = false puts native before the GPU decoders (media-foundation, nvdec) for VP8/VP9/AV1.
• A decoder that can't decode a stream hands it to the next backend automatically, e.g. a GPU without 10-bit H.264 or 4:4:4 support. GPU decoders don't even take such streams when the file's setup data shows the variant up front.
• Automatic fallback: if a decoder falls behind (more than 100 ms late for 1.5 s), the stream switches once to the next backend that can decode it, at the same position. auto_fallback = false turns this off.
• backend_order = Some(vec!["ffmpeg-cli", "native"]) tries backends in your order. For full control, build a backend::Registry and pass it as PlayerConfig::registry.
• player.stats().video_backend tells you which one is decoding.

ffmpeg is looked for in PlayerConfig::ffmpeg.path, then $ALHAZEN_FFMPEG, then PATH, and on macOS also Homebrew's and MacPorts' locations (Finder-launched apps don't get the shell's PATH). On Windows it runs inside a Job Object, so closing the player also kills the real ffmpeg behind a Chocolatey/Scoop shim.

Cargo features

Configuration

PlayerConfig (all fields public; start from PlayerConfig::default()):

Events and state

PlayerState: Loading → Buffering → Playing ⇄ Paused → Ended, or Error(e) at any point. Seeking passes through Buffering.

Environment variables

Platform requirements

• Rust 1.92+.
• x86-64 with the native feature: nasm on PATH; rav1d assembles its fast AV1 code with it (apt install nasm, brew install nasm, winget install NASM.NASM).
• Linux with audio-output: ALSA headers (libasound2-dev / alsa-lib).
• Windows: Windows 10 or 11. Media Foundation is missing from "N" editions until the Media Feature Pack is installed; the engine then falls back to the other backends.
• Linux GPU decoding: an NVIDIA GPU and its proprietary driver; nothing else at build or run time. Without it, playback uses the other backends. (Intel/AMD through VA-API is planned.) The GPU is only woken up once a video needs it, so audio-only playback leaves a laptop's discrete GPU asleep.
• ffmpeg (optional, at runtime): 4.0+. CI tests 6–9 on Linux, Windows and macOS.

Performance

• VP9: vp9-mt decodes tile columns in parallel, bit-identical to libvpx: 45 fps vs 17.5 fps single-threaded on a 4K 10-bit stream.
• GPU on Windows: H.264/HEVC/VP9/AV1 decode on the GPU's video engine; frames are copied back once (NV12/P010) and converted on the CPU.
• GPU on Linux (NVIDIA): frames are decoded and scaled to the display size on the GPU, so only display-sized frames are copied back and converted.
• Scale first: a 4K frame shown at 1280×720 is scaled down in YUV before conversion.
• Catch-up: after a stall, frames more than 50 ms late are skipped (one is still shown per 100 ms), so video rejoins the audio quickly.

Development

cargo test                                        # everything (ffmpeg tests skip without ffmpeg)
cargo clippy --all-targets -- -D warnings
python3 scripts/fetch_vp9_vectors.py target/vp9-vectors
VP9_VECTORS_DIR=target/vp9-vectors cargo test --release --test vp9_conformance

• Test fixtures and the ffmpeg commands that made them: tests/fixtures/README.md.
• On Windows, cargo run --release --example mf-check -- tests/fixtures reports, per codec, which Media Foundation decoder ran (GPU or software) and whether it decoded correctly. mf-check --native-type <file> prints the media types Windows' own demuxer reports.
• CI runs on Linux, Windows and macOS, and checks that the default build stays permissively licensed.

License

Apache License, Version 2.0 (LICENSE). If you distribute software that contains alhazen-core, include the NOTICE text (Apache-2.0 §4(d)), for example in an "About" or "Licenses" screen.

The default build depends only on permissively licensed crates (MIT, Apache-2.0, BSD). AAC patents: AAC-LC's have largely expired; HE-AAC and Parametric Stereo patents may still apply in some countries, and rusty_aac's licence grants no patent rights. Apps with that concern can turn off native-aac. ffmpeg-cli links nothing: ffmpeg's licence applies to the ffmpeg program the user installed, not to your app; don't bundle an ffmpeg build without checking its licence and patent terms.

Part of Alhazen.