# media-pp
A small, GStreamer-flavored media pipeline library in Rust, built on
[`ffmpeg-next`](https://github.com/zmwangx/rust-ffmpeg). `lib/` is the
library (crate name `media-pp`); `examples/` holds one independent crate
per demo pipeline, grouped into subdirectories by theme (`core/`,
`render/`, `rtsp/`, `vision/`, `webrtc/`).
## Architecture
Everything is built from a handful of primitives in `lib/src/`:
- **`Element`** (`element.rs`) — a named node in the graph. Just identity;
says nothing about input/output.
- **`Sink`** (`element.rs`) — anything that can receive a buffer
(`consume(&mut self, buf: MediaBuffer)`). The only "connection"
primitive in the pipeline. By default, consuming a buffer is a plain
function call on the caller's thread — zero overhead.
- **`Source`** (`element.rs`) — anything with one or more output ports
(`src_pads()`). An element with more than one src pad *is* a tee — fan-out
needs no separate primitive (see `FileDemuxer`, which exposes one pad per
container stream).
- **`SourceElement`** (`element.rs`) — a pure source that drives its own
thread via `run()` (e.g. `FileDemuxer`, wrapping blocking file I/O).
- **`Filter`** (`element.rs`) — anything that's both a `Source` and a
`Sink` (decoder, pacer, ...). Auto-implemented for any `T: Source + Sink`;
no separate "processing element" trait needed.
- **`SrcPad`** (`pad.rs`) — an output port with exactly one runtime peer.
Applications connect it through `Context::attach`, so the runtime peer
and topology graph cannot diverge.
- **`MediaBuffer`** (`buffer.rs`) — the unit of data flowing between
elements (`Packet` / `Video` / `Audio` / `Eos`). Payloads are
`Arc`-wrapped, so cloning a buffer (e.g. to fan it out) is a refcount
bump, never a copy of the encoded/decoded data.
- **`Queue`** (`queue.rs`) — the explicit thread boundary between stages
in a dataflow chain.
Wrapping a `Sink` in a `Queue` hands buffers off through a bounded
channel to a dedicated worker thread. Directly-linked stages stay on
their upstream caller's thread; the top-level `Pipeline` and
`DriverRunner` separately own the background threads that drive a source
or driver.
- **`Bus` / `BusEvent`** (`bus.rs`) — a cross-thread event channel. Once a
buffer crosses a `Queue` boundary, errors can't propagate with `?`
anymore, so they're posted here instead (`Error`, `Eos`, `Dropped`).
`BusReceiver::iter_with_ids()` pairs each event with its stable graph
`ElementId`; `log_events()` drains and prints them in a default format.
- **`PpLog`** (`pp_log.rs`) — the contextual log identity stored privately by
every element. `pp_info!`, `pp_debug!`, `pp_warn!`, `pp_error!`, and `pp_trace!` attach
that identity to records sent to the opt-in private file logger;
`element_pp_log()` keeps `pipeline_id`, element type, and instance name as
separate fields used throughout the graph.
- **`Pipeline` / `ChainBuilder` / `PipelineBuilder`** (`pipeline.rs`) —
`ctx.branch()` builds one linear, detached chain (`.pipe(filter)` for
same-thread stages, `.queue(name, capacity)` for a thread boundary,
`.to(sink)` to terminate). `ctx.attach(source, pad, branch)` commits the
runtime connection and graph in one operation. `Pipeline::run()` drives
a `SourceElement` on a background thread and returns immediately;
draining its bus waits for that source and every reachable `Queue`
worker to finish, provided the application has not retained an extra
`Context`/`Bus` sender. `Pipeline::new` is the
single-source case; `PipelineBuilder::new(id).add_source(source, wire)…`
combines more than one independent `SourceElement` (e.g. a video capture
and an audio capture both feeding one `Mp4Muxer`) into one `Pipeline` —
each source gets its own thread, but they share one `Bus`/`Clock`/
graph, and `run`/`pause`/`resume`/`stop`/`seek` reach every source
from a single call.
- **`PipelineGraph`** (`graph.rs`) — the live node/edge graph. Elements,
edges, and dynamic branches use stable `ElementId`/`EdgeId`/`BranchId`
values; names are display labels only. `Pipeline::graph()` returns a
revisioned, consistent snapshot, and `Pipeline::topology()` renders it.
A detached branch never appears until attachment succeeds.
- **`Clock`** (`clock.rs`) — a shared wall-clock anchor (`Arc<Clock>`) so
multiple `Pacer`s (e.g. one per stream) agree on the same t=0.
- **`PlaybackClock`** (`playback_clock.rs`) — a shared media-position clock.
Video-only playback derives it from `Clock`; a bound audio renderer can
take over with its actual played-sample position without moving the
timeline backwards.
## Elements (`lib/src/elements/`)
One-line index only — each element's own doc comment (`cargo doc --open`)
has the full rationale (why it's built the way it is, what to watch out
for); this table isn't meant to duplicate that.
### Sources
| Element | What it does |
|---|---|
| `FileDemuxer` | Demuxes a file; one src pad per container stream |
| `AppSource` | Application code pushes buffers in via a handle, from any thread — GStreamer's `appsrc` equivalent |
| `RtspSource` | Demuxes a live RTSP stream (the receive counterpart to `RtspSink`) — no internal retry/reconnect on a dropped connection, fails fast instead; the caller rebuilds a fresh one to reconnect |
| `TestVideoSource` | Generates a synthetic moving-gradient `Pixel::YUV420P` stream — GStreamer's `videotestsrc` equivalent, no file/camera/decoder needed |
| `TestAudioSource` | Generates a synthetic sine-tone `Sample::F32(Packed)` audio stream — the audio sibling of `TestVideoSource`, no file/microphone/decoder needed |
| `DxgiCaptureSource` (`dxgi-capture`) | Captures the desktop live via DXGI Desktop Duplication — GStreamer's `d3d11screencapturesrc` equivalent. Pushes `Pixel::BGRA` untouched (chain a `Scaler` for YUV420P); emits at a constant `fps` (default 30, same convention as `TestVideoSource`) rather than one push per real desktop change — repeats the latest captured image if nothing changed since the last tick, since a variable-rate/push-on-change version of this turned out to cause visible judder against a vsync-locked renderer. `CaptureMode::Cpu` (default, optional cursor compositing) or `CaptureMode::Gpu` — the GPU mode resolves the capture adapter, creates its own `ID3D11Device`, and returns that device from `open()` so the renderer and other D3D11 stages can share it; capture then emits zero-copy `Pixel::D3D11` textures with no `Map`/CPU pixel copy (no cursor support yet in this mode) |
| `WasapiCaptureSource` (`wasapi-capture`) | Captures audio live via WASAPI — either a playback endpoint's own outgoing mix (loopback, i.e. system audio — the audio counterpart to record alongside `DxgiCaptureSource`) or a microphone, picked from `WasapiCaptureSource::list_devices()` |
| `AudioMixer`¹ | Live-mixes any number of inputs, attachable/detachable while running via `MixerHandle::add_source`/`remove_source` (`add_source` returns a terminal `Sink` that a different pipeline can pass to `ctx.branch().to(...)`) — the fan-in counterpart to `Tee`'s fan-out |
| `VideoCompositor` | Composites the latest frames from independently-driven input pipelines into a fixed-rate BGRA output, entirely on the CPU via `libswscale` + a hand-written alpha blend. Each latest-frame slot uses an atomic `ArcSwapOption` instead of a Mutex; `add_source` returns both a terminal Sink and a `VideoLayerHandle` for runtime position, size, opacity, visibility, fit, and z-order changes. `VideoRect`/`VideoLayer`/`VideoFit` and the `layer_geometry` math live in a shared `video_layer` module; colors use the crate-wide `Color` type so `D3d11VideoCompositor` uses the exact same layer-control API |
| `D3d11VideoCompositor` (`d3d11-renderer`) | The GPU sibling of `VideoCompositor` — same `add_source`/`VideoLayerHandle` API, but every input must already be a `Pixel::D3D11` texture (BGRA or NV12) and compositing happens via a D3D11 pixel shader into an offscreen render target, never touching the CPU. Draws each layer as a screen-covering triangle (no vertex buffer) clipped by viewport+scissor to its `VideoRect`; output textures are returned to a growable pool only after the last downstream frame reference is dropped, so queued/Tee'd frames are never overwritten. NV12 conversion follows each frame's color-space/range metadata (with an SD=BT.601, HD=BT.709 limited-range fallback when unspecified) |
| `WebRtcPeer` (`webrtc`) | Drives one str0m `Rtc` session on its own thread. Not a `Pipeline` source itself — `WebRtcHandle::add_track`/`next_track()` mint a `WebRtcTrackSink`+`WebRtcTrackSource` pair per track (see below), symmetric for tracks either side added, so one `Direction::SendRecv` track carries both directions |
| `WebRtcTrackSource` (`webrtc`) | The receive side of one WebRTC track — a plain `SourceElement`, same shape as `AppSource`; obtained via `WebRtcHandle::next_track()`, not constructed directly |
¹ Each input is driven from wherever it was attached — typically a *different* `Pipeline`/thread than the one the `AudioMixer` itself is the source of, which is the whole point (e.g. a capture pipeline feeding a mixer that another pipeline reads from). For combining a fixed, known-up-front set of live sources into one output instead (no dynamic attach/detach needed), see `PipelineBuilder` — a simpler fit for e.g. one video capture + one audio capture feeding a single `Mp4Muxer`.
### Filters
| Element | What it does |
|---|---|
| `SwDecoder` | Decodes `Packet`s into `Video`/`Audio` frames (software) |
| `D3d12vaDecoder` (`d3d12-renderer`) | Decodes into GPU-resident `Video` frames via D3D12VA hardware acceleration |
| `D3d11Decoder` (`d3d11-renderer`) | Decodes into GPU-resident `Video` frames via D3D11VA hardware acceleration — the D3D11 sibling of `D3d12vaDecoder`. `extra_hw_frames` matters here in a way it doesn't for D3D12: D3D11VA's decode surface pool is fixed-size, sized once at open time, so it must cover the deepest downstream queue/buffer or decode itself starts failing once the pool runs out |
| `D3d11Upload` (`d3d11-renderer`) | Uploads CPU-resident `Pixel::NV12` frames to a GPU-resident `Pixel::D3D11` texture — the D3D11 sibling of `D3d12Upload`. Doesn't go through FFmpeg's own hwframe-pool machinery at all (an earlier version that did corrupted memory); builds the `ID3D11Texture2D` directly via plain `windows-rs` calls instead |
| `D3d11Download` (`d3d11-renderer`) | The mirror of `D3d11Upload` — downloads a GPU-resident `Pixel::D3D11` BGRA texture (e.g. from `D3d11VideoCompositor`) back to a CPU-resident `Pixel::BGRA` frame via `CopySubresourceRegion`/`Map` into a cached staging texture, including a selected slice of a texture array. Needed because `SwEncoder` is software-only and has no zero-copy GPU input path; chain a `Scaler` after this for whatever pixel format the encoder actually needs |
| `SwEncoder` | Encodes `Video` frames into `Packet`s (software only) — `VideoCodec` picks H.264/H.265/VP8/VP9/AV1 across GPL (`libx264`/`libx265`) and non-GPL (`libopenh264`/`libkvazaar`/`libvpx`/`libaom-av1`/`libsvtav1`) encoders; fails with a clear error, not a panic, if the linked ffmpeg build doesn't have the one you asked for |
| `SwAudioEncoder` | Encodes `Audio` frames into `Packet`s (software `aac`) — resamples to whatever format/channel layout the codec actually needs, built lazily from the first frame it sees |
| `AudioResampler` | Converts decoded `Audio` sample format/rate/channels through `libswresample`; its explicit input time base preserves the media PTS across conversion, and it flushes delayed samples at EOS |
| `AudioVolume` | Applies runtime-adjustable gain/mute through `AudioVolumeHandle`; uses a configurable 10 ms default ramp to prevent clicks and preserves the input audio format/timestamps |
| `Pacer` | Releases buffers at real playback speed (PTS + a shared `Clock`) — `new` rejects an invalid `time_base` with a typed `PacerError` rather than panicking, since it comes from a demuxed/externally supplied stream |
| `VideoSynchronizer` | Replaces `Pacer` for A/V playback: uses the pipeline wall clock for video-only playback, then automatically waits/drops video against a registered audio playback master |
| `Scaler` | Converts pixel format and resizes `Video` frames in one pass (`libswscale`) |
| `Tee`² | Fans one input out to multiple branches; `TeeBuilder` defines the initial fan-out and `TeeHandle::attach`/`detach` changes runtime branches by stable `BranchId` |
² Doesn't actually implement `Source` — its pads live in individually locked branch slots instead of a plain `&mut [SrcPad]`. `consume` only holds the branch-list lock long enough to clone an `Arc` snapshot, so a slow downstream does not block unrelated `TeeHandle::attach`/`detach` operations. Detach prevents a push that has not started yet; one already executing downstream call may finish. See its own doc comment.
### Sinks
| Element | What it does |
|---|---|
| `FrameCounter` / `PacketCounter` | Count decoded frames / raw packets, expose the count via `Arc<AtomicUsize>` |
| `Mp4Muxer`³ | Muxes one or more `Packet` streams — encoder output (`SwEncoder`/`SwAudioEncoder`) or a `FileDemuxer`'s own streams for a pure remux — into an MP4 file, one or more tracks |
| `SegmentedMp4Muxer`⁴ | Same shape as `Mp4Muxer`, but cuts to a new file every so often (`SegmentPolicy::Duration`) instead of writing one file for the whole recording — e.g. `rec_000.mp4`, `rec_001.mp4`, ... — so a crash mid-recording only loses the currently-open segment |
| `HlsMuxer`⁵ | Muxes one or more encoded packet streams into an HLS media playlist with MPEG-TS or fMP4 segments; supports sliding live windows, EVENT/VOD playlists, atomic manifest replacement, and optional deletion of expired live segments |
| `D3d12Renderer` (`d3d12-renderer`) | Submits frames to a `D3d12FrameRenderer` impl — zero-copy for `D3d12vaDecoder`'s frames. `media-pp` only defines the trait (plus `RawPlane`/`SubmitError`); the actual DX12 window rendering lives in `examples/render/render_common`'s own `D3d12WindowRenderer` |
| `D3d11Renderer` (`d3d11-renderer`) | Submits frames to a `D3d11FrameRenderer` impl — zero-copy for `D3d11Upload`/`D3d11Decoder`/`DxgiCaptureSource`'s GPU mode. No fence, no `keep_alive` (unlike `D3d12FrameRenderer`): every producer in this crate's D3D11 stack shares one `ID3D11Device`+context, and D3D11's own driver-deferred resource destruction means the runtime — not this crate — keeps a texture alive for as long as the GPU still needs it. `examples/render/render_common`'s own `D3d11WindowRenderer` is the concrete implementation |
| `WasapiRenderer` (`wasapi-renderer`) | Plays decoded audio through a WASAPI shared-mode render endpoint. `open()` returns the endpoint's `AudioFormat`; place `AudioResampler` before it and a `Queue` at the blocking device boundary |
| `RtspSink` | Publishes one compressed packet stream to an already-running RTSP server; it remuxes rather than re-encoding and works with any server that accepts RTSP publishing |
| `AppSink` | Hands buffers (and, optionally, control messages) to plain closures — GStreamer's `appsink` equivalent |
| `OrtDetector` (`ort`) | Runs a YOLOv8/v11-style ONNX model on each frame via `ort`, hands decoded/NMS-filtered detections to a closure |
| `WebRtcTrackSink` (`webrtc`) | The send side of one WebRTC track — `consume()` hands off to its `WebRtcPeer`'s own thread; handed out by `WebRtcHandle::next_track()`, not `WebRtcHandle::add_track` (which only returns a `TrackId`) |
³ Not a plain `Sink` itself — `Mp4Muxer::create`/`add_stream`/`open` is a two-phase builder, since a container's header has to describe every track's codec parameters before it can be written at all. `create` opens the file, `add_stream` registers one track at a time (name + `codec::Parameters` + `time_base`), and `open` writes the header and returns one real `Sink` per track, in registration order — all sharing one lock around the file, so tracks fed from independently-threaded branches (e.g. one video encode chain, one audio encode chain) can write concurrently without racing. The trailer is written once *every* track reports done (`Eos` or `Stop`), not on whichever finishes first. See its own doc comment, and `PipelineBuilder` for wiring two independent live sources (e.g. video + audio capture) into the tracks it expects.
⁴ Same two-phase builder shape as `Mp4Muxer` (`create`/`add_stream`/`open`), plus a naming closure (`FnMut(u64) -> PathBuf`, called with the segment index) instead of one fixed path. A rotation only actually cuts once the configured duration has elapsed *and* the video track's next packet is a keyframe — never mid-GOP — so every segment file is independently decodable from its own frame 0, closing the outgoing segment (writing its trailer) via the exact same all-tracks-report-done mechanism `Mp4Muxer` already uses for a normal `Eos`/`Stop`. Building this is what surfaced a real gap in `SwEncoder`: it now always sets a ~2-second keyframe interval itself, since at least one codec (`libopenh264`) was found to otherwise go an entire recording without a second keyframe against smoothly-changing content — relying on scene-change detection alone, which would have meant a `SegmentedMp4Muxer` using it might never rotate at all.
⁵ `HlsOptions::new` defaults to live fMP4 with two-second target segments and a six-entry sliding window. `HlsMode` selects live/EVENT/VOD behavior and `HlsSegmentFormat` selects fMP4 or MPEG-TS. Like the other muxers, `HlsMuxer::open` returns one sink per registered track and writes `#EXT-X-ENDLIST` only after every track reports `Eos` or `Stop`. Segment timing is media-timestamp based inside FFmpeg's HLS muxer; video segments cut on keyframes, so the encoder GOP should be close to the requested segment duration.
## Examples (`examples/`)
Each is its own crate so per-example dependencies (e.g. `winit` for
`sw_decode_render`) don't leak into the others. Every file-based example takes
its media path as a required argument — there is no default and no media is
checked in, so supply your own file; run one with no arguments to see its usage
line. Live capture, WebRTC, and synthetic-source examples have their own
arguments or need none.
### Core concepts
| Crate | Pipeline | Demonstrates |
|---|---|---|
| `decode` | Demux → SwDecoder → FrameCounter | `SwDecoder` actually decodes, direct (same-thread) chaining |
| `probe` | Demux → Queue → PacketCounter | An explicit `Queue` thread boundary |
| `fanout` | Demux → {Queue → PacketCounter} × 2 | Multi-pad fan-out at the source (video + audio to separate branches) |
| `pace` | Demux → SwDecoder → Queue → Pacer → FrameCounter | `Pacer` releasing frames at real playback speed — compare its `wall time` output against `decode`'s near-instant run |
| `tee` | Demux → Tee → {SwDecoder → FrameCounter, PacketCounter} | `TeeBuilder` committing a fixed initial fan-out as one subgraph |
| `dynamic_tee` | TestVideoSource → Tee → {FrameCounter, runtime FrameCounter} | `TeeHandle` attaching and detaching a branch while frames flow |
| `app_sink` | Demux → SwDecoder → AppSink | Same chain as `decode`, but the terminal sink is a plain closure instead of a bespoke `FrameCounter` |
| `app_source` | AppSource → SwDecoder → FrameCounter | A background thread feeds packets in via `AppSourceHandle`, standing in for whatever a real external producer would push from |
| `audio_record` | TestAudioSource → SwAudioEncoder → Mp4Muxer | Encodes a synthetic sine tone straight into a playable `.mp4` — `Mp4Muxer`'s single-track path, the audio counterpart to `transcode_render`'s `SwEncoder` proof |
| `audio_playback` (`wasapi-renderer`) | TestAudioSource → AudioResampler → AudioVolume → Queue → WasapiRenderer | Lists render endpoints and demonstrates runtime gain/mute changes while playing a three-second tone in the selected device's native mix format |
| `video_compositor` | TestVideoSource × 2 → VideoCompositor → Scaler → SwEncoder → Mp4Muxer | Composites two independently-paced inputs and moves one layer at runtime without changing source connections |
| `hls` | TestVideoSource → SwEncoder → HlsMuxer | Writes a live fMP4 `index.m3u8`, `init.mp4`, and keyframe-aligned `.m4s` segments with a sliding playlist window |
| `remux` | FileDemuxer → Mp4Muxer (one track per kept stream) | Remuxes a file's video + audio streams into a new `.mp4` with no decode/re-encode — `Mp4Muxer`'s multi-track builder driven by a single source's multiple `src_pads`, packets passed through untouched |
### Recording (Windows only)
| Crate | Pipeline | Demonstrates |
|---|---|---|
| `audio_capture` (`wasapi-capture`) | WasapiCaptureSource → FrameCounter | Lists WASAPI endpoints, captures ~3s from one (system-audio loopback by default, or a microphone), reports how many buffers came through |
| `screen_record` (`dxgi-capture`) | DxgiCaptureSource → Scaler → SwEncoder → Mp4Muxer | Headless desktop recording straight to `.mp4` — no window, no renderer (compare `screen_capture`, which renders instead of encoding) |
| `screen_audio_record` (`dxgi-capture` + `wasapi-capture`) | DxgiCaptureSource + WasapiCaptureSource → Mp4Muxer | Desktop + system-audio recording combined into one file — two independent live sources driven by one `PipelineBuilder`-built `Pipeline`, both tracks finalized together; stops on `q` + Enter in the terminal |
### Playback (Windows only)
| Crate | Pipeline | Demonstrates |
|---|---|---|
| `sw_decode_render` | Demux → SwDecoder → Queue → Pacer → D3d12Renderer | End-to-end playback in a native window, CPU decode + CPU-upload render |
| `av_playback` | Demux → {SwDecoder → Queue → VideoSynchronizer → D3d12Renderer, dynamic Tee → SwDecoder → AudioResampler → Queue → WasapiRenderer} | Starts video-only, then accepts terminal commands to attach/detach WASAPI audio and seek; video switches between wall-clock pacing and the played-audio master without rebuilding the pipeline |
| `hw_decode_render` | Demux → D3d12vaDecoder → Queue → Pacer → D3d12Renderer | Same, but GPU decode feeding the renderer zero-copy — no decoded pixel ever touches system memory |
| `d3d11_decode_render` | Demux → D3d11Decoder → Queue → Pacer → D3d11Renderer | The D3D11 sibling of `hw_decode_render` — GPU decode via D3D11VA, zero-copy render. What actually proved `D3d11Decoder` safe on real hardware: `D3d11Decoder` never touches FFmpeg's `hw_frames_ctx` struct layout itself (only `bind_flags`, via the documented `avcodec_get_hw_frames_parameters` API, from inside `get_format`) — unlike an earlier, abandoned attempt at manual `AVD3D11VAFramesContext` construction, which corrupted memory |
| `test_video` | TestVideoSource → Queue → D3d12Renderer | A synthetic moving-gradient stream rendered directly (no file/camera/decoder, no `Pacer`) — proves `TestVideoSource`'s frames and `D3d12Renderer`'s CPU-upload path work end to end. Confirmed smooth without a `Pacer`: `TestVideoSource` self-paces on a drift-free absolute schedule, which turned out to be what actually mattered (see `screen_capture`, which confirmed the same thing even with a `Scaler` in between); `transcode_render` (below) keeps one, since its `SwEncoder`/`SwDecoder` stages have their own real per-frame variance, untested without |
| `transcode_render` | TestVideoSource → Queue → SwEncoder → Queue → SwDecoder → Queue → Pacer → D3d12Renderer | Encodes the synthetic stream (`libopenh264`) and decodes it straight back, no container/mux involved — proves `SwEncoder`'s `Packet`s are actually valid, decodable bitstream, not just "opened successfully" |
| `seek_render` | Demux → SwDecoder → Queue → Pacer → D3d12Renderer | Same chain as `sw_decode_render`, plus a terminal prompt that calls `Pipeline::seek` while the window is open |
| `screen_capture` | DxgiCaptureSource (CPU mode) → Queue → Scaler → Queue → D3d12Renderer | Live desktop capture (DXGI Desktop Duplication, cursor included) at a constant frame rate, converted/resized to the window's own size and rendered directly, no `Pacer`. Confirmed smooth without one: an earlier, variable-rate version of `DxgiCaptureSource` measurably needed a `Pacer` here to avoid judder, but once it moved to constant-rate, drift-free-scheduled emission (same pattern as `TestVideoSource`), `Scaler` alone wasn't enough to bring the judder back |
| `screen_capture_gpu` | DxgiCaptureSource (GPU mode) → Queue → D3d11Renderer | The zero-copy sibling of `screen_capture`: captures straight to a GPU-resident `Pixel::D3D11` BGRA texture on the renderer's own `ID3D11Device` — no `Map`, no CPU pixel copy, no `Scaler` (desktop content is already BGRA/RGB). No cursor (`CaptureMode::Gpu` doesn't support it yet) |
| `d3d12_upload` | TestVideoSource → Queue → Scaler → Queue → D3d12Upload → Queue → D3d12Renderer | A CPU `Pixel::YUV420P` stream converted to `Pixel::NV12` on the CPU, then uploaded to a GPU `Pixel::D3D12` texture on the renderer's own device via `D3d12Upload` before being presented zero-copy — proves `D3d12Upload`'s frames are structurally identical to `D3d12vaDecoder`'s own, so `D3d12Renderer` takes its zero-copy path unmodified even though nothing here ever decoded anything |
| `d3d11_upload` | TestVideoSource → Queue → Scaler → Queue → D3d11Upload → Queue → D3d11Renderer | The D3D11 sibling of `d3d12_upload`, same proof for `D3d11Upload`/`D3d11Renderer` |
| `gpu_video_compositor` | TestVideoSource × 2 → Scaler(NV12) → D3d11Upload → D3d11VideoCompositor → Tee → {D3d11Renderer, D3d11Download → Scaler → SwEncoder → Mp4Muxer} | The GPU sibling of `video_compositor`: composites two GPU-resident inputs with a moving PiP layer entirely via shader, then fans the composited output to a live window *and* a recording, proving one `D3d11VideoCompositor` frame serves both a display consumer and a CPU-readback consumer without being recomposed |
The D3D12 examples above build their `D3d12Renderer`, and the D3D11 ones their `D3d11Renderer`, through `render_common` (`examples/render/render_common`) — a small shared crate holding its own minimal window renderers (`D3d12GpuContext`/`D3d12WindowRenderer` for D3D12, `D3d11GpuContext`/`D3d11WindowRenderer` for D3D11) instead of each example hand-copying them. `media-pp` has no dependency on any *window*-rendering crate — only `render_common` depends on `windows`' DXGI swap-chain bindings to actually present to an `HWND`. `D3d11VideoCompositor` is the one exception to "no shader code in `media-pp`": compositing is window-independent (pure texture-to-texture), so its D3D11 pipeline/shader setup lives directly in `lib` rather than being pushed out to example code the way window presentation is. The D3D11/D3D12 stacks remain independent (separate device, separate shader set) — nothing shares a device across them.
### RTSP publishing
| Crate | Pipeline | Demonstrates |
|---|---|---|
| `rtsp_serve` | Demux → Queue → Pacer → RtspSink | Publishes a file's video to an already-running RTSP server; pass the file and publishing URL as arguments |
| `rtsp_serve_seek` | Demux → Queue → Pacer → RtspSink | Same, plus terminal commands for pause, resume, seek, and stop while publishing |
### RTSP client (no extra feature — just `ffmpeg-next`)
| Crate | Pipeline | Demonstrates |
|---|---|---|
| `rtsp_source` | RtspSource → Queue → PacketCounter | Connects to a real RTSP server/camera (TCP transport by default), counts video packets for a fixed window, then stops — `RtspSource` is the receive counterpart to `RtspSink` |
### Inference-pipeline building blocks
| Crate | Pipeline | Demonstrates |
|---|---|---|
| `scale` | Demux → SwDecoder → Queue → Scaler → (verify) | `Scaler` converting decoded frames to a fixed RGB24 640x640 — prints the first scaled frame's actual format/size to prove the conversion really happened |
| `detect` | Demux → SwDecoder → Queue → Scaler → OrtDetector | `OrtDetector` running a YOLOv8/v11 ONNX model on the scaled frames and printing every detection |
### WebRTC (`webrtc` feature)
| Crate | Pipeline | Demonstrates |
|---|---|---|
| `webrtc_loopback` | Two `WebRtcPeer`s over loopback UDP | One `Direction::SendRecv` track (opened by `WebRtcHandle::add_track` on one side, accepted via `WebRtcHandle::accept_remote_offer` on the other) carrying data both ways over the *same* `Mid` — no second negotiation for the reverse direction. No browser/signaling server: real ICE/DTLS-SRTP over loopback UDP |
| `webrtc_av_loopback` | TestVideoSource → SwEncoder → WebRtcTrackSink, TestAudioSource → SwAudioEncoder → WebRtcTrackSink (two `PipelineBuilder` sources) | Two tracks — one video, one audio — negotiated onto the *same* `WebRtcPeer` connection (two sequential `add_track` renegotiations, one `Rtc`/socket/peer pair), each carrying real encoded media; peer-b counts packets per track to prove they arrive independently, no cross-contamination |
Examples that read a file take its path as a required argument — none of them
carry a default, so no media is checked into this repository. Run one with no
arguments to see its usage line.
```sh
cargo run -p decode -- path/to/video.mp4
cargo run -p sw_decode_render -- path/to/video.mp4 # d3d12-renderer is already enabled in its own Cargo.toml
```
## Logging
`media-pp` logging is disabled by default. Applications can opt into a private,
daily-rotating file logger without installing or interacting with the
process-global `log`/`tracing` subscriber:
```rust
let _log_guard = media_pp::log::init(
"media-pp",
"./logs",
media_pp::log::Level::Info,
7,
)?;
```
Keep the returned `LogGuard` alive while logging is needed. `media_pp` records
never enter the embedding application's logger, and the application's records
never enter these files. If the writer's 4096-line queue fills, it drops new
records instead of blocking a media thread; `LogGuard::dropped_lines()` reports
how many were dropped.
Dropping the guard permanently rejects log calls that begin afterwards — a record
already being emitted on another thread may still complete — and makes a
*bounded* attempt to flush what is still queued: at most 100 ms to hand the
worker a shutdown message, then
at most one second for it to report back that it has drained. That is enough
whenever the log file is writable at normal speed, but a stalled writer that
keeps the queue full will make the drop return with records still unwritten (and
`tracing-appender` prints one line to stdout on that path, which this crate
cannot suppress). If final records matter, drop the guard before anything else
that could saturate the queue, and check `dropped_lines()`.
`init` succeeds at most once per process. A second call returns
`LogInitError::AlreadyInitialized` even after the first `LogGuard` has been
dropped, so the log directory and level are chosen once at startup and logging
cannot be re-enabled later. Anything needing this logger under `cargo test`
consequently needs its own integration-test file, because one test binary is one
process.
Each record uses explicit identity fields so it can be read without parsing a
compressed target string:
```text
2026-08-15T15:52:24.068+09:00 INFO [thread=pipeline:source#2] [pipeline_id=app-sink] [element=FileDemuxer] [name=demux] started
```
Records emitted before an element is attached to a pipeline omit only the
`pipeline_id` field. Timestamps use local time with a numeric UTC offset and
millisecond precision.
`thread` names the thread the record came from, numbered in the order threads
first log so that a pipeline's two `pipeline:source` threads stay distinguishable.
It sits ahead of the identity fields deliberately: `[element=…] [name=…]` and the
message remain adjacent, so grepping for one element's records still returns what
that element said.
When a pipeline starts it writes one `Info` record whose body is a pad-aware
flow diagram of the graph it just started. Stable element IDs distinguish
different elements that happen to share a name, while each downstream connector
begins under its upstream element so fan-out remains visually clear:
```text
INFO [thread=main#1] [pipeline_id=app-sink] [element=Pipeline] [name=app-sink] run
FileDemuxer(demux)#1
└── [src_0] → Tee(tee)#2
├── [tee_src0] → AppSink(preview)#3
└── [tee_src1] → AppSink(record)#4
```
The diagram shares the event's record rather than following it as a second one.
Only the lines within a single record are guaranteed to stay together — a
`Queue` worker that `run()` itself just started can write in between two records
— so a separate diagram would merely tend to sit next to the event that caused
it.
A runtime `Tee` change writes the same kind of record under the `Tee`'s own
identity, with `attach` or `detach` in place of `run` and the updated diagram as
its body. EOS and control messages use `Trace` records at
every element boundary with explicit phases (`sending`/`sent`,
`requested`/`received`/`forwarding`/`completed`). Ordinary video, audio, and
packet buffers are not logged per buffer.
Every checked-in executable example enables this logger at `Trace` level, writes
to `./logs`, and uses its Cargo package name as the file prefix.
## Feature flags
- `d3d12-renderer` (on `media-pp`) — pulls in `windows` and enables
`D3d12Renderer`, `D3d12FrameRenderer`, `RawPlane`, `SubmitError`,
`D3d12vaDecoder`, and `D3d12Upload`. `media-pp` itself has no dependency
on any rendering crate at all — `D3d12Renderer` takes a `Box<dyn
D3d12FrameRenderer>`, and it's each example's own job to provide a
concrete renderer (`examples/render/render_common`'s own
`D3d12WindowRenderer`) implementing that trait. Off by default so
consumers that don't render to a window never build DX12/Windows-only
code. The D3D12-based render examples turn it on in their own
`Cargo.toml`; D3D11-only examples enable `d3d11-renderer` instead.
- `d3d11-renderer` (on `media-pp`) — pulls in `windows` (including
`Win32_Graphics_Direct3D_Fxc`, needed for `D3d11VideoCompositor`'s own
runtime `D3DCompile` calls — the one place this crate compiles HLSL
itself, everywhere else that's `render_common`'s job) and enables
`D3d11Renderer`, `D3d11FrameRenderer`, `D3d11Decoder`, `D3d11Upload`,
`D3d11Download`, `D3d11VideoCompositor` (+ handle types), and
`SubmitError` (shared with `d3d12-renderer`). Independent of
`d3d12-renderer` — separate device, separate shader set, nothing shared
between the two stacks. Off by default, same reasoning as
`d3d12-renderer`. Every `d3d11_*`/`screen_capture_gpu`/
`gpu_video_compositor` example crate turns it on in its own
`Cargo.toml`.
- `dxgi-capture` (on `media-pp`) — pulls in `windows` (DXGI sub-features)
and enables `DxgiCaptureSource`/`CaptureMode`. Requires `d3d11-renderer`
(`DxgiCaptureOptions`' `CaptureMode::Gpu` produces a `Pixel::D3D11` frame
the same way `D3d11Upload` does, via the same shared helper) — enabling
`dxgi-capture` pulls `d3d11-renderer` in automatically. Windows-only.
`screen_capture`/`screen_capture_gpu` turn it on in their own
`Cargo.toml` (alongside `d3d12-renderer`/`d3d11-renderer` respectively,
to actually render what they capture).
- `wasapi-capture` (on `media-pp`) — pulls in `windows` (WASAPI/Core Audio
sub-features) and enables `WasapiCaptureSource`/`WasapiCaptureOptions`/
`WasapiDevice`/`WasapiDeviceKind`. Independent of `dxgi-capture`/
`d3d11-renderer`/`d3d12-renderer` — capturing audio needs none of them —
but commonly turned on alongside `dxgi-capture` for a combined
desktop+audio recording (see `screen_audio_record`). Windows-only (WASAPI
itself is a Windows API). `audio_capture`/`screen_audio_record` turn it
on in their own `Cargo.toml`.
- `wasapi-renderer` (on `media-pp`) — pulls in the same Windows Core Audio
bindings and enables `WasapiRenderer`/`WasapiRendererOptions`. It shares
`WasapiDevice`/`WasapiDeviceKind` with `wasapi-capture`, but is otherwise
independent. `audio_playback` enables it and converts into the selected
endpoint's returned `AudioFormat` with `AudioResampler`.
- `ort` (on `media-pp`) — pulls in the `ort` crate (ONNX Runtime bindings;
downloads a prebuilt onnxruntime binary at build time) and `ndarray`, and
enables `OrtDetector`. `detect` turns it on in its own `Cargo.toml`.
- `webrtc` (on `media-pp`) — pulls in `str0m` (sans-I/O WebRTC, `wincrypto`
backend — native Windows crypto, no OpenSSL vendoring) and enables
`WebRtcPeer`/`WebRtcHandle`/`WebRtcTrackSink`/`WebRtcTrackSource`. The initial SDP
offer/answer and ICE candidate setup happen via str0m directly, in the
caller's own code, *before* constructing a `WebRtcPeer`; there's no
signaling server built in. `webrtc_loopback` turns it on in its own
`Cargo.toml`.
There is no `default` feature set. The [docs.rs] build enables the Windows
backend features and targets Windows so their public types remain visible. Its
Linux build host uses its installed FFmpeg metadata and parses those headers
for the Linux host ABI so `ffmpeg-sys-next` can generate bindings while
cross-documenting the Rust API; rustdoc does not link a Windows executable. To
build the same API docs on Windows locally:
```sh
cargo doc -p media-pp --open --features d3d11-renderer,d3d12-renderer,dxgi-capture,wasapi-capture,wasapi-renderer,webrtc
```
[docs.rs]: https://docs.rs/media-pp
## Requirements
- ffmpeg installed and discoverable by `ffmpeg-sys-next` (see that crate's
build requirements). `D3d12vaDecoder`/`D3d11Decoder` additionally need an
ffmpeg build with `d3d12va`/`d3d11va` hwaccel support respectively (check
`ffmpeg -hwaccels`) and a GPU/driver that supports it.
- `rtsp_serve` and `rtsp_serve_seek` require an external RTSP server that
accepts publishing at the supplied URL. MediaMTX is one compatible option,
but it is not bundled or managed by `media-pp`.
- No media is checked in, so tests that need a real video read its path from
`MEDIA_PP_TEST_VIDEO` and skip with a printed reason when it is unset. Point
it at any container ffmpeg can open, holding a video stream and at least a few
seconds long. **A skipped test still reports as passing**, so set the variable
when a change touches demuxing, seeking, or decoding:
```sh
MEDIA_PP_TEST_VIDEO=/path/to/video.mp4 cargo test -p media-pp
```
- Windows-backed examples (`audio_capture`, `audio_playback`, the
`examples/render/*` window/capture examples, and the current
WebRTC loopbacks) keep their runtime dependencies behind `cfg(windows)`.
They build as unsupported stubs on other targets and print a clear message
when run; their actual pipelines still run only on Windows.
- Windows backend modules and public re-exports are guarded by both their
Cargo feature and `target_os = "windows"`; enabling one of those features
for another target does not expose the Windows-specific element types.
- `D3d12vaDecoder` hand-mirrors a few structs from FFmpeg's
`libavutil/hwcontext_d3d12va.h` that `ffmpeg-sys-next` doesn't bind
(see the doc comment at the top of
`elements/filter/decoder/windows/d3d12va_decoder.rs`) — sourced from FFmpeg
n8.0's header. A future FFmpeg version changing that header's layout
would silently break this with no compile-time warning.
`D3d11Decoder`/`D3d11Upload`/`DxgiCaptureSource`'s GPU mode do the same
for a couple of small D3D11VA-specific structs
(`elements/filter/decoder/windows/d3d11va_decoder.rs`),
but deliberately touch only a handful of already-initialized fields
(never construct FFmpeg's `AVHWFramesContext` from scratch) — an earlier
version that did corrupted memory badly enough to trip `/GS`
(`STATUS_STACK_BUFFER_OVERRUN`), for a reason never fully root-caused;
see that file's own doc comments for the history.
- `D3d11Decoder`'s decode surface pool is fixed-size (unlike D3D12VA's) —
its `extra_hw_frames` parameter must cover whatever the deepest
downstream queue/buffer can hold, or decode itself starts failing once
the pool runs out (see its own doc comment).
- Every D3D11 element in one pipeline (`D3d11Decoder`, `D3d11Upload`,
`D3d11Renderer`, `DxgiCaptureSource`'s GPU mode) must share exactly one
`ID3D11Device` — that's what lets this stack skip explicit GPU-side
fences entirely, unlike the D3D12 side (see `D3d11Renderer`'s own doc
comment for why).
## License
Licensed under either of the [Apache License, Version 2.0](LICENSE-APACHE)
or the [MIT License](LICENSE-MIT), at your option.
`media-pp` does not bundle FFmpeg. Users are responsible for complying with
the license of the FFmpeg build and optional codecs they link against.