media-pp

A small, GStreamer-flavored media pipeline library in Rust, built on ffmpeg-next. lib/ is the library (crate name media-pp); examples/ holds one independent crate per demo pipeline, grouped into subdirectories by theme (core/, render/, rtsp/, vision/, webrtc/).

Architecture

Everything is built from a handful of primitives in lib/src/:

• Element (element.rs) — a named node in the graph. Just identity; says nothing about input/output.
• Sink (element.rs) — anything that can receive a buffer (consume(&mut self, buf: MediaBuffer)). The only "connection" primitive in the pipeline. By default, consuming a buffer is a plain function call on the caller's thread — zero overhead.
• Source (element.rs) — anything with one or more output ports (src_pads()). An element with more than one src pad is a tee — fan-out needs no separate primitive (see FileDemuxer, which exposes one pad per container stream).
• SourceElement (element.rs) — a pure source that drives its own thread via run() (e.g. FileDemuxer, wrapping blocking file I/O).
• Filter (element.rs) — anything that's both a Source and a Sink (decoder, pacer, ...). Auto-implemented for any T: Source + Sink; no separate "processing element" trait needed.
• SrcPad (pad.rs) — an output port with exactly one runtime peer. Applications connect it through Context::attach, so the runtime peer and topology graph cannot diverge.
• MediaBuffer (buffer.rs) — the unit of data flowing between elements (Packet / Video / Audio / Eos). Payloads are Arc-wrapped, so cloning a buffer (e.g. to fan it out) is a refcount bump, never a copy of the encoded/decoded data.
• Queue (queue.rs) — the explicit thread boundary between stages in a dataflow chain. Wrapping a Sink in a Queue hands buffers off through a bounded channel to a dedicated worker thread. Directly-linked stages stay on their upstream caller's thread; the top-level Pipeline and DriverRunner separately own the background threads that drive a source or driver.
• Bus / BusEvent (bus.rs) — a cross-thread event channel. Once a buffer crosses a Queue boundary, errors can't propagate with ? anymore, so they're posted here instead (Error, Eos, Dropped). BusReceiver::iter_with_ids() pairs each event with its stable graph ElementId; log_events() drains and prints them in a default format.
• PpLog (pp_log.rs) — the contextual log identity stored privately by every element. pp_info!, pp_debug!, pp_warn!, pp_error!, and pp_trace! attach that identity to records sent to the opt-in private file logger; element_pp_log() keeps pipeline_id, element type, and instance name as separate fields used throughout the graph.
• Pipeline / ChainBuilder / PipelineBuilder (pipeline.rs) — ctx.branch() builds one linear, detached chain (.pipe(filter) for same-thread stages, .queue(name, capacity) for a thread boundary, .to(sink) to terminate). ctx.attach(source, pad, branch) commits the runtime connection and graph in one operation. Pipeline::run() drives a SourceElement on a background thread and returns immediately; draining its bus waits for that source and every reachable Queue worker to finish, provided the application has not retained an extra Context/Bus sender. Pipeline::new is the single-source case; PipelineBuilder::new(id).add_source(source, wire)… combines more than one independent SourceElement (e.g. a video capture and an audio capture both feeding one Mp4Muxer) into one Pipeline — each source gets its own thread, but they share one Bus/Clock/ graph, and run/pause/resume/stop/seek reach every source from a single call.
• PipelineGraph (graph.rs) — the live node/edge graph. Elements, edges, and dynamic branches use stable ElementId/EdgeId/BranchId values; names are display labels only. Pipeline::graph() returns a revisioned, consistent snapshot, and Pipeline::topology() renders it. A detached branch never appears until attachment succeeds.
• Clock (clock.rs) — a shared wall-clock anchor (Arc<Clock>) so multiple Pacers (e.g. one per stream) agree on the same t=0.
• PlaybackClock (playback_clock.rs) — a shared media-position clock. Video-only playback derives it from Clock; a bound audio renderer can take over with its actual played-sample position without moving the timeline backwards.

Elements (lib/src/elements/)

One-line index only — each element's own doc comment (cargo doc --open) has the full rationale (why it's built the way it is, what to watch out for); this table isn't meant to duplicate that.

Sources

¹ Each input is driven from wherever it was attached — typically a different Pipeline/thread than the one the AudioMixer itself is the source of, which is the whole point (e.g. a capture pipeline feeding a mixer that another pipeline reads from). For combining a fixed, known-up-front set of live sources into one output instead (no dynamic attach/detach needed), see PipelineBuilder — a simpler fit for e.g. one video capture + one audio capture feeding a single Mp4Muxer.

Filters

² Doesn't actually implement Source — its pads live in individually locked branch slots instead of a plain &mut [SrcPad]. consume only holds the branch-list lock long enough to clone an Arc snapshot, so a slow downstream does not block unrelated TeeHandle::attach/detach operations. Detach prevents a push that has not started yet; one already executing downstream call may finish. See its own doc comment.

Sinks

³ Not a plain Sink itself — Mp4Muxer::create/add_stream/open is a two-phase builder, since a container's header has to describe every track's codec parameters before it can be written at all. create opens the file, add_stream registers one track at a time (name + codec::Parameters + time_base), and open writes the header and returns one real Sink per track, in registration order — all sharing one lock around the file, so tracks fed from independently-threaded branches (e.g. one video encode chain, one audio encode chain) can write concurrently without racing. The trailer is written once every track reports done (Eos or Stop), not on whichever finishes first. See its own doc comment, and PipelineBuilder for wiring two independent live sources (e.g. video + audio capture) into the tracks it expects.

⁴ Same two-phase builder shape as Mp4Muxer (create/add_stream/open), plus a naming closure (FnMut(u64) -> PathBuf, called with the segment index) instead of one fixed path. A rotation only actually cuts once the configured duration has elapsed and the video track's next packet is a keyframe — never mid-GOP — so every segment file is independently decodable from its own frame 0, closing the outgoing segment (writing its trailer) via the exact same all-tracks-report-done mechanism Mp4Muxer already uses for a normal Eos/Stop. Building this is what surfaced a real gap in SwEncoder: it now always sets a ~2-second keyframe interval itself, since at least one codec (libopenh264) was found to otherwise go an entire recording without a second keyframe against smoothly-changing content — relying on scene-change detection alone, which would have meant a SegmentedMp4Muxer using it might never rotate at all.

⁵ HlsOptions::new defaults to live fMP4 with two-second target segments and a six-entry sliding window. HlsMode selects live/EVENT/VOD behavior and HlsSegmentFormat selects fMP4 or MPEG-TS. Like the other muxers, HlsMuxer::open returns one sink per registered track and writes #EXT-X-ENDLIST only after every track reports Eos or Stop. Segment timing is media-timestamp based inside FFmpeg's HLS muxer; video segments cut on keyframes, so the encoder GOP should be close to the requested segment duration.

Examples (examples/)

Each is its own crate so per-example dependencies (e.g. winit for sw_decode_render) don't leak into the others. Every file-based example takes its media path as a required argument — there is no default and no media is checked in, so supply your own file; run one with no arguments to see its usage line. Live capture, WebRTC, and synthetic-source examples have their own arguments or need none.

Core concepts

Recording (Windows only)

Playback (Windows only)

The D3D12 examples above build their D3d12Renderer, and the D3D11 ones their D3d11Renderer, through render_common (examples/render/render_common) — a small shared crate holding its own minimal window renderers (D3d12GpuContext/D3d12WindowRenderer for D3D12, D3d11GpuContext/D3d11WindowRenderer for D3D11) instead of each example hand-copying them. media-pp has no dependency on any window-rendering crate — only render_common depends on windows' DXGI swap-chain bindings to actually present to an HWND. D3d11VideoCompositor is the one exception to "no shader code in media-pp": compositing is window-independent (pure texture-to-texture), so its D3D11 pipeline/shader setup lives directly in lib rather than being pushed out to example code the way window presentation is. The D3D11/D3D12 stacks remain independent (separate device, separate shader set) — nothing shares a device across them.

RTSP publishing

RTSP client (no extra feature — just ffmpeg-next)

Inference-pipeline building blocks

WebRTC (webrtc feature)

Examples that read a file take its path as a required argument — none of them carry a default, so no media is checked into this repository. Run one with no arguments to see its usage line.

cargo run -p decode -- path/to/video.mp4

cargo run -p sw_decode_render -- path/to/video.mp4   # d3d12-renderer is already enabled in its own Cargo.toml

Logging

media-pp logging is disabled by default. Applications can opt into a private, daily-rotating file logger without installing or interacting with the process-global log/tracing subscriber:

let _log_guard = media_pp::log::init(
    "media-pp",
    "./logs",
    media_pp::log::Level::Info,
    7,
)?;

Keep the returned LogGuard alive while logging is needed. media_pp records never enter the embedding application's logger, and the application's records never enter these files. If the writer's 4096-line queue fills, it drops new records instead of blocking a media thread; LogGuard::dropped_lines() reports how many were dropped.

Dropping the guard permanently rejects log calls that begin afterwards — a record already being emitted on another thread may still complete — and makes a bounded attempt to flush what is still queued: at most 100 ms to hand the worker a shutdown message, then at most one second for it to report back that it has drained. That is enough whenever the log file is writable at normal speed, but a stalled writer that keeps the queue full will make the drop return with records still unwritten (and tracing-appender prints one line to stdout on that path, which this crate cannot suppress). If final records matter, drop the guard before anything else that could saturate the queue, and check dropped_lines().

init succeeds at most once per process. A second call returns LogInitError::AlreadyInitialized even after the first LogGuard has been dropped, so the log directory and level are chosen once at startup and logging cannot be re-enabled later. Anything needing this logger under cargo test consequently needs its own integration-test file, because one test binary is one process.

Each record uses explicit identity fields so it can be read without parsing a compressed target string:

2026-08-15T15:52:24.068+09:00 INFO [thread=pipeline:source#2] [pipeline_id=app-sink] [element=FileDemuxer] [name=demux] started

Records emitted before an element is attached to a pipeline omit only the pipeline_id field. Timestamps use local time with a numeric UTC offset and millisecond precision.

thread names the thread the record came from, numbered in the order threads first log so that a pipeline's two pipeline:source threads stay distinguishable. It sits ahead of the identity fields deliberately: [element=…] [name=…] and the message remain adjacent, so grepping for one element's records still returns what that element said.

When a pipeline starts it writes one Info record whose body is a pad-aware flow diagram of the graph it just started. Stable element IDs distinguish different elements that happen to share a name, while each downstream connector begins under its upstream element so fan-out remains visually clear:

INFO [thread=main#1] [pipeline_id=app-sink] [element=Pipeline] [name=app-sink] run
FileDemuxer(demux)#1
└── [src_0] → Tee(tee)#2
              ├── [tee_src0] → AppSink(preview)#3
              └── [tee_src1] → AppSink(record)#4

The diagram shares the event's record rather than following it as a second one. Only the lines within a single record are guaranteed to stay together — a Queue worker that run() itself just started can write in between two records — so a separate diagram would merely tend to sit next to the event that caused it.

A runtime Tee change writes the same kind of record under the Tee's own identity, with attach or detach in place of run and the updated diagram as its body. EOS and control messages use Trace records at every element boundary with explicit phases (sending/sent, requested/received/forwarding/completed). Ordinary video, audio, and packet buffers are not logged per buffer.

Every checked-in executable example enables this logger at Trace level, writes to ./logs, and uses its Cargo package name as the file prefix.

Feature flags

• d3d12-renderer (on media-pp) — pulls in windows and enables D3d12Renderer, D3d12FrameRenderer, RawPlane, SubmitError, D3d12vaDecoder, and D3d12Upload. media-pp itself has no dependency on any rendering crate at all — D3d12Renderer takes a Box<dyn D3d12FrameRenderer>, and it's each example's own job to provide a concrete renderer (examples/render/render_common's own D3d12WindowRenderer) implementing that trait. Off by default so consumers that don't render to a window never build DX12/Windows-only code. The D3D12-based render examples turn it on in their own Cargo.toml; D3D11-only examples enable d3d11-renderer instead.
• d3d11-renderer (on media-pp) — pulls in windows (including Win32_Graphics_Direct3D_Fxc, needed for D3d11VideoCompositor's own runtime D3DCompile calls — the one place this crate compiles HLSL itself, everywhere else that's render_common's job) and enables D3d11Renderer, D3d11FrameRenderer, D3d11Decoder, D3d11Upload, D3d11Download, D3d11VideoCompositor (+ handle types), and SubmitError (shared with d3d12-renderer). Independent of d3d12-renderer — separate device, separate shader set, nothing shared between the two stacks. Off by default, same reasoning as d3d12-renderer. Every d3d11_*/screen_capture_gpu/ gpu_video_compositor example crate turns it on in its own Cargo.toml.
• dxgi-capture (on media-pp) — pulls in windows (DXGI sub-features) and enables DxgiCaptureSource/CaptureMode. Requires d3d11-renderer (DxgiCaptureOptions' CaptureMode::Gpu produces a Pixel::D3D11 frame the same way D3d11Upload does, via the same shared helper) — enabling dxgi-capture pulls d3d11-renderer in automatically. Windows-only. screen_capture/screen_capture_gpu turn it on in their own Cargo.toml (alongside d3d12-renderer/d3d11-renderer respectively, to actually render what they capture).
• wasapi-capture (on media-pp) — pulls in windows (WASAPI/Core Audio sub-features) and enables WasapiCaptureSource/WasapiCaptureOptions/ WasapiDevice/WasapiDeviceKind. Independent of dxgi-capture/ d3d11-renderer/d3d12-renderer — capturing audio needs none of them — but commonly turned on alongside dxgi-capture for a combined desktop+audio recording (see screen_audio_record). Windows-only (WASAPI itself is a Windows API). audio_capture/screen_audio_record turn it on in their own Cargo.toml.
• wasapi-renderer (on media-pp) — pulls in the same Windows Core Audio bindings and enables WasapiRenderer/WasapiRendererOptions. It shares WasapiDevice/WasapiDeviceKind with wasapi-capture, but is otherwise independent. audio_playback enables it and converts into the selected endpoint's returned AudioFormat with AudioResampler.
• ort (on media-pp) — pulls in the ort crate (ONNX Runtime bindings; downloads a prebuilt onnxruntime binary at build time) and ndarray, and enables OrtDetector. detect turns it on in its own Cargo.toml.
• webrtc (on media-pp) — pulls in str0m (sans-I/O WebRTC, wincrypto backend — native Windows crypto, no OpenSSL vendoring) and enables WebRtcPeer/WebRtcHandle/WebRtcTrackSink/WebRtcTrackSource. The initial SDP offer/answer and ICE candidate setup happen via str0m directly, in the caller's own code, before constructing a WebRtcPeer; there's no signaling server built in. webrtc_loopback turns it on in its own Cargo.toml.

There is no default feature set. The docs.rs build enables the Windows backend features and targets Windows so their public types remain visible. Its Linux build host uses its installed FFmpeg metadata only to let ffmpeg-sys-next generate and type-check bindings; rustdoc does not link a Windows executable. To build the same API docs on Windows locally:

cargo doc -p media-pp --open --features d3d11-renderer,d3d12-renderer,dxgi-capture,wasapi-capture,wasapi-renderer,webrtc

Requirements

• ffmpeg installed and discoverable by ffmpeg-sys-next (see that crate's build requirements). D3d12vaDecoder/D3d11Decoder additionally need an ffmpeg build with d3d12va/d3d11va hwaccel support respectively (check ffmpeg -hwaccels) and a GPU/driver that supports it.

• rtsp_serve and rtsp_serve_seek require an external RTSP server that accepts publishing at the supplied URL. MediaMTX is one compatible option, but it is not bundled or managed by media-pp.

• No media is checked in, so tests that need a real video read its path from MEDIA_PP_TEST_VIDEO and skip with a printed reason when it is unset. Point it at any container ffmpeg can open, holding a video stream and at least a few seconds long. A skipped test still reports as passing, so set the variable when a change touches demuxing, seeking, or decoding:

  MEDIA_PP_TEST_VIDEO=/path/to/video.mp4 cargo test -p media-pp
  
  

• Windows-backed examples (audio_capture, audio_playback, the examples/render/* window/capture examples, and the current WebRTC loopbacks) keep their runtime dependencies behind cfg(windows). They build as unsupported stubs on other targets and print a clear message when run; their actual pipelines still run only on Windows.

• Windows backend modules and public re-exports are guarded by both their Cargo feature and target_os = "windows"; enabling one of those features for another target does not expose the Windows-specific element types.

• D3d12vaDecoder hand-mirrors a few structs from FFmpeg's libavutil/hwcontext_d3d12va.h that ffmpeg-sys-next doesn't bind (see the doc comment at the top of elements/filter/decoder/windows/d3d12va_decoder.rs) — sourced from FFmpeg n8.0's header. A future FFmpeg version changing that header's layout would silently break this with no compile-time warning. D3d11Decoder/D3d11Upload/DxgiCaptureSource's GPU mode do the same for a couple of small D3D11VA-specific structs (elements/filter/decoder/windows/d3d11va_decoder.rs), but deliberately touch only a handful of already-initialized fields (never construct FFmpeg's AVHWFramesContext from scratch) — an earlier version that did corrupted memory badly enough to trip /GS (STATUS_STACK_BUFFER_OVERRUN), for a reason never fully root-caused; see that file's own doc comments for the history.

• D3d11Decoder's decode surface pool is fixed-size (unlike D3D12VA's) — its extra_hw_frames parameter must cover whatever the deepest downstream queue/buffer can hold, or decode itself starts failing once the pool runs out (see its own doc comment).

• Every D3D11 element in one pipeline (D3d11Decoder, D3d11Upload, D3d11Renderer, DxgiCaptureSource's GPU mode) must share exactly one ID3D11Device — that's what lets this stack skip explicit GPU-side fences entirely, unlike the D3D12 side (see D3d11Renderer's own doc comment for why).

License

Licensed under either of the Apache License, Version 2.0 or the MIT License, at your option.

media-pp does not bundle FFmpeg. Users are responsible for complying with the license of the FFmpeg build and optional codecs they link against.