Crate fframes

Expand description

fframes renders videos from Rust code. A video is a type that implements Video. It declares the size, frame rate and duration once and returns an SVG tree (Svgr) for every frame from the svgr! macro. The renderer rasterises the frames on all cores, mixes the audio and encodes the result with FFmpeg.

§Pipeline

Video::duration, Video::define_scenes and Video::audio are resolved into a frame timeline once per render. Scenes (Scene, Scenes, Overlap) are placed back to back, audio tracks (AudioMap, AudioTrack) are placed at their sample position.
For every frame the renderer builds a Frame (index, time, scene offset) and a FFramesContext (media lookups, scene info, video size) and calls Video::render_frame. Frames are rendered concurrently, so the method must be pure: read precomputed data from self, no I/O, no panics.
The returned Svgr becomes a usvgr::Tree. With the compile-time-svgtree feature the macro emits the tree at compile time and every subtree without {} interpolation carries a static hash, which lets the backends cache its rasterisation across frames. Without the feature the markup is a string parsed per frame.
A rendering backend (FFramesRenderBackend) rasterises the tree. The built-in cpu::CpuRenderingBackend renders one video segment per thread with tiny-skia. The Skia backend in the fframes_skia_renderer crate walks the tree on the GPU and also executes Shader layers.
Segments are encoded through FFmpeg (EncoderOptions), concatenated, and muxed with the audio mix (AudioMixOptions: summing, ducking, fades, master limiter).

§Minimal video

use fframes::{AudioMap, Duration, FFramesContext, Frame, RenderOptions, Svgr, Video, animation::Easing};

// Every file in the folder becomes a field; fonts are registered by family name.
fframes::include_media_dir!(pub struct Media, "media");

struct Hello<'a> {
    media: &'a Media,
    title: &'a str,
}

impl Video for Hello<'_> {
    const FPS: usize = 30;
    const WIDTH: usize = 1920;
    const HEIGHT: usize = 1080;

    fn duration(&self) -> Duration<'_> {
        Duration::Seconds(3.0)
    }

    fn audio(&self) -> AudioMap<'_> {
        AudioMap::none()
    }

    fn render_frame<'a>(&'a self, frame: Frame, _ctx: &FFramesContext<'a, '_>) -> Svgr<'a> {
        let opacity = frame.animate(&fframes::timeline!(
            at 0.0 => 0.5, animate 0.0_f32 => 1.0, Easing::EaseOut
        ));
        fframes::svgr!(
            <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1920 1080"
                 width={Self::WIDTH} height={Self::HEIGHT}>
                <text x="120" y="560" font-family="DM Sans" font-size="150" fill="#fff"
                      opacity={opacity}>
                    "Hello " {self.title}
                </text>
            </svg>
        )
    }
}

fn main() -> std::process::ExitCode {
    let media = Media::prepare().expect("embedded media");
    let video = Hello { media: &media, title: "world" };
    // `render`, `frame`, `strip`, `inspect`, `audio` and more (feature `cli`).
    fframes::cli::new(&video, RenderOptions { media: Some(&media), ..Default::default() }).run()
}

Without the cli feature call render with the output path, a backend and RenderOptions, or render single frames through Previewer.

§Feature flags

cpu_renderer (default): the tiny-skia based cpu::CpuRenderingBackend and the encoding pipeline. Disable it for a wasm32 build.
cli: the cli module and its clap dependency.
compile-time-svgtree: svgr! builds the SVG tree at compile time and hashes static subtrees. Required by the Skia backend and by Shader.
exif: EXIF orientation of loaded images.
3d: three_d, 3D scenes with the building blocks of three.js, rendered into an <image> of the frame. Runs on the CPU, also in the editor.
wgpu: wgpu_layer puts textures rendered with wgpu into frames, and the Skia Vulkan backend draws them without a readback. With 3d, the 3D renders on the GPU.
Codecs h264, h265, aac, mp3lame, opus, vpx: compile the library into the static FFmpeg build. Some of them need libav-agree-gpl, libav-agree-nonfree or libav-agree-version3, which state that you accept the corresponding FFmpeg license terms.
Hardware acceleration videotoolbox, audiotoolbox, vaapi, nvidia, qsv, vulkan, mediacodec: enable the platform encoders and decoders in FFmpeg.

§`FFmpeg`

Decoding, encoding and muxing use the FFmpeg libraries through ffmpeg-sys-fframes, re-exported as ffmpeg_sys_fframes. On Linux and macOS FFmpeg is linked statically: a prebuilt build for the target and codec features is downloaded during cargo build, and compiled from source when there is none, which needs the toolchain listed in the repository README (nasm, clang). The codec features link the system codec libraries (x264, x265, …), so their dev packages must be installed either way. On Windows a prebuilt FFmpeg 9 is linked through FFMPEG_DIR or vcpkg and the codec features are not available.

Re-exports§

pub use crate::usvgr::roxmltree;
pub use fframes_media as media;
pub use lazy_static;
pub use lru;
pub use media::bytemuck;
pub use media::ffmpeg_sys_fframes;
pub use serde;
pub use ttf_parser;
pub use usvgr;
pub use media::exif;
pub use rayon;

Modules§

animation
cli: A ready-made command line for video binaries (feature cli). One call gives every video the same tools for rendering and for checking the result without watching it:
concatenator
continuity: Finds pops: parts of a picture that change in a single frame between frames in which they stand still. An element that appears, vanishes or jumps in front of another element without moving there is easy to miss in a contact sheet but obvious when the video plays.
cpu
diagnostics: Frame diagnostics: problems that do not fail a render but produce a wrong picture, such as a missing image, an unknown font family or text pushed off the canvas. The collector reports them without rasterizing anything, from three sources:
element_labels: Outlines and labels the elements of a frame that have an id, to tell which shape is which in a rendered PNG (frame --ids).
error
fframes_logger
log
sheet: Compositing several frames into one image: labelled contact sheets and onion skins, which show a movement in a single picture.
three_d: 3D scenes for fframes, with an API after three.js. fframes re-exports this crate as fframes::three_d with its 3d feature: fframes = { features = ["3d", "wgpu"] }.

Macros§

MKTAG
ffmpeg_action
ffmpeg_loggable_action
include_media_dir: Embed the contents of a directory in your crate.
log
option_duration
option_literal
svgr
timeline

Structs§

AbortSignal: A signal that can be used to abort the rendering process from the other thread to stop the rendering process or to abort the rendering process inside the rendering without panicking.
ActiveTrack: A track playing at some moment, see AudioMixer::active_tracks_at.
AnimateRuntimeInput
AudioMap: Audio map represents when and how long each audio file should be played within a video or a scene. In case of scene audio map is always relative to the scene timestamp (which is resolved based on the Video::define_scenes()).
AudioMixOptions: Master bus settings.
AudioMixer: Mixes the resolved audio map block by block. Blocks are expected in order (the limiter is stateful); jumping elsewhere resets the limiter.
AudioReport: Everything fframes audio analyze reports.
AudioTimelineFrames
AudioTimelineSamples
AudioTrack: One audio file placed on the timeline, with its mix settings.
AudioTrackReport
AvBuffer: A counted reference to a libav buffer (AVBufferRef). Hardware device and frames contexts are passed around this way.
BreakLinesOpts
Color: The basic rgba color type. It is limited to the range of 0-255 because the most of video codecs only support 8-bit color depth.
CombinedMediaProvider: Combines several allocated media providers into one. Allows to combine static and dynamic media providers.
CompactFFramesLogger
Compressor: Feed-forward compressor with a peak detector, linked on both channels.
CpuFrameRenderer: The built-in tiny-skia rasterizer as a FrameRenderer.
Ducking: Lowers a track while voice tracks (see AudioTrack::voice) play. Driven by the timeline, not the signal: the level drops before the voice starts and comes back after it ends.
DynamicMediaProvider: Media loaded at runtime. Start with DynamicMediaProvider::default() and add files with the with_* methods:
Encoder
EncoderFrame
EncoderInput: What the video encoders of a render are opened with, the outcome of FFramesRenderBackend::negotiate_encoder_input. Every VideoFrame submitted to the render has to match it.
EncoderOptions
EstimateTextWidthOptions
FFramesContext
FontQuery: Define a query that will resolve a font from the provided media source. The values from the struct could be passed directly to the <text> element attributes like:
Frame: Contains all the temporal information about the current frame and the mutable links to the intermidient caches.
FrameClaim: A frame handed to a worker by the [FrameScheduler].
FrameConvertOptions
FrameLayout: Where the planes of a software frame are inside its pixel buffer.
FramePanic: Where and why Video::render_frame panicked.
FramePool: Hands out software VideoFrames of one format and size. The pixel buffers go back to the pool once the encoder is done with a frame, so a render does not allocate per frame.
FrameReport: What the diagnostics found in one frame.
LimiterOptions
LinesLogger: Backs FFramesLoggerVariant::Lines and FFramesLoggerVariant::Json.
LoudnessAnalysis: Per 100 ms energies of a K-weighted stereo signal, the base of every loudness value.
MediaDirectory: A struct representing owned media directory which can be used to process and fill the media provider. Basically represented as the vector of bytes where every media file is owned bytes vector.
NamedRange
PreviewRequest: What the CLI’s preview command asks a real-time player to do.
Previewer: A long lived session for rendering arbitrary frames of a video.
QuietLogger
RawFontData
RenderOptions: All the final render-specific options applies to the final video rendering pipeline including media resolution, logging, rendering backend, and encoding.
ReportedDiagnostic
ResizeVideoFrame
ResolvedAudioMap: The resolved audio_map contain each audio file position and duration in specified units.
ResolvedAudioTrack: A track with its position on the timeline resolved.
ResolvedRenderingTimeline
ResolvedScenesTimeline
RgbaEncoderFrameRenderer: An EncoderFrameRenderer for any FrameRenderer: renders RGBA and converts it on the CPU.
RgbaFrame: Straight (not premultiplied) RGBA8 pixels of one frame.
RgbaFrameConverter: Converts RGBA8 pixels into frames of a software pixel format on the CPU: the built-in SIMD converter for yuv420p, swscale for every other format.
Rotate
Scale
SceneInfo: Represents current scene position and duration within a video.
SceneReport
SceneWithAudio
Scenes
ScenesWithAudio
SectionLoudness
Shader: A compiled-once GPU shader definition. Cheap to clone.
ShaderDraw: Everything a renderer needs to execute one queued Shader::draw.
ShaderUniforms: Named uniform values for one Shader::draw call.
SilentLogger
Svgr
SvgrImages: The svgr images of one render: the rasterizer of the backend and the images of static trees, which render once.
SyncVideoFrameInput
TimeBase
TimelineIndex: Everything needed to resolve time specs of one video.
TimelineReport: The resolved structure of a video: size, duration, scenes and audio tracks.
TimelineScene: A scene as seen by the addressing: its name and resolved frames (overlaps included).
TrackMix: How a track is mixed. Everything defaults to “play the file as is”.
Transform: SVG transform attribute that can be used directly as transform={...} argument using svgr! macro
VideoDecodersWorker
VideoEncoderInfo: The video encoder of a render as a backend sees it while negotiating the frames it will deliver. The codec is resolved the same way Encoder does it, nothing is opened yet.
VideoFrame: A frame on its way to the video encoder: an owned libav AVFrame.
VideoSize
VisualizeFrameInput
WasmEditorVideoFrameFallback: This is used by the editor as a fallback for the static image
WrappedTextLine
WrappedTextStructure

Enums§

AVPixelFormat
AudioData
AudioTimestamp
Duration
Effect: One processing step of a track, see the AudioTrack methods with the same names.
EncoderOutput: What an Encoder writes.
FFramesLoggerVariant: Different options for logging rendering process.
FFramesMode
FFramesRendererError
FadeCurve: Shape of a fade.
FontStretch
FontStyle
FontVariant
Overlap
RenderEncodingError: Thread or Chunk level error which can happen during parallelized rendering
SampleSize
ShaderLanguage: The language the shader source was written in.
ShaderUniformValue: A single value bound to a named uniform of a Shader.
TextAlign
TextOverflow: How text that does not fit into its box is shortened, the equivalent of the css text-overflow property.
TimeSpecError
WindowFunction
YuvMatrix: The matrix every RGB to YCbCr conversion of the encoder input uses, limited range either way. The encoder tags the stream with it, so players decode the colors that were rendered.

Traits§

AudioTimelineUnit
EncoderFrameRenderer: Rasterizes frames straight into what the video encoder takes. One per rendering thread; implementations keep their surfaces, caches and converters between frames.
FFramesLogger
FFramesRenderBackend
FFramesSyncedVideoFrame
FontFace
FontSource
FrameRenderer: Rasterizes converted frames. Implementations keep their surfaces and caches between calls, so rendering many frames through one renderer is cheap.
MediaProvider
Scene
StaticMediaProvider
SvgrRasterizer: Renders converted svgr trees into images for FFramesContext::render_svgr. A backend shares one rasterizer between all threads of a render.
Video: The base fframes video trait. It represents how to render a video for a struct which becomes an input of the video.

Functions§

FFMPEG_AVERROR
analyze_audio: Analyzes a stereo mix. sections are named sample ranges (usually the scenes).
apply_window_function
av_error_to_string
center_spectrum_low_frequencies: Prettifies audio spectrum by moving low frequencies (first elements) in the middle and all the other elements to be proportionally positioned to the edges from the middle
char_to_digit
encode_wav: Encodes stereo samples as a WAV file: 16-bit PCM with TPDF dither (plays everywhere) or 32-bit float (keeps levels above full scale).
estimate_text_width: Estimates the width of the text in pixels. Returns None if the font is not resolved.
fit_transform: The transform that fits a tree into the output size. Videos that hardcode their WIDTH as the <svg width> are scaled this way when rendering with scale_resolution.
get_thread_count: Convenient function that can be used by different backend implementation to save the reused about of threads for the rendering process. Allowing to override the default thread count using the FFRAMES_NUN_THREADS environment variable.
get_visualization
hamming_window: Applies a Hamming window (https://en.wikipedia.org/wiki/Window_function#Hann_and_Hamming_windows) to an array of samples.
hann_window: Applies a Hann window (https://en.wikipedia.org/wiki/Window_function#Hann_and_Hamming_windows) to an array of samples.
hardware_device: Opens a libav hardware device (av_hwdevice_ctx_create). Fails when FFmpeg was built without it or the machine has no such device.
image_pixels: The pixels of image. An image on the GPU (fframes::wgpu_layer) is read back, which waits for the GPU. None when that fails. Other images are their own pixels.
is_hardware_pixel_format: true for formats that reference a hardware surface (AV_PIX_FMT_VIDEOTOOLBOX, AV_PIX_FMT_VULKAN, …) instead of carrying pixels.
make_logger
render: Renders fframes video to the output destination path. Uses the provided video and media implementations as long as the reneder backend implementation.
render_frame: Renders a single frame into the output image buffer. Prints all the rendering warns and errors for the frame along with the svg file itself. Returns a byte representation specific to the render backend used. For CpuRenderBackend it is a RGBA image of the video size.
render_frame_guarded: Calls Video::render_frame, turning a panic into an error that says which frame, second and scene it happened in. Rendering backends should call the video through this function.
rendered_image: An image of premultiplied RGBA pixels, width * 4 bytes per row. Each image gets a new id, so a cache keyed by the id never returns an old image.
resolve_shader_draw: Returns the shader draw request if image is a placeholder created by Shader::draw. Called by rendering backends for every raster image.
resolve_timeline
seconds_to_frames
short_scene_name: my_video::scenes::Intro<'_> -> Intro

Type Aliases§

FFramesRendererResult
RenderEncodingResult

Crate fframes

Crate fframes Copy item path

§Pipeline

§Minimal video

§Feature flags

§FFmpeg

Re-exports§

Modules§

Macros§

Structs§

Enums§

Traits§

Functions§

Type Aliases§

Crate fframes

§`FFmpeg`