framefinery 0.0.4

Safe Rust media pipeline toolkit and FrameFinery command-line encoder.
Documentation

FrameFinery Engine

FrameFinery Engine is a safe Rust media pipeline toolkit. The project is built around composable stages:

input -> decode -> filter -> encode -> output

The initial focus is experimental video encoding and validation infrastructure, with AV2 and VVC as the first planned codec families. The repository is a software-only sibling of the FrameFinery Engine hardware project: FrameFinery Engine remains the RTL, synthesis, and hardware-validation workspace, while this repository is free to optimize for software APIs, usability, codec quality, and safe Rust performance.

This repository is in bootstrap state. It currently provides project structure, shared media primitives, a CLI, and local experimental AV2/VVC software encoders.

The public Rust API is still in the 0.0.x line. It is intended to be useful for early integration and validation work, but names and behavior may break before 0.1.0 as the API contract cools down.

Goals

  • Provide safe Rust media pipeline components.
  • Keep codec implementations modular and independently selectable at build time.
  • Validate generated bitstreams and reconstructions with strict, reproducible tests.
  • Support open-source use and separately licensed commercial/private use.
  • Grow from codec and validation foundations into a broader media toolkit without forcing premature abstractions.

Quick Start

Requirements:

  • Rust toolchain with Cargo.
  • make.

Check the local toolchain:

make check-tools

Build and test:

make build
make test

make build creates a release binary at:

./ff

For a debug build, use:

make debug

Run the CLI:

make run ARGS="--help"

The crates.io package is intended to be framefinery. By default it includes the public library facade, the ff CLI binary, AV2, VVC, and the product filter catalog:

cargo install framefinery
ff --help

API documentation is published by docs.rs after crates.io publication:

https://docs.rs/framefinery

Generate the same API reference locally from the Rust /// and //! documentation comments:

make api-docs

Open the generated facade reference at:

target/doc/framefinery/index.html

Before release, run the strict documentation audit:

make api-docs-strict

That target fails when exported API is missing Rustdoc comments, so the generated reference stays tied to the code instead of becoming a stale manual.

The CLI guide lives in docs/cli.md.

Library API Sketch

The public Rust facade is codec-neutral. Select the codec once through the fluent encoder("...") builder or through a lower-level VideoEncoderConfig; callers should not fetch a codec manifest and then pass a separate config for a different codec. Manifest lookup is for discovery, catalogs, help text, and capability checks:

use framefinery::{encoder, find_encoder_manifest, Frame, FrameInfo, PixelFormat, Result};

fn encode_one_black_frame() -> Result<Vec<u8>> {
    let info = FrameInfo::new(16, 16, PixelFormat::Yuv420p8)?;

    let manifest = find_encoder_manifest("av2").expect("av2 is enabled in the default build");
    assert!(manifest.accepts_frame_info(info));

    let output = encoder("av2")?
        .input(info)
        .lossless()
        .reconstruction_frames()
        .encode_frame(Frame::blank(info))?;
    Ok(output
        .chunks
        .into_iter()
        .flat_map(|chunk| chunk.data)
        .collect())
}

For longer raw streams, call .into_config()? on the builder and pass that to encode_source(&config, source, output, recon, metrics). The source callback fills one frame buffer at a time and returns Ok(false) at clean EOF, so file and capture adapters do not need to preload a whole video in memory.

Generate a standalone Rust module/code browser:

make code-browser

The generated HTML is written under verification/generated/code_browser/. To overlay gated wall-time profiling data on the same browser, run:

make profile-hotspots HOTSPOT_CODECS="av2 vvc" HOTSPOT_VISUALIZE=1

That workflow builds only the requested compile-gated stats features and writes the matrix, wall-time summary, raw JSONL traces, and heatmap browser under verification/generated/profiling/hotspots/.

The installed command name is intended to be short:

./ff --help
./ff --help codecs
./ff --help filters
./ff --help filters pattern
./ff --help pixfmt
./ff --help settings
./ff --help settings qp
./ff --help presets

Run the default local quality gate:

make ci
make release-check

make ci is the same gate used by GitHub Actions. It runs the release check, the strict exported-API documentation audit, a wasm32-unknown-unknown type-check, and required-reference AV2/VVC validation. Portable generated fixtures cover lossless/QP24, intra/predictive/reset GOPs, native depths and chroma formats, RGB, padding, filters, and complete per-frame session output. Both reference decoders must already be available locally; missing tools fail the gate. CI builds pinned AVM/VTM revisions before running it. The suite records all failing groups and retains their artifacts for investigation. It does not require private media files; the full six-vector release check remains an additional local gate. make release-check is kept as the explicit release-oriented target name. Use make package-verify with current Cargo to build the actual archives and verify their license files and dependency versions before publication.

The first target-practice browser WASM demo is intentionally not a published web package. Build it and serve the local screen-capture/upload example with:

rustup target add wasm32-unknown-unknown
make wasm-build
make wasm-screen-demo

The demo writes uploaded elementary streams under verification/generated/wasm_screen_capture/.

Release candidates should also run the local AOM CTC release manifest and save a performance table for version-to-version comparison:

make validate-release-aomctc AOMCTC_ROOT=/path/to/aomctc
make release-performance-table AOMCTC_ROOT=/path/to/aomctc
make pre-release-validation AOMCTC_ROOT=/path/to/aomctc

Those targets require AOMCTC_ROOT to point at the local A5/B2 Y4M tree, read the source files directly, avoid decompression, and clean encoded, reconstruction, and generated raw-vector artifacts after metrics are collected.

Generate and run the current software encode fixtures:

make test-vector-sets
make validate-set CODEC=av2 VALIDATION_SET=smoke
make validate-set CODEC=vvc VALIDATION_SET=smoke
make validate-set CODEC=av2 VALIDATION_SET=smoke VALIDATION_SOURCE_FILTERS=1
make validate-set CODEC=av2 VALIDATION_SET=pipeline-smoke

Reference decoders are required for validation. Declared reference toolchains can be listed and built with:

make reference-list
make reference-setup
make reference-setup REFERENCE_CODEC=av2

Validation uses VALIDATION_REFERENCE_MODE=required by default: the runner decodes the FrameFinery Engine bitstream and compares the reference output against the encoder's internal reconstruction. A missing decoder is a failure. Explicit auto and off remain available for diagnostic runs; those runs cannot establish release compatibility. CI always requires external decoding.

Build-Time Composition

Codec and filter availability is selected at build time. The published framefinery package enables the normal product feature set by default, so cargo install framefinery and make build include every codec and product filter stage currently intended for the developer CLI without compiling analysis-only instrumentation.

Override CARGO_FEATURES to build a smaller or more specialized binary:

make build CARGO_FEATURES=all
make build CARGO_FEATURES="all-codecs all-filters"
make build CARGO_FEATURES="av2 filter-pattern filter-identity"
make build CARGO_FEATURES=

CARGO_FEATURES=all means all normal product features. The user-facing av2 and vvc features, plus the compatibility codec-av2 and codec-vvc features, enable the local experimental software encoders. The filter-* features map to reusable filter manifests and implementations in framefinery-api. The default product-filters feature enables executable pattern, identity, crop, and scale stages; all-filters is kept as a compatibility alias for the complete filter catalog. Custom builds can disable default features and re-enable only the filters they want. Analysis-only features, such as AV2 superblock bit accounting and codec wall-time profiling, are enabled through dedicated Makefile switches like AV2_SB_BITS=1, AV2_STATS=1, and VVC_STATS=1 so the normal product build is not slowed by instrumentation.

CLI Shape

The CLI entry point is ff. The initial interface is centered on stage discovery and a single encode action:

ff --help
ff --help codecs
ff --help filters
ff --help filters pattern
ff --help pixfmt
ff --help settings
ff --help settings qp
ff --help presets
ff codecs
ff filters
ff encode input.yuv --video 640x360:yuv444p \
  --encode av2:output.obu --set lossless
ff encode input.y4m --encode av2:output.obu --set lossless
ff encode input.y4m --encode av2:output.obu --set qp=24
ff encode --filter pattern=checker --video 64x64:yuv444p \
  --frames 1 --encode av2:pattern.obu
ff encode input_640x360_30_1f_yuv444p8.yuv \
  --filter identity --encode av2:output.obu
ff encode input_640x360_30_1f_yuv444p8.yuv \
  --encode av2:output.obu --recon output_recon.yuv
ff encode input_640x360_30_1f_yuv444p8.yuv \
  --encode av2:output.obu --set qp=24 --psnr

The commands validate command-line structure and report stage availability. When built with av2/codec-av2 or vvc/codec-vvc, ff encode can encode raw YUV inputs and Y4M inputs through the imported software model for that codec. Y4M files are demuxed by the shared input reader before frames reach AV2 or VVC.

Current option placement and inference rules:

  • Input options, such as --video, --fps, and --frames, belong after the input path and override metadata inferred from filenames or Y4M headers.
  • If --frames and filename frame-count metadata are both omitted for a file input, ff encode processes whole frames until the raw input file or Y4M stream reaches EOF.
  • If --frames is larger than the number of complete frames in a file, ff encode stops at EOF instead of failing.
  • Source filters require explicit --frames because they do not have a file EOF.
  • Output/encoder options, such as --recon output.yuv, --psnr, --preset, and repeated --set key[=value], belong after --encode codec:output.
  • --recon <path> writes the encoder's internal reconstructed raw stream for debugging and reference validation. --psnr calculates per-frame PSNR from that same internal reconstruction without writing the raw reconstruction stream.
  • Bare --set keys imply true. Accepted settings and setting-specific defaults/contracts are listed by ff --help settings and ff --help settings <name>.
  • VVC accepts --set profile=auto|main-10-444|main-12-444|main-10|main-12. The default auto keeps palette-capable profile signalling unless a tighter profile is explicitly requested.

Global accepted settings are listed by ff --help settings; codec-specific settings are listed with the codec that owns them. Temporal prediction defaults to gop=-1, meaning one intra frame followed by unbounded predictive frames. Use --set gop=0 for intra-only coding, or a positive value such as --set gop=30 to reset codec reference state at fixed GOP boundaries.

The positional input is optional when the first filter is a source. The initial source filter is pattern=<name>, with black, checker, gradient, and color_blocks patterns. Source filters require explicit --video metadata because there is no filename to infer dimensions or pixel format from. The identity transform filter is executable for file inputs and source-filter inputs. crop extracts rectangular regions and scale performs deterministic nearest-neighbor resizing. The reusable filter catalog and implementations live in framefinery-api; the ff CLI reads that manifest and maps command-line strings onto the core pipeline API. Each filter manifest also points at a typed spec manifest, so ff --help filters <name> can print the accepted forms, parameters, examples, and notes without duplicating filter-specific prose in the CLI.

Default filter capability:

Filter Kind Status
pattern=<name> source executable generated input
identity transform executable no-op frame pass-through
crop transform executable rectangular crop
scale transform executable nearest-neighbor resize

Raw video metadata uses a compact WxH:pixfmt spelling when it cannot be inferred from the input filename or Y4M header, or when it needs to be overridden. File names imply metadata with *_<WxH>[_<fps>][_<frames>f][_<pixfmt>].yuv, for example input_640x360_30_1f_yuv444p8.yuv. Y4M headers provide width, height, frame rate, and planar YUV pixel format. Short 8-bit aliases such as yuv444p and yuv420p are accepted and normalized to yuv444p8 and yuv420p8 internally. Planar YUV and gray input formats accept checked numeric bit depths from 8 through 16, for example yuv420p9le, yuv444p12le, and gray16le. If a .yuv filename has dimensions but no pixel-format token, the CLI assumes yuv420p8. Encode endpoints must name the codec and output path together, such as --encode av2:output.obu.

The raw input CLI/API contract is documented in docs/raw-input-formats.md, including the current codec/input support matrix for tested chroma layouts, bit depths, lossless paths, fallback behavior, and geometry limits. The initial molten Rust/API contract is documented in docs/api-v0.md.

Repository Layout

crates/
  framefinery-cli/   Published as package `framefinery`; library facade plus `ff`.
  framefinery-api/  Shared frame, packet, pipeline, source, and filter APIs.
  framefinery-codecs/  Local experimental AV2/VVC software encoders.
docs/                     Architecture and validation notes.
tests/                    Future shared integration tests and fixtures.
tools/                    Future development and validation helper scripts.

Current Limitations

  • Compressed input decode is not implemented yet; ff encode accepts raw YUV and Y4M inputs.
  • AV2 and VVC encoders are experimental software implementations, not production codec implementations.
  • VVC signals level 15.5/high tier under H.266 (09/2023), reflecting that the encoder does not enforce ordinary level bitrate/resource ceilings. Decoders must assess the stream's actual requirements; bounded-level compatibility is not promised.
  • crop requires chroma-aligned rectangles for subsampled planar formats.
  • scale currently uses nearest-neighbor sampling rather than a resampling kernel intended for display-quality resizing.
  • Validation defaults to required reference decoding. CI and release claims require the decoders; explicit auto/off diagnostic runs do not establish reference compatibility.

Safety Posture

FrameFinery Engine should use safe Rust. Performance work should start with safe Rust, better algorithms, optimizer-friendly data layout, and compiler-supported optimizations. Optimized implementations that replace correctness-critical kernels should be proven bit-exact against simple scalar implementations.

Validation Direction

Validation should remain strict and reproducible:

  • lossless paths must reconstruct exactly;
  • lossy paths should report PSNR and bitrate;
  • reference decoders should validate generated bitstreams when available;
  • checksums and bitstream sizes should be recorded for regressions;
  • generated test vectors should be deterministic.

The first batch fixtures live under verification/test_vector_sets/. They are generated on demand into verification/generated/test_vectors/ and encoded by scripts/run_validation_set.py, which records per-vector logs, output sizes, and SHA-256 checksums under verification/generated/.

License

FrameFinery Engine is offered under a dual-license model:

  • Open-source distribution under the GNU Affero General Public License version 3 or later (AGPL-3.0-or-later). See LICENSE.
  • Commercial licensing under separate written agreement from the applicable copyright holder(s). See COMMERCIAL-LICENSE.md.

Unless a file states otherwise, the repository source code, documentation, tests, fixtures, and validation scripts are covered by AGPL-3.0-or-later. Third-party dependencies, external reference tools, generated artifacts, and local vendor tooling remain under their own licenses.

Codec patent rights are separate from source-code copyright licensing. Users are responsible for evaluating any codec patent or deployment obligations that apply to their use case and jurisdiction.