FrameFinery Engine
FrameFinery Engine is a safe Rust media pipeline toolkit. The project is built around composable stages:
input -> decode -> filter -> encode -> output
The initial focus is experimental video encoding and validation infrastructure, with AV2 and VVC as the first planned codec families. The repository is a software-only sibling of the FrameFinery Engine hardware project: FrameFinery Engine remains the RTL, synthesis, and hardware-validation workspace, while this repository is free to optimize for software APIs, usability, codec quality, and safe Rust performance.
This repository is in bootstrap state. It currently provides project structure, shared media primitives, a CLI, and local experimental AV2/VVC software encoders.
The public Rust API is still in the 0.0.x line. It is intended to be useful
for early integration and validation work, but names and behavior may break
before 0.1.0 as the API contract cools down.
Goals
- Provide safe Rust media pipeline components.
- Keep codec implementations modular and independently selectable at build time.
- Validate generated bitstreams and reconstructions with strict, reproducible tests.
- Support open-source use and separately licensed commercial/private use.
- Grow from codec and validation foundations into a broader media toolkit without forcing premature abstractions.
Quick Start
Requirements:
- Rust toolchain with Cargo.
make.
Check the local toolchain:
Build and test:
make build creates a release binary at:
For a debug build, use:
Run the CLI:
The crates.io package is intended to be framefinery. By default it includes
the public library facade, the ff CLI binary, AV2, VVC, and the product filter
catalog:
API documentation is published by docs.rs after crates.io publication:
https://docs.rs/framefinery
Generate the same API reference locally from the Rust /// and //!
documentation comments:
Open the generated facade reference at:
target/doc/framefinery/index.html
Before release, run the strict documentation audit:
That target fails when exported API is missing Rustdoc comments, so the generated reference stays tied to the code instead of becoming a stale manual.
The CLI guide lives in
docs/cli.md.
Library API Sketch
The public Rust facade is codec-neutral. Select the codec once through the
fluent encoder("...") builder or through a lower-level
VideoEncoderConfig; callers should not fetch a codec manifest and then pass a
separate config for a different codec. Manifest lookup is for discovery,
catalogs, help text, and capability checks:
use ;
For longer raw streams, call .into_config()? on the builder and pass that to
encode_source(&config, source, output, recon, metrics). The source callback
fills one frame buffer at a time and returns Ok(false) at clean EOF, so file
and capture adapters do not need to preload a whole video in memory.
Generate a standalone Rust module/code browser:
The generated HTML is written under verification/generated/code_browser/.
To overlay gated wall-time profiling data on the same browser, run:
That workflow builds only the requested compile-gated stats features and writes
the matrix, wall-time summary, raw JSONL traces, and heatmap browser under
verification/generated/profiling/hotspots/.
The installed command name is intended to be short:
Run the default local quality gate:
make ci is the same gate used by GitHub Actions. It runs the release check,
the strict exported-API documentation audit, a wasm32-unknown-unknown
type-check, and required-reference AV2/VVC validation. Portable generated
fixtures cover lossless/QP24, intra/predictive/reset GOPs, native depths and
chroma formats, RGB, padding, filters, and complete per-frame session output.
Both reference decoders must already be available locally; missing tools fail
the gate. CI builds pinned AVM/VTM revisions before running it. The suite
records all failing groups and retains their artifacts for investigation.
It does not require private media files; the full six-vector release check
remains an additional local gate.
make release-check is kept as the explicit release-oriented target name.
Use make package-verify with current Cargo to build the actual archives and
verify their license files and dependency versions before publication.
The first target-practice browser WASM demo is intentionally not a published web package. Build it and serve the local screen-capture/upload example with:
The demo writes uploaded elementary streams under
verification/generated/wasm_screen_capture/.
Release candidates should also run the local AOM CTC release manifest and save a performance table for version-to-version comparison:
Those targets require AOMCTC_ROOT to point at the local A5/B2 Y4M tree, read
the source files directly, avoid decompression, and clean encoded,
reconstruction, and generated raw-vector artifacts after metrics are collected.
Generate and run the current software encode fixtures:
Reference decoders are required for validation. Declared reference toolchains can be listed and built with:
Validation uses VALIDATION_REFERENCE_MODE=required by default: the runner
decodes the FrameFinery Engine bitstream and compares the reference output
against the encoder's internal reconstruction. A missing decoder is a failure.
Explicit auto and off remain available for diagnostic runs; those runs
cannot establish release compatibility. CI always requires external decoding.
Build-Time Composition
Codec and filter availability is selected at build time. The published
framefinery package enables the normal product feature set by default, so
cargo install framefinery and make build include every codec and product
filter stage currently intended for the developer CLI without compiling
analysis-only instrumentation.
Override CARGO_FEATURES to build a smaller or more specialized binary:
CARGO_FEATURES=all means all normal product features. The user-facing av2
and vvc features, plus the compatibility codec-av2 and codec-vvc
features, enable the local experimental software encoders. The filter-*
features map to reusable filter manifests and implementations in
framefinery-api. The default product-filters feature enables executable
pattern, identity, crop, and scale stages; all-filters is kept as a
compatibility alias for the complete filter catalog. Custom builds can disable
default features and re-enable only the filters they want. Analysis-only
features, such as AV2 superblock bit accounting and codec wall-time profiling,
are enabled through dedicated Makefile switches like AV2_SB_BITS=1,
AV2_STATS=1, and VVC_STATS=1 so the normal product build is not slowed by
instrumentation.
CLI Shape
The CLI entry point is ff. The initial interface is centered on stage
discovery and a single encode action:
The commands validate command-line structure and report stage availability.
When built with av2/codec-av2 or vvc/codec-vvc, ff encode can encode
raw YUV inputs and Y4M inputs through the imported software model for that codec. Y4M
files are demuxed by the shared input reader before frames reach AV2 or VVC.
Current option placement and inference rules:
- Input options, such as
--video,--fps, and--frames, belong after the input path and override metadata inferred from filenames or Y4M headers. - If
--framesand filename frame-count metadata are both omitted for a file input,ff encodeprocesses whole frames until the raw input file or Y4M stream reaches EOF. - If
--framesis larger than the number of complete frames in a file,ff encodestops at EOF instead of failing. - Source filters require explicit
--framesbecause they do not have a file EOF. - Output/encoder options, such as
--recon output.yuv,--psnr,--preset, and repeated--set key[=value], belong after--encode codec:output. --recon <path>writes the encoder's internal reconstructed raw stream for debugging and reference validation.--psnrcalculates per-frame PSNR from that same internal reconstruction without writing the raw reconstruction stream.- Bare
--setkeys implytrue. Accepted settings and setting-specific defaults/contracts are listed byff --help settingsandff --help settings <name>. - VVC accepts
--set profile=auto|main-10-444|main-12-444|main-10|main-12. The defaultautokeeps palette-capable profile signalling unless a tighter profile is explicitly requested.
Global accepted settings are listed by ff --help settings; codec-specific
settings are listed with the codec that owns them. Temporal prediction defaults
to gop=-1, meaning one intra frame followed by unbounded predictive frames.
Use --set gop=0 for intra-only coding, or a positive value such as
--set gop=30 to reset codec reference state at fixed GOP boundaries.
The positional input is optional when the first filter is a source. The initial
source filter is pattern=<name>, with black, checker, gradient, and
color_blocks patterns. Source filters require explicit --video metadata
because there is no filename to infer dimensions or pixel format from. The
identity transform filter is executable for file inputs and source-filter
inputs. crop extracts rectangular regions and scale performs deterministic
nearest-neighbor resizing.
The reusable filter catalog and implementations live in framefinery-api; the
ff CLI reads that manifest and maps command-line strings onto the core
pipeline API. Each filter manifest also points at a typed spec manifest, so
ff --help filters <name> can print the accepted forms, parameters, examples,
and notes without duplicating filter-specific prose in the CLI.
Default filter capability:
| Filter | Kind | Status |
|---|---|---|
pattern=<name> |
source | executable generated input |
identity |
transform | executable no-op frame pass-through |
crop |
transform | executable rectangular crop |
scale |
transform | executable nearest-neighbor resize |
Raw video metadata uses a compact WxH:pixfmt spelling when it cannot be
inferred from the input filename or Y4M header, or when it needs to be
overridden. File names imply metadata with
*_<WxH>[_<fps>][_<frames>f][_<pixfmt>].yuv, for example
input_640x360_30_1f_yuv444p8.yuv. Y4M headers provide width, height, frame
rate, and planar YUV pixel format. Short 8-bit aliases such as yuv444p and
yuv420p are accepted and normalized to yuv444p8 and yuv420p8 internally.
Planar YUV and gray input formats accept checked numeric bit depths from 8
through 16, for example yuv420p9le, yuv444p12le, and gray16le.
If a .yuv filename has dimensions but no pixel-format token, the CLI assumes
yuv420p8. Encode endpoints must name the codec and output path together, such
as --encode av2:output.obu.
The raw input CLI/API contract is documented in
docs/raw-input-formats.md, including the current
codec/input support matrix for tested chroma layouts, bit depths, lossless
paths, fallback behavior, and geometry limits.
The initial molten Rust/API contract is documented in
docs/api-v0.md.
Repository Layout
crates/
framefinery-cli/ Published as package `framefinery`; library facade plus `ff`.
framefinery-api/ Shared frame, packet, pipeline, source, and filter APIs.
framefinery-codecs/ Local experimental AV2/VVC software encoders.
docs/ Architecture and validation notes.
tests/ Future shared integration tests and fixtures.
tools/ Future development and validation helper scripts.
Current Limitations
- Compressed input decode is not implemented yet;
ff encodeaccepts raw YUV and Y4M inputs. - AV2 and VVC encoders are experimental software implementations, not production codec implementations.
- VVC signals level 15.5/high tier under H.266 (09/2023), reflecting that the encoder does not enforce ordinary level bitrate/resource ceilings. Decoders must assess the stream's actual requirements; bounded-level compatibility is not promised.
croprequires chroma-aligned rectangles for subsampled planar formats.scalecurrently uses nearest-neighbor sampling rather than a resampling kernel intended for display-quality resizing.- Validation defaults to required reference decoding. CI and release claims
require the decoders; explicit
auto/offdiagnostic runs do not establish reference compatibility.
Safety Posture
FrameFinery Engine should use safe Rust. Performance work should start with safe Rust, better algorithms, optimizer-friendly data layout, and compiler-supported optimizations. Optimized implementations that replace correctness-critical kernels should be proven bit-exact against simple scalar implementations.
Validation Direction
Validation should remain strict and reproducible:
- lossless paths must reconstruct exactly;
- lossy paths should report PSNR and bitrate;
- reference decoders should validate generated bitstreams when available;
- checksums and bitstream sizes should be recorded for regressions;
- generated test vectors should be deterministic.
The first batch fixtures live under verification/test_vector_sets/. They are
generated on demand into verification/generated/test_vectors/ and encoded by
scripts/run_validation_set.py, which records per-vector logs, output sizes,
and SHA-256 checksums under verification/generated/.
License
FrameFinery Engine is offered under a dual-license model:
- Open-source distribution under the GNU Affero General Public License version
3 or later (
AGPL-3.0-or-later). SeeLICENSE. - Commercial licensing under separate written agreement from the applicable
copyright holder(s). See
COMMERCIAL-LICENSE.md.
Unless a file states otherwise, the repository source code, documentation,
tests, fixtures, and validation scripts are covered by AGPL-3.0-or-later.
Third-party dependencies, external reference tools, generated artifacts, and
local vendor tooling remain under their own licenses.
Codec patent rights are separate from source-code copyright licensing. Users are responsible for evaluating any codec patent or deployment obligations that apply to their use case and jurisdiction.