Syeve – Simple yet efficient video encoding

This is a lossless video codec, aiming at screen capture and anything else you'd want to format in PNG instead of JPEG. It is the equivalent of PNG for video.

As PNG, there are two steps: filtering, and compressing.

The filter transforms the pixels into the differences between adjacent pixels. This way adjacent pixels with the same color have the value 0, and linear gradients become constant series. The filtered image is now easier to compress with a conventional algorithm.

The difference with PNG is that the filter also takes the difference across time, so that if the frames are almost the same, they will be full of zeros.

Example

Run a P2P video-conferencing example:

# Run these commands in two different terminals or machines
cargo run --release -p conference -- server 0.0.0.0:8080
cargo run --release -p conference -- client 127.0.0.1:8080

(change 127.0.0.1 to the server address)

// Let `input_raw_frame: Vec<u8>` be an RGB24 bitmap of size 1920*1080

let mut buf = Vec::new();
let mut encoder = syeve::Encoder::new((1920, 1080), 3, syeve::Compression::Brotli(4), 30);
encoder.encode(&mut input_raw_frame, &mut buf).unwrap();

let mut decoder = syeve::Decoder::new();
let output_raw_frame = decoder.decode(&buf).unwrap().pixels;

Motivation

Lossy video formats are very efficient, but:

they are super complex and involve lots of mathematical concepts that you can't understand unless it's your research field;
hence, they have a very few implementations;
none of these implementations are easy to use for a programmer;
they perform badly on screen captures, both in bandwidth and legibility;
video processing is fun.

Status

⚠ unstable, not benchmarked, not fuzz-tested... but promising 😎

Divide image into individual blocks
Remove unsafe code
Optional lossy optimization? (posterization)
Benchmark
More tests (frame loss)
Avoid allocation on decoding

Single frame encoding is faster than with image-rs/image-png (thanks to SIMD optimization).

Any help with optimization (potentially lossy) or benchmark or else would be greatly appreciated. (PR/issue/email welcome)

Protocol

The protocol is extremely simple: comparable to PNG, but each frame depends on the previous frame.

sequence number (u32 BE)
frame number (u32 BE)
width (u32 BE)
height (u32 BE)
pixel size (u8)
compression algo (u8)
data

"data" is the filtered bitmap image, compressed with the specified algo. Its length is width*height*pixel_size. (pixel_size is the number of bytes per pixel, generally 3 for RGB and 4 for RGBA)

Compression algos:

brotli
deflate

The first frame should have frame number to zero. Frame number must be incremented by one between each frame. Because each frame depends on the previous frame, a frame loss may result in a broken frame for the decoder. This problem is reduced by sending periodically a fully self-described frame. This frame should have frame number to zero, and a sequence number different from the previous frame (typically incremented by one). The sequence number of the first frame can be random.

Sources

Donate on LiberaPay

License

CopyLeft 2021 Pascal Engélibert

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3 of the License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

syeve 0.1.0