EdgeFirst Codec

Zero-allocation image decoding framework for pre-allocated tensor buffers in real-time vision pipelines.

Overview

edgefirst-codec decodes JPEG and PNG images directly into pre-allocated Tensor<T> or TensorDyn buffers, supporting strided memory layouts (GPU pitch-aligned DMA-BUF, PBO). This eliminates per-frame allocations in the hot loop — the primary design goal.

The decoder emits each image in its native pixel format and never colour- converts, rotates, or resizes. Colour conversion and geometry are the job of ImageProcessor::convert() downstream:

The decoder configures the destination tensor's dimensions and pixel format to match the decoded image (within the tensor's existing allocation), so a single tensor sized for the largest expected frame can receive smaller images without reallocating.

JPEG decoding uses a custom from-scratch baseline decoder with reusable state, achieving zero heap allocations after the first decode at each resolution. SIMD-optimized IDCT kernels (NEON on AArch64, SSE4.1/SSE2 on x86-64) are selected automatically at init via dynamic dispatch. On Linux, an optional V4L2 hardware backend offloads JPEG decode to a SoC accelerator when one is present. PNG decoding uses zune-png.

Quick Start

use edgefirst_codec::{ImageDecoder, ImageLoad};
use edgefirst_tensor::{CpuAccess, Tensor, PixelFormat, TensorMemory};

// Allocate once at init (prefer ImageProcessor::create_image() for DMA/PBO).
// A colour JPEG decodes to NV12, so allocate an NV12 tensor. The decoder
// CPU-writes the pixels: declare CpuAccess::Write.
let mut tensor = Tensor::<u8>::image(1920, 1080, PixelFormat::Nv12,
    Some(TensorMemory::Mem), CpuAccess::Write).unwrap();
let mut decoder = ImageDecoder::new();

// Decode in the hot loop — zero allocations after warmup for JPEG.
let jpeg_bytes = std::fs::read("frame.jpg").unwrap();
let info = tensor.load_image(&mut decoder, &jpeg_bytes).unwrap();
println!("Decoded {}x{} {:?}", info.width, info.height, info.format);
// info.rotation_degrees / info.flip_horizontal carry the EXIF orientation
// the caller should apply downstream (the codec never rotates — see below).

Recommended Pattern

For maximum performance, use tensors allocated by ImageProcessor::create_image() and convert the native decode into the format your pipeline needs:

use edgefirst_image::{ImageProcessor, ImageProcessorTrait, Crop, Rotation, Flip};
use edgefirst_codec::{ImageDecoder, ImageLoad};

let mut processor = ImageProcessor::new()?;
// Source tensor holds the codec's native NV12; destination is the RGB the
// model consumes.
let mut src =
    processor.create_image(1920, 1080, PixelFormat::Nv12, DType::U8, None, CpuAccess::ReadWrite)?;
let mut dst =
    processor.create_image(640, 640, PixelFormat::Rgb, DType::U8, None, CpuAccess::ReadWrite)?;
let mut decoder = ImageDecoder::new();

loop {
    let bytes = capture_frame();
    let info = src.load_image(&mut decoder, &bytes)?;
    // convert() performs colour conversion (NV12 → RGB), resize, and any EXIF
    // rotation/flip the decode reported.
    processor.convert(&src, &mut dst, Rotation::None, Flip::None,
        Crop::new(0, 0, info.width, info.height))?;
}

Benefits of ImageProcessor::create_image() tensors:

• DMA-BUF backing for zero-copy GPU import (and the V4L2 zero-copy decode path — see below)
• PBO backing when OpenGL is the active transfer path
• GPU pitch alignment (64-byte for Mali compatibility)

Free-standing tensors work but cannot use PBO and may lack GPU-aligned pitch.

EXIF Orientation: Reported, Never Applied

The decoder reports the source's EXIF orientation in ImageInfo but writes the pixels and dimensions exactly as stored — it never rotates or flips. This keeps the decode path branch-free and lets the GPU convert() apply orientation for free alongside colour conversion and resize.

ImageInfo carries the transform the caller should apply:

• rotation_degrees: clockwise rotation in degrees (0 / 90 / 180 / 270)
• flip_horizontal: whether to also mirror horizontally

Apply the rotation by passing it to ImageProcessor::convert() (mapping rotation_degrees to Rotation and flip_horizontal to Flip). When the image has no EXIF orientation both fields are 0 / false.

Hardware Acceleration (V4L2)

On Linux, the v4l2 feature (enabled by default) adds a hardware JPEG-decode backend that drives any device exposing a JPEG decoder through the standard V4L2 mem2mem (M2M) API — the lead target is i.MX mxc-jpeg, but discovery is purely capability-based, so no device node, driver name, or output format is hardcoded.

The backend is probed lazily on the first JPEG decode and is tried before the software decoder. Anything it cannot drive transparently — no JPEG M2M device present, an unsupported capture format, a per-image hardware failure — falls back to the from-scratch CPU decoder, producing identical native output. After repeated failures a circuit breaker demotes the device to CPU for the rest of the session.

The deps (nix, libc) are pulled in only on Linux targets and all backend code is gated #[cfg(all(target_os = "linux", feature = "v4l2"))], so off Linux the feature compiles to nothing.

When the destination is a DMA-backed tensor with MCU(16)-aligned dimensions and the driver accepts a single-plane contiguous capture at the tensor pitch, the hardware decodes straight into the tensor's dmabuf — a true zero-copy path. Otherwise the driver buffers are mapped and the decoded planes are copied (cropped to the logical image) into the destination.

Supported Formats

Need Rgb/Rgba/Bgra from a JPEG, or a resized/rotated result? Decode to the native format, then call ImageProcessor::convert().

Decoder Limitations

The codec decodes a strict subset of the JPEG / PNG specs. Inputs that fall outside the supported subset surface a typed CodecError::Unsupported(...) variant so callers can pattern-match programmatically (no string parsing required).

JPEG

PNG

PNG decoding goes through zune-png; the codec writes the native colorspace (Luma/LumaA → Grey, RGB → Rgb, RGBA → Rgba) into the tensor with stride-aware row copies and optional bit-depth/dtype conversion.

Data Types

JPEG decodes to u8 only (its native Nv12/Grey are byte layouts). PNG supports the full set of tensor element types:

API Reference

ImageDecoder

Reusable decoder with internal scratch buffers (and, on Linux, the lazily-probed V4L2 backend state). Create once, reuse across frames — scratch buffers and the hardware streaming session amortize after the first decode.

ImageLoad Extension Trait

Implemented for both Tensor<T> and TensorDyn:

• load_image(&mut self, decoder, data) — decode from &[u8]
• load_image_read(&mut self, decoder, reader) — decode from Read
• load_image_file(&mut self, decoder, path) — decode from file path

Each configures the tensor's dimensions and format to the decoded native format and returns an ImageInfo. Returns CodecError::InsufficientCapacity if the decoded image is larger than the tensor's allocation.

ImageInfo

Returned by all decode methods:

• width, height: decoded image size (the source's true, unrotated dimensions)
• format: native pixel format written to the tensor
• row_stride: row stride in bytes used when writing into the tensor
• rotation_degrees: EXIF clockwise rotation the caller should apply (0/90/180/270)
• flip_horizontal: whether the caller should also flip horizontally

peek_image_info(data) returns the same metadata without decoding pixels.

License

Apache-2.0