EdgeFirst Codec

Zero-allocation image decoding framework for pre-allocated tensor buffers in real-time vision pipelines.

Overview

edgefirst-codec decodes JPEG and PNG images directly into pre-allocated Tensor<T> or TensorDyn buffers, supporting strided memory layouts (GPU pitch-aligned DMA-BUF, PBO). This eliminates per-frame allocations in the hot loop — the primary design goal.

The decoder emits each image in its native pixel format and never colour- converts, rotates, or resizes. Colour conversion and geometry are the job of ImageProcessor::convert() downstream:

A colour JPEG lands on the NV format that matches its own chroma sampling, so nothing is resampled on the way out. Only exotic subsamplings (4:1:1, 4:1:0, mismatched Cb/Cr) get downsampled, and they fall back to Nv12.

The decoder configures the destination tensor's dimensions and pixel format to match the decoded image (within the tensor's existing allocation), so a single tensor sized for the largest expected frame can receive smaller images without reallocating.

JPEG decoding uses a custom from-scratch baseline decoder with reusable state, achieving zero heap allocations after the first decode at each resolution. SIMD-optimized IDCT kernels (NEON on AArch64, SSE4.1/SSE2 on x86-64) are selected automatically at init via dynamic dispatch. On Linux two optional hardware backends sit in front of it — V4L2 for SoC JPEG accelerators and nvJPEG for CUDA GPUs — and either falls back to the CPU decoder transparently. PNG decoding uses zune-png.

Feature Flags

Both default features are fully #[cfg]-gated to Linux, so off Linux they compile to nothing. nvjpeg is pure dlopen and adds no link-time CUDA dependency, so enabling it does not constrain where the binary runs.

Quick Start

use edgefirst_codec::{ImageDecoder, ImageLoad};
use edgefirst_tensor::{CpuAccess, PixelFormat, Tensor, TensorMemory};

// Allocate once at init (prefer ImageProcessor::create_image() for DMA/PBO).
// A 4:2:0 colour JPEG decodes to NV12, so allocate an NV12 tensor. The
// decoder CPU-writes the pixels: declare CpuAccess::Write.
let mut tensor = Tensor::<u8>::image(1920, 1080, PixelFormat::Nv12,
    Some(TensorMemory::Mem), CpuAccess::Write).unwrap();
let mut decoder = ImageDecoder::new();

// Decode in the hot loop — zero allocations after warmup for JPEG.
let jpeg_bytes = std::fs::read("frame.jpg").unwrap();
let info = tensor.load_image(&mut decoder, &jpeg_bytes).unwrap();
println!("Decoded {}x{} {:?}", info.width, info.height, info.format);
// info.rotation_degrees / info.flip_horizontal carry the EXIF orientation
// the caller should apply downstream (the codec never rotates — see below).

Don't know the image's dimensions or chroma sampling up front? peek_info parses the headers without touching pixels, so you can size the tensor to the format the decode will actually produce:

use edgefirst_codec::{peek_info, ImageDecoder, ImageLoad};
use edgefirst_tensor::{CpuAccess, Tensor, TensorMemory};

let data = std::fs::read("frame.jpg").unwrap();
let peek = peek_info(&data).unwrap();
let mut tensor = Tensor::<u8>::image(peek.width, peek.height, peek.format,
    Some(TensorMemory::Mem), CpuAccess::Write).unwrap();
let mut decoder = ImageDecoder::new();
let info = tensor.load_image(&mut decoder, &data).unwrap();

Recommended Pattern

For maximum performance, use tensors allocated by ImageProcessor::create_image() and convert the native decode into the format your pipeline needs:

use edgefirst_codec::{ImageDecoder, ImageLoad};
use edgefirst_image::{Crop, Flip, ImageProcessor, ImageProcessorTrait, Rotation};
use edgefirst_tensor::{CpuAccess, DType, PixelFormat};

let mut processor = ImageProcessor::new()?;
// Source tensor holds the codec's native NV12; destination is the RGB the
// model consumes.
let mut src =
    processor.create_image(1920, 1080, PixelFormat::Nv12, DType::U8, None, CpuAccess::ReadWrite)?;
let mut dst =
    processor.create_image(640, 640, PixelFormat::Rgb, DType::U8, None, CpuAccess::ReadWrite)?;
let mut decoder = ImageDecoder::new();

loop {
    let bytes = capture_frame();
    let info = src.load_image(&mut decoder, &bytes)?;
    // convert() performs colour conversion (NV12 → RGB) and resize; the codec
    // reports EXIF orientation in `info` but does not apply it, so pass it on.
    let rot = Rotation::from_degrees_clockwise(info.rotation_degrees as usize);
    let flip = if info.flip_horizontal { Flip::Horizontal } else { Flip::None };
    processor.convert(&src, &mut dst, rot, flip, Crop::new())?;
}

Benefits of ImageProcessor::create_image() tensors:

• DMA-BUF backing for zero-copy GPU import (and the V4L2 zero-copy decode path — see below)
• PBO backing when OpenGL is the active transfer path
• GPU pitch alignment (64-byte for Mali compatibility)

Free-standing tensors work but cannot use PBO and may lack GPU-aligned pitch.

EXIF Orientation: Reported, Never Applied

The decoder reports the source's EXIF orientation in ImageInfo but writes the pixels and dimensions exactly as stored — it never rotates or flips. This keeps the decode path branch-free and lets the GPU convert() apply orientation for free alongside colour conversion and resize.

ImageInfo carries the transform the caller should apply:

• rotation_degrees: clockwise rotation in degrees (0 / 90 / 180 / 270)
• flip_horizontal: whether to also mirror horizontally

Apply the rotation by passing it to ImageProcessor::convert() (mapping rotation_degrees to Rotation and flip_horizontal to Flip). When the image has no EXIF orientation both fields are 0 / false.

Hardware Acceleration (V4L2)

On Linux, the v4l2 feature (enabled by default) adds a hardware JPEG-decode backend that drives any device exposing a JPEG decoder through the standard V4L2 mem2mem (M2M) API — the lead target is i.MX mxc-jpeg, but discovery is purely capability-based, so no device node, driver name, or output format is hardcoded.

The backend is probed lazily on the first JPEG decode and is tried before the software decoder. Anything it cannot drive transparently — no JPEG M2M device present, an unsupported capture format, a per-image hardware failure — falls back to the from-scratch CPU decoder, producing identical native output. After repeated failures a circuit breaker demotes the device to CPU for the rest of the session.

The deps (nix, libc) are pulled in only on Linux targets and all backend code is gated #[cfg(all(target_os = "linux", feature = "v4l2"))], so off Linux the feature compiles to nothing.

When the destination is a DMA-backed tensor with MCU(16)-aligned dimensions and the driver accepts a single-plane contiguous capture at the tensor pitch, the hardware decodes straight into the tensor's dmabuf — a true zero-copy path. Otherwise the driver buffers are mapped and the decoded planes are copied (cropped to the logical image) into the destination.

GPU Acceleration (nvJPEG)

On Linux with CUDA, the nvjpeg feature (compiled in by default) adds a GPU JPEG-decode backend for NVIDIA platforms, with Jetson Orin as the lead target. It sits ahead of V4L2 and CPU in the dispatch order, but it is off unless you opt in:

The default is off because nvJPEG decodes on the same GPU your inference runs on. Sharing the device can cost a concurrent TensorRT engine more than the decode speedup returns, so it stays opt-in for decode-bound workloads or machines with no concurrent GPU compute. V4L2 is the mirror image — a separate hardware block that contends with nothing, so it is opt-out via EDGEFIRST_DISABLE_V4L2.

Two things to know before enabling it:

• It emits Rgb, not an NV format — a deliberate exception to the native-format contract. nvJPEG does YCbCr→RGB on the GPU at near-zero marginal cost and the result is GPU-resident, which removes the downstream NV12→RGB step. (The L4T nvJPEG 12.3.3 build has no NV12 output mode anyway.)
• It only engages for a CUDA-backed destination — in practice a TensorMemory::Pbo tensor from ImageProcessor::create_image() on Jetson. Anything else is left untouched and falls through to V4L2 or CPU.

The backend is pure dlopen with no link-time CUDA dependency, so one binary runs on Jetson (nvJPEG), i.MX (V4L2), and a laptop (CPU). nvjpeg_available() reports whether it is live — useful for benchmarks and for consumers that branch on backend availability. Transient nvJPEG failures restore the native format and fall through to the CPU decoder, which also covers the progressive and non-baseline JPEGs nvJPEG rejects.

Supported Formats

Need Rgb/Rgba/Bgra from a JPEG, or a resized/rotated result? Decode to the native format, then call ImageProcessor::convert().

The one exception is the nvJPEG GPU backend, which emits packed Rgb — see below.

Decoder Limitations

The codec decodes a strict subset of the JPEG / PNG specs. Inputs that fall outside the supported subset surface a typed CodecError::Unsupported(...) variant so callers can pattern-match programmatically (no string parsing required).

JPEG

PNG

PNG decoding goes through zune-png; the codec writes the native colorspace (Luma/LumaA → Grey, RGB → Rgb, RGBA → Rgba) into the tensor with stride-aware row copies and optional bit-depth/dtype conversion.

Data Types

JPEG decodes to u8 only (its native NV formats and Grey are byte layouts). PNG supports the full set of tensor element types:

API Reference

ImageDecoder

Reusable decoder with internal scratch buffers (and, on Linux, the lazily-probed V4L2 backend state). Create once, reuse across frames — scratch buffers and the hardware streaming session amortize after the first decode.

ImageLoad Extension Trait

Implemented for both Tensor<T> and TensorDyn:

• load_image(&mut self, decoder, data) — decode from &[u8]
• load_image_read(&mut self, decoder, reader) — decode from Read
• load_image_file(&mut self, decoder, path) — decode from file path

Each configures the tensor's dimensions and format to the decoded native format and returns an ImageInfo. Returns CodecError::InsufficientCapacity if the decoded image is larger than the tensor's allocation.

ImageInfo

Returned by all decode methods:

• width, height: decoded image size (the source's true, unrotated dimensions)
• format: native pixel format written to the tensor
• row_stride: row stride in bytes used when writing into the tensor
• rotation_degrees: EXIF clockwise rotation the caller should apply (0/90/180/270)
• flip_horizontal: whether the caller should also flip horizontally

peek_info

peek_info(data) returns the same ImageInfo from the headers alone, without decoding pixels. Use it in one-shot flows to size a tensor to the image (see Quick Start). Its row_stride is the natural pitch for the format, since no destination tensor is involved yet.

Errors

Every decode entry point returns Result<ImageInfo, CodecError>:

License

Apache-2.0