Struct ImageProcessor 

pub struct ImageProcessor {
    pub cpu: Option<CPUProcessor>,
    pub g2d: Option<G2DProcessor>,
    pub opengl: Option<GLProcessorThreaded>,
    /* private fields */
}

Image converter that uses available hardware acceleration or CPU as a fallback.

Fields

cpu: Option<CPUProcessor>

CPU-based image converter as a fallback. This is only None if the EDGEFIRST_DISABLE_CPU environment variable is set.

g2d: Option<G2DProcessor>

G2D-based image converter for Linux systems. This is only available if the EDGEFIRST_DISABLE_G2D environment variable is not set and libg2d.so is available.

opengl: Option<GLProcessorThreaded>

OpenGL-based image converter for Linux systems. This is only available if the EDGEFIRST_DISABLE_GL environment variable is not set and OpenGL ES is available.

Implementations

impl ImageProcessor

pub fn alloc_tile_batch( &self, n: usize, cfg: &TilingConfig, format: PixelFormat, dtype: DType, memory: Option<TensorMemory>, access: CpuAccess, ) -> Result<TensorDyn>

Allocate the tall packed batch destination [tile_w, n*tile_h] — a single GL-importable parent that stacks n tiles vertically. Uses ImageProcessor::create_image so DMA pitch alignment and Dma>Pbo>Mem selection apply. Caller-owned for pool reuse.

access declares planned CPU involvement (see edgefirst_tensor::CpuAccess); use None for NPU-bound tile batches and ReadWrite when the batch will be CPU-mapped. As for create_image, what this processor’s own convert may do on the CPU is added; tiles are written through views, so any predicted CPU write is added as ReadWrite.

Errors

Returns an error if cfg is invalid (overlap not in [0,1), or zero tile size) or if the underlying allocation fails.

pub fn plan_tiles( &self, src_w: usize, src_h: usize, cfg: &TilingConfig, ) -> Result<Vec<TilePlacement>>

Compute the tile grid and per-tile TilePlacement metadata for a src_w×src_h frame without touching the GPU. A pipelined runtime calls this once per frame to size pools and drive its tile stream.

The realized tile count can exceed the naive ceil(frame / tile): the grid treats cfg.overlap_ratio as a minimum and redistributes the surplus evenly, so allocate from len() rather than estimating.

The returned placements are consumed by edgefirst_decoder::tiling to lift per-tile detections back to full-frame coordinates.

Errors

Returns an error if cfg is invalid (overlap not in [0,1), or zero tile size).

pub fn tile_into( &mut self, src: &TensorDyn, dst_batched: &mut TensorDyn, cfg: &TilingConfig, ) -> Result<Vec<TilePlacement>>

Render every tile of src into dst_batched (a tall packed parent from Self::alloc_tile_batch), one deferred convert per tile, then a single flush. Returns the per-tile TilePlacement in tile-index order.

The source crop is selected via Crop::with_source (a sampled sub-rect of the whole-frame src) — not src.view() — so all tiles share one source import. Each destination tile is a view row-band of the shared parent.

Examples

let mut processor = ImageProcessor::new()?;
let cfg = TilingConfig::new(640, 640).with_overlap(0.2);

// Pure geometry — size the batch before touching the GPU.
let planned = processor.plan_tiles(3840, 2160, &cfg)?;
let mut batch = processor.alloc_tile_batch(
    planned.len(), &cfg, PixelFormat::Rgba, DType::U8, None, CpuAccess::None,
)?;

// One import, one flush, N tiles.
let placements = processor.tile_into(src, &mut batch, &cfg)?;
assert_eq!(placements.len(), planned.len());

Backend note

The GL band lowering covers single-pass geometry only. A zero-copy (DMA) dst_batched in packed Rgb or a planar layout reinterprets the render-target shape, which the band path does not yet compute, so GL declines and each tile falls back to CPU. Rgba, Bgra, and Grey batches stay on the batched GPU path.

Errors

Returns an error if cfg is invalid, if src/dst_batched are not images, if dst_batched is shorter than count * tile_h, or if a tile convert fails.

pub fn tile_one( &mut self, src: &TensorDyn, dst_slot: &mut TensorDyn, placement: &TilePlacement, cfg: &TilingConfig, ) -> Result<()>

Render exactly one tile of src into dst_slot (a single model-input sized destination, e.g. a slot of a caller-owned ring). Deferred — the caller flushes on its own cadence so tiles overlap with inference. Holds no frame-wide state; all geometry rides in placement (from Self::plan_tiles).

A deferred tile is not CPU/CUDA-readable until ImageProcessorTrait::flush (mirrors ImageProcessorTrait::convert_deferred).

Errors

Returns an error if cfg is invalid or the tile convert fails.

impl ImageProcessor

pub fn new() -> Result<Self>

Creates a new ImageProcessor instance, initializing available hardware converters based on the system capabilities and environment variables.

Examples

let image = std::fs::read("zidane.jpg")?;
// The codec emits the source's native format (a colour JPEG decodes to
// NV12) and configures the destination tensor during the decode.
let info = peek_info(&image).expect("peek");
let mut src = Tensor::<u8>::image(info.width, info.height, info.format,
                                   Some(TensorMemory::Mem), edgefirst_tensor::CpuAccess::ReadWrite)?;
let mut decoder = ImageDecoder::new();
src.load_image(&mut decoder, &image).expect("decode");
let mut converter = ImageProcessor::new()?;
let mut dst =
    converter.create_image(640, 480, PixelFormat::Rgb, DType::U8, None, CpuAccess::ReadWrite)?;
converter.convert(&src.into(), &mut dst, None)?;

pub fn convert_fallback_count(&self) -> u64

Number of converts where the auto chain’s GL backend declined and the frame fell through toward G2D/CPU — each one silently lost any zero-copy guarantee. A pipeline that expects to stay on the GPU asserts this stays flat across its steady-state loop; pair with GLProcessorThreaded::convert_stats to see how the frames that DID reach GL were fed. Always 0 under a forced backend (no chain).

pub fn compression_fallback_count(&self) -> u64

Number of Compression::Any requests since process start that resolved to a linear layout (process-wide mirror of edgefirst_tensor::compression_fallback_count, surfaced here beside convert_fallback_count so pipeline telemetry reads from one place). A pipeline that expects compressed convert destinations asserts this stays flat after warmup.

pub fn convert_with_fence( &mut self, src: &TensorDyn, dst: &mut TensorDyn, options: Option<ConvertOptions>, ) -> Result<Option<CompletionFence>>

Convert and return a native fence that signals when the GPU work completes, instead of blocking the CPU — the GL→NPU handoff (EGL_ANDROID_native_fence_sync on Android). On Windows the handle is a manual-reset event set when the GPU work completes; wait on it with WaitForSingleObject.

Ok(Some(fence)): the destination buffer is still in flight; hand the fence to the consumer (e.g. ANeuralNetworksExecution_startComputeWithDependencies on Android, or WaitForSingleObject on Windows) or poll it before reading. Ok(None): the convert completed with the normal blocking contract (no native fence on this platform, or a non-GL backend handled the frame) — the destination is already safe. Semantics are otherwise identical to convert, including the GL→G2D→CPU fallback chain.

On Windows, for a texture destination, the event and the destination’s own Tensor::gpu_completion name the same fence value: one flush and one signal cover the convert, and the value recorded on dst is the one this event waits for, so a consumer may use either. A destination that is not a texture records nothing, and the event then carries a value of its own that only it can be waited on for.

pub fn supported_render_dtypes(&self) -> RenderDtypeSupport

Report which float dtypes the GPU can render to.

Probes GL_EXT_color_buffer_half_float and GL_EXT_color_buffer_float once at ImageProcessor::new() time and caches the result. Call this once at startup to decide whether to request F16 or F32 destination tensors; create_image uses the result internally to auto-select float PBO when supported.

Returns RenderDtypeSupport { f32: false, f16: false } when no OpenGL backend is active or the opengl feature is disabled.

pub fn with_config(config: ImageProcessorConfig) -> Result<Self>

Creates a new ImageProcessor with the given configuration.

When ImageProcessorConfig::backend is set to a specific backend, environment variables are ignored and the processor initializes the requested backend plus CPU as a fallback.

When Auto, the existing EDGEFIRST_FORCE_BACKEND and EDGEFIRST_DISABLE_* environment variables apply.

pub fn set_colorimetry_mode(&mut self, mode: ColorimetryMode) -> Result<()>

Sets the colorimetry/performance trade-off (see ColorimetryMode) on the OpenGL backend. No-op if OpenGL is not available. The EDGEFIRST_COLORIMETRY environment variable takes precedence — when it is set, this call logs and keeps the env-selected mode.

pub fn set_interpolation(&mut self, mode: Interpolation) -> Result<()>

Sets how convert() resamples when it resizes (see Interpolation) on every backend this processor holds. The EDGEFIRST_INTERPOLATION environment variable takes precedence: when it is set, this call logs and keeps the env-selected mode.

pub fn set_int8_interpolation_mode( &mut self, mode: Int8InterpolationMode, ) -> Result<()>

Sets the interpolation mode for int8 proto textures on the OpenGL backend. No-op if OpenGL is not available.

pub fn create_image_desc(&self, desc: &ImageDesc) -> Result<TensorDyn>

Allocate an image tensor from a declarative ImageDesc request — the full-featured variant of create_image.

Without a compression or contiguous request this is exactly create_image (the processor’s memory negotiation applies). With one, the allocation rides the tensor desc path un-negotiated: the layout decision belongs to the platform allocator, and the request’s guards and fallback counting live there (see Tensor::image_desc). A contiguous request is a CMA DMA-BUF at the same 64-byte GPU pitch create_image uses, or an error; it never falls back to a PBO or host memory. Its CPU access gets the same widening as create_image; a compression request’s CpuAccess::None does not.

pub fn create_image( &self, width: usize, height: usize, format: PixelFormat, dtype: DType, memory: Option<TensorMemory>, access: CpuAccess, ) -> Result<TensorDyn>

Create a TensorDyn image with the best available memory backend.

Priority: DMA-buf → float PBO (F16/F32) → u8/i8 PBO → system memory.

Use this method instead of TensorDyn::image() when the tensor will be used with ImageProcessor::convert(). It selects the optimal memory backing (including PBO for GPU zero-copy) which direct allocation cannot achieve.

This method is on ImageProcessor rather than ImageProcessorTrait because optimal allocation requires knowledge of the active compute backends (e.g. the GL context handle for PBO allocation). Individual backend implementations (CPUProcessor, etc.) do not have this cross-backend visibility.

Float dtype behaviour: when dtype is F16 or F32 and supported_render_dtypes reports the GPU supports that type, memory: None auto-selects a float PBO (Linux) or IOSurface (macOS F16 only). If GPU float support is absent the allocation falls through to TensorMemory::Mem; convert then uses the CPU path. Passing memory: Some(TensorMemory::DmaBuf) with dtype: F32 always returns Error::NotSupported — no 32-bit-float DRM fourcc exists.

Arguments

• width - Image width in pixels
• height - Image height in pixels
• format - Pixel format
• dtype - Element data type (e.g. DType::U8, DType::F16, DType::F32)
• memory - Optional memory type override; when None, the best available backend is selected automatically.
• access - Declares the CPU access you plan on (ReadWrite is the natural choice). Use CpuAccess::None for a destination that only hardware reads (an NPU input). Hardware access is always implied. What this processor’s own convert may do to the image as a destination on the CPU is added to the declaration so that convert succeeds: Write when it may fill the whole destination on the CPU (the CPU backend, host memory, a format or float dtype the GPU path does not render), ReadWrite when a pass reads back what was written or writes only part of the buffer (an I8 destination through the CPU or G2D, a Bgra destination through the CPU, a G2D Rgb letterbox border). The returned tensor’s cpu_access may therefore be wider than access. Any other CPU map beyond the declaration fails with CpuAccessDenied. The widening covers the destination role of convert only: a tensor later used as a convert source needs Read declared by the caller, and a CPU mask draw blends into its destination and needs ReadWrite, so declare it when the CPU may draw. A destination written through views (a batch) needs ReadWrite for a CPU convert, which alloc_tile_batch predicts. A destination left without CPU access (None on a GPU-rendered format) is written only by the GPU, so a convert into it needs the GPU to accept the source as well; when it declines, the CPU fallback is refused with Error::CpuFallbackDenied carrying the GPU’s reason. A PBO destination is always ReadWrite: the GL readback maps it on the CPU itself, and a narrower declaration would buy no cheaper mapping.

Returns

A TensorDyn backed by the highest-performance memory type available on this system.

Examples

let processor = ImageProcessor::new()?;

// Model input the NPU reads directly — no CPU mapping planned.
let input = processor.create_image(
    640, 640, PixelFormat::Rgb, DType::U8, None, CpuAccess::None,
)?;

Pitch alignment for DMA-backed allocations

DMA-BUF imports into the GL backend (Mali Valhall on i.MX 95 specifically) require every row pitch to be a multiple of GPU_DMA_BUF_PITCH_ALIGNMENT_BYTES (currently 64). When this method lands on TensorMemory::DmaBuf, the underlying allocation is silently padded so the row stride satisfies that requirement.

The user-requested width is preserved — tensor.width() returns the same value you passed in. The padding is carried by TensorDyn::row_stride / effective_row_stride(), which the GL backend reads when importing the buffer as an EGLImage. Callers that compute byte offsets from the tensor must use the stride, not width × bytes_per_pixel; the CPU mapping spans the full stride × height bytes.

Pre-aligned widths (640, 1280, 1920, 3008, 3840 …) allocate exactly width × bpp × height bytes with no padding. PBO and Mem fallbacks never pad — they don’t go through EGLImage import.

See also align_width_for_gpu_pitch for an advisory helper that external callers (GStreamer plugins, video pipelines) can use to size their own DMA-BUFs for GL compatibility.

Errors

Returns an error if all allocation strategies fail.

pub fn import_image( &self, image: PlaneDescriptor, chroma: Option<PlaneDescriptor>, width: usize, height: usize, format: PixelFormat, dtype: DType, colorimetry: Option<Colorimetry>, ) -> Result<TensorDyn>

Import an external DMA-BUF image.

Each [PlaneDescriptor] owns an already-duped fd; this method consumes the descriptors and takes ownership of those fds (whether the call succeeds or fails).

The caller must ensure the DMA-BUF allocation is large enough for the specified width, height, format, and any stride/offset on the plane descriptors. No buffer-size validation is performed; an undersized buffer may cause GPU faults or EGL import failure.

Arguments

• image - Plane descriptor for the primary (or only) plane
• chroma - Optional plane descriptor for the UV chroma plane (required for multiplane NV12)
• width - Image width in pixels
• height - Image height in pixels
• format - Pixel format of the buffer
• dtype - Element data type (e.g. DType::U8)

Returns

A TensorDyn configured as an image.

Errors

• Error::NotSupported if chroma is Some for a non-semi-planar format, or multiplane NV16 (not yet supported), or the fd is not DMA-backed
• Error::InvalidShape if NV12 height is odd

Platform

Linux only.

Examples

use edgefirst_tensor::PlaneDescriptor;

// Single-plane RGBA
let pd = PlaneDescriptor::new(fd.as_fd())?;
let src = proc.import_image(pd, None, 1920, 1080, PixelFormat::Rgba, DType::U8, None)?;

// Multi-plane NV12 with stride
let y_pd = PlaneDescriptor::new(y_fd.as_fd())?.with_stride(2048);
let uv_pd = PlaneDescriptor::new(uv_fd.as_fd())?.with_stride(2048);
let src = proc.import_image(y_pd, Some(uv_pd), 1920, 1080,
                            PixelFormat::Nv12, DType::U8, None)?;

pub fn materialize_masks( &mut self, detect: &[DetectBox], proto_data: &ProtoData, letterbox: Option<[f32; 4]>, resolution: MaskResolution, ) -> Result<Vec<Segmentation>>

Materialize per-instance segmentation masks from raw prototype data.

Computes mask_coeff @ protos with sigmoid activation for each detection, producing compact masks at prototype resolution (e.g., 160×160 crops). Mask values are continuous sigmoid confidence outputs quantized to u8 (0 = background, 255 = full confidence), NOT binary thresholded.

The returned Vec<Segmentation> can be:

• Inspected or exported for analytics, IoU computation, etc.
• Passed directly to ImageProcessorTrait::draw_decoded_masks for GPU-interpolated rendering.

Performance Note

Calling materialize_masks + draw_decoded_masks separately prevents the HAL from using its internal fused optimization path. For render-only use cases, prefer ImageProcessorTrait::draw_proto_masks which selects the fastest path automatically (currently 1.6×–27× faster on tested platforms). Use this method when you need access to the intermediate masks.

Errors

Returns Error::NoConverter if the CPU backend is not available.

Trait Implementations

impl Debug for ImageProcessor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl ImageProcessorTrait for ImageProcessor

fn convert( &mut self, src: &TensorDyn, dst: &mut TensorDyn, options: Option<ConvertOptions>, ) -> Result<()>

Converts the source image to the destination image format and size. The image is cropped first, then flipped, then rotated

Prefer hardware accelerators when available, falling back to CPU if necessary.

fn convert_deferred( &mut self, src: &TensorDyn, dst: &mut TensorDyn, options: Option<ConvertOptions>, ) -> Result<()>

Like convert, but does not wait for the GPU to finish.

fn flush(&mut self) -> Result<()>

Complete all work enqueued by convert_deferred since the last flush, issuing a single GPU synchronization.

fn draw_decoded_masks( &mut self, dst: &mut TensorDyn, detect: &[DetectBox], segmentation: &[Segmentation], overlay: MaskOverlay<'_>, ) -> Result<()>

Draw pre-decoded detection boxes and segmentation masks onto dst.

fn draw_proto_masks( &mut self, dst: &mut TensorDyn, detect: &[DetectBox], proto_data: &ProtoData, overlay: MaskOverlay<'_>, ) -> Result<()>

Draw masks from proto data onto image (fused decode+draw).

fn set_class_colors(&mut self, colors: &[[u8; 4]]) -> Result<()>

Sets the colors used for rendering segmentation masks. Up to 20 colors can be set.

impl Send for ImageProcessor

impl Sync for ImageProcessor