edgefirst-decoder

High-performance ML model output decoding for object detection and segmentation.

This crate provides efficient post-processing for YOLO and ModelPack model outputs, supporting both floating-point and quantized inference results.

Role in edgefirst-hal

edgefirst-decoder sits between the inference engine and the image-rendering side of the EdgeFirst HAL workspace:

• Depends on edgefirst-tensor for reading model output buffers.
• Optionally depends on edgefirst-tracker (feature tracker) for decode_tracked().
• Consumed by edgefirst-image — its DetectBox, Segmentation, and proto-data outputs feed the draw_decoded_masks / draw_proto_masks / draw_masks_tracked rendering APIs.
• Bridged to C via edgefirst-decoder-capi (libedgefirst_decoder, edgefirst/decoder.h).
• Bridged to Python via crates/python-decoder.

Supported Models

Features

• Quantized decoding - Direct int8/uint8 processing without dequantization overhead
• Configurable NMS - Class-agnostic or class-aware non-maximum suppression
• Batch processing - Efficient handling of batched model outputs
• Builder pattern - Flexible configuration with sensible defaults

Quick Start

use edgefirst_decoder::{DecoderBuilder, DetectBox, Segmentation};
use edgefirst_tensor::TensorDyn;

// Build the decoder once, from the model's config document.
let decoder = DecoderBuilder::new()
    .with_score_threshold(0.25)
    .with_iou_threshold(0.7)
    .with_config_json_str(model_config_json)   // String
    .build()?;

// Then decode once per inference frame. `decode` dispatches to the
// quantized or float kernel from the tensor dtype — there is no separate
// entry point per dtype.
let mut detections: Vec<DetectBox> = Vec::with_capacity(100);
let mut masks: Vec<Segmentation> = Vec::with_capacity(100);

let outputs: Vec<&TensorDyn> = model_outputs.iter().collect();
decoder.decode(&outputs, &mut detections, &mut masks)?;

// Process results
for det in &detections {
    println!("Class {} at [{:.1}, {:.1}, {:.1}, {:.1}] score={:.2}",
        det.label, det.bbox.xmin, det.bbox.ymin, det.bbox.xmax, det.bbox.ymax, det.score);
}

Both Vecs are cleared on entry, and their capacity is an allocation hint rather than a cap — the detection count is bounded by max_det (default 300), not by what you pre-allocated.

Note: every model decode kernel in the yolo and modelpack modules is crate-private, so Decoder is the entire public decoding surface — there is no supported way to decode a model output without going through it. The float and byte modules do export reusable primitives (nms_float, nms_class_aware_int, iou_value, ios_value, box_area, and friends) if you need NMS or box geometry on your own candidates.

Configuration

Decoders can be configured via JSON/YAML matching the model's output specification:

{
  "decoder": "ultralytics",
  "shape": [1, 84, 8400],
  "quantization": [0.012345, 26],
  "normalized": true
}

NMS Modes

• Auto - Use the model config's nms mode, falling back to ClassAware (builder default)
• ClassAware - Only suppress boxes with the same class label (the default concrete mode, matching trainers and COCO evaluation)
• ClassAgnostic - Suppress overlapping boxes regardless of class
• None - Bypass NMS (for models with built-in NMS)

Multi-Label Decode

By default the decoder keeps one box per anchor, labelled with its highest-scoring class. Multi-label decode instead emits a box for every class whose score clears score_threshold, which is how trainer validators and COCO-style evaluation count detections. Use it for validation and mAP runs, not deployment.

Enable it with DecoderBuilder::with_multi_label(true), the Python multi_label=True constructor argument, or ef_decoder_params_set_multi_label(p, 1). A model can also declare it with the optional top-level nms_multi_label key in edgefirst.json; an explicit API value overrides the key. Multi-label candidates go through the decoder's NMS mode: the class-aware default keeps every class of an anchor, while an explicit class-agnostic mode suppresses across classes, as Ultralytics does with agnostic=True.

decode_tracked and decode_proto_tracked ignore multi-label and always decode one label per box: the tracker matches on IoU only, so per-class duplicates of one anchor would become phantom tracks. Callers that feed their own tracker should use decode_for_tracking, which applies the same rule. Each logs a warning once per decoder if the decoder has multi-label enabled. A decoder whose multi-label comes from nms_multi_label in the model metadata also logs a warning when it is built, since the model file then changes what decode returns.

Pre-NMS Top-K: Validation vs Deployment

The decoder's pre_nms_top_k parameter caps how many score-passing candidates enter NMS, bounding its O(N²) cost via an O(N) partial sort; 0 means no limit. It applies to every NMS decode path, detection-only models included, and equal scores at the cut keep the lower anchor index. The default of 300 is tuned for deployment — but it must be raised (or set to 0 for no limit) for mAP evaluation. With multi-label decode on (from the API or the model's nms_multi_label) and no explicit cap, the default is 30 000, Ultralytics' max_nms, since multi-label emits up to anchors × classes candidates. Decoder::pre_nms_top_k is None until a cap is set; the default is then resolved per decode (Decoder::pre_nms_top_k_for), so the argmax tracking entry points of a multi-label decoder keep 300. A cap set on the builder or assigned as Some(n) after build applies to every decode.

Why it matters

With COCO's low threshold, most of the 8 400 YOLO anchors pass the score filter. The default top-K of 300 silently truncates the candidate pool, causing ~9 pp box mAP loss — a measurement artifact, not a model quality issue. The decoder math is correct in both cases.

Recommended settings

// Deployment (real-time inference)
let decoder = DecoderBuilder::new()
    .with_config_json_str(config)
    .with_score_threshold(0.25)
    // pre_nms_top_k = 300 (default) — appropriate
    .build()?;

// COCO mAP evaluation
let decoder = DecoderBuilder::new()
    .with_config_json_str(config)
    .with_score_threshold(0.001)
    .with_pre_nms_top_k(0)      // no limit: every candidate reaches NMS
    .with_max_det(300)           // COCO detection cap applied post-NMS
    .build()?;

Performance trade-off

Post-processing latency scales with the number of candidates entering NMS. At deployment thresholds (≥ 0.25), the candidate count is already small regardless of the top-K setting, so raising it has negligible cost. At validation thresholds (0.001), the increase is measurable — but necessary for correct recall across the full precision-recall curve.

End-to-End Models (YOLO26)

YOLO26 models embed NMS directly in the model architecture (one-to-one matching heads), eliminating the need for external NMS post-processing.

Configure via the decoder_version field in the model config:

{
  "decoder": "ultralytics",
  "decoder_version": "yolo26",
  "shape": [1, 300, 6],
  "quantization": [0.012345, 26],
  "normalized": true
}

When decoder_version is "yolo26", the decoder:

• Bypasses NMS entirely (the nms config field is ignored)
• Expects post-NMS output format: [batch, N, 6+] where columns are [x1, y1, x2, y2, conf, class, ...]
• Supports both detection-only and detection+segmentation variants

For non-end-to-end YOLO26 exports (end2end=false), use decoder_version: "yolov8" with explicit NMS configuration.

Non-End-to-End Mode

Models exported with end2end=false require external NMS, configurable via the nms field:

{
  "decoder": "ultralytics",
  "decoder_version": "yolov8",
  "nms": "class_agnostic",
  "shape": [1, 84, 8400]
}

Semantic Segmentation API

decode_semantic() writes the class map of a semantic segmentation model into a tensor you allocate: [H, W], u8 for up to 256 classes or u16 beyond, at any resolution (usually the original image's). Pass the letterbox content region so the padding is cropped, and pick the decode mode per call.

use edgefirst_decoder::{schema::DecodeMode, semantic::SemanticOptions};

let mut class_map = TensorDyn::new(&[image_h, image_w], DType::U8, Some(TensorMemory::Mem), None)?;
let options = SemanticOptions::new()
    .with_mode(DecodeMode::Fast)           // default: the model's decode_mode
    .with_letterbox([0.0, 0.0, 1.0, 0.75]); // content region of the model input
decoder.decode_semantic(&outputs, &mut class_map, None, &options)?;

DecodeMode::Accurate upsamples the class scores bilinearly and takes the argmax, matching the PyTorch reference; DecodeMode::Fast takes the argmax at the model's resolution and upsamples the class map. On SegFormer-B1 (ADE20K) fast costs about 0.12 mIoU points. An optional confidence tensor receives the winning class's probability for overlays.

Proto Mask API

For segmentation models, decode_proto() returns the mask prototypes and per-detection coefficients instead of materialized pixel masks. Prefer it when the masks are headed for a GPU rendering pipeline (e.g. ImageProcessor::draw_proto_masks()): the GPU evaluates sigmoid(coeffs @ protos) per output pixel, so the CPU never pays for full-resolution masks.

It returns Ok(None) for detection-only and ModelPack models, and Ok(Some(ProtoData)) for YOLO segmentation models. Like decode, it picks the quantized or float kernel from the tensor dtype.

// GPU rendering path: decode proto data, pass to GL for fused rendering
let mut detections: Vec<DetectBox> = Vec::with_capacity(100);
let proto_data = decoder.decode_proto(&outputs, &mut detections)?;

// Detection-only models return None — there is nothing to overlay.
if let Some(proto_data) = proto_data {
    processor.draw_proto_masks(&mut frame, &detections, &proto_data)?;
}

Model Type Variants

The decoder automatically selects the appropriate model type from the output schema, supporting YOLO (detection / segmentation, with or without end-to-end NMS and split outputs) and ModelPack (detection, segmentation, split variants). The full variant matrix, output-shape disambiguation rules, the 2-way split format used by TFLite INT8 segmentation models, and the nc=28 edge case are documented in ARCHITECTURE.md § Model-type selection.

Schema Inference for Ultralytics Exports

This crate decodes the output tensors of YOLO-family models. It contains no upstream model code: the decoders, NMS and mask kernels here are an independent Rust implementation, and nothing from any model-training project is vendored, linked or redistributed. Exporting a model is something you do with your own tooling; this crate reads what came out.

A vanilla Ultralytics YOLOv8/YOLO11/YOLO26 export (ONNX, TFLite, or CoreML; detection or segmentation) carries no edgefirst.json, but its own metadata and tensor shapes are enough to derive one. infer_ultralytics_schema reads the model's names/task/end2end metadata plus its I/O tensor shapes and dtypes, and returns a SchemaV2 ready to hand to DecoderBuilder, along with the ordered class labels. Metadata and shapes are cross-checked, never guessed past — a mismatch (e.g. a class count that doesn't fit the output width) is a typed InferError, not a silent best-effort classification.

use edgefirst_decoder::{infer_ultralytics_schema, DecoderBuilder};

// `signals: ModelSignals` comes from the runtime that loaded the model.
let inferred = infer_ultralytics_schema(&signals)?;
let decoder = DecoderBuilder::new().with_schema(inferred.schema).build()?;
// `inferred.labels` are the class names in index order.

Rust callers hand the SchemaV2 straight to with_schema; there is no reason to serialize it. Nor does Python: its binding returns the schema as a dict, which Decoder(schema) takes directly. The JSON rendering exists for the C API alone, which has no dict to hand across the boundary.

The inferred schema pins the NMS mode but not the thresholds. Ultralytics runs NMS class-aware (agnostic=False), and the schema says so explicitly rather than relying on the builder's fallback. with_nms still overrides. YOLO26 end-to-end exports perform NMS in-graph and carry no mode.

Box normalization follows the export format, not a fixed convention: Ultralytics ONNX exports report pixel-space boxes (normalized: false), while TFLite exports report boxes normalized to [0, 1] (normalized: true) — the inferred schema always matches the tensors the model actually produces. A CoreML export (no-NMS) is pixel-space too, matching ONNX, established by source-tracing the Ultralytics exporter and confirmed by the same runtime measurement that pinned the other two. ModelSource::Other is refused rather than defaulted, because this is the one field no tensor shape reveals and picking wrong scales every box by the input size. The schema's decoder_version (yolov8 or yolo26) is always set explicitly rather than left for the builder to infer from output shape, because shape-based inference of end-to-end vs. pre-NMS layout is ambiguous in the general case; inference here settles it once, from the model's own end2end metadata flag when present.

Tiled Inference (SAHI)

When a frame is preprocessed into an overlapping tile grid (see edgefirst-image's tiled preprocessing), each tile is decoded independently and its detections are lifted back to full-frame coordinates and merged. A single object split across a tile seam has low IoU but high IoS (intersection-over-smaller), so the default merge metric is Ios with a 0.5 threshold. The merge is a greedy, class-aware pass that by default keeps the highest-scoring box of each matched group unchanged (MergeMode::KeepBest); the enclosing-union merge of the original GREEDYNMM reference is MergeMode::Union, opt-in.

The input and output sides share one TilePlacement (produced by ImageProcessor::plan_tiles/tile_into) describing how each tile was cut.

use edgefirst_decoder::{DecoderBuilder, DetectBox, Nms, Segmentation};
use edgefirst_decoder::tiling::{MergeConfig, TiledFrameAccumulator};

// The per-tile decoder is deliberately permissive: a fragment clipped at a
// seam scores low, and a high per-tile threshold would discard it before the
// merge could rebuild the object. Gate the final scores in MergeConfig instead.
let decoder = DecoderBuilder::new()
    .with_config_json_str(model_config_json)
    .with_score_threshold(0.05)
    .with_nms(Some(Nms::ClassAware))
    .build()?;

// One accumulator per in-flight frame; collect tiles in any order.
let mut acc = TiledFrameAccumulator::new(
    (frame_w as f32, frame_h as f32),
    placements.len(),   // tiles_total — the fan-in fence
    MergeConfig::default(),
    16,                 // estimated detections per tile (capacity hint)
);

for (tile_outputs, placement) in tiles {
    let mut boxes: Vec<DetectBox> = Vec::new();
    let mut masks: Vec<Segmentation> = Vec::new();
    decoder.decode(&tile_outputs, &mut boxes, &mut masks)?;
    // Lifts to full-frame pixels and appends. Returns false for a tile this
    // accumulator has already seen, which is how retries stay harmless.
    acc.push_tile(boxes, &placement);
}

// Once every tile has arrived, merge and normalize to [0,1] for the tracker.
assert!(acc.is_complete());
let detections = acc.finalize_normalized(); // full-frame, deduplicated

push_tile is idempotent per placement.index, so out-of-order and at-least-once tile delivery both converge to the same result — the merge runs once at finalize, never per push. This is the "collect after the final tile" contract a pipelined runtime needs: plan_tiles sizes the ring up front, tile_one streams tiles through inference, and is_complete()/remaining() fence the frame.

The free functions lift_tile_boxes and merge_tiled_detections expose the two stages directly if you need to merge without the accumulator. MergeConfig tunes the metric (Ios, default, or Iou), the match threshold (0.5), class_agnostic (false), max_det (300), a final score_threshold, and the mode (KeepBest, default, or Union). score_threshold defaults to 0.0 on purpose: per-tile decode is the real flood control, and this is where you put the score gate once fragments have been joined. The same machinery supports standard SAHI with Ultralytics YOLO models. Merged boxes can be fed straight to the tracker (next section), which expects normalized coordinates, hence finalize_normalized.

Note: MergeMode::KeepBest suppresses — the best box of each matched group comes back unchanged, never grown — so an object larger than a single tile is reported as its best fragment. MergeMode::Union grows that box to the group's enclosing union instead, but on the Ocean Cleanup ADIS 4K validation (TOP2-836) the union cost about 0.05 AP50 on every frame, tiled or not (whole frame: 0.491 AP50 with plain NMS, 0.437 after the union, 0.490 with keep-best; 28-tile pipeline: 0.500 keep-best vs 0.442 union), which is why keep-best is the default. Add an optional full-frame downscaled pass (another push_tile at origin=(0,0), crop_size=frame_dims) for mixed-scale datasets.

Tracked Decoding

The tracker feature adds decode_tracked to integrate object tracking directly into the decode step. Each decoded detection box is matched to a persistent track and assigned a stable UUID for the lifetime of the track.

Enable the feature in Cargo.toml:

edgefirst-decoder = { version = "0.30", features = ["tracker"] }

Usage

use edgefirst_decoder::{DecoderBuilder, DetectBox, Segmentation, TrackInfo};
use edgefirst_decoder::Tracker; // re-exported from edgefirst-tracker
use edgefirst_tracker::ByteTrackBuilder;

let decoder = DecoderBuilder::new()
    .with_score_threshold(0.25)
    .with_iou_threshold(0.7)
    .with_config_json_str(model_config_json)
    .build()?;

let mut tracker = ByteTrackBuilder::new()
    .track_high_conf(0.5)
    .track_update(0.1)
    .build();

let mut detections: Vec<DetectBox> = Vec::with_capacity(100);
let mut masks: Vec<Segmentation> = Vec::with_capacity(100);
let mut tracks: Vec<TrackInfo> = Vec::with_capacity(100);

decoder.decode_tracked(
    &mut tracker,
    timestamp,          // u64 frame timestamp
    &model_outputs,     // &[&TensorDyn]
    &mut detections,
    &mut masks,
    &mut tracks,
)?;

// detections[i] and tracks[i] correspond to the same object
for (det, track) in detections.iter().zip(tracks.iter()) {
    println!(
        "Track {} class {} score={:.2} at [{:.1}, {:.1}, {:.1}, {:.1}]",
        track.uuid, det.label, det.score,
        det.bbox.xmin, det.bbox.ymin, det.bbox.xmax, det.bbox.ymax,
    );
}

TrackInfo Fields

What decode_tracked does to detections

decode_tracked does not simply append track metadata to the decoded boxes. It rewrites detections to be the set of active tracks, which changes two things worth planning around:

• Every det.bbox is overwritten with that track's Kalman-smoothed tracked_location. After decode_tracked, det.bbox and track.tracked_location hold the same coordinates; the raw pre-smoothing box is not returned. Use decode if you need the unsmoothed detections.
• Tracks still alive but unmatched this frame are included, so detections can contain entries that no detection in this frame produced. They carry a coasting Kalman prediction, and on the segmentation path they have no mask. track.last_updated (older than the current timestamp) identifies them.

detections[i] and tracks[i] describe the same track in both cases, so the zip above stays valid.

Documentation

• Architecture overview: ARCHITECTURE.md
• Testing guide: TESTING.md
• Full API reference: docs.rs/edgefirst-decoder
• Project README: ../../README.md

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.