nanoglia 0.3.1

Run TensorFlow Lite models loaded at runtime on microcontrollers: one no_std API, CMSIS-NN / ESP-NN accelerated kernels, every byte from your allocator.
Documentation

nanoglia

Run TensorFlow Lite models loaded at run time on microcontrollers, in no_std Rust, with the best kernels of each chip.

nanoglia is a TinyML inference engine: it reads a .tflite (LiteRT) model from flash, RAM, an SD card or the network, without copying its weights, and runs it with Arm CMSIS-NN on Cortex-M, Espressif ESP-NN on the ESP32-S3 (feature esp-nn), and pure Rust kernels for the rest. No RTOS, no HAL, no vendor SDK, and every byte of memory comes from the allocator you pass, a static buffer included. Results are bit-exact with TensorFlow Lite.

use nanoglia::{Arena, Encoding, Graph, Target};

static MODEL: &[u8] = include_bytes!("model.tflite"); // on a 16-byte boundary in practice

let mut memory = [0u8; 24 * 1024]; // MemoryRequirements tells how much a model needs
let arena = Arena::new(&mut memory);

let graph = Graph::load_in(&[MODEL], Encoding::Autodetect, Target::Auto, &arena)?;
let mut ctx = graph.init_execution_context_in(&arena)?;

ctx.input_mut(0)?.as_mut_slice::<i8>()?.copy_from_slice(&features);
ctx.compute()?; // allocates nothing
let scores = ctx.output(0)?.as_slice::<i8>()?;
  • Models loaded at run time: update or swap a model without reflashing the firmware.
  • Your memory only: a Graph makes one allocation, each ExecutionContext one more, compute none. They come from the global allocator (feature alloc), your own, or an Arena over a static buffer, sized ahead of time with MemoryRequirements.
  • Checked before flashing: ModelReport lists every operator of a model with the kernel that runs it or the reason none does, and the memory the model needs, where loading stops at the first operator it cannot run.
  • Accelerated: CMSIS-NN (Cortex-M DSP and Helium), ESP-NN (ESP32-S3 SIMD), portable code everywhere else, WebAssembly included.
  • Bit-exact with TensorFlow Lite: the outputs match the model run in Python, checked on every commit against golden outputs, including the MLPerf Tiny models.
  • Extensible: a Registry of your own puts your backends first; their kernels may borrow your drivers. An Observer passed to compute_with sees each operator, to time it, read its output or stop at a deadline.
  • Non-blocking: step() runs one operator at a time, compute_async() yields to any async executor between operators.
  • Safe on untrusted models: the flatbuffer is fully verified before use; errors carry a wasi-nn ErrorCode and a precise Detail, never a panic.
Cargo feature Default Effect
cmsis-nn ✅ CMSIS-NN kernels (Arm DSP/Helium on Cortex-M, portable C elsewhere)
kernels ✅ Pure Rust kernels for the operators the C libraries lack
float ✅ float32 operators
int16 ✅ int16 (16x8) operators
all-ops ✅ Every operator: the op-* features, one per operator (op-conv-2d, op-softmax...)
esp-nn ESP-NN kernels, tried first (ESP32-S3 SIMD on xtensa-esp32s3)
alloc Graph::load and friends, with the Rust global allocator
single-rounding TFLITE_SINGLE_ROUNDING requantization instead of TensorFlow Lite Micro's double rounding
defmt defmt::Format for the error types

An int8-only firmware can use default-features = false, features = ["cmsis-nn", "kernels", "all-ops"].

Models are loaded at run time, so the linker keeps the kernels of every enabled operator. To link only those of your models, replace all-ops with the feature of each of their operators:

nanoglia = { version = "0.3", default-features = false, features = [
  "cmsis-nn", "kernels",
  "op-conv-2d", "op-depthwise-conv-2d", "op-average-pool-2d", "op-reshape", "op-fully-connected", "op-softmax",
] }

A model with another operator then fails to load with Detail::UnsupportedOp. See Reduce the code size.

Owned models and persistent contexts

OwnedGraph::load_in takes an aligned nanoglia::core::Block filled by the caller, retaining its allocation without copying model bytes. GraphExecutionContext::new_in accepts an owned graph handle (including alloc::sync::Arc with feature alloc) and allocates its arena once. ExecutionContext is the context that borrows its Graph. See the workspace documentation's memory reference for ownership and allocation details.

Learn more: the repository, the first-inference tutorial, supported operators, sizing the memory, running on a microcontroller and writing a backend.

Licensed under Apache-2.0.