nanoglia
Run TensorFlow Lite models loaded at run time on microcontrollers, in no_std Rust, with the
best kernels of each chip.
nanoglia is a TinyML inference engine: it reads a .tflite (LiteRT) model from flash, RAM, an
SD card or the network, without copying its weights, and runs it with Arm
CMSIS-NN on Cortex-M, Espressif
ESP-NN on the ESP32-S3 (feature esp-nn), and pure Rust
kernels for the rest. No RTOS, no HAL, no vendor SDK, and every byte of memory comes from the
allocator you pass, a static buffer included. Results are bit-exact with TensorFlow Lite.
use ;
static MODEL: & = include_bytes!; // on a 16-byte boundary in practice
let mut memory = ; // MemoryRequirements tells how much a model needs
let arena = new;
let graph = load_in?;
let mut ctx = graph.init_execution_context_in?;
ctx.input_mut?.?.copy_from_slice;
ctx.compute?; // allocates nothing
let scores = ctx.output?.?;
- Models loaded at run time: update or swap a model without reflashing the firmware.
- Your memory only: a
Graphmakes one allocation, eachExecutionContextone more,computenone. They come from the global allocator (featurealloc), your own, or anArenaover a static buffer, sized ahead of time withMemoryRequirements. - Checked before flashing:
ModelReportlists every operator of a model with the kernel that runs it or the reason none does, and the memory the model needs, where loading stops at the first operator it cannot run. - Accelerated: CMSIS-NN (Cortex-M DSP and Helium), ESP-NN (ESP32-S3 SIMD), portable code everywhere else, WebAssembly included.
- Bit-exact with TensorFlow Lite: the outputs match the model run in Python, checked on every commit against golden outputs, including the MLPerf Tiny models.
- Extensible: a
Registryof your own puts your backends first; their kernels may borrow your drivers. AnObserverpassed tocompute_withsees each operator, to time it, read its output or stop at a deadline. - Non-blocking:
step()runs one operator at a time,compute_async()yields to any async executor between operators. - Safe on untrusted models: the flatbuffer is fully verified before use; errors carry a
wasi-nn
ErrorCodeand a preciseDetail, never a panic.
| Cargo feature | Default | Effect |
|---|---|---|
cmsis-nn |
✅ | CMSIS-NN kernels (Arm DSP/Helium on Cortex-M, portable C elsewhere) |
kernels |
✅ | Pure Rust kernels for the operators the C libraries lack |
float |
✅ | float32 operators |
int16 |
✅ | int16 (16x8) operators |
all-ops |
✅ | Every operator: the op-* features, one per operator (op-conv-2d, op-softmax...) |
esp-nn |
ESP-NN kernels, tried first (ESP32-S3 SIMD on xtensa-esp32s3) |
|
alloc |
Graph::load and friends, with the Rust global allocator |
|
single-rounding |
TFLITE_SINGLE_ROUNDING requantization instead of TensorFlow Lite Micro's double rounding |
|
defmt |
defmt::Format for the error types |
An int8-only firmware can use default-features = false, features = ["cmsis-nn", "kernels", "all-ops"].
Models are loaded at run time, so the linker keeps the kernels of every enabled operator. To link
only those of your models, replace all-ops with the feature of each of their operators:
= { = "0.3", = false, = [
"cmsis-nn", "kernels",
"op-conv-2d", "op-depthwise-conv-2d", "op-average-pool-2d", "op-reshape", "op-fully-connected", "op-softmax",
] }
A model with another operator then fails to load with Detail::UnsupportedOp. See
Reduce the code size.
Owned models and persistent contexts
OwnedGraph::load_in takes an aligned nanoglia::core::Block filled by the caller,
retaining its allocation without copying model bytes. GraphExecutionContext::new_in
accepts an owned graph handle (including alloc::sync::Arc with feature alloc) and
allocates its arena once. ExecutionContext is the context that borrows its Graph.
See the workspace documentation's memory reference for ownership and allocation details.
Learn more: the repository, the first-inference tutorial, supported operators, sizing the memory, running on a microcontroller and writing a backend.
Licensed under Apache-2.0.