Expand description
§nanoglia
Run TensorFlow Lite models loaded at run time on microcontrollers, in no_std Rust, with the
best kernels of each chip.
nanoglia is a TinyML inference engine: it reads a .tflite (LiteRT) model from flash, RAM, an
SD card or the network, without copying its weights, and runs it with Arm
CMSIS-NN on Cortex-M, Espressif
ESP-NN on the ESP32-S3 (feature esp-nn), and pure Rust
kernels for the rest. No RTOS, no HAL, no vendor SDK, and every byte of memory comes from the
allocator you pass, a static buffer included. Results are bit-exact with TensorFlow Lite.
use nanoglia::{Arena, Encoding, Graph, Target};
static MODEL: &[u8] = include_bytes!("model.tflite"); // on a 16-byte boundary in practice
let mut memory = [0u8; 24 * 1024]; // MemoryRequirements tells how much a model needs
let arena = Arena::new(&mut memory);
let graph = Graph::load_in(&[MODEL], Encoding::Autodetect, Target::Auto, &arena)?;
let mut ctx = graph.init_execution_context_in(&arena)?;
ctx.input_mut(0)?.as_mut_slice::<i8>()?.copy_from_slice(&features);
ctx.compute()?; // allocates nothing
let scores = ctx.output(0)?.as_slice::<i8>()?;- Models loaded at run time: update or swap a model without reflashing the firmware.
- Your memory only: a
Graphmakes one allocation, eachExecutionContextone more,computenone. They come from the global allocator (featurealloc), your own, or anArenaover a static buffer, sized ahead of time withMemoryRequirements. - Checked before flashing:
ModelReportlists every operator of a model with the kernel that runs it or the reason none does, and the memory the model needs, where loading stops at the first operator it cannot run. - Accelerated: CMSIS-NN (Cortex-M DSP and Helium), ESP-NN (ESP32-S3 SIMD), portable code everywhere else, WebAssembly included.
- Bit-exact with TensorFlow Lite: the outputs match the model run in Python, checked on every commit against golden outputs, including the MLPerf Tiny models.
- Extensible: a
Registryof your own puts your backends first; their kernels may borrow your drivers. AnObserverpassed tocompute_withsees each operator, to time it, read its output or stop at a deadline. - Non-blocking:
step()runs one operator at a time,compute_async()yields to any async executor between operators. - Safe on untrusted models: the flatbuffer is fully verified before use; errors carry a
wasi-nn
ErrorCodeand a preciseDetail, never a panic.
| Cargo feature | Default | Effect |
|---|---|---|
cmsis-nn | ✅ | CMSIS-NN kernels (Arm DSP/Helium on Cortex-M, portable C elsewhere) |
kernels | ✅ | Pure Rust kernels for the operators the C libraries lack |
float | ✅ | float32 operators |
int16 | ✅ | int16 (16x8) operators |
all-ops | ✅ | Every operator: the op-* features, one per operator (op-conv-2d, op-softmax…) |
esp-nn | ESP-NN kernels, tried first (ESP32-S3 SIMD on xtensa-esp32s3) | |
alloc | Graph::load and friends, with the Rust global allocator | |
single-rounding | TFLITE_SINGLE_ROUNDING requantization instead of TensorFlow Lite Micro’s double rounding | |
defmt | defmt::Format for the error types |
An int8-only firmware can use default-features = false, features = ["cmsis-nn", "kernels", "all-ops"].
Models are loaded at run time, so the linker keeps the kernels of every enabled operator. To link
only those of your models, replace all-ops with the feature of each of their operators:
nanoglia = { version = "0.3", default-features = false, features = [
"cmsis-nn", "kernels",
"op-conv-2d", "op-depthwise-conv-2d", "op-average-pool-2d", "op-reshape", "op-fully-connected", "op-softmax",
] }A model with another operator then fails to load with Detail::UnsupportedOp. See
Reduce the code size.
§Owned models and persistent contexts
OwnedGraph::load_in takes an aligned nanoglia::core::Block filled by the caller,
retaining its allocation without copying model bytes. GraphExecutionContext::new_in
accepts an owned graph handle (including alloc::sync::Arc with feature alloc) and
allocates its arena once. ExecutionContext is the context that borrows its Graph.
See the workspace documentation’s memory reference for ownership and allocation details.
Learn more: the repository, the first-inference tutorial, supported operators, sizing the memory, running on a microcontroller and writing a backend.
Licensed under Apache-2.0.
Re-exports§
pub use nanoglia_core as core;
Structs§
- Alloc
Error - The
AllocErrorerror indicates an allocation failure that may be due to resource exhaustion or to something wrong when combining the given input arguments with this allocator. - Arena
- A bump allocator over a caller-provided buffer, typically a
staticarray. - Builtin
Op - A TFLite builtin operator code (
BuiltinOperatorin the TFLite schema). - Dims
- The shape of a tensor, stored inline (no allocation).
- Error
- The error type of every fallible nanoglia operation. It is
Copyand never allocates. - Global
- The global memory allocator.
- Graph
- A loaded model, ready to create execution contexts.
- Graph
Execution Context - The mutable state of one inference: the arena holding the activations and the scratch memory.
- Memory
Breakdown - Where the memory of a model goes: the sizes of
MemoryRequirementsand what they are made of. Reported byModelReportfor a model that loads. - Memory
Requirements - Memory a model needs, computed without loading it.
- Model
Report - Whether a model runs, and what it needs: the answer to “will
Graph::loadaccept this model?” with every operator that stands in the way, where loading stops at the first. - OpInfo
- An operator, as an
Observersees it duringExecutionContext::compute_with. - Operator
Report - What
ModelReport::check_withfound for one operator of a model. - Owned
Graph - A prepared graph owning an aligned model allocation, without copying its weights.
- Quantization
- The quantization parameters of a tensor, borrowed from the model.
- Registry
- The ordered list of backends a graph picks its kernels from: for each operator, the first backend that supports it wins.
- Selected
Kernel - The kernel selected for an operator, in an
OperatorReport. - Tensor
Desc - Everything known before execution about a tensor of the graph.
- Tensor
Mut - A writable view of an input tensor of an execution context.
- Tensor
Ref - A read-only view of a tensor of an execution context.
Enums§
- Activation
- Activation fused into an operator (TFLite
ActivationFunctionType). - Data
Type - The element type of a tensor. Discriminants are the TFLite
TensorTypecodes. - Detail
- The precise reason of an
Error, with the indices needed to locate it in the model. - Encoding
- How the model bytes are encoded, as in wasi-nn’s
graph-encoding. - Error
Code - The category of an
Error, identical to theerror-codeenum of wasi-nn so a wasi-nn host can forward it unchanged. - OpCode
- The code identifying an operator.
- Step
- Progress of an inference run step by step (
ExecutionContext::step). - Target
- Where the graph should run, as in wasi-nn’s
execution-target.
Traits§
- Allocator
- An implementation of
Allocatorcan allocate, grow, shrink, and deallocate arbitrary blocks of data described viaLayout. - Element
- Rust element types matching a
DataType. - Graph
Handle - A stable graph reference or owner retained by an execution context.
- Observer
- Watches the operators of
ExecutionContext::compute_with: to time them, trace them, or check intermediate tensors.
Type Aliases§
- Execution
Context - An execution context that borrows its graph; no ownership allocation is introduced.
- Result
- The result type of nanoglia.