Skip to main content

Crate nanoglia

Crate nanoglia 

Source
Expand description

§nanoglia

Run TensorFlow Lite models loaded at run time on microcontrollers, in no_std Rust, with the best kernels of each chip.

nanoglia is a TinyML inference engine: it reads a .tflite (LiteRT) model from flash, RAM, an SD card or the network, without copying its weights, and runs it with Arm CMSIS-NN on Cortex-M, Espressif ESP-NN on the ESP32-S3 (feature esp-nn), and pure Rust kernels for the rest. No RTOS, no HAL, no vendor SDK, and every byte of memory comes from the allocator you pass, a static buffer included. Results are bit-exact with TensorFlow Lite.

ⓘ
use nanoglia::{Arena, Encoding, Graph, Target};

static MODEL: &[u8] = include_bytes!("model.tflite"); // on a 16-byte boundary in practice

let mut memory = [0u8; 24 * 1024]; // MemoryRequirements tells how much a model needs
let arena = Arena::new(&mut memory);

let graph = Graph::load_in(&[MODEL], Encoding::Autodetect, Target::Auto, &arena)?;
let mut ctx = graph.init_execution_context_in(&arena)?;

ctx.input_mut(0)?.as_mut_slice::<i8>()?.copy_from_slice(&features);
ctx.compute()?; // allocates nothing
let scores = ctx.output(0)?.as_slice::<i8>()?;
  • Models loaded at run time: update or swap a model without reflashing the firmware.
  • Your memory only: a Graph makes one allocation, each ExecutionContext one more, compute none. They come from the global allocator (feature alloc), your own, or an Arena over a static buffer, sized ahead of time with MemoryRequirements.
  • Checked before flashing: ModelReport lists every operator of a model with the kernel that runs it or the reason none does, and the memory the model needs, where loading stops at the first operator it cannot run.
  • Accelerated: CMSIS-NN (Cortex-M DSP and Helium), ESP-NN (ESP32-S3 SIMD), portable code everywhere else, WebAssembly included.
  • Bit-exact with TensorFlow Lite: the outputs match the model run in Python, checked on every commit against golden outputs, including the MLPerf Tiny models.
  • Extensible: a Registry of your own puts your backends first; their kernels may borrow your drivers. An Observer passed to compute_with sees each operator, to time it, read its output or stop at a deadline.
  • Non-blocking: step() runs one operator at a time, compute_async() yields to any async executor between operators.
  • Safe on untrusted models: the flatbuffer is fully verified before use; errors carry a wasi-nn ErrorCode and a precise Detail, never a panic.
Cargo featureDefaultEffect
cmsis-nn✅CMSIS-NN kernels (Arm DSP/Helium on Cortex-M, portable C elsewhere)
kernels✅Pure Rust kernels for the operators the C libraries lack
float✅float32 operators
int16✅int16 (16x8) operators
all-ops✅Every operator: the op-* features, one per operator (op-conv-2d, op-softmax…)
esp-nnESP-NN kernels, tried first (ESP32-S3 SIMD on xtensa-esp32s3)
allocGraph::load and friends, with the Rust global allocator
single-roundingTFLITE_SINGLE_ROUNDING requantization instead of TensorFlow Lite Micro’s double rounding
defmtdefmt::Format for the error types

An int8-only firmware can use default-features = false, features = ["cmsis-nn", "kernels", "all-ops"].

Models are loaded at run time, so the linker keeps the kernels of every enabled operator. To link only those of your models, replace all-ops with the feature of each of their operators:

nanoglia = { version = "0.3", default-features = false, features = [
  "cmsis-nn", "kernels",
  "op-conv-2d", "op-depthwise-conv-2d", "op-average-pool-2d", "op-reshape", "op-fully-connected", "op-softmax",
] }

A model with another operator then fails to load with Detail::UnsupportedOp. See Reduce the code size.

§Owned models and persistent contexts

OwnedGraph::load_in takes an aligned nanoglia::core::Block filled by the caller, retaining its allocation without copying model bytes. GraphExecutionContext::new_in accepts an owned graph handle (including alloc::sync::Arc with feature alloc) and allocates its arena once. ExecutionContext is the context that borrows its Graph. See the workspace documentation’s memory reference for ownership and allocation details.

Learn more: the repository, the first-inference tutorial, supported operators, sizing the memory, running on a microcontroller and writing a backend.

Licensed under Apache-2.0.

Re-exports§

pub use nanoglia_core as core;

Structs§

AllocError
The AllocError error indicates an allocation failure that may be due to resource exhaustion or to something wrong when combining the given input arguments with this allocator.
Arena
A bump allocator over a caller-provided buffer, typically a static array.
BuiltinOp
A TFLite builtin operator code (BuiltinOperator in the TFLite schema).
Dims
The shape of a tensor, stored inline (no allocation).
Error
The error type of every fallible nanoglia operation. It is Copy and never allocates.
Global
The global memory allocator.
Graph
A loaded model, ready to create execution contexts.
GraphExecutionContext
The mutable state of one inference: the arena holding the activations and the scratch memory.
MemoryBreakdown
Where the memory of a model goes: the sizes of MemoryRequirements and what they are made of. Reported by ModelReport for a model that loads.
MemoryRequirements
Memory a model needs, computed without loading it.
ModelReport
Whether a model runs, and what it needs: the answer to “will Graph::load accept this model?” with every operator that stands in the way, where loading stops at the first.
OpInfo
An operator, as an Observer sees it during ExecutionContext::compute_with.
OperatorReport
What ModelReport::check_with found for one operator of a model.
OwnedGraph
A prepared graph owning an aligned model allocation, without copying its weights.
Quantization
The quantization parameters of a tensor, borrowed from the model.
Registry
The ordered list of backends a graph picks its kernels from: for each operator, the first backend that supports it wins.
SelectedKernel
The kernel selected for an operator, in an OperatorReport.
TensorDesc
Everything known before execution about a tensor of the graph.
TensorMut
A writable view of an input tensor of an execution context.
TensorRef
A read-only view of a tensor of an execution context.

Enums§

Activation
Activation fused into an operator (TFLite ActivationFunctionType).
DataType
The element type of a tensor. Discriminants are the TFLite TensorType codes.
Detail
The precise reason of an Error, with the indices needed to locate it in the model.
Encoding
How the model bytes are encoded, as in wasi-nn’s graph-encoding.
ErrorCode
The category of an Error, identical to the error-code enum of wasi-nn so a wasi-nn host can forward it unchanged.
OpCode
The code identifying an operator.
Step
Progress of an inference run step by step (ExecutionContext::step).
Target
Where the graph should run, as in wasi-nn’s execution-target.

Traits§

Allocator
An implementation of Allocator can allocate, grow, shrink, and deallocate arbitrary blocks of data described via Layout.
Element
Rust element types matching a DataType.
GraphHandle
A stable graph reference or owner retained by an execution context.
Observer
Watches the operators of ExecutionContext::compute_with: to time them, trace them, or check intermediate tensors.

Type Aliases§

ExecutionContext
An execution context that borrows its graph; no ownership allocation is introduced.
Result
The result type of nanoglia.