sensor-db
Append-only sensor database for NOR flash on embedded devices. Built for the RP2350 (Raspberry Pi Pico 2) but works with any embedded-storage-compatible NOR flash.
#![no_std] · #![no_alloc] · zero-copy typed API · sync + async
Is this the right crate?
Probably the first question worth answering, because the embedded flash-storage crates that show up next to this one solve a different problem. They are key-value stores; this is an append-only time series.
sensor-db |
sequential-storage |
ekv |
tickv |
|
|---|---|---|---|---|
| Data model | append-only log of (sensor_id, timestamp, payload) |
key→value map, and a byte-slice FIFO queue | key→value map (LSM-tree) | key→value map, keys hashed to u64 |
| The question it answers | "what did sensor 3 read between t₁ and t₂?" | "what is the value of key K?" | "what is the value of key K?" | "what is the value of key K?" |
| Range / time queries | yes — query_range, latest, oldest, count, per-sensor |
no | no (ordered key scan, not time) | no |
| Update / delete a record | no, by design — retention is compact |
yes (remove_item needs MultiwriteNorFlash) |
yes | yes |
| Typed records | yes — #[derive(SensorData)], zero-copy &T out of the read buffer, schema version tagged per payload |
byte slices (Key/Value traits) |
byte slices | byte slices |
| Flash trait | NorFlash only, no MultiwriteNorFlash anywhere |
NorFlash; MultiwriteNorFlash for removal |
its own flash trait | its own FlashController |
| Store-and-forward to removable media | yes — replication cursor lives on the card | no | no | no |
Reach for sensor-db when the record is a reading with a timestamp, you
only ever append, and you want to ask for a time range or the latest sample
per sensor — optionally with an SD card or removable flash acting as the bulk
archive that the device drains into.
Reach for something else when:
- Config, calibration, mutable state — anything with a "current value per
key". Use
sequential-storage's map,ekv, ortickv. This crate has no update and no delete; simulating one by appending and reading the latest works but wastes erase cycles and RAM index space on data you do not keep. - A work queue — jobs pushed and popped, not readings retained. That is
sequential-storage'squeue. - You need both on one chip. That is not an either/or — see Sharing a chip below.
- Payloads larger than a page. An entry never spans a page, so a record is
capped at
PAGE_SIZE - 20bytes (- 21typed) — 235 bytes on a 256-byte-page part. Blobs belong somewhere else. - You need a filesystem. No paths, no directories, no files. See
embedded-sdmmc.
Why not a B-tree?
B-trees incur write amplification on NOR flash: page splits rewrite entire nodes, and in-place updates require erase-before-write cycles that wear the medium. Sensor data is append-only and time-ordered — a log-structured store eliminates these overheads entirely.
Architecture
┌─────────────────────────────────────────────────────────────┐
│ SensorLogger (clock injection) │
│ generic over the ports below — never over a log type │
├─────────────────────────────────────────────────────────────┤
│ LogSink / LogSource (capability ports) │
│ append() · append_run() · sync() | latest() · oldest() │
│ count() · query_range() — typed API in the *Ext mixins │
├─────────────────────────────────────────────────────────────┤
│ TieredDb / TieredDbAsync (optional: removable media) │
│ routes append() between onboard flash and a card │
│ attach() · detach() · spool_step() · drain() │
├─────────────────────────────────────────────────────────────┤
│ Spooler / SpoolerAsync (optional: replication) │
│ copies onboard → card, owns the on-card cursor │
├─────────────────────────────────────────────────────────────┤
│ SensorDb / SensorDbAsync (high-level API) │
│ append() · sync() · latest() · query_range() │
│ compact() · format() · stats() · mount() │
│ Typed API: append_sensor() · latest_sensor() · ... │
├─────────────────────────────────────────────────────────────┤
│ Circular sector ring │
│ ┌────────┬────────┬────────┬───┬────────┬───────┐ │
│ │SEALED │SEALED │SEALED │...│ ACTIVE │ERASED │ │
│ │seq=3 │seq=4 │seq=5 │ │seq=N+1 │ │ │
│ └────────┴────────┴────────┴───┴────────┴───────┘ │
│ ↑ tail (oldest) ↑ head (write) │
├─────────────────────────────────────────────────────────────┤
│ Page-buffered entry log │
│ Page 0: [SectorHdr 28B][Entry][Entry][pad 0xFF] │
│ Page 1: [Entry][Entry][Entry][pad 0xFF] │
│ ... │
│ Page K: [Entry][pad 0xFF] │
├─────────────────────────────────────────────────────────────┤
│ embedded-storage NorFlash / embedded-storage-async │
│ (RP2350 internal QSPI · external SPI · SD card · mock) │
└─────────────────────────────────────────────────────────────┘
The removable-media layers are opt-in. A device with no card uses SensorDb
directly and pays nothing for them — and because everything above the ports
names a capability rather than a type, adding a card later does not change a
line of the code that records samples.
Flash Layout for RP2350
0x0000_0000 ┌──────────────────────┐
│ Firmware (2 MB) │
0x0020_0000 ├──────────────────────┤
│ Sensor DB (2 MB) │ 512 sectors × 4 KB
0x0040_0000 └──────────────────────┘
Adjust base_offset and sector_count to your firmware size.
Entry Wire Format (v4)
Offset Size Field
────── ──── ───────────────────────────────────
0 1 marker 0xAA one sample, 0xAB a run (≠ 0xFF erased)
1 1 flags 0xFF=uncommitted, 0x00=committed
2 2 sensor_id u16 LE
4 8 timestamp u64 LE (caller epoch)
12 2 payload_len u16 LE
14 2 header_crc CRC-16/IBM-SDLC over bytes 0..14, flags masked
16 N payload raw bytes
16+N 4 crc32c CRC-32C (same as btrfs, ext4)
Total: 20 + N bytes per entry. Entries never span page boundaries.
The header CRC is checked before payload_len is trusted, so a 0xAA byte
occurring naturally inside a payload is rejected structurally rather than
desyncing the scan.
The typed API frames one more byte inside the payload — the schema version of
the stored type — so a typed entry is 21 + size_of::<T>() bytes. The raw
append / query_range API is untagged and unaffected. See
Schema evolution.
Run entries (marker 0xAB)
A single f32 reading spends 20 bytes of framing plus a schema byte to carry 4
bytes of data — 84% overhead. A run stores a whole burst under one header:
Offset Size Field
────── ──── ───────────────────────────────────
0 1 schema_version
1 8 interval u64 LE — timestamp step between samples
9 2 count u16 LE — samples, never zero
11 N*S samples S = size_of::<T>()
The entry header's timestamp is the first sample's, so sample i is at
base + i × interval. Thirty-two f32s cost 159 bytes instead of 800.
A run is invisible to every read. query_range, latest, oldest and
count expand it back into the samples it holds, delivering byte for byte what
appending them one at a time would have — same headers, same payloads, same
order, same counts. Nothing above the storage layer has to know it happened, so
you can start using runs without changing a single read.
use Run;
let burst: = sample_at_1_khz;
db.append_run?; // one entry
// …and it reads back as thirty-two.
assert_eq!;
Only replication sees the run as one entry: read_from streams entries, and the
receiving log stores it back as a run, so the amortisation survives the trip to
a removable card rather than being unpacked on arrival.
Why a marker rather than a flag in the payload: the raw append path stores
arbitrary bytes, so a payload flag would misfire on any payload that happened
to carry the pattern — inventing a time span for an entry that is not a run.
The marker byte is covered by the header CRC and only this crate writes it.
Sector Header (SD05, 28 bytes)
Offset Size Field
────── ──── ───────────────────────────────────
0 4 magic "SD05"
4 1 version 5
5 3 reserved 0
8 4 log_id u32 LE — identity of the owning log (0 = anonymous)
12 4 epoch u32 LE — the run that created this sector
16 4 sequence u32 LE — monotonic sector counter
20 4 erase_count u32 LE
24 4 header_crc CRC-32C over the preceding 24 bytes
Only SD05 is supported. A sector whose magic or header CRC does not validate is
treated as free and never scanned; one that validates but carries a different
log_id aborts the mount rather than being reclaimed — see
Removable media.
Sector Footer (52 bytes, in the sector's last page)
Sealing a sector writes its in-RAM summary into the sector's last page,
which append never uses:
Offset Size Field
────── ──── ───────────────────────────────────
0 4 magic "SDF2"
4 1 version 2
5 3 reserved 0
8 4 sequence u32 LE — the sector this describes
12 4 entry_count u32 LE — entries; a run counts once
16 4 sample_count u32 LE — readable samples; a run counts all of its own
20 4 write_offset u32 LE — byte after the last entry
24 8 min_timestamp u64 LE
32 8 max_timestamp u64 LE — newest *sample*, so a run's span end
40 8 sensor_bloom u64 LE
48 4 footer_crc CRC-32C over the preceding 48 bytes
This is what makes a queryable mount cost O(sector_count) instead of
O(bytes stored) — one page read per sealed sector rather than a scan of every
entry. It is never a correctness dependency: a footer that is missing, torn, or
names a different sequence is ignored and the sector's body is scanned exactly
as before, which is also what happens to the sector a crashed run left active.
MountMode::Verify ignores footers entirely and re-reads the whole medium.
The reserved page costs PAGE_SIZE / ERASE_SIZE of raw capacity — 6.25% at 256
byte pages and 4 KB sectors. It gets its own page rather than sharing the header
page so that a footer write torn by power loss can damage nothing but the
footer.
RAM Usage
| Component | Size (RP2350 config) |
|---|---|
| Page buffer | 256 B (1 page) |
| Sector index | 512 × 48 B ≈ 24 KB |
| Scalars/state | ~80 B |
| Total | ~24 KB of 520 KB |
Adding a Spooler costs one further page buffer (for cursor records) plus about
64 bytes.
Quick Start
Sync API
use SensorDb;
// RP2350: 256-byte pages, up to 512 sectors
let mut db: = new?;
// First boot: format. Subsequent boots: mount.
db.format?;
// — or —
db.mount?;
Async API (Embassy)
use SensorDbAsync;
let mut db: = new?;
db.format.await?;
Write
let timestamp = get_time_us;
let temp: f32 = read_temperature;
db.append?;
Typed Write (zero-copy)
use SensorData;
use ;
// version defaults to 0; bump it when the struct changes
db.append_sensor?;
SensorData needs the derive feature. Without it, write the impl out — it is
all the derive expands to:
id and version take any constant expression, so the central ID module that
actually prevents collisions works directly:
Two things the derive deliberately does not do. It does not emit the zerocopy
derives — a derive macro cannot add attributes to its own item, and zerocopy's
derives expand to absolute ::zerocopy:: paths, so zerocopy has to be in your
Cargo.toml either way; injecting them would make that requirement invisible
and its absence inscrutable. And it does not check for padding, because
IntoBytes already refuses to derive for a padded type and reports the exact
number of wasted bytes:
error[E0277]: `Reading` has 3 total byte(s) of padding
types with padding cannot implement `IntoBytes`
Import zerocopy's traits through sensor_db::zerocopy as above. SensorData is
bounded on them, so zerocopy is a public dependency of this crate, and a
version skew between your Cargo.toml and this one otherwise surfaces as
the trait bound `Temperature: FromBytes` is not satisfied on a type that
plainly derives it.
Checking the sensor ID space
Nothing stops two types from claiming the same sensor ID. What happens then depends on how they differ, and one case is undetectable:
| The other type differs in | Read reports |
|---|---|
| schema version | SchemaMismatch |
| payload size | SizeMismatch (or BufferTooSmall if the stored entry is wider) |
| neither | nothing — the bytes are reinterpreted |
A same-sized collision is silent by construction: the entry is intact, its CRC
is correct, and Humidity { percent: 41.0 } reads back as
Temperature { celsius: 41.0 }. Because the mistake is in what was written,
it cannot be repaired afterwards.
sensor_registry! turns that into a build failure. List every type the firmware
writes, next to where the IDs are defined:
use sensor_registry;
sensor_registry!
error[E0080]: evaluation panicked: sensor registry: `Temperature` and
`Humidity` share a SENSOR_ID at the same SCHEMA_VERSION. ...
Sharing an ID at different versions is schema evolution, not a collision, so
the registry accepts it. The macro needs no feature and costs nothing at build
time — it expands to an anonymous const block of assertions and defines
nothing, so it can appear in any module, as often as you like. It only knows
about the types you list: it cannot see across crates, and a type left off the
list is not checked.
Adapting a driver's reading type
A sensor driver hands you its own struct, or a tuple of f32s. You cannot
implement SensorData for it: from your crate both the trait and the type are
foreign, so the orphan rule refuses — and the driver's type would not carry
zerocopy's derives anyway.
Declare the record you want on flash instead, and convert at the point of sampling. That is one small struct per sensor, and it is the same shape for every driver, which is why this appears once here rather than once per crate:
use SensorData;
use ;
// The driver's type — 12 channels plus the settings they were taken at:
// struct Reading { channels: [f32; 12], gain: Gain, integration_ms: f32 }
/// What actually goes on flash.
db.append_sensor?;
Three decisions in that struct are worth copying.
Fixed point, not f32. The sensor reported integers; the driver widened
them for you. Storing the counts is exact, halves the payload — the channels are
24 bytes rather than 48 — and avoids soft-float on parts with no FPU. Where a
scale is genuinely needed, put it in the field name: celsius_x100: i16 states
its own units and covers −327.68 … 327.67 in two bytes.
Store what was measured, plus whatever is needed to interpret it. Gain and integration time are in the record because a calibration applied on the device is a number you can never un-apply. The host can always recompute; the device cannot recover what it discarded.
Order fields widest first. [u16; 12] followed by two u8s is 26 bytes with
no padding. Drop one of the u8s and repr(C) pads the struct back to 26 — at
which point IntoBytes refuses to derive, and says so exactly:
error[E0277]: `Spectrum` has 1 total byte(s) of padding
types with padding cannot implement `IntoBytes`
That is the check working. Padding is uninitialised memory, and every byte this crate writes goes under a CRC.
examples/bme280.rs is this worked end to end against the real
bme280 crate — driver conversation, the
conversion above, a remount, and an assertion that every reading came back
exactly as measured:
Read Latest
// Raw API
let mut buf = ;
if let Some = db.latest?
// Typed API — zero-copy deserialization
if let Some = db.?
Range Query
let count = db.query_range?;
SensorLogger (auto-timestamped)
use SensorLogger;
let mut logger = new;
logger.log?;
// ...or over a tier, unchanged — the logger wraps any `LogSink`.
let mut logger = new;
logger.log?;
logger.sink_mut.attach?;
Writing against the capability, not the type
LogSink (append + sync) and LogSource (latest, oldest, count, query_range)
are implemented by both SensorDb and TieredDb, with LogSinkAsync /
LogSourceAsync for the async pair. The typed API lives in blanket-implemented
extension traits over them, so it is defined once and reaches both:
use ;
Both core ports are object-safe, so &mut dyn LogSink works too. Reads take
&mut self: SensorDb can answer through &self via its RefCell, but a
TieredDb query has to stream the onboard log from the replication watermark.
Maintenance
// Flush write buffer to flash (for durability across power loss)
db.sync?;
// Erase the 5 oldest sectors
db.compact?;
// Check health
let stats = db.stats;
// stats.total_entries, stats.used_sectors, stats.free_sectors, ...
Power-Loss Safety
- Two-phase commit: entries are written with
flags=0xFF(uncommitted), then the same page is re-written withflags=0x00(committed). NOR flash only clears bits (1→0), so the second write is always valid. On mount, uncommitted entries are discarded. - CRC-32C per entry (~1-in-4-billion collision rate). CRC includes the entry's offset within the sector to detect transplanted entries after partial erases.
- Sector headers use a monotonic sequence number to reconstruct ring order.
- The write buffer (one page) is the blast radius: at most one page of entries can be lost on unexpected power loss. Call
sync()before critical checkpoints. - No in-place flash updates — fully compatible with strict
NorFlash(noMultiwriteNorFlashrequired).
What it costs
Measured, not estimated. Every number below is counted from real NorFlash
calls by MockFlash, and regenerated with:
They are operations, not seconds. A wall-clock figure from a desktop
measures a Vec<u8>; flash operations transfer across parts. On the NOR flash
this crate targets a sector erase is 100 ms or more, a 256-byte page program
is around a millisecond, and a read is bus-bound — so multiply by your
datasheet's numbers and you have the answer for your hardware.
Geometry throughout: 256-byte pages, 4 KB sectors, 40-byte payloads (60 bytes per entry, so 59 entries per sector).
One call
| Operation | Reads | Bytes read | Page writes | Erases | |
|---|---|---|---|---|---|
append, buffered |
0 | 0 | 0 | 0 | the common case — the entry goes to the RAM page buffer |
append that fills a page |
0 | 0 | 2 | 0 | the data pass, then the commit pass |
append that seals a sector |
0 | 0 | 4 | 1 | flush, footer, erase, header — the one that costs 100 ms+ |
sync, buffer non-empty |
0 | 0 | 2 | 0 | the same two-phase page write |
sync, buffer empty |
0 | 0 | 0 | 0 | free |
count(None) |
0 | 0 | 0 | 0 | answered from the in-RAM index |
latest, sensor absent |
0 | 0 | 0 | 0 | every sector rejected by its bloom filter |
latest, sensor present |
51 | 1.4 KiB | 0 | 0 | newest sector first, stopping at the first that has it |
count(Some(_)) over 200 entries |
447 | 12.5 KiB | 0 | 0 | CRC-validated, which is why it agrees with query_range |
query_range, same 200 entries |
447 | 12.5 KiB | 0 | 0 | exactly the same reads |
compact(1, All) |
0 | 0 | 0 | 1 |
So a sampling loop is free until it fills a page, and pays for an erase once every 59 samples. Nothing on the read path ever writes.
One burst, stored both ways
A run of n readings of T costs 20 + 11 + n×size_of::<T>() bytes against
n × (20 + 1 + size_of::<T>()) appended one at a time. For a 4-byte reading:
| Samples | Bytes, one at a time | Bytes, as a run | Saving | Page writes, singles | Page writes, run |
|---|---|---|---|---|---|
| 8 | 200 | 63 | 69% | 2 | 2 |
| 16 | 400 | 95 | 77% | 4 | 2 |
| 32 | 800 | 159 | 81% | 8 | 2 |
| 56 | 1,400 | 255 | 82% | 12 | 4 |
56 is the most a 256-byte page holds. Reads get cheaper in step — the samples share a header, a CRC and usually a page — and the API does not change, because a run expands transparently.
mount on a 2 MB region (512 × 4 KB)
| Fill | Mode | Entries indexed | Reads | Bytes read | Sector bodies scanned |
|---|---|---|---|---|---|
| ¼ full | Full |
8,048 | 722 | 24.6 KiB | 1 |
| ¼ full | Verify |
8,048 | 18,538 | 518 KiB | 137 |
| ¼ full | HeadersOnly |
— | 586 | 18.2 KiB | 1 |
| ½ full | Full |
16,096 | 900 | 30.8 KiB | 1 |
| ½ full | Verify |
16,096 | 36,532 | 1018 KiB | 273 |
| ½ full | HeadersOnly |
— | 628 | 18.1 KiB | 1 |
| full | Full |
30,187 | 1,122 | 42.2 KiB | 1 |
| full | Verify |
30,187 | 68,063 | 1.85 MiB | 512 |
| full | HeadersOnly |
— | 611 | 18.3 KiB | 1 |
This is what the sector footer
buys. A fully queryable mount of a brim-full 2 MB region reads 42 KiB
instead of 1.85 MiB — 45× less — and grows with the sectors written rather
than with the data in them: quadrupling the stored data takes Full from 24.6
to 42.2 KiB while Verify goes from 518 KiB to 1.85 MiB. HeadersOnly indexes
nothing and so answers no queries; its cost is the floor, and Full is now
close to it.
tests/cost.rs pins these — the per-call rows as exact equalities, the mount
ratios with headroom — so the table cannot rot unnoticed.
Performance
- Per-sector bloom filter: 64-bit Fibonacci-hashed sensor ID bloom filter skips sectors that definitely don't contain the target sensor. Near-zero false positives for ≤16 distinct sensor types.
- Wear leveling via erase counts:
find_next_sector()prefers erased sectors with lowest erase count, stored in SD05 sector headers and persisted across remount. - Sector footers: a sealed sector carries its own summary in its last page, so
mount()rebuilds a fully queryable index inO(sector_count)— one 28-byte header plus one 52-byte footer per sector — instead of reading every byte of stored data. Only the sector the previous run left active is scanned. On the 2 MB RP2350 configuration that is ~512 short page reads per boot instead of 2 MB over QSPI. Falls back to a body scan wherever a footer is missing or fails to validate, so it is purely an optimisation. See Sector Footer. MountMode::HeadersOnly: skips the footers too, reading 28 bytes per sector and nothing else. Now a smaller win than it used to be, and still the only mode whose cost isO(sector_count)unconditionally — it never falls back to a body scan. Queries returnReadsUnavailablerather than answering from a partial index.MountMode::Verify: the opposite trade — re-reads and CRC-checks every stored entry at mount and reportscorrupt_entries. Cost is proportional to the data stored. This is whatFulldid before footers existed.
Removable media
For a logger with no network link: samples are collected continuously, and a card is periodically pulled and carried to a computer. While the card is out, collection continues onboard; when it returns, the backlog lands on it.
use ;
use LogId;
let log = new.unwrap;
// The card reserves two sectors for the replication cursor; the rest is a
// normal SensorDb region.
let spooler = new?;
let mut db = new?;
// Collect. Never fails because of the card, present or not.
db.append?;
db.sync?;
// Card goes in: the backlog drains, then samples go straight to it.
db.attach?;
db.drain?;
// Card comes out. Collection continues onboard.
let card = db.detach;
OnboardFirst |
CardFirst |
|
|---|---|---|
| Every sample written | onboard, then copied | to the card while it is in |
| Onboard wear/space while card is in | full | ~none |
| Copies of a sample | two, until the ring reclaims one | one |
| Power cut in steady state | loses one onboard page | loses one card page |
| Card yanked mid-write | nothing to lose | nothing lost — RAM buffer rescued |
What it guarantees
- No loss. The onboard watermark advances only after the card's copy is durable, so an interruption re-copies rather than skips. A yank is not a power loss — the processor keeps running — so samples still in the card's RAM page buffer are rescued into the onboard log.
- Bounded duplication. A power cut or unclean removal replays at most the
in-flight batch:
SpoolBudget::max_entriesentries. That value is the bound. - In practice, no duplication. The bound is the fallback. On reinsertion the
interrupted batch's intent record says where on the card it began, so the card
is read forward from there to find how much of it survived, and exactly that
much is skipped on the source. The skip is checked against a fingerprint of
both runs and abandoned if they disagree — every fallback duplicates rather
than skips, because duplication is bounded and visible while a wrong skip is
silent loss.
Attached::recovered_entriesreports what was found. - Exact de-duplication offline. Each commit record maps a region of the card to a source position range, so a reader spots a replayed batch by its overlapping range and keeps the later one. This works regardless of the clock — it needs no unique timestamps.
- A card from another logger is refused, not overwritten. Every sector
carries a
log_id; mounting a foreign one fails withForeignVolumebefore anything is written. Without this the "no valid sectors" path would erase a sector to bootstrap the log, destroying another device's data.
How the cursor works
Two reserved erase-sectors on the card, ping-ponged, one record per page-sized slot. A batch writes an intent before copying and a commit after the data is durable — two records rather than an in-place flag flip, because a commit carries values the intent did not have and NOR can only clear bits.
On reinsertion the resume point is the highest position any record justifies, counting an intent only for where its batch started. A torn commit is therefore recovered from the next intent, which is what bounds duplicates to a single batch.
That bound is then closed. An intent also records dest, the position on the
card its first entry landed at, and nothing is appended after a batch except the
next batch — which never started, or its intent would be the newest record. So
reading the card forward from dest counts exactly what the interrupted batch
delivered, and Spooler::reconcile skips that many source entries before copying
resumes. Both runs are fingerprinted (length, plus the first and last
(sensor_id, timestamp)) and the skip is abandoned unless they match, so a
mismatch — or a source ring that has wrapped past those entries — falls back to
replaying the batch rather than stepping over data no card ever received.
The batch counter is a u32, so it eventually wraps — 136 years at one batch per
second, but the cursor is meant to outlive the device. Records are ordered by
serial-number arithmetic rather than <, which is unambiguous here because a
card's whole cursor region holds only a few hundred records and so spans a tiny
window of the counter. Ordering them numerically would name the wrong sector
active and, on the next wrap, erase the one holding the true resume point.
Telling the operator what the card is doing
Two things about a card are invisible from the write path, and neither is an error when it happens:
- Erases. An erase is 100 ms or more on many NOR parts, against microseconds
for a page write.
SpoolProgress.volume_erasedreports the sectors a replication pass erased on the card — its log rotating, plus the cursor region wrapping — so a caller with a deadline elsewhere can spool less often or with a smaller budget. UnderCardFirstthe same spike lands onappendinstead;SensorDb::sectors_erasedon the card is the counter that sees those. It counts what this instance erased, unlikeDbStats::total_erase_count, which is the wear history read back from the sector headers. - A card filling up.
TieredDb::volume_fillgivesused_sectors,free_sectors,total_sectorsandentries, plusused_permille()andwill_overwrite(). The last isfree_sectors == 0: the final moment at which swapping the card still costs nothing. This matters most underCardFirst, where the card holds the only copy of everything written since it took over, so wrapping is real loss rather than the loss of a redundant copy. A ring overwriting its own tail is normal operation for a log and is never reported as an error, so this is the whole warning.
See examples/removable_card.rs for the full cycle, and WritePolicy's
documentation for the costs of each mode.
Schema evolution
A SensorData type binds a struct to a sensor ID. Change the struct and every
entry already on flash is still stored under that ID, and is no longer a T. A
length check catches the easy half of that — a field added or removed — and
misses the dangerous half: a change that keeps the size and alters the meaning
(celsius becoming fahrenheit, two fields swapped, a scale factor changed).
Those read back as plausible garbage.
So a typed payload carries its version in front of the value:
payload = [ schema_version : u8 ] [ T's bytes ]
To migrate, keep the old struct, give the new one the next version, and let both name the same sensor ID:
query_sensor skips entries of another version, so each type reads its own
era out of one log and a firmware update needs no flash migration pass. The
single-entry reads instead report which era they found:
| Result | Means |
|---|---|
SchemaMismatch { expected, found } |
Another era, named. Dispatch on it. |
SizeMismatch { .. } |
This version, a different length — the struct changed and SCHEMA_VERSION did not. |
BufferTooSmall |
The stored entry is longer than a T; its payload never lands, so its tag cannot be read. Try the largest candidate type first. |
The tag lives in the payload rather than the entry header on purpose. The header
is storage framing — what the scanner needs to walk the log — and two things
follow from putting an application concept there instead. The only spare room is
the flags byte, which both CRCs mask to 0xFF so a committed and an
uncommitted entry hash alike, making it the one field nothing protects. And
replication copies an entry as (sensor_id, timestamp, payload), so a header
field would have to be threaded through read_from's callback and both spool
drivers to reach the card. A payload tag is CRC-covered and rides along
untouched.
Internal vs External Flash
The RP2350's internal QSPI flash shares the XIP bus with firmware. Flash operations briefly disable XIP (both cores stall). For high-frequency logging or latency-sensitive applications, consider an external SPI flash (e.g., W25Q64 on SPI1). The database API is identical — just swap the NorFlash implementation:
// Internal flash (embassy-rp)
type InternalDb<'a> = ;
// External SPI flash (your driver)
type ExternalDb<S> = ; // 8MB = 1024 sectors
Sharing a chip
One flash part usually holds more than one thing. Partition<F> is a NorFlash
view of a sub-range — pure address arithmetic over your driver, knowing nothing
about which driver it is:
use ;
// Firmware below, config in the middle, the log in the top 2 MB.
let region = new?;
let mut db: = new?;
The database now addresses the partition, not the chip: offset 0 is 0x0020_0000,
capacity() reports 2 MB, and an access past the end is
PartitionError::OutOfBounds rather than a write into whoever lives next door.
Erase, write and read granularity pass through unchanged.
This is also how you run sensor-db alongside sequential-storage (which takes
a Range<u32> for the same reason) or a firmware slot: give each a disjoint
range and nothing needs to trust anything.
Reaching the rest of the chip
SensorDb owns the driver by value, so something has to broker access to the
neighbours. Two methods, with different jobs:
with_region(base, len, |p| …) |
Flash access bounded to [base, base+len), handed to the closure as a Partition. A region intersecting the database's own sectors is refused with Error::RegionOverlap. |
driver_mut() |
The driver itself, for capabilities that are not addresses: presence detection, wear counters, chip id, power modes. Unbounded — do not address flash through it. |
The replication cursor uses the first: Spooler writes its records through
with_region, so a cursor_base that overlaps the card's own database region
now fails at attach instead of silently overwriting replicated entries.
The async driver has region(base, len) -> Partition<&mut F> instead of a
closure, because a closure cannot return a future that borrows its own argument.
Sharing one log between tasks or cores
SensorDb and SensorDbAsync are Send + Sync whenever the driver is, so two
Embassy tasks — or both RP2350 cores — can log to one database. Wrap it in a
mutex and share it:
use CriticalSectionRawMutex;
use Mutex;
type Db = ;
static DB: = new;
async
Four things are worth knowing before you build on that.
Mutex, not RwLock. An RwLock compiles and buys nothing. Reading flash
goes through ReadNorFlash::read(&mut self), so every method — latest and
count as much as append — needs &mut, which is the write half of the lock.
An RwLock whose read half is never taken is a Mutex with extra steps.
Never hold the lock across a wait. An embedded executor is cooperative: a
task holding the mutex across Timer::after(..).await blocks every other task
for that whole duration. Take it, do the operation, drop it.
Put sync() in one place. It is normally two page writes — microseconds —
but the one that happens to seal a sector also erases, and an erase is 100 ms
or more on many NOR parts. Whoever is waiting on the mutex waits for that too.
Calling it from a single task on a known schedule makes that latency
predictable instead of arriving whenever a sampler happens to fill a page.
Durability is not visibility. Reads see the RAM page buffer as well as
flash, so an un-synced append from another task is already queryable.
sync() is only about surviving a power cut.
Two runnable examples, both asserting the properties above rather than merely printing:
multithread_async.rs also carries a table mapping each piece of its desktop
scaffolding to the RP2350 equivalent, so the task bodies transfer unchanged.
Features
| Feature | Description |
|---|---|
std (default) |
Enables MockFlash for desktop testing |
async |
Enables SensorDbAsync, SpoolerAsync and TieredDbAsync via embedded-storage-async 0.4 (for Embassy, etc.). The async ports and the code generic over them are available without it. |
derive |
Enables #[derive(SensorData)]. Pulls in syn/quote, at build time only — nothing reaches the target. |
defmt |
Enables defmt::Format impls for embedded debugging |
Design Decisions
| Decision | Rationale |
|---|---|
| Entries can't span pages | Simplifies recovery (each page is self-contained) |
| No in-place state updates | Strict NorFlash compliance, no MultiwriteNorFlash needed |
| CRC-32C per entry | ~1-in-4-billion detection rate, same algorithm as btrfs/ext4 |
| Two-phase commit | Power-loss atomic writes without MultiwriteNorFlash |
| Sector-level index only | Keeps RAM bounded at MAX_SECTORS × 48 bytes, and keeps a query's flash reads proportional to the sectors the bloom filter admits rather than to the region — see What it costs |
| Circular allocation | Natural FIFO retention; no external GC policy needed |
| Callback-based queries | Avoids lifetime issues with flash borrows in iterators |
| Sync + async as parallel types | Zero abstraction overhead; no generic async/sync trait gymnastics |
| Replication cursor on the card, not the device | The replica is the only thing that knows what it received. Two cards can be rotated against one log with no local state — whichever is inserted says where it got to. |
| At-least-once replication, reconciled to exactly-once on reinsert | Exactly-once during a batch needs the copy and the acknowledgement to be one atomic act, which two independent media cannot provide. So the protocol guarantees at-least-once with a one-batch bound, and closes it afterwards: the card is asked how much of the interrupted batch it kept, and only the remainder is copied. A reconciliation that cannot be verified is abandoned back to the bound. |
| A run is a distinct entry marker, not a payload flag | The raw append path stores arbitrary bytes, so a payload flag misfires on any payload that happens to carry the pattern — inventing a time span for an entry that is not a run. An earlier attempt shipped exactly that and was reverted when the model-based tests caught it on a workload with no typed data in it. The marker byte is covered by the header CRC and only this crate writes it. |
| Runs expand at the storage layer, not in the typed API | query_range, latest, oldest and count all speak samples, so a run is byte-for-byte indistinguishable from the same single appends and no reader has to opt in. Replication is the one exception: it copies entries, and a replica should keep the amortisation rather than unpack it. |
| Schema version tagged into the payload, not the entry header | The header is storage framing; a schema version is application meaning that nothing in the crate reads. The only spare header room is the flags byte, which both CRCs mask so the commit flip does not change them — a version there would be unprotected. And replication copies (sensor_id, timestamp, payload), so a header field would have to be threaded through the spool drivers to survive the trip to the card. |
Typed API defined over capability ports, not over SensorDb |
One definition serves a single log and a tier. It also has to be unoverridable — if an implementor could customise how a SensorData value becomes payload bytes, the same entry would read back differently depending on which log wrote it — so it lives in a blanket-implemented mixin rather than in the port. |
| Presence detection is a port, not a driver | Removable is one method. Card-detect wiring is irreducibly board-specific, so it stays out of the crate. |
Testing
# Run all tests (sync + async + derive)
# Verify no_std compilation
# Verify no_std + async
# Run examples
# Regenerate the cost tables above
534 tests: 101 unit + 126 integration + 12 power-loss + 5 power-loss fuzz +
147 removable-media (spool, identity, tiered, card_first,
tiered_async) + 3 cursor-protocol fuzz + 6 model-based + 11 capability-port +
19 schema-evolution + 27 partition + 20 sector-footer + 15 flash-op budget +
24 run-encoding + 13 derive + 5 registry.
The examples are run in CI as well as built. Each ends by asserting the claim it
demonstrates, so power_loss is a check that no synced reading is ever lost to a
torn write, and bursts a check that a run and the same single appends are
observationally identical.
Power-loss and removal are tested deterministically, not by hope: MockFlash
injects a clean or partial write failure at a chosen write index — or a torn
erase, leaving a blank head and a stale tail as a real interrupted erase does
— and RemovableFlash makes a medium vanish, failing reads as well as writes,
either on command or after n further writes. The async driver is checked
against the blocking one by running the same scenario through both and comparing
the resulting media byte for byte, cursor records included.
The two *_fuzz suites stop choosing where to interrupt. power_loss_fuzz
replays a workload once per (fault site × byte offset) pair — every write and
every erase it performs, torn after every possible number of bytes — and after
each one asserts that the log mounts, that the survivors are a contiguous run of
what was appended, that nothing a successful sync acknowledged is missing, and
that the log still accepts writes which then survive another cycle.
spool_fuzz does the same to the replication cursor: every card write torn at
every byte offset, every yank point, and repeated interruption across
insert/remove cycles, asserting the whole series reaches the card exactly once.
Between them that is some ten thousand recoveries, in under three seconds.
tests/cost.rs pins the numbers in What it costs, so a
published figure cannot rot unnoticed: the per-call costs as exact equalities,
the mount aggregates as ratios with headroom.
tests/run.rs is almost entirely differential, because the claim it checks is
an equivalence: the same series is stored once as a
run and once as single appends, and the two logs are
compared through every read path there is — count, latest, oldest, the raw
and typed queries, and a set of windows that cut a run in half at either end.
Anything the two disagree about is a defect by construction, whichever is right.
Runs are in the model-based workload too, where they are expanded into ordinary
entries in the Vec oracle, which is the same claim stated a second way.
License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.