sensor-db 0.1.0

Append-only sensor database for NOR flash on embedded devices
Documentation

sensor-db

CI License

Append-only sensor database for NOR flash on embedded devices. Built for the RP2350 (Raspberry Pi Pico 2) but works with any embedded-storage-compatible NOR flash.

#![no_std] · #![no_alloc] · zero-copy typed API · sync + async

Is this the right crate?

Probably the first question worth answering, because the embedded flash-storage crates that show up next to this one solve a different problem. They are key-value stores; this is an append-only time series.

sensor-db sequential-storage ekv tickv
Data model append-only log of (sensor_id, timestamp, payload) key→value map, and a byte-slice FIFO queue key→value map (LSM-tree) key→value map, keys hashed to u64
The question it answers "what did sensor 3 read between t₁ and t₂?" "what is the value of key K?" "what is the value of key K?" "what is the value of key K?"
Range / time queries yes — query_range, latest, oldest, count, per-sensor no no (ordered key scan, not time) no
Update / delete a record no, by design — retention is compact yes (remove_item needs MultiwriteNorFlash) yes yes
Typed records yes — #[derive(SensorData)], zero-copy &T out of the read buffer, schema version tagged per payload byte slices (Key/Value traits) byte slices byte slices
Flash trait NorFlash only, no MultiwriteNorFlash anywhere NorFlash; MultiwriteNorFlash for removal its own flash trait its own FlashController
Store-and-forward to removable media yes — replication cursor lives on the card no no no

Reach for sensor-db when the record is a reading with a timestamp, you only ever append, and you want to ask for a time range or the latest sample per sensor — optionally with an SD card or removable flash acting as the bulk archive that the device drains into.

Reach for something else when:

  • Config, calibration, mutable state — anything with a "current value per key". Use sequential-storage's map, ekv, or tickv. This crate has no update and no delete; simulating one by appending and reading the latest works but wastes erase cycles and RAM index space on data you do not keep.
  • A work queue — jobs pushed and popped, not readings retained. That is sequential-storage's queue.
  • You need both on one chip. That is not an either/or — see Sharing a chip below.
  • Payloads larger than a page. An entry never spans a page, so a record is capped at PAGE_SIZE - 20 bytes (- 21 typed) — 235 bytes on a 256-byte-page part. Blobs belong somewhere else.
  • You need a filesystem. No paths, no directories, no files. See embedded-sdmmc.

Why not a B-tree?

B-trees incur write amplification on NOR flash: page splits rewrite entire nodes, and in-place updates require erase-before-write cycles that wear the medium. Sensor data is append-only and time-ordered — a log-structured store eliminates these overheads entirely.

Architecture

┌─────────────────────────────────────────────────────────────┐
│  SensorLogger               (clock injection)               │
│    generic over the ports below — never over a log type     │
├─────────────────────────────────────────────────────────────┤
│  LogSink / LogSource        (capability ports)              │
│    append() · append_run() · sync() | latest() · oldest()   │
│    count() · query_range() — typed API in the *Ext mixins   │
├─────────────────────────────────────────────────────────────┤
│  TieredDb / TieredDbAsync   (optional: removable media)     │
│    routes append() between onboard flash and a card         │
│    attach() · detach() · spool_step() · drain()             │
├─────────────────────────────────────────────────────────────┤
│  Spooler / SpoolerAsync     (optional: replication)         │
│    copies onboard → card, owns the on-card cursor           │
├─────────────────────────────────────────────────────────────┤
│  SensorDb / SensorDbAsync  (high-level API)                 │
│    append() · sync() · latest() · query_range()             │
│    compact() · format() · stats() · mount()                 │
│    Typed API: append_sensor() · latest_sensor() · ...       │
├─────────────────────────────────────────────────────────────┤
│  Circular sector ring                                       │
│    ┌────────┬────────┬────────┬───┬────────┬───────┐        │
│    │SEALED  │SEALED  │SEALED  │...│ ACTIVE │ERASED │        │
│    │seq=3   │seq=4   │seq=5   │   │seq=N+1 │       │        │
│    └────────┴────────┴────────┴───┴────────┴───────┘        │
│     ↑ tail (oldest)              ↑ head (write)             │
├─────────────────────────────────────────────────────────────┤
│  Page-buffered entry log                                    │
│    Page 0: [SectorHdr 28B][Entry][Entry][pad 0xFF]          │
│    Page 1: [Entry][Entry][Entry][pad 0xFF]                  │
│    ...                                                      │
│    Page K: [Entry][pad 0xFF]                                │
├─────────────────────────────────────────────────────────────┤
│  embedded-storage NorFlash / embedded-storage-async         │
│    (RP2350 internal QSPI · external SPI · SD card · mock)   │
└─────────────────────────────────────────────────────────────┘

The removable-media layers are opt-in. A device with no card uses SensorDb directly and pays nothing for them — and because everything above the ports names a capability rather than a type, adding a card later does not change a line of the code that records samples.

Flash Layout for RP2350

  0x0000_0000  ┌──────────────────────┐
               │  Firmware  (2 MB)    │
  0x0020_0000  ├──────────────────────┤
               │  Sensor DB (2 MB)    │  512 sectors × 4 KB
  0x0040_0000  └──────────────────────┘

Adjust base_offset and sector_count to your firmware size.

Entry Wire Format (v4)

Offset  Size  Field
──────  ────  ───────────────────────────────────
  0      1    marker        0xAA one sample, 0xAB a run (≠ 0xFF erased)
  1      1    flags         0xFF=uncommitted, 0x00=committed
  2      2    sensor_id     u16 LE
  4      8    timestamp     u64 LE (caller epoch)
 12      2    payload_len   u16 LE
 14      2    header_crc    CRC-16/IBM-SDLC over bytes 0..14, flags masked
 16      N    payload       raw bytes
 16+N    4    crc32c        CRC-32C (same as btrfs, ext4)

Total: 20 + N bytes per entry. Entries never span page boundaries.

The header CRC is checked before payload_len is trusted, so a 0xAA byte occurring naturally inside a payload is rejected structurally rather than desyncing the scan.

The typed API frames one more byte inside the payload — the schema version of the stored type — so a typed entry is 21 + size_of::<T>() bytes. The raw append / query_range API is untagged and unaffected. See Schema evolution.

Run entries (marker 0xAB)

A single f32 reading spends 20 bytes of framing plus a schema byte to carry 4 bytes of data — 84% overhead. A run stores a whole burst under one header:

Offset  Size  Field
──────  ────  ───────────────────────────────────
  0      1    schema_version
  1      8    interval      u64 LE — timestamp step between samples
  9      2    count         u16 LE — samples, never zero
 11      N*S  samples       S = size_of::<T>()

The entry header's timestamp is the first sample's, so sample i is at base + i × interval. Thirty-two f32s cost 159 bytes instead of 800.

A run is invisible to every read. query_range, latest, oldest and count expand it back into the samples it holds, delivering byte for byte what appending them one at a time would have — same headers, same payloads, same order, same counts. Nothing above the storage layer has to know it happened, so you can start using runs without changing a single read.

use sensor_db::Run;

let burst: [Temperature; 32] = sample_at_1_khz();
db.append_run(&Run::of(t0, 1_000, &burst)?)?;   // one entry

// …and it reads back as thirty-two.
assert_eq!(Temperature::count(&mut db)?, 32);

Only replication sees the run as one entry: read_from streams entries, and the receiving log stores it back as a run, so the amortisation survives the trip to a removable card rather than being unpacked on arrival.

Why a marker rather than a flag in the payload: the raw append path stores arbitrary bytes, so a payload flag would misfire on any payload that happened to carry the pattern — inventing a time span for an entry that is not a run. The marker byte is covered by the header CRC and only this crate writes it.

Sector Header (SD05, 28 bytes)

Offset  Size  Field
──────  ────  ───────────────────────────────────
  0      4    magic         "SD05"
  4      1    version       5
  5      3    reserved      0
  8      4    log_id        u32 LE — identity of the owning log (0 = anonymous)
 12      4    epoch         u32 LE — the run that created this sector
 16      4    sequence      u32 LE — monotonic sector counter
 20      4    erase_count   u32 LE
 24      4    header_crc    CRC-32C over the preceding 24 bytes

Only SD05 is supported. A sector whose magic or header CRC does not validate is treated as free and never scanned; one that validates but carries a different log_id aborts the mount rather than being reclaimed — see Removable media.

Sector Footer (52 bytes, in the sector's last page)

Sealing a sector writes its in-RAM summary into the sector's last page, which append never uses:

Offset  Size  Field
──────  ────  ───────────────────────────────────
  0      4    magic         "SDF2"
  4      1    version       2
  5      3    reserved      0
  8      4    sequence      u32 LE — the sector this describes
 12      4    entry_count   u32 LE — entries; a run counts once
 16      4    sample_count  u32 LE — readable samples; a run counts all of its own
 20      4    write_offset  u32 LE — byte after the last entry
 24      8    min_timestamp u64 LE
 32      8    max_timestamp u64 LE — newest *sample*, so a run's span end
 40      8    sensor_bloom  u64 LE
 48      4    footer_crc    CRC-32C over the preceding 48 bytes

This is what makes a queryable mount cost O(sector_count) instead of O(bytes stored) — one page read per sealed sector rather than a scan of every entry. It is never a correctness dependency: a footer that is missing, torn, or names a different sequence is ignored and the sector's body is scanned exactly as before, which is also what happens to the sector a crashed run left active. MountMode::Verify ignores footers entirely and re-reads the whole medium.

The reserved page costs PAGE_SIZE / ERASE_SIZE of raw capacity — 6.25% at 256 byte pages and 4 KB sectors. It gets its own page rather than sharing the header page so that a footer write torn by power loss can damage nothing but the footer.

RAM Usage

Component Size (RP2350 config)
Page buffer 256 B (1 page)
Sector index 512 × 48 B ≈ 24 KB
Scalars/state ~80 B
Total ~24 KB of 520 KB

Adding a Spooler costs one further page buffer (for cursor records) plus about 64 bytes.

Quick Start

Sync API

use sensor_db::SensorDb;

// RP2350: 256-byte pages, up to 512 sectors
let mut db: SensorDb<_, 256, 512> = SensorDb::new(
    flash,          // your NorFlash implementation
    0x0020_0000,    // base offset in flash
    512,            // sectors to use
    2,              // reserved headroom sectors
)?;

// First boot: format. Subsequent boots: mount.
db.format()?;
// — or —
db.mount()?;

Async API (Embassy)

use sensor_db::SensorDbAsync;

let mut db: SensorDbAsync<_, 256, 512> = SensorDbAsync::new(
    flash,          // your async NorFlash implementation
    0x0020_0000,
    512,
    2,
)?;

db.format().await?;

Write

let timestamp = get_time_us();
let temp: f32 = read_temperature();
db.append(SENSOR_TEMP, timestamp, &temp.to_le_bytes())?;

Typed Write (zero-copy)

use sensor_db::SensorData;
use sensor_db::zerocopy::{FromBytes, IntoBytes, Immutable, KnownLayout};

#[derive(FromBytes, IntoBytes, KnownLayout, Immutable, SensorData)]
#[sensor(id = 0x01, version = 1)]   // version defaults to 0; bump it when the struct changes
#[repr(C)]
struct Temperature { celsius: f32 }

db.append_sensor(timestamp, &Temperature { celsius: 22.5 })?;

SensorData needs the derive feature. Without it, write the impl out — it is all the derive expands to:

impl SensorData for Temperature {
    const SENSOR_ID: u16 = 0x01;
    const SCHEMA_VERSION: u8 = 1;
}

id and version take any constant expression, so the central ID module that actually prevents collisions works directly:

mod ids { pub const TEMPERATURE: u16 = 0x01; }

#[derive(FromBytes, IntoBytes, KnownLayout, Immutable, SensorData)]
#[sensor(id = ids::TEMPERATURE, version = 1)]
#[repr(C)]
struct Temperature { celsius: f32 }

Two things the derive deliberately does not do. It does not emit the zerocopy derives — a derive macro cannot add attributes to its own item, and zerocopy's derives expand to absolute ::zerocopy:: paths, so zerocopy has to be in your Cargo.toml either way; injecting them would make that requirement invisible and its absence inscrutable. And it does not check for padding, because IntoBytes already refuses to derive for a padded type and reports the exact number of wasted bytes:

error[E0277]: `Reading` has 3 total byte(s) of padding
   types with padding cannot implement `IntoBytes`

Import zerocopy's traits through sensor_db::zerocopy as above. SensorData is bounded on them, so zerocopy is a public dependency of this crate, and a version skew between your Cargo.toml and this one otherwise surfaces as the trait bound `Temperature: FromBytes` is not satisfied on a type that plainly derives it.

Checking the sensor ID space

Nothing stops two types from claiming the same sensor ID. What happens then depends on how they differ, and one case is undetectable:

The other type differs in Read reports
schema version SchemaMismatch
payload size SizeMismatch (or BufferTooSmall if the stored entry is wider)
neither nothing — the bytes are reinterpreted

A same-sized collision is silent by construction: the entry is intact, its CRC is correct, and Humidity { percent: 41.0 } reads back as Temperature { celsius: 41.0 }. Because the mistake is in what was written, it cannot be repaired afterwards.

sensor_registry! turns that into a build failure. List every type the firmware writes, next to where the IDs are defined:

use sensor_db::sensor_registry;

sensor_registry! {
    Temperature,
    Humidity,
    Pressure,
}
error[E0080]: evaluation panicked: sensor registry: `Temperature` and
`Humidity` share a SENSOR_ID at the same SCHEMA_VERSION. ...

Sharing an ID at different versions is schema evolution, not a collision, so the registry accepts it. The macro needs no feature and costs nothing at build time — it expands to an anonymous const block of assertions and defines nothing, so it can appear in any module, as often as you like. It only knows about the types you list: it cannot see across crates, and a type left off the list is not checked.

Adapting a driver's reading type

A sensor driver hands you its own struct, or a tuple of f32s. You cannot implement SensorData for it: from your crate both the trait and the type are foreign, so the orphan rule refuses — and the driver's type would not carry zerocopy's derives anyway.

Declare the record you want on flash instead, and convert at the point of sampling. That is one small struct per sensor, and it is the same shape for every driver, which is why this appears once here rather than once per crate:

use sensor_db::SensorData;
use sensor_db::zerocopy::{FromBytes, Immutable, IntoBytes, KnownLayout};

// The driver's type — 12 channels plus the settings they were taken at:
//   struct Reading { channels: [f32; 12], gain: Gain, integration_ms: f32 }

/// What actually goes on flash.
#[derive(FromBytes, IntoBytes, KnownLayout, Immutable, SensorData)]
#[sensor(id = ids::SPECTRUM, version = 1)]
#[repr(C)]
struct Spectrum {
    /// Raw ADC counts, unscaled.
    channels: [u16; 12],
    /// Gain code and integration time, so the counts can be interpreted later.
    gain: u8,
    integration_ms: u8,
}

impl From<&driver::Reading> for Spectrum {
    fn from(r: &driver::Reading) -> Self {
        Self {
            channels: core::array::from_fn(|i| r.channels[i] as u16),
            gain: r.gain as u8,
            integration_ms: r.integration_ms as u8,
        }
    }
}

db.append_sensor(now(), &Spectrum::from(&reading))?;

Three decisions in that struct are worth copying.

Fixed point, not f32. The sensor reported integers; the driver widened them for you. Storing the counts is exact, halves the payload — the channels are 24 bytes rather than 48 — and avoids soft-float on parts with no FPU. Where a scale is genuinely needed, put it in the field name: celsius_x100: i16 states its own units and covers −327.68 … 327.67 in two bytes.

Store what was measured, plus whatever is needed to interpret it. Gain and integration time are in the record because a calibration applied on the device is a number you can never un-apply. The host can always recompute; the device cannot recover what it discarded.

Order fields widest first. [u16; 12] followed by two u8s is 26 bytes with no padding. Drop one of the u8s and repr(C) pads the struct back to 26 — at which point IntoBytes refuses to derive, and says so exactly:

error[E0277]: `Spectrum` has 1 total byte(s) of padding
   types with padding cannot implement `IntoBytes`

That is the check working. Padding is uninitialised memory, and every byte this crate writes goes under a CRC.

examples/bme280.rs is this worked end to end against the real bme280 crate — driver conversation, the conversion above, a remount, and an assertion that every reading came back exactly as measured:

cargo run --example bme280

Read Latest

// Raw API
let mut buf = [0u8; 64];
if let Some((hdr, n)) = db.latest(SENSOR_TEMP, &mut buf)? {
    let temp = f32::from_le_bytes(buf[..4].try_into().unwrap());
}

// Typed API — zero-copy deserialization
if let Some((hdr, temp)) = db.latest_sensor::<Temperature>()? {
    println!("{}°C at ts={}", temp.celsius, hdr.timestamp);
}

Range Query

let count = db.query_range(
    Some(SENSOR_TEMP),       // or None for all sensors
    start_ts, end_ts,
    |hdr, payload| {
        // process each matching entry
        true  // return false to stop early
    },
)?;

SensorLogger (auto-timestamped)

use sensor_db::SensorLogger;

let mut logger = SensorLogger::new(db, || get_time_us());
logger.log(&Temperature { celsius: 22.5 })?;

// ...or over a tier, unchanged — the logger wraps any `LogSink`.
let mut logger = SensorLogger::new(tiered, || get_time_us());
logger.log(&Temperature { celsius: 22.5 })?;
logger.sink_mut().attach(card)?;

Writing against the capability, not the type

LogSink (append + sync) and LogSource (latest, oldest, count, query_range) are implemented by both SensorDb and TieredDb, with LogSinkAsync / LogSourceAsync for the async pair. The typed API lives in blanket-implemented extension traits over them, so it is defined once and reaches both:

use sensor_db::{LogSink, LogSinkExt};

fn record<L: LogSink>(log: &mut L, t: u64, c: f32) -> Result<(), Error> {
    log.append_sensor(t, &Temperature { celsius: c })?;
    log.sync()
}

Both core ports are object-safe, so &mut dyn LogSink works too. Reads take &mut self: SensorDb can answer through &self via its RefCell, but a TieredDb query has to stream the onboard log from the replication watermark.

Maintenance

// Flush write buffer to flash (for durability across power loss)
db.sync()?;

// Erase the 5 oldest sectors
db.compact(5, CompactScope::All)?;

// Check health
let stats = db.stats();
// stats.total_entries, stats.used_sectors, stats.free_sectors, ...

Power-Loss Safety

  • Two-phase commit: entries are written with flags=0xFF (uncommitted), then the same page is re-written with flags=0x00 (committed). NOR flash only clears bits (1→0), so the second write is always valid. On mount, uncommitted entries are discarded.
  • CRC-32C per entry (~1-in-4-billion collision rate). CRC includes the entry's offset within the sector to detect transplanted entries after partial erases.
  • Sector headers use a monotonic sequence number to reconstruct ring order.
  • The write buffer (one page) is the blast radius: at most one page of entries can be lost on unexpected power loss. Call sync() before critical checkpoints.
  • No in-place flash updates — fully compatible with strict NorFlash (no MultiwriteNorFlash required).

What it costs

Measured, not estimated. Every number below is counted from real NorFlash calls by MockFlash, and regenerated with:

cargo run --example flash_cost

They are operations, not seconds. A wall-clock figure from a desktop measures a Vec<u8>; flash operations transfer across parts. On the NOR flash this crate targets a sector erase is 100 ms or more, a 256-byte page program is around a millisecond, and a read is bus-bound — so multiply by your datasheet's numbers and you have the answer for your hardware.

Geometry throughout: 256-byte pages, 4 KB sectors, 40-byte payloads (60 bytes per entry, so 59 entries per sector).

One call

Operation Reads Bytes read Page writes Erases
append, buffered 0 0 0 0 the common case — the entry goes to the RAM page buffer
append that fills a page 0 0 2 0 the data pass, then the commit pass
append that seals a sector 0 0 4 1 flush, footer, erase, header — the one that costs 100 ms+
sync, buffer non-empty 0 0 2 0 the same two-phase page write
sync, buffer empty 0 0 0 0 free
count(None) 0 0 0 0 answered from the in-RAM index
latest, sensor absent 0 0 0 0 every sector rejected by its bloom filter
latest, sensor present 51 1.4 KiB 0 0 newest sector first, stopping at the first that has it
count(Some(_)) over 200 entries 447 12.5 KiB 0 0 CRC-validated, which is why it agrees with query_range
query_range, same 200 entries 447 12.5 KiB 0 0 exactly the same reads
compact(1, All) 0 0 0 1

So a sampling loop is free until it fills a page, and pays for an erase once every 59 samples. Nothing on the read path ever writes.

One burst, stored both ways

A run of n readings of T costs 20 + 11 + n×size_of::<T>() bytes against n × (20 + 1 + size_of::<T>()) appended one at a time. For a 4-byte reading:

Samples Bytes, one at a time Bytes, as a run Saving Page writes, singles Page writes, run
8 200 63 69% 2 2
16 400 95 77% 4 2
32 800 159 81% 8 2
56 1,400 255 82% 12 4

56 is the most a 256-byte page holds. Reads get cheaper in step — the samples share a header, a CRC and usually a page — and the API does not change, because a run expands transparently.

mount on a 2 MB region (512 × 4 KB)

Fill Mode Entries indexed Reads Bytes read Sector bodies scanned
¼ full Full 8,048 722 24.6 KiB 1
¼ full Verify 8,048 18,538 518 KiB 137
¼ full HeadersOnly — 586 18.2 KiB 1
½ full Full 16,096 900 30.8 KiB 1
½ full Verify 16,096 36,532 1018 KiB 273
½ full HeadersOnly — 628 18.1 KiB 1
full Full 30,187 1,122 42.2 KiB 1
full Verify 30,187 68,063 1.85 MiB 512
full HeadersOnly — 611 18.3 KiB 1

This is what the sector footer buys. A fully queryable mount of a brim-full 2 MB region reads 42 KiB instead of 1.85 MiB — 45× less — and grows with the sectors written rather than with the data in them: quadrupling the stored data takes Full from 24.6 to 42.2 KiB while Verify goes from 518 KiB to 1.85 MiB. HeadersOnly indexes nothing and so answers no queries; its cost is the floor, and Full is now close to it.

tests/cost.rs pins these — the per-call rows as exact equalities, the mount ratios with headroom — so the table cannot rot unnoticed.

Performance

  • Per-sector bloom filter: 64-bit Fibonacci-hashed sensor ID bloom filter skips sectors that definitely don't contain the target sensor. Near-zero false positives for ≤16 distinct sensor types.
  • Wear leveling via erase counts: find_next_sector() prefers erased sectors with lowest erase count, stored in SD05 sector headers and persisted across remount.
  • Sector footers: a sealed sector carries its own summary in its last page, so mount() rebuilds a fully queryable index in O(sector_count) — one 28-byte header plus one 52-byte footer per sector — instead of reading every byte of stored data. Only the sector the previous run left active is scanned. On the 2 MB RP2350 configuration that is ~512 short page reads per boot instead of 2 MB over QSPI. Falls back to a body scan wherever a footer is missing or fails to validate, so it is purely an optimisation. See Sector Footer.
  • MountMode::HeadersOnly: skips the footers too, reading 28 bytes per sector and nothing else. Now a smaller win than it used to be, and still the only mode whose cost is O(sector_count) unconditionally — it never falls back to a body scan. Queries return ReadsUnavailable rather than answering from a partial index.
  • MountMode::Verify: the opposite trade — re-reads and CRC-checks every stored entry at mount and reports corrupt_entries. Cost is proportional to the data stored. This is what Full did before footers existed.

Removable media

For a logger with no network link: samples are collected continuously, and a card is periodically pulled and carried to a computer. While the card is out, collection continues onboard; when it returns, the backlog lands on it.

use sensor_db::spool::{SpoolBudget, Spooler, TieredDb, WritePolicy};
use sensor_db::sector::LogId;

let log = LogId::new(0xC0FFEE).unwrap();

// The card reserves two sectors for the replication cursor; the rest is a
// normal SensorDb region.
let spooler = Spooler::new(Some(log), 0, SECTOR as u32)?;
let mut db = TieredDb::new(onboard, spooler, WritePolicy::CardFirst)?;

// Collect. Never fails because of the card, present or not.
db.append(TEMPERATURE, timestamp, &reading)?;
db.sync()?;

// Card goes in: the backlog drains, then samples go straight to it.
db.attach(card)?;
db.drain(SpoolBudget::default(), 64)?;

// Card comes out. Collection continues onboard.
let card = db.detach();
OnboardFirst CardFirst
Every sample written onboard, then copied to the card while it is in
Onboard wear/space while card is in full ~none
Copies of a sample two, until the ring reclaims one one
Power cut in steady state loses one onboard page loses one card page
Card yanked mid-write nothing to lose nothing lost — RAM buffer rescued

What it guarantees

  • No loss. The onboard watermark advances only after the card's copy is durable, so an interruption re-copies rather than skips. A yank is not a power loss — the processor keeps running — so samples still in the card's RAM page buffer are rescued into the onboard log.
  • Bounded duplication. A power cut or unclean removal replays at most the in-flight batch: SpoolBudget::max_entries entries. That value is the bound.
  • In practice, no duplication. The bound is the fallback. On reinsertion the interrupted batch's intent record says where on the card it began, so the card is read forward from there to find how much of it survived, and exactly that much is skipped on the source. The skip is checked against a fingerprint of both runs and abandoned if they disagree — every fallback duplicates rather than skips, because duplication is bounded and visible while a wrong skip is silent loss. Attached::recovered_entries reports what was found.
  • Exact de-duplication offline. Each commit record maps a region of the card to a source position range, so a reader spots a replayed batch by its overlapping range and keeps the later one. This works regardless of the clock — it needs no unique timestamps.
  • A card from another logger is refused, not overwritten. Every sector carries a log_id; mounting a foreign one fails with ForeignVolume before anything is written. Without this the "no valid sectors" path would erase a sector to bootstrap the log, destroying another device's data.

How the cursor works

Two reserved erase-sectors on the card, ping-ponged, one record per page-sized slot. A batch writes an intent before copying and a commit after the data is durable — two records rather than an in-place flag flip, because a commit carries values the intent did not have and NOR can only clear bits.

On reinsertion the resume point is the highest position any record justifies, counting an intent only for where its batch started. A torn commit is therefore recovered from the next intent, which is what bounds duplicates to a single batch.

That bound is then closed. An intent also records dest, the position on the card its first entry landed at, and nothing is appended after a batch except the next batch — which never started, or its intent would be the newest record. So reading the card forward from dest counts exactly what the interrupted batch delivered, and Spooler::reconcile skips that many source entries before copying resumes. Both runs are fingerprinted (length, plus the first and last (sensor_id, timestamp)) and the skip is abandoned unless they match, so a mismatch — or a source ring that has wrapped past those entries — falls back to replaying the batch rather than stepping over data no card ever received.

The batch counter is a u32, so it eventually wraps — 136 years at one batch per second, but the cursor is meant to outlive the device. Records are ordered by serial-number arithmetic rather than <, which is unambiguous here because a card's whole cursor region holds only a few hundred records and so spans a tiny window of the counter. Ordering them numerically would name the wrong sector active and, on the next wrap, erase the one holding the true resume point.

Telling the operator what the card is doing

Two things about a card are invisible from the write path, and neither is an error when it happens:

  • Erases. An erase is 100 ms or more on many NOR parts, against microseconds for a page write. SpoolProgress.volume_erased reports the sectors a replication pass erased on the card — its log rotating, plus the cursor region wrapping — so a caller with a deadline elsewhere can spool less often or with a smaller budget. Under CardFirst the same spike lands on append instead; SensorDb::sectors_erased on the card is the counter that sees those. It counts what this instance erased, unlike DbStats::total_erase_count, which is the wear history read back from the sector headers.
  • A card filling up. TieredDb::volume_fill gives used_sectors, free_sectors, total_sectors and entries, plus used_permille() and will_overwrite(). The last is free_sectors == 0: the final moment at which swapping the card still costs nothing. This matters most under CardFirst, where the card holds the only copy of everything written since it took over, so wrapping is real loss rather than the loss of a redundant copy. A ring overwriting its own tail is normal operation for a log and is never reported as an error, so this is the whole warning.

See examples/removable_card.rs for the full cycle, and WritePolicy's documentation for the costs of each mode.

Schema evolution

A SensorData type binds a struct to a sensor ID. Change the struct and every entry already on flash is still stored under that ID, and is no longer a T. A length check catches the easy half of that — a field added or removed — and misses the dangerous half: a change that keeps the size and alters the meaning (celsius becoming fahrenheit, two fields swapped, a scale factor changed). Those read back as plausible garbage.

So a typed payload carries its version in front of the value:

payload = [ schema_version : u8 ] [ T's bytes ]

To migrate, keep the old struct, give the new one the next version, and let both name the same sensor ID:

#[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct TemperatureV0 { celsius: f32 }
impl SensorData for TemperatureV0 {
    const SENSOR_ID: u16 = 0x01;
    const SCHEMA_VERSION: u8 = 0;
}

#[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Temperature { millikelvin: u32 }
impl SensorData for Temperature {
    const SENSOR_ID: u16 = 0x01;
    const SCHEMA_VERSION: u8 = 1;
}

query_sensor skips entries of another version, so each type reads its own era out of one log and a firmware update needs no flash migration pass. The single-entry reads instead report which era they found:

Result Means
SchemaMismatch { expected, found } Another era, named. Dispatch on it.
SizeMismatch { .. } This version, a different length — the struct changed and SCHEMA_VERSION did not.
BufferTooSmall The stored entry is longer than a T; its payload never lands, so its tag cannot be read. Try the largest candidate type first.

The tag lives in the payload rather than the entry header on purpose. The header is storage framing — what the scanner needs to walk the log — and two things follow from putting an application concept there instead. The only spare room is the flags byte, which both CRCs mask to 0xFF so a committed and an uncommitted entry hash alike, making it the one field nothing protects. And replication copies an entry as (sensor_id, timestamp, payload), so a header field would have to be threaded through read_from's callback and both spool drivers to reach the card. A payload tag is CRC-covered and rides along untouched.

Internal vs External Flash

The RP2350's internal QSPI flash shares the XIP bus with firmware. Flash operations briefly disable XIP (both cores stall). For high-frequency logging or latency-sensitive applications, consider an external SPI flash (e.g., W25Q64 on SPI1). The database API is identical — just swap the NorFlash implementation:

// Internal flash (embassy-rp)
type InternalDb<'a> = SensorDb<Flash<'a, FLASH, Async, FLASH_SIZE>, 256, 512>;

// External SPI flash (your driver)
type ExternalDb<S> = SensorDb<W25Q64<S>, 256, 1024>;  // 8MB = 1024 sectors

Sharing a chip

One flash part usually holds more than one thing. Partition<F> is a NorFlash view of a sub-range — pure address arithmetic over your driver, knowing nothing about which driver it is:

use sensor_db::{Partition, SensorDb};

// Firmware below, config in the middle, the log in the top 2 MB.
let region = Partition::new(flash, 0x0020_0000, 0x0020_0000)?;
let mut db: SensorDb<_, 256, 512> = SensorDb::new(region, 0, 512, 2)?;

The database now addresses the partition, not the chip: offset 0 is 0x0020_0000, capacity() reports 2 MB, and an access past the end is PartitionError::OutOfBounds rather than a write into whoever lives next door. Erase, write and read granularity pass through unchanged.

This is also how you run sensor-db alongside sequential-storage (which takes a Range<u32> for the same reason) or a firmware slot: give each a disjoint range and nothing needs to trust anything.

Reaching the rest of the chip

SensorDb owns the driver by value, so something has to broker access to the neighbours. Two methods, with different jobs:

with_region(base, len, |p| …) Flash access bounded to [base, base+len), handed to the closure as a Partition. A region intersecting the database's own sectors is refused with Error::RegionOverlap.
driver_mut() The driver itself, for capabilities that are not addresses: presence detection, wear counters, chip id, power modes. Unbounded — do not address flash through it.

The replication cursor uses the first: Spooler writes its records through with_region, so a cursor_base that overlaps the card's own database region now fails at attach instead of silently overwriting replicated entries.

The async driver has region(base, len) -> Partition<&mut F> instead of a closure, because a closure cannot return a future that borrows its own argument.

Sharing one log between tasks or cores

SensorDb and SensorDbAsync are Send + Sync whenever the driver is, so two Embassy tasks — or both RP2350 cores — can log to one database. Wrap it in a mutex and share it:

use embassy_sync::blocking_mutex::raw::CriticalSectionRawMutex;
use embassy_sync::mutex::Mutex;

type Db = SensorDbAsync<Flash<'static, FLASH, Async, FLASH_SIZE>, 256, 512>;
static DB: StaticCell<Mutex<CriticalSectionRawMutex, Db>> = StaticCell::new();

#[embassy_executor::task]
async fn sampler(db: &'static Mutex<CriticalSectionRawMutex, Db>) {
    loop {
        {
            let mut db = db.lock().await;
            db.append_sensor(Instant::now().as_micros(), &read_sensor()).await.ok();
        } // released before the wait — never hold it across a Timer
        Timer::after_millis(100).await;
    }
}

Four things are worth knowing before you build on that.

Mutex, not RwLock. An RwLock compiles and buys nothing. Reading flash goes through ReadNorFlash::read(&mut self), so every method — latest and count as much as append — needs &mut, which is the write half of the lock. An RwLock whose read half is never taken is a Mutex with extra steps.

Never hold the lock across a wait. An embedded executor is cooperative: a task holding the mutex across Timer::after(..).await blocks every other task for that whole duration. Take it, do the operation, drop it.

Put sync() in one place. It is normally two page writes — microseconds — but the one that happens to seal a sector also erases, and an erase is 100 ms or more on many NOR parts. Whoever is waiting on the mutex waits for that too. Calling it from a single task on a known schedule makes that latency predictable instead of arriving whenever a sampler happens to fill a page.

Durability is not visibility. Reads see the RAM page buffer as well as flash, so an un-synced append from another task is already queryable. sync() is only about surviving a power cut.

Two runnable examples, both asserting the properties above rather than merely printing:

cargo run --example multithread                    # threads, sync API
cargo run --example multithread_async --features async   # tasks, the Embassy shape

multithread_async.rs also carries a table mapping each piece of its desktop scaffolding to the RP2350 equivalent, so the task bodies transfer unchanged.

Features

Feature Description
std (default) Enables MockFlash for desktop testing
async Enables SensorDbAsync, SpoolerAsync and TieredDbAsync via embedded-storage-async 0.4 (for Embassy, etc.). The async ports and the code generic over them are available without it.
derive Enables #[derive(SensorData)]. Pulls in syn/quote, at build time only — nothing reaches the target.
defmt Enables defmt::Format impls for embedded debugging

Design Decisions

Decision Rationale
Entries can't span pages Simplifies recovery (each page is self-contained)
No in-place state updates Strict NorFlash compliance, no MultiwriteNorFlash needed
CRC-32C per entry ~1-in-4-billion detection rate, same algorithm as btrfs/ext4
Two-phase commit Power-loss atomic writes without MultiwriteNorFlash
Sector-level index only Keeps RAM bounded at MAX_SECTORS × 48 bytes, and keeps a query's flash reads proportional to the sectors the bloom filter admits rather than to the region — see What it costs
Circular allocation Natural FIFO retention; no external GC policy needed
Callback-based queries Avoids lifetime issues with flash borrows in iterators
Sync + async as parallel types Zero abstraction overhead; no generic async/sync trait gymnastics
Replication cursor on the card, not the device The replica is the only thing that knows what it received. Two cards can be rotated against one log with no local state — whichever is inserted says where it got to.
At-least-once replication, reconciled to exactly-once on reinsert Exactly-once during a batch needs the copy and the acknowledgement to be one atomic act, which two independent media cannot provide. So the protocol guarantees at-least-once with a one-batch bound, and closes it afterwards: the card is asked how much of the interrupted batch it kept, and only the remainder is copied. A reconciliation that cannot be verified is abandoned back to the bound.
A run is a distinct entry marker, not a payload flag The raw append path stores arbitrary bytes, so a payload flag misfires on any payload that happens to carry the pattern — inventing a time span for an entry that is not a run. An earlier attempt shipped exactly that and was reverted when the model-based tests caught it on a workload with no typed data in it. The marker byte is covered by the header CRC and only this crate writes it.
Runs expand at the storage layer, not in the typed API query_range, latest, oldest and count all speak samples, so a run is byte-for-byte indistinguishable from the same single appends and no reader has to opt in. Replication is the one exception: it copies entries, and a replica should keep the amortisation rather than unpack it.
Schema version tagged into the payload, not the entry header The header is storage framing; a schema version is application meaning that nothing in the crate reads. The only spare header room is the flags byte, which both CRCs mask so the commit flip does not change them — a version there would be unprotected. And replication copies (sensor_id, timestamp, payload), so a header field would have to be threaded through the spool drivers to survive the trip to the card.
Typed API defined over capability ports, not over SensorDb One definition serves a single log and a tier. It also has to be unoverridable — if an implementor could customise how a SensorData value becomes payload bytes, the same entry would read back differently depending on which log wrote it — so it lives in a blanket-implemented mixin rather than in the port.
Presence detection is a port, not a driver Removable is one method. Card-detect wiring is irreducibly board-specific, so it stays out of the crate.

Testing

# Run all tests (sync + async + derive)
cargo test --lib --tests --features async,derive

# Verify no_std compilation
cargo build --no-default-features

# Verify no_std + async
cargo build --features async --no-default-features

# Run examples
cargo run --example quickstart        # format, append, read, remount
cargo run --example power_loss        # tears a write at every phase
cargo run --example bursts            # run encoding vs single appends
cargo run --example schema_evolution  # a firmware update mid-deployment
cargo run --example removable_card    # collect/insert/drain/pull/yank
cargo run --example multithread
cargo run --example multithread_async --features async

# Regenerate the cost tables above
cargo run --example flash_cost

534 tests: 101 unit + 126 integration + 12 power-loss + 5 power-loss fuzz + 147 removable-media (spool, identity, tiered, card_first, tiered_async) + 3 cursor-protocol fuzz + 6 model-based + 11 capability-port + 19 schema-evolution + 27 partition + 20 sector-footer + 15 flash-op budget + 24 run-encoding + 13 derive + 5 registry.

The examples are run in CI as well as built. Each ends by asserting the claim it demonstrates, so power_loss is a check that no synced reading is ever lost to a torn write, and bursts a check that a run and the same single appends are observationally identical.

Power-loss and removal are tested deterministically, not by hope: MockFlash injects a clean or partial write failure at a chosen write index — or a torn erase, leaving a blank head and a stale tail as a real interrupted erase does — and RemovableFlash makes a medium vanish, failing reads as well as writes, either on command or after n further writes. The async driver is checked against the blocking one by running the same scenario through both and comparing the resulting media byte for byte, cursor records included.

The two *_fuzz suites stop choosing where to interrupt. power_loss_fuzz replays a workload once per (fault site × byte offset) pair — every write and every erase it performs, torn after every possible number of bytes — and after each one asserts that the log mounts, that the survivors are a contiguous run of what was appended, that nothing a successful sync acknowledged is missing, and that the log still accepts writes which then survive another cycle. spool_fuzz does the same to the replication cursor: every card write torn at every byte offset, every yank point, and repeated interruption across insert/remove cycles, asserting the whole series reaches the card exactly once. Between them that is some ten thousand recoveries, in under three seconds.

tests/cost.rs pins the numbers in What it costs, so a published figure cannot rot unnoticed: the per-call costs as exact equalities, the mount aggregates as ratios with headroom.

tests/run.rs is almost entirely differential, because the claim it checks is an equivalence: the same series is stored once as a run and once as single appends, and the two logs are compared through every read path there is — count, latest, oldest, the raw and typed queries, and a set of windows that cut a run in half at either end. Anything the two disagree about is a defect by construction, whichever is right. Runs are in the model-based workload too, where they are expanded into ordinary entries in the Vec oracle, which is the same claim stated a second way.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.