Expand description
Store-and-forward spooling from an onboard log to a removable volume.
The problem this solves: a device with no network link collects sensor data continuously, and the only way to get the data off it is to pull a card and carry it to a computer. While the card is out, samples must keep landing somewhere; when it comes back, the backlog must land on it.
§Layering
SensorDb single-device append-only log
└─ Spooler owns the card, copies onboard → card, owns the on-card cursor
└─ TieredDb routes append() between card and onboardEach layer is independently useful, and a device with no card slot uses none
of them — plain SensorDb is unchanged and costs nothing.
§Module organisation
cursor— the on-medium record that says how far a copy got. Pure.policy— batch planning, resume computation, and the spooler’s own state machine. Pure.tier— which mediumappendwrites to, and when that moves. Pure.port— capability traits the spooler needs from drivers. Pure.blocking— the blocking driver.
The pure modules carry no feature gates and no I/O dependencies; only the
driver modules are gated. Between them they hold every decision the
spool layer makes, so each driver contributes only its reads and writes:
the two are unrelated trait families, and tests/tiered_async.rs compares
the media they produce byte for byte, which means a decision written twice
is one that suite calls correct by construction.
§Guarantees
- No loss. The source’s watermark advances only after the replica’s copy is durable, so an interruption re-copies rather than skips.
- Bounded duplication. A power cut or unclean removal replays at most the
in-flight batch —
SpoolBudget::max_entriesentries. - In practice, no duplication. That bound is the fallback. On reinsert, the interrupted batch’s intent record says where on the replica it began, so the replica itself is read to find how much of it survived, and exactly that much is skipped on the source. The skip is verified against a fingerprint of both runs and abandoned if they disagree, so a wrong reconciliation costs duplicates rather than data.
- Exact de-duplication is possible offline. Each commit record maps a region of the replica to a source position range, so a reader can spot a replayed batch by its overlapping range and keep the later one.
§Example
use sensor_db::db::DbConfig;
use sensor_db::mock::MockFlash;
use sensor_db::sector::LogId;
use sensor_db::spool::{SpoolBudget, Spooler, TieredDb, WritePolicy};
use sensor_db::SensorDb;
const PAGE: usize = 256;
const SECTOR: usize = 4096;
const SECTORS: usize = 8;
let log = LogId::new(0xC0FFEE).unwrap();
let config = DbConfig::new(0, SECTORS as u16, 1)?.with_log_id(log);
// Onboard log.
let mut onboard: SensorDb<_, PAGE, SECTORS> =
SensorDb::with_config(MockFlash::<PAGE, SECTOR>::new(SECTORS * SECTOR), config)?;
onboard.format()?;
// The card: two sectors of cursor region, then the replica database.
let cursor_base = 0;
let card_config = DbConfig::new(2 * SECTOR as u32, (SECTORS - 2) as u16, 1)?
.with_log_id(log);
let mut card: SensorDb<_, PAGE, SECTORS> =
SensorDb::with_config(MockFlash::<PAGE, SECTOR>::new(SECTORS * SECTOR), card_config)?;
card.format()?;
let spooler = Spooler::new(Some(log), cursor_base, SECTOR as u32)?;
let mut db = TieredDb::new(onboard, spooler, WritePolicy::OnboardFirst)?;
// Collect while the card is out.
for i in 0..100 {
db.append(1, i, &[i as u8])?;
}
db.sync()?;
// Card goes in: the backlog drains onto it.
db.attach(card)?;
let progress = db.drain(SpoolBudget::default(), 64)?.unwrap();
assert_eq!(progress.entries_copied, 100);
assert!(progress.caught_up);Re-exports§
pub use blocking::Spooler;pub use blocking::TieredDb;pub use cursor::CursorKind;pub use cursor::CursorRegion;pub use cursor::SpoolCursorWire;pub use policy::Attached;pub use policy::RunFingerprint;pub use policy::SpoolBudget;pub use policy::SpoolProgress;pub use policy::VolumeFill;pub use port::Removable;pub use tier::WriteMode;pub use tier::WritePolicy;pub use asynch::SpoolerAsync;asyncpub use asynch::TieredDbAsync;async
Modules§
- asynch
async - Async store-and-forward drivers:
SpoolerAsyncandTieredDbAsync. - blocking
- Blocking store-and-forward drivers:
SpoolerandTieredDb. - cursor
- On-medium replication cursor: the record that says how far a copy got.
- policy
- Replication policy: what to copy next, and where the cursor record goes.
- port
- Ports: capabilities the spooler needs from the outside world.
- tier
- Routing a tiered log: which medium
appendwrites to, and when that moves.