Skip to main content

Module spool

Module spool 

Source
Expand description

Store-and-forward spooling from an onboard log to a removable volume.

The problem this solves: a device with no network link collects sensor data continuously, and the only way to get the data off it is to pull a card and carry it to a computer. While the card is out, samples must keep landing somewhere; when it comes back, the backlog must land on it.

§Layering

SensorDb        single-device append-only log
  └─ Spooler    owns the card, copies onboard → card, owns the on-card cursor
       └─ TieredDb   routes append() between card and onboard

Each layer is independently useful, and a device with no card slot uses none of them — plain SensorDb is unchanged and costs nothing.

§Module organisation

  • cursor — the on-medium record that says how far a copy got. Pure.
  • policy — batch planning, resume computation, and the spooler’s own state machine. Pure.
  • tier — which medium append writes to, and when that moves. Pure.
  • port — capability traits the spooler needs from drivers. Pure.
  • blocking — the blocking driver.

The pure modules carry no feature gates and no I/O dependencies; only the driver modules are gated. Between them they hold every decision the spool layer makes, so each driver contributes only its reads and writes: the two are unrelated trait families, and tests/tiered_async.rs compares the media they produce byte for byte, which means a decision written twice is one that suite calls correct by construction.

§Guarantees

  • No loss. The source’s watermark advances only after the replica’s copy is durable, so an interruption re-copies rather than skips.
  • Bounded duplication. A power cut or unclean removal replays at most the in-flight batch — SpoolBudget::max_entries entries.
  • In practice, no duplication. That bound is the fallback. On reinsert, the interrupted batch’s intent record says where on the replica it began, so the replica itself is read to find how much of it survived, and exactly that much is skipped on the source. The skip is verified against a fingerprint of both runs and abandoned if they disagree, so a wrong reconciliation costs duplicates rather than data.
  • Exact de-duplication is possible offline. Each commit record maps a region of the replica to a source position range, so a reader can spot a replayed batch by its overlapping range and keep the later one.

§Example

use sensor_db::db::DbConfig;
use sensor_db::mock::MockFlash;
use sensor_db::sector::LogId;
use sensor_db::spool::{SpoolBudget, Spooler, TieredDb, WritePolicy};
use sensor_db::SensorDb;

const PAGE: usize = 256;
const SECTOR: usize = 4096;
const SECTORS: usize = 8;

let log = LogId::new(0xC0FFEE).unwrap();
let config = DbConfig::new(0, SECTORS as u16, 1)?.with_log_id(log);

// Onboard log.
let mut onboard: SensorDb<_, PAGE, SECTORS> =
    SensorDb::with_config(MockFlash::<PAGE, SECTOR>::new(SECTORS * SECTOR), config)?;
onboard.format()?;

// The card: two sectors of cursor region, then the replica database.
let cursor_base = 0;
let card_config = DbConfig::new(2 * SECTOR as u32, (SECTORS - 2) as u16, 1)?
    .with_log_id(log);
let mut card: SensorDb<_, PAGE, SECTORS> =
    SensorDb::with_config(MockFlash::<PAGE, SECTOR>::new(SECTORS * SECTOR), card_config)?;
card.format()?;

let spooler = Spooler::new(Some(log), cursor_base, SECTOR as u32)?;
let mut db = TieredDb::new(onboard, spooler, WritePolicy::OnboardFirst)?;

// Collect while the card is out.
for i in 0..100 {
    db.append(1, i, &[i as u8])?;
}
db.sync()?;

// Card goes in: the backlog drains onto it.
db.attach(card)?;
let progress = db.drain(SpoolBudget::default(), 64)?.unwrap();
assert_eq!(progress.entries_copied, 100);
assert!(progress.caught_up);

Re-exports§

pub use blocking::Spooler;
pub use blocking::TieredDb;
pub use cursor::CursorKind;
pub use cursor::CursorRegion;
pub use cursor::SpoolCursorWire;
pub use policy::Attached;
pub use policy::RunFingerprint;
pub use policy::SpoolBudget;
pub use policy::SpoolProgress;
pub use policy::VolumeFill;
pub use port::Removable;
pub use tier::WriteMode;
pub use tier::WritePolicy;
pub use asynch::SpoolerAsync;async
pub use asynch::TieredDbAsync;async

Modules§

asynchasync
Async store-and-forward drivers: SpoolerAsync and TieredDbAsync.
blocking
Blocking store-and-forward drivers: Spooler and TieredDb.
cursor
On-medium replication cursor: the record that says how far a copy got.
policy
Replication policy: what to copy next, and where the cursor record goes.
port
Ports: capabilities the spooler needs from the outside world.
tier
Routing a tiered log: which medium append writes to, and when that moves.