Module observability 

Production-friendly observability layer (replaces legacy Timeline + Invariant). Trace-based observability for moonpool simulations.

Processes and workloads emit ordinary tracing events — the exact same instrumentation production observability consumes (fmt, OpenTelemetry, Loki, …). In simulation, SimulationLayer captures those events into a timeline and registered Invariants cross-validate them after every simulation step. The point: anomalies like dual leaders or metastable failures are detected in simulation with the same signals you would use to find them in production — traces.

Emission convention

Emit a plain tracing event with a constant message (the event name) and structured fields. No special markers, no derives:

tracing::info!(target: "raft", term, leader = %my_ip, "leader_elected");

• Use % (Display) for strings and IPs — ? (Debug) on a String includes quotes.
• The message must be a constant name ("leader_elected"), not an interpolated sentence: events are grouped and queried by name.

An event is captured iff it is INFO or more severe, carries a non-empty message, and fires inside a process/workload span (the orchestrator wraps every actor task in a span carrying its ip, which becomes the event’s source). In production, where no SimulationLayer is installed, the same emission flows to whatever subscriber is configured.

Querying

Invariants receive a TraceQuery and read events by name, extracting typed fields by key — the same way you would query a production trace store:

fn observe(&self, q: &dyn TraceQuery, _sim_time_ms: u64) {
    for e in q.since("leader_elected", &self.cursor) {
        let term = e.u64("term");
        let leader = e.str("leader");
        // assert one leader per term...
    }
}

Sim-injected faults (partitions, kills, storage corruption) are recorded by the runner under the crate::SIM_FAULT_EVENT_NAME name with a kind field and source = "sim", so invariants can correlate application anomalies with infrastructure faults in one timeline.

Re-exports

pub use event::FieldValue;

pub use event::TraceEvent;

pub use fmt::Clock;

pub use fmt::SimTime;

pub use init::init_sim_tracing;

pub use invariant::Invariant;

pub use invariant::invariant_fn;

pub use layer::InstallGuard;

pub use layer::SimulationLayer;

pub use layer::SimulationLayerHandle;

pub use query::TraceQuery;

Modules

event

Captured trace events and field values.

fmt

tracing_subscriber::fmt integration that uses simulation time as the timestamp prefix.

init

Convenience initializer for example/sim binaries.

invariant

Cross-process invariant trait checked after every simulation step.

layer

Custom tracing layer that captures plain trace events for invariants.

query

Read-side trait used by invariants to inspect captured trace events.