Skip to main content

Crate bitrep

Crate bitrep 

Source
Expand description

§bitrep — Any order. Any hardware. Same bits.

Order-invariant, bit-identical floating-point reductions.

Add floats in any order, on any number of threads, across any shard split, on any architecture — the result is the exactly rounded sum, and its bytes are identical everywhere. fp64 fixes your decisions; it can’t fix your hashes. This crate fixes your hashes.

§How

A SumF64 is a fixed-point superaccumulator (a 2176-bit two’s-complement integer, in units of 2⁻¹⁰⁷⁴) wide enough to hold every finite f64 exactly, with headroom for 2⁶³ additions. Integer addition is associative and commutative, so the accumulated state — not just the rounded result — is independent of insertion order by construction. One rounding happens at the very end, and it is correct rounding (round-to-nearest, ties-to-even).

This is the classic long-accumulator idea (Kulisch; see also Neal’s superaccumulators, arXiv:1505.05571, and Demmel–Nguyen reproducible summation). The primitives are textbook; what this crate packages is the distributed contract: accumulators are mergeable and serializable, so shards computed on different machines combine — in any order — into the same bytes.

§Example

use bitrep::SumF64;

let data = [0.5_f64, 1e100, -1e100, 0.25, 0.125, -0.875, 1e-300];

// Sequential, reversed, and sharded-then-merged: identical state, identical bits.
let a: SumF64 = data.iter().copied().collect();

let mut b = SumF64::new();
for x in data.iter().rev() { b.add(*x); }

let (left, right) = data.split_at(3);
let mut c: SumF64 = left.iter().copied().collect();
c.merge(&right.iter().copied().collect::<SumF64>());

assert_eq!(a.to_bytes(), b.to_bytes());
assert_eq!(a.to_bytes(), c.to_bytes());
assert_eq!(a.value().to_bits(), b.value().to_bits());
// And the value is the exactly rounded sum (naive summation is not):
assert_eq!(a.value(), 1e-300);

§What becomes possible

Each of these was previously blocked by one missing property — order-proof float addition with a mergeable, canonically-encoded state:

  • Float counter CRDTs. Counter CRDTs (G-Counter, PN-Counter) have been integer-only for fifteen years because CRDT merge must commute and associate, and float addition does neither. SumF64::merge restores both, exactly — the standard counter recipe now works for floats, with convergence provable by hash instead of epsilon. The construction’s convergence laws (join semilattice, delivery-schedule invariance, exact reads) are machine-checked in proofs/FloatGCounter.lean; the README’s CRDT section gives the recipe.
  • Floats in replicated state machines. Deterministic-simulation-testing shops ban floats in replicated state because reduction order differs across replicas and the states drift. Aggregates routed through an accumulator produce identical bytes on every replica.
  • Retry-immune distributed aggregation. Parallel frameworks sum partitions in whatever order execution happens to deliver, so the same job can return different answers run to run. Merge order stops mattering: any merge tree, any retry, any straggler — same bytes, exactly rounded.
  • Numeric outputs you can sign or content-address. “Recompute this anywhere and the hash matches” is the property that makes signatures, receipts and content-addressing meaningful for float pipelines — and it holds across languages, via the canonical encoding (FORMAT.md).

§What this costs (honest numbers)

Exactness is not free: expect roughly an order of magnitude over a naive scalar loop for random data (see benches/, run them on your hardware). Use it where bit-identity or exactness matters — replicated state, signed/hashed outputs, cross-machine aggregation, ill-conditioned sums — not in your inner render loop.

§Scope and named limits

  • SumF64 / SumF32: exact, order-invariant sums. no_std compatible.
  • FastSumF64: a high-throughput streaming front-end (Neal’s small-accumulator technique) that finishes into the same canonical SumF64 bytes — differentially verified against the direct path.
  • DotF64 (feature std): exact dot products via FMA two-products. Named limit: each partial product a*b must not overflow, and must not fall in the range where FMA two-products lose exactness (|a·b| < ~2⁻⁹⁶⁹); see DotF64 docs. Inputs outside that domain are detected and reported, never silently wrong.
  • MomentsF64 / Moments4F64 / CovF64 (feature stats): convergent statistics — mergeable, order-invariant moment states with exactly rounded reads (mean, variance, kurtosis, covariance, regression slope/intercept, R²), derived from the exact integer state with a single final rounding.
  • NaN/±∞ are tracked as flags (any NaN, or +∞ and −∞ together, yields NaN; a single infinity sign is preserved). An exactly-zero sum returns +0.0 (canonical zero: -0.0 inputs are sign-preserving in IEEE addition only for empty-ish cases; a canonical result keeps bytes stable).

§Non-goals

Reproducing your existing float pipeline bit-for-bit (that depends on your kernels’ order); this crate replaces order-sensitive reductions with order-free ones. It is also not a general bignum: it holds sums of floats, nothing else.

Structs§

ConvergentMap
A keyed family of mergeable states: GROUP BY key for the convergent world. Merging merges per key; encoding is canonical (keys sorted, from the BTreeMap).
CovF64
Exact, mergeable bivariate state: covariance, correlation, and simple least-squares regression y ≈ intercept + slope·x — with exactly rounded covariance, slope, intercept and r_squared.
CovMatrixF64
Exact, mergeable d-dimensional second-moment state: covariance matrices bit-identical across any sharding, and deterministic multiple regression.
Deltas
Delta-state transport for additive states: keep a full state and a pending delta; ship only the delta since the last sync. The receiver simply merges the delta into its copy — correct because merge is additive and associative (delta-state CRDTs, Almeida–Shoker–Baquero).
DotF64
An exact, order-invariant, mergeable dot product of f64 pairs.
ExtremaF64
Exact, mergeable running minimum and maximum (plus count).
FastSumF64
A fast streaming accumulator that finishes into a canonical SumF64.
HistogramF64
An exact, mergeable fixed-bucket histogram.
Moments4F64
Exact, mergeable moments through the 4th: adds skewness and kurtosis.
MomentsF64
Exact, mergeable first and second moments: mean, variance, stddev.
PnMomentsF64
Moments with exact retraction: add(x) and remove(x), PN-counter style (two grow-only states; the reads are computed on their exact difference). Inserting then removing a sample returns the derived statistics to byte-identical values — the primitive incremental view maintenance needs.
Replicated
The per-replica CRDT wrapper: replica id → that replica’s own state, joined per entry by highest count wins.
SumF32
An exact, order-invariant, mergeable sum of f32 values.
SumF64
An exact, order-invariant, mergeable sum of f64 values.
WeightedMomentsF64
Exact, mergeable weighted moments: weighted mean and variance with exactly rounded reads.

Enums§

DotError
The dot product could not be computed exactly (see DotF64 docs).
StatsError
Why a statistic could not be produced. Mirrors crate::DotError’s philosophy: degraded meaning is reported, never silently returned.

Traits§

Mergeable
A mergeable, canonically-encodable accumulator state.

Functions§

dot
Convenience: the exactly rounded dot product of two slices.
state_hash
The SHA-256 of a state’s canonical encoding.