Crate cram_codec 

Error-bounded compression of scientific arrays with a bit-accounted quantized tensor train (QTT) codec.

A grid of N samples is tensorized into a d-way tensor with mode sizes of mostly 2 (tensorize), decomposed by TT-SVD in the canonical left-orthogonal gauge (tt), and coded by codec: per-column equal-slope quantization with error feedback (each core is refit against the already-quantized prefix), ranks picked by rate–distortion rather than a separate tolerance, and static rANS over transmitted frequency tables (rans). The stream is self-describing and every byte counts toward the reported rate — header, shape, ranks, step exponents, tables, payload, and (in max-abs mode) the sparse residual-correction stream.

use cram_codec::{decode, encode_f32, Config, Decoded, ErrorBound};
let x: Vec<f32> = (0..4096).map(|i| (i as f32 * 0.01).sin()).collect();
let (bytes, stats) = encode_f32(&x, &[64, 64], &Config::new(ErrorBound::MaxAbs(1e-3)));
let Decoded::F32 { data, .. } = decode(&bytes).unwrap() else { unreachable!() };
assert!(x.iter().zip(&data).all(|(a, b)| (a - b).abs() <= 1e-3));
assert_eq!(stats.bytes, bytes.len());

Other codecs — wavelet (wavcodec), Tucker (tucker), interpolation (interp) and the brick-parallel GPU layouts (gpuwav, gpuwav3) — share the ErrorBound vocabulary, and decode_any decodes any of them by magic.

Features and targets

• Default features are empty: CPU encode and decode only.
• gpu adds gpudec::GpuDecoder, a wgpu decoder for tc-gpu / TCG3 streams whose output is bit-identical to the CPU decoder.
• The crate builds for wasm32-unknown-unknown; the wasm-bindgen browser front end is the (unpublished) cram-codec-wasm crate in the tang repository.

Numerical notes

Truncated SVDs use the Gram matrix of the smaller unfolding side with tang_la::SymmetricEigen and tang_la GEMMs (row-major, NEON on aarch64), rather than tang_la::Svd (one-sided Jacobi), which is far too slow on 2 × 2^23 unfoldings. Squaring the spectrum resolves singular values down to roughly 1e-7·σ₁ in f64 — finer than any step the codec uses at PSNR ≤ ~120 dB. Encoding and decoding are single threaded and deterministic: the same input yields the same bytes, and decoding the same bytes yields the same array, run to run.

Re-exports

pub use codec::decode;

pub use codec::encode_f32;

pub use codec::encode_f64;

pub use codec::Config;

pub use codec::Decoded;

pub use codec::Dtype;

pub use codec::EncodeStats;

pub use codec::ErrorBound;

pub use tensorize::DigitOrder;

pub use tensorize::Layout;

pub use tensorize::PadPolicy;

pub use tt::tt_svd;

pub use tt::TensorTrain;

pub use tt::Truncation;

Modules

abac

Adaptive binary arithmetic coding — integer only, bit-exact everywhere.

bitio

Byte- and bit-level serialization primitives.

codec

The bit-accounted QTT codec.

coefcode

Context-modelled coding of quantized wavelet coefficients.

gpuwav

tc-gpu (stream magic TCG1): a GPU-decodable, random-access variant of the wavelet codec whose decode is integer only, so a CPU and a GPU decoder produce the same bytes.

gpuwav3

tc-gpu v3 (stream magic TCG3): one global integer CDF 9/7 transform over the whole field, with the coefficients entropy-coded in independently decodable code-blocks (JPEG2000-style).

img

Overfitted image codec (Phase 2, Track D): Cool-chic 5 style decoder in exact 32-bit integer arithmetic, so the CPU reference and the GPU kernels ([crate::imggpu]) decode the same bytes to the same pixels.

interp

SZ3-style multilevel interpolation codec for pointwise (max-abs) bounds, and for RMSE / PSNR targets through rate control of the bound (encode_rmse_f32).

irdec

The TCG3 rANS lane decoder as tang-expr integer graphs.

kvcodec

KV-cache codec: append-friendly, error-bounded coding of attention K/V.

kvgpu

Host side of the GPU KV decoder: packs GPU-layout TKV streams (crate::kvcodec, gpu = true) into the flat tables the device kernel reads (tang-compute tkv_decode, MSL source in tang-compute/src/kernels/kvdec_msl.rs). No GPU dependency here.

learned

Learned coefficient models: a small MLP predicts each residual’s distribution — a two-component discretized logistic mixture — from the same causal context features the hand model uses (Feat: neighbour and parent magnitudes, neighbour signs, prediction magnitude, band class and orientation).

linalg

Small dense helpers on row-major f64 buffers.

lmodel3

Learned probabilities for TCG3 code-blocks, in 32-bit integer arithmetic end to end, so a WGSL kernel (no i64) reproduces every probability.

lorenzo

Lorenzo-predictor codec for pointwise (max-abs) bounds, and for RMSE / PSNR targets through rate control of the bound.

rans

Deterministic static rANS over a hybrid-integer alphabet.

shipdec

Streams that carry their own decoder (TCS1).

tensorize

Quantized tensorization: an n₁×…×n_D grid → a d-way tensor with tiny mode sizes (mostly 2).

tt

Tensor trains: TT-SVD with error-budget / Gavish–Donoho truncation, and full contraction.

tucker

Tucker (HOSVD) codec, TTHRESH-style, on the shared quantizer / context coder / range coder stack.

wavcodec

The wavelet codec: CDF 9/7 pyramid → deadzone quantizer → context model → adaptive binary range coding, with a SPERR-style outlier stream for pointwise bounds and an integer 5/3 lossless mode.

wavelet

Separable multi-level 3D wavelet transforms by lifting.

Structs

AutoReport

What the portfolio encoder did.

Enums

Candidate

A portfolio candidate.

Constants

QTT_PROBE_CAP

QTT probe rank cap (qtt_probe with stride 2): full QTT runs only when the probe’s ranks stay below it.

TAG_INTERP

Portfolio tag: interpolation stream follows.

TAG_LORENZO

Portfolio tag: Lorenzo stream follows.

TAG_QTT

Portfolio tag: QTT stream follows.

TAG_TUCKER

Portfolio tag: Tucker stream follows.

TAG_WAVELET

Portfolio tag: wavelet stream follows.

Functions

center_crop

Centered crop with every axis of length ≥ 16 halved (1D: the middle eighth), and its shape.

decimate

Every-stride-th sample along each axis longer than 4·stride (row-major), with the new shape.

decode_any

Decode any cram-codec stream (QTT, wavelet, Tucker, interpolation or Lorenzo), by magic; a leading portfolio tag byte is skipped.

encode_auto_f32

Automatic codec choice: encode_portfolio_f32 with the pre-screen on. Returns the tagged stream and the chosen candidate.

encode_candidate_f32

Encode with one candidate (no tag byte).

encode_candidate_fast_f32

Encode with one candidate’s fast variant (no tag byte): step from a formula, one interpolation preset, the configured (pretrained or hand) model and never an in-stream overfit. QTT has no fast variant.

encode_fast_f32

Fast encode (tc --fast): the single candidate fast_pick names, coded with encode_candidate_fast_f32. Same tagged stream format as encode_auto_f32.

encode_portfolio_f32

Portfolio encode: encode with every candidate the pre-screen keeps (all applicable ones when screen is false) and keep the smallest stream. The output is one tag byte (counted in the rate) followed by the stream.

fast_pick

Fast mode’s single candidate: each fast variant that applies (max-abs: wavelet, interpolation, Lorenzo, QTT residual-only; PSNR/RMSE: wavelet, plus Tucker on 3D) is trial-encoded on center_crop (an eighth of the field) at the same absolute bound, and the smallest stream names the pick.

prescreen

Which candidates the pre-screen keeps for a bound, before any full encode.

qtt_probe

QTT rate probe: row-major QTT on a stride-decimated copy at the same absolute bound (PSNR is turned into the full field’s RMSE), ranks capped at cap. Returns (probe bytes, max rank, seconds).