1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
//! The `DataArray` implementation for data buffer manipulation used for, amongst other things,
//! mapping numeric data to and from optionally compressed little endian base-64 byte arrays with
//! as few data copies as possible, algebraically.
//!
//! Also covers the `ArrayType` enum that covers the standardized array flavors telling us which
//! dimension an array describes, `BinaryArrayMap` collections of (`ArrayType`, `DataArray`), and
//! traits for converting to and from `BinaryArrayMap` based upon the set of available `ArrayType`.
//! This includes a generalization for `BinaryArrayMap3D` when ion mobility is available.
//!
//! ## Compression
//!
//! `mzdata` supports several kinds of data compression methods with tradeoffs
//!
//! ### Lossless methods
//! - `zlib` compression [`BinaryCompressionType::Zlib`]: A well known, widely available general-purpose compressor codec with many implementations,
//! with varying speed and size tradeoffs taken from [`flate2`].
//! - `zstd` compression [`BinaryCompressionType::Zstd`]: A newer general-purpose codec that is not widely used in mzML but very popular in other applications.
//! Much faster and often better compression compared to `zlib`.
//! Requires feature `zstd`
//! - `byte-shuffled zstd` compression [`BinaryCompressionType::ShuffleZstd`]: This first applies a byte shuffle transform to the data to compress
//! the `n`th byte of each value contiguously, and then apply `zstd`. This makes sorted data compress more effectively.
//! Requires feature `zstd`
//! - `dictionary encoded zstd` [`BinaryCompressionType::ZstdDict`]: Another elaboration of `byte-shuffled zstd` where the values are first dictionary-encoded
//! so that repeated values are denoted by indices rather than the full size value itself. The dictionary is sorted and then both the dictionary and the
//! indices are byte shuffle encoded and then `zstd` compressed. This is even more effective than previous codecs for data with repeated, semi-sorted values
//! like m/z and ion mobility.
//! Requires feature `zstd`
//!
//! ### Lossy methods
//! - `numpress-linear` compression [`BinaryCompressionType::NumpressLinear`]: A combination simple linear model fit and byte packing codec that stores the model
//! coefficients as float64 and the residuals as unsigned 16 or 32-bit integers. With an optimal fixed point, this will produce a median loss of accuracy below
//! of about 2e-8 absolute units, but may vary widely from dataset to dataset. This codec **MUST** be used with sorted data only, and is only suitable for
//! float64 data. It may be combined with `zlib` [`BinaryCompressionType::NumpressLinearZlib`] or `zstd` [`BinaryCompressionType::NumpressLinearZstd`] for even
//! greater space savings. See details at <https://doi.org/10.1074/mcp.O114.037879>. /// Requires feature `numpress`
//! - `numpress-slof` compression [`BinaryCompressionType::NumpressSLOF`]: A lossy numerical approximation that log transforms and scales values to make them very
//! close to 16-bit integers. This codec is lossier the larger the value is, and is only appropriate for count data like intensities. It can also combine with
//! `zlib` [`BinaryCompressionType::NumpressSLOFZlib`] and `zstd` [`BinaryCompressionType::NumpressSLOFZstd`]. See details at <https://doi.org/10.1074/mcp.O114.037879>
//! Requires feature `numpress`
//! - `numpress-pic` compression [`BinaryCompressionType::NumpressPIC`]: This simply rounds a floating point value and then packs it
//! into a truncated integer. May be suitable for intensity values, but not consistently better than [`BinaryCompressionType::NumpressSLOF`].
//! See details at <https://doi.org/10.1074/mcp.O114.037879>
//! Requires feature `numpress`
//! - Linear prediction compression [`BinaryCompressionType::LinearPrediction`]: The same concept as [`BinaryCompressionType::NumpressLinear`] without the truncation
//! All of the same concerns apply though it loses less precision. This does not perform any form of compression. It was uses as part of mzMLb which applies
//! container-level compression. Only suitable for float64 data.
//! - Delta encoding [`BinaryCompressionType::DeltaPrediction`]: This computes first order difference of a sorted array of values and stores the difference
//! and the starting point. It is sometimes more accurate than [`BinaryCompressionType::LinearPrediction`]. Like [`BinaryCompressionType::LinearPrediction`],
//! it does not do any actual compression. It was meant for use with mzMLb, which compresses the entire container. Only suitable for float64 data.
//! - Delta-shuffled `zstd` [`BinaryCompressionType::DeltaShuffleZstd`]: Combines [`BinaryCompressionType::DeltaPrediction`] with [`BinaryCompressionType::ShuffleZstd`].
//! Requires feature `zstd`
//!
//!
//! ## Supported Data types
//!
//! `mzdata` supports five basic binary data types via the [`BinaryDataArrayType`] enum. Data that doesn't follow any of these types
//! may still be transported with the [`BinaryDataArrayType::Unknown`] variant, but it will not be directly usable.
//!
//! These values may be decoded on the fly or stored for faster repeated access. Coercion with [`DataArray::coerce`] relies on
//! [`bytemuck`] to do the heavy lifting. Conversion through [`DataArray::convert`] maps between different types but makes copies.
//!
//!
//!
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;