Crate mzdata_bindata

Expand description

The DataArray implementation for data buffer manipulation used for, amongst other things, mapping numeric data to and from optionally compressed little endian base-64 byte arrays with as few data copies as possible, algebraically.

Also covers the ArrayType enum that covers the standardized array flavors telling us which dimension an array describes, BinaryArrayMap collections of (ArrayType, DataArray), and traits for converting to and from BinaryArrayMap based upon the set of available ArrayType. This includes a generalization for BinaryArrayMap3D when ion mobility is available.

§Compression

mzdata supports several kinds of data compression methods with tradeoffs

§Lossless methods

zlib compression BinaryCompressionType::Zlib: A well known, widely available general-purpose compressor codec with many implementations, with varying speed and size tradeoffs taken from flate2.
zstd compression BinaryCompressionType::Zstd: A newer general-purpose codec that is not widely used in mzML but very popular in other applications. Much faster and often better compression compared to zlib. Requires feature zstd
byte-shuffled zstd compression BinaryCompressionType::ShuffleZstd: This first applies a byte shuffle transform to the data to compress the nth byte of each value contiguously, and then apply zstd. This makes sorted data compress more effectively. Requires feature zstd
dictionary encoded zstd BinaryCompressionType::ZstdDict: Another elaboration of byte-shuffled zstd where the values are first dictionary-encoded so that repeated values are denoted by indices rather than the full size value itself. The dictionary is sorted and then both the dictionary and the indices are byte shuffle encoded and then zstd compressed. This is even more effective than previous codecs for data with repeated, semi-sorted values like m/z and ion mobility. Requires feature zstd

§Lossy methods

numpress-linear compression BinaryCompressionType::NumpressLinear: A combination simple linear model fit and byte packing codec that stores the model coefficients as float64 and the residuals as unsigned 16 or 32-bit integers. With an optimal fixed point, this will produce a median loss of accuracy below of about 2e-8 absolute units, but may vary widely from dataset to dataset. This codec MUST be used with sorted data only, and is only suitable for float64 data. It may be combined with zlib BinaryCompressionType::NumpressLinearZlib or zstd BinaryCompressionType::NumpressLinearZstd for even greater space savings. See details at https://doi.org/10.1074/mcp.O114.037879. /// Requires feature numpress
numpress-slof compression BinaryCompressionType::NumpressSLOF: A lossy numerical approximation that log transforms and scales values to make them very close to 16-bit integers. This codec is lossier the larger the value is, and is only appropriate for count data like intensities. It can also combine with zlib BinaryCompressionType::NumpressSLOFZlib and zstd BinaryCompressionType::NumpressSLOFZstd. See details at https://doi.org/10.1074/mcp.O114.037879 Requires feature numpress
numpress-pic compression BinaryCompressionType::NumpressPIC: This simply rounds a floating point value and then packs it into a truncated integer. May be suitable for intensity values, but not consistently better than BinaryCompressionType::NumpressSLOF. See details at https://doi.org/10.1074/mcp.O114.037879 Requires feature numpress
Linear prediction compression BinaryCompressionType::LinearPrediction: The same concept as BinaryCompressionType::NumpressLinear without the truncation All of the same concerns apply though it loses less precision. This does not perform any form of compression. It was uses as part of mzMLb which applies container-level compression. Only suitable for float64 data.
Delta encoding BinaryCompressionType::DeltaPrediction: This computes first order difference of a sorted array of values and stores the difference and the starting point. It is sometimes more accurate than BinaryCompressionType::LinearPrediction. Like BinaryCompressionType::LinearPrediction, it does not do any actual compression. It was meant for use with mzMLb, which compresses the entire container. Only suitable for float64 data.
Delta-shuffled zstd BinaryCompressionType::DeltaShuffleZstd: Combines BinaryCompressionType::DeltaPrediction with BinaryCompressionType::ShuffleZstd. Requires feature zstd

§Supported Data types

mzdata supports five basic binary data types via the BinaryDataArrayType enum. Data that doesn’t follow any of these types may still be transported with the BinaryDataArrayType::Unknown variant, but it will not be directly usable.

These values may be decoded on the fly or stored for faster repeated access. Coercion with DataArray::coerce relies on bytemuck to do the heavy lifting. Conversion through DataArray::convert maps between different types but makes copies.

Modules§

utils

Structs§

BinaryArrayMap: A collection of DataArrays that are identified by name.
BinaryArrayMap3D: Represent a set of BinaryArrayMap that has been split across an ion mobility dimension.
DataArray: Represents a data array that holds a byte buffer that may be compressed, base64 encoded, or raw little endian bytes, and provides views of those bytes as a small range of supported types.
DataArraySlice: Represent a slice of a DataArray that manages offsets and decoding automatically.

Enums§

ArrayRetrievalError: A high level set of failure modes that an operation to retrieve a typed memory buffer from a BinaryArrayMap might encounter. May also be used to represented conversion during reading or writing.
ArrayType: The kinds of data arrays found in mass spectrometry data files governed by the PSI-MS controlled vocabulary.
ArraysAvailable: Whether or not sufficient arrays are available in a BinaryArrayMap to construct a peak list using BuildFromArrayMap or BuildFromArrayMap3D
BinaryCompressionType: The range of compression and encoding states that a raw byte buffer might be in during different stages of decoding. Other than Decoded, these states may or may not include intermediate base64 encoding.
BinaryDataArrayType: The canonical primitive data types found in MS data file formats supported by the PSI-MS controlled vocabulary

Traits§

BuildArrayMap3DFrom: An extension of BuildArrayMapFrom that produces BinaryArrayMap3D instead.
BuildArrayMapFrom: A trait for converting a slice of peaks or features into a BinaryArrayMap.
BuildFromArrayMap: A trait for reconstructing a Vec of peaks or features from a BinaryArrayMap.
BuildFromArrayMap3D: An extension of BuildFromArrayMap that consumes [BinaryArray3D] to build a Vec of Self.
ByteArrayView: A byte array that may be encoded and compressed as [BinaryCompressionType], or unpacked in memory and re-interpreted as native types appropriate to BinaryDataArrayType.
ByteArrayViewMut: A mutable byte array

Functions§

as_bytes: Reinterpret a slice of a bytemuck::Pod type as raw bytes without copying or converting endianness.
delta_decoding: The decoder for BinaryCompressionType::DeltaPrediction.
delta_encoding: The encoder for BinaryCompressionType::DeltaPrediction.
linear_prediction_decoding: The decoder for BinaryCompressionType::LinearPrediction.
linear_prediction_encoding: The encoder for BinaryCompressionType::LinearPrediction.
to_bytes: Convert the values in data into an owned buffer of little-endian bytes
vec_as_bytes: Convert an owned Vec of a bytemuck::Pod type into an owned buffer of bytes.

Type Aliases§

Bytes