Crate npyz

Expand description

Serialize and deserialize the NumPy’s *.npy binary format.

§Overview

NPY is a simple binary data format. It stores the type, shape and endianness information in a header, which is followed by a flat binary data field. This crate offers a simple, mostly type-safe way to read and write *.npy files. Files are handled using iterators, so they don’t need to fit in memory.

§Optional cargo features

No features are enabled by default. Here is the list of existing features:

There are a couple of features which enable support for serialization/deserialization of foreign types. These require opt-in because they can be stability hazards; a major version bump of npyz may introduce a major version bump of one of these crates. (NOTE: to ease this issue somewhat, npyz will re-export the versions of the crates it uses)
- "complex" enables the use of num_complex::Complex.
- "half" enables the use of half::f16.
- "arrayvec" enables the use of arrayvec::ArrayVec and arrayvec::ArrayString as alternatives to Vec and String for some string types.
"derive" enables derives of traits for working with structured arrays.
"npz" enables adapters for working with NPZ files (including scipy sparse matrices), adding a public dependency on the zip crate. This requires opt-in because zip has a fair number of transitive dependencies. (note that some npz-related helper functions are available even without the feature)

§Reading

Let’s create a simple *.npy file in Python:

import numpy as np
a = np.array([1, 3.5, -6, 2.3])
np.save('test-data/plain.npy', a)

Now, we can load it in Rust using NpyFile:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let bytes = std::fs::read("test-data/plain.npy")?;

    // Note: In addition to byte slices, this accepts any io::Read
    let npy = npyz::NpyFile::new(&bytes[..])?;
    for number in npy.data::<f64>()? {
        let number = number?;
        eprintln!("{}", number);
    }
    Ok(())
}

And we can see our data:

1
3.5
-6
2.3

§Inspecting properties of the array

NpyFile provides methods that let you inspect the array.

fn main() -> std::io::Result<()> {
    let bytes = std::fs::read("test-data/c-order.npy")?;

    let data = npyz::NpyFile::new(&bytes[..])?;
    assert_eq!(data.shape(), &[2, 3, 4]);
    assert_eq!(data.order(), npyz::Order::C);
    assert_eq!(data.strides(), &[12, 4, 1]);

    // convenience method for reading to vec
    println!("{:?}", data.into_vec::<f64>());
    Ok(())
}

§Writing

The primary interface for writing npy files is the WriterBuilder trait.

use npyz::WriterBuilder;

fn main() -> std::io::Result<()> {
    // Any io::Write is supported.  For this example we'll
    // use Vec<u8> to serialize in-memory.
    let mut out_buf = vec![];
    let mut writer = {
        npyz::WriteOptions::new()
            .default_dtype()
            .shape(&[2, 3])
            .writer(&mut out_buf)
            .begin_nd()?
    };

    writer.push(&100)?;
    writer.push(&101)?;
    writer.push(&102)?;
    // you can also write multiple items at once
    writer.extend(vec![200, 201, 202])?;
    writer.finish()?;

    eprintln!("{:02x?}", out_buf);
    Ok(())
}

§Supported dtypes

A complete description of the supported numpy dtypes and the corresponding rust types can be found on the crate::type_matchup_docs module.

§Working with `ndarray`

Using the ndarray crate? No problem! At the time, no conversion API is provided by npyz, but one can easily be written:

// Example of parsing to an array with fixed NDIM.
fn to_array_3<T>(data: Vec<T>, shape: Vec<u64>, order: npyz::Order) -> ndarray::Array3<T> {
    use ndarray::ShapeBuilder;

    let shape = match shape[..] {
        [i1, i2, i3] => [i1 as usize, i2 as usize, i3 as usize],
        _  => panic!("expected 3D array"),
    };
    let true_shape = shape.set_f(order == npyz::Order::Fortran);

    ndarray::Array3::from_shape_vec(true_shape, data)
        .unwrap_or_else(|e| panic!("shape error: {}", e))
}

// Example of parsing to an array with dynamic NDIM.
fn to_array_d<T>(data: Vec<T>, shape: Vec<u64>, order: npyz::Order) -> ndarray::ArrayD<T> {
    use ndarray::ShapeBuilder;

    let shape = shape.into_iter().map(|x| x as usize).collect::<Vec<_>>();
    let true_shape = shape.set_f(order == npyz::Order::Fortran);

    ndarray::ArrayD::from_shape_vec(true_shape, data)
        .unwrap_or_else(|e| panic!("shape error: {}", e))
}

pub fn main() -> std::io::Result<()> {
    let bytes = std::fs::read("test-data/c-order.npy")?;
    let reader = npyz::NpyFile::new(&bytes[..])?;
    let shape = reader.shape().to_vec();
    let order = reader.order();
    let data = reader.into_vec::<i64>()?;

    println!("{:?}", to_array_3(data.clone(), shape.clone(), order));
    println!("{:?}", to_array_d(data.clone(), shape.clone(), order));
    Ok(())
}

Likewise, here is a function that can be used to write an ndarray:

use std::io;
use std::fs::File;

use ndarray::Array;
use npyz::WriterBuilder;

// Example of writing an array with unknown shape.  The output is always C-order.
fn write_array<T, S, D>(writer: impl io::Write, array: &ndarray::ArrayBase<S, D>) -> io::Result<()>
where
    T: Clone + npyz::AutoSerialize,
    S: ndarray::Data<Elem=T>,
    D: ndarray::Dimension,
{
    let shape = array.shape().iter().map(|&x| x as u64).collect::<Vec<_>>();
    let c_order_items = array.iter();

    let mut writer = npyz::WriteOptions::new().default_dtype().shape(&shape).writer(writer).begin_nd()?;
    writer.extend(c_order_items)?;
    writer.finish()
}

pub fn main() -> io::Result<()> {
    let array = Array::from_shape_fn((6, 7, 8), |(i, j, k)| 100*i as i32 + 10*j as i32 + k as i32);
    // even weirdly-ordered axes and non-contiguous arrays are fine
    let view = array.view(); // shape (6, 7, 8), C-order
    let view = view.reversed_axes(); // shape (8, 7, 6), fortran order
    let view = view.slice(ndarray::s![.., .., ..;2]); // shape (8, 7, 3), non-contiguous
    assert_eq!(view.shape(), &[8, 7, 3]);

    let mut file = io::BufWriter::new(File::create("examples/output/ndarray.npy")?);
    write_array(&mut file, &view)
}

§Structured arrays

npyz supports structured arrays! Consider the following structured array created in Python:

import numpy as np
a = np.array([(1,2.5,4), (2,3.1,5)], dtype=[('a', 'i4'),('b', 'f4'),('c', 'i8')])
np.save('test-data/simple.npy', a)

To load this in Rust, we need to create a corresponding struct. There are three derivable traits we can define for it:

Deserialize — Enables easy reading of .npy files.
AutoSerialize — Enables easy writing of .npy files. (in a default format)
Serialize — Supertrait of AutoSerialize that allows one to specify a custom DType.

Enable the "derive" feature in Cargo.toml, and make sure the field names and types all match up:

// make sure to add `features = ["derive"]` in Cargo.toml!
#[derive(npyz::Deserialize, Debug)]
struct Struct {
    a: i32,
    b: f32,
    c: i64,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let bytes = std::fs::read("test-data/structured.npy")?;

    let npy = npyz::NpyFile::new(&bytes[..])?;
    for row in npy.data::<Struct>()? {
        let row = row?;
        eprintln!("{:?}", row);
    }
    Ok(())
}

The output is:

Array { a: 1, b: 2.5, c: 4 }
Array { a: 2, b: 3.1, c: 5 }

§`.npz` files

To work with .npz files in general, see the npz module.
To work with scipy.sparse matrices see the sparse module.

Re-exports§

pub use num_complex;
pub use arrayvec;
pub use half;

Modules§

npz: Utilities for working with npz files.
sparse: Tools for reading and writing Scipy sparse matrices in NPZ format.
type_matchup_docs: DType to/from rust type documentation.
write_options: Types and traits related to the implementation of WriteOptions.

Structs§

DTypeError: Indicates that a particular rust type does not support serialization or deserialization as a given DType.
Field: A field of a structured array dtype
FixedSizeBytes: Wrapper around [u8; N] that can serialize as |VN. The size must match exactly.
NpyDataDeprecated: Legacy type for reading npy files.
NpyFile: Object for reading an npy file.
NpyHeader: Represents the parsed header portion of an npy file.
NpyReader: Iterator returned by NpyFile::data which reads elements of type T from the data portion of an NPY file.
NpyWriter: Interface for writing an NPY file to a data stream.
ParseTypeStrError: Error type returned by <TypeStr as FromStr>::parse.
TypeStr: Represents an Array Interface type-string.
WriteOptions: Represents an almost-empty configuration for an NpyWriter.

Enums§

DType: Representation of a Numpy type
Endianness: Represents the first character in a TypeStr, which describes endianness.
Order: Order of axes in a file.
TimeUnits: Represents the units of the m and M datatypes in a TypeStr.
TypeChar: Represents the second character in a TypeStr.

Traits§

AutoSerialize: Subtrait of Serialize for types which have a reasonable default DType.
Deserialize: Trait that permits reading a type from an .npy file.
Serialize: Trait that permits writing a type to an .npy file.
TypeRead: Like some sort of for<R: io::Read> Fn(R) -> io::Result<T>.
TypeReadDyn: The proper trait to use for trait objects of TypeRead.
TypeWrite: Like some sort of for<W: io::Write> Fn(W, &T) -> io::Result<()>.
TypeWriteDyn: The proper trait to use for trait objects of TypeWrite.
WriterBuilder: Trait that provides methods on WriteOptions.

Functions§

to_fileDeprecated: Serialize an iterator over a struct to a NPY file.
to_file_1d: Serialize an iterator over a struct to a NPY file.

Type Aliases§

OutFileDeprecated: NpyWriter that writes an entire file.

Derive Macros§

AutoSerialize
Deserialize
Serialize

Crate npyzCopy item path