1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116

#![warn(missing_docs)]

/*!
Serialize and deserialize the NumPy's
[*.npy binary format](https://docs.scipy.org/doc/numpy-dev/neps/npy-format.html).

# Overview

[**NPY**](https://docs.scipy.org/doc/numpy-dev/neps/npy-format.html) is a simple binary data format.
It stores the type, shape and endianness information in a header,
which is followed by a flat binary data field. This crate offers a simple, mostly type-safe way to
read and write *.npy files. Files are handled using iterators, so they don't need to fit in memory.

Only one-dimensional [structured arrays](https://docs.scipy.org/doc/numpy/user/basics.rec.html) are
supported at the moment as they map well to Rust structs.

To successfully import an array from NPY using the `#[derive(NpyData)]` mechanism, the target struct
must contain:

* corresponding number of fields in the same order,
* corresponding names of fields,
* compatible field types.

Currently, all the primitive numeric types and arrays of up to 16 elements are supported, though
they work only with little-endian. To deserialize other types or big-endian values, one must
manually implement [`Serializable`](trait.Serializable.html). A very common object that (right now)
requires a manual `impl` is a vector, as illustrated in
[an example](https://github.com/potocpav/npy-rs/tree/master/examples/vector.rs).

# Examples

More examples can be found in the [examples](https://github.com/potocpav/npy-rs/tree/master/examples)
directory.

Let's create a simple *.npy file in Python:

```python
import numpy as np
a = np.array([(1,2.5,4), (2,3.1,5)], dtype=[('a', 'i4'),('b', 'f4'),('c', 'i8')])
np.save('examples/simple.npy', a)
```

Now, we can load it in Rust:

```
#[macro_use]
extern crate npy_derive;
extern crate npy;

use std::io::Read;

#[derive(NpyData, Debug)]
struct Array {
    a: i32,
    b: f32,
    c: i64,
}

fn main() {
    let mut buf = vec![];
    std::fs::File::open("examples/simple.npy").unwrap()
        .read_to_end(&mut buf).unwrap();

    for arr in npy::from_bytes::<Array>(&buf).unwrap() {
        println!("{:?}", arr);
    }
}
```

The output is:

```text
Array { a: 1, b: 2.5, c: 4 }
Array { a: 2, b: 3.1, c: 5 }
```
*/

extern crate byteorder;
#[macro_use]
extern crate nom;

mod header;
mod serializable;
mod npy_data;
mod out_file;

pub use serializable::Serializable;
pub use header::DType;
pub use npy_data::{NpyData, NpyIterator, from_bytes};
pub use out_file::{to_file, OutFile};

#[cfg(test)]
mod tests {
    // use super::header::*;
    // use super::header::Value::*;
    // use super::nom::*;

    // #[test]
    // #[derive(NpyData)]
    // struct S {
    //     batchId: i32,
    //     hostHash: i64,
    //     user: i64,
    //     aggregate: f64,
    //     label: i8,
    // }

    //
    // #[test]
    // fn from_file() {
    //     let file_mmap = Mmap::open_path("test/file.npy", Protection::Read).unwrap();
    //     let bytes: &[u8] = unsafe { file_mmap.as_slice() }; // No concurrent modification allowed
    //     let res: Vec<_> = S::from_bytes(bytes).unwrap().collect();
    //     println!("{:?}", res);
    // }
}