Skip to main content

Crate pyferrite

Crate pyferrite 

Source
Expand description

§pyferrite

crates.io docs.rs GitHub licence: MIT

Source code: https://github.com/89605502155/pyferrite, release v0.0.2.

Read and write Python machine-learning artefacts from Rust — .npy, .npz, .pkl, .pt, .joblib and .h5 — with no C, no C++, no FFI and no compiled artefacts from any other language. If you have rustc, you can build it, including on bare-metal and embedded targets.

.forbid(unsafe_code) is enforced crate-wide.

[dependencies]
pyferrite = "0.0.2"

§Why

Moving a model between Python and Rust usually means either linking against libhdf5 and libtorch, or reimplementing the file formats badly. pyferrite does the second job properly, so a Rust inference stack can load reference weights, and a Rust training loop can hand results back to the Python tooling your team already uses. That makes it practical to cross-check a Rust implementation against the Python original on byte-identical data.

§A tour in four lines

use pyferrite::prelude::*;

let mut weights = read("model.pt")?;          // dict of tensors
for a in weights.arrays_mut() {               // everything is owned, so mutable
    if let Some(f) = a.as_f32_mut() {
        f.iter_mut().for_each(|x| *x *= 0.5); // rescale before loading
    }
}
write("model.h5", &weights)?;                 // and out again, any format

§What it does

Formats.npy .npz .pkl/.pickle .pt/.pth .joblib .h5/.hdf5
Python typesdict list tuple set str bytes int (arbitrary precision) float complex bool None
Arraysnp.ndarray and torch.Tensor → ndarray::ArrayD, both directions
Tablespandas / polars frames via Frame, with a feature-gated polars bridge
Element typesbool, i8–i64, u8–u64, f16, bf16, f32, f64, f128, x87 80-bit, complex64/128/256, and 8-bit floats (e4m3, e5m2)
Streamingchunked reading and writing for files larger than RAM
Castingpick the stored width on the way in and on the way out

§Choosing numeric types

Both directions are configurable, which is the point of the *_as options:

use pyferrite::prelude::*;

// "I know these should be f32, whatever the file says."
let opts = ReadOptions::new().float_as(FloatKind::F32).int_as(IntKind::I32);
let v = read_with("weights.npz", &opts)?;

// And symmetrically on the way out, clamping instead of failing.
let out = WriteOptions::new()
    .float_as(FloatKind::F16)
    .cast_policy(CastPolicy::Saturate);
write_with("small.npz", &v, &out)?;

CastPolicy::Strict is the default and refuses any conversion that would lose information; Saturate clamps to the target range; Wrap truncates like a C cast. Note that f8 here means an eight-bit float, unlike numpy’s f8, which means an eight-byte one.

§Streaming a file that does not fit in memory

use pyferrite::prelude::*;

let mut r = read_lazy("huge.npy", &ReadOptions::new().chunk_elements(1 << 16))?;
while let Some(chunk) = r.next_chunk()? {
    // chunk.data is a small owned array; chunk.offset says where it belongs
    // inside chunk.full_shape.
}

let mut w = write_lazy("huge_out.npy", DType::F32, &[10_000, 512], &WriteOptions::new())?;
for _ in 0..10_000 {
    w.push(&block)?;
}
w.finish()?;   // fails if fewer elements arrived than were declared

§Writing is validated

Shapes and types are checked before anything reaches the disk, because numpy distinguishes a scalar 2 from [2] from [[2]]:

use pyferrite::prelude::*;

// Declare the exact shape numpy should see.
write_with("x.npy", &value, &WriteOptions::new().shape([1, 6]))?;

// And a string is not an array, so this is an error, not a corrupt file:
assert!(write("x.npy", &Value::Str("hello".into())).is_err());

§Safety

Pickle is an executable format, and pyferrite does not execute it. The virtual machine resolves an allow-list of numpy, torch and collections constructors; anything else becomes an inert PyObject that records the class name and its arguments without calling anything. References to os, sys, subprocess, eval and similar are refused unless ReadOptions::allow_unknown_globals asks to capture them inertly for inspection. Allocation is bounded by ReadOptions::max_alloc, so a hostile header cannot exhaust memory.

§Dataframes

Frame is the neutral table type. A numpy record array becomes one on read; a pickled pandas.DataFrame is reassembled from its BlockManager by interop::pandas::to_frame; and with the polars-interop feature, Frame converts to and from a polars::DataFrame.

let v = read("frame.pkl")?;
if let Some(frame) = pyferrite::interop::pandas::to_frame(&v)? {
    println!("{} rows x {} columns", frame.height(), frame.width());
}

§Documentation

Full per-function, per-parameter documentation with worked examples:

§Feature flags

FlagDefaultEffect
deflateyescompressed .npz, gzip/zlib .joblib, deflate-filtered HDF5
polars-interopnoconverts Frame to and from a polars::DataFrame

Turning deflate off removes the only non-trivial dependency; uncompressed files still work.

§Status

Version 0.0.2 is an early release: complete enough to move real models between Python and Rust, but the API may still change before 0.1.0.

The minimum supported Rust version is 1.70 for the default features and for --no-default-features. On a toolchain that old, resolve dependencies with CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback cargo update (or Cargo’s MSRV-aware resolver) so that transitive crates are not newer than the compiler. The polars-interop feature follows the requirements of polars 0.35 itself and needs a more recent compiler.

§Licence

MIT, see LICENSE.

Authors, Bryansk State Engineering Technological University (BGITU), Bryansk, Russia:

  • Andrey O. Ferubko, postgraduate student, Department of Information Technologies, ferubko1999@yandex.ru
  • Oleg D. Kazakov, Candidate of Economic Sciences, Associate Professor, Head of the Department of Information Technologies (scientific supervisor), kazakov@bgitu.ru

Авторы: Ферубко Андрей Олегович, аспирант кафедры информационных технологий, и Казаков Олег Дмитриевич, канд. экон. наук, доцент, заведующий кафедрой информационных технологий (научный руководитель), Брянский государственный инженерно-технологический университет (БГИТУ).

Re-exports§

pub use api::convert;
pub use api::flatten_arrays;
pub use api::read;
pub use api::read_lazy;
pub use api::read_with;
pub use api::write;
pub use api::write_lazy;
pub use api::write_with;
pub use error::Error;
pub use error::Result;
pub use options::ReadOptions;
pub use options::WriteOptions;
pub use value::Array;
pub use value::Dict;
pub use value::Frame;
pub use value::PyObject;
pub use value::Series;
pub use value::Value;

Modules§

api
The five functions most users need: read(), write(), read_lazy(), write_lazy() and convert().
dtype
Description of element types shared by numpy, torch, HDF5 and this crate.
error
Error and result types used across the whole crate.
formats
Container formats understood by the crate, plus sniffing helpers.
interop
Bridges to the Rust and Python dataframe ecosystems.
lazy
Chunk-at-a-time reading and writing for files that do not fit in RAM.
num
Numeric primitives that Rust’s standard library does not provide but that Python / ML ecosystems use every day: 8-bit and 16-bit floats, quad precision, x87 extended precision, and complex numbers.
options
Read/write knobs: laziness, numeric down-casting, safety limits.
prelude
use pyferrite::prelude::*; pulls in everything needed for day-to-day work.
util
Small, dependency-free helpers: byte cursors, CRC-32 and deflate glue.
value
The dynamic value model: everything a Python artefact can contain.
zip
Minimal, pure-Rust ZIP reader/writer for .npz and .pt containers.