pyferrite
Source code: https://github.com/89605502155/pyferrite, release v0.0.1.
Read and write Python machine-learning artefacts from Rust — .npy, .npz,
.pkl, .pt, .joblib and .h5 — with no C, no C++, no FFI and no
compiled artefacts from any other language. If you have rustc, you can
build it, including on bare-metal and embedded targets.
.forbid(unsafe_code) is enforced crate-wide.
[]
= "0.0.1"
Why
Moving a model between Python and Rust usually means either linking against
libhdf5 and libtorch, or reimplementing the file formats badly. pyferrite
does the second job properly, so a Rust inference stack can load reference
weights, and a Rust training loop can hand results back to the Python tooling
your team already uses. That makes it practical to cross-check a Rust
implementation against the Python original on byte-identical data.
A tour in four lines
use *;
let mut weights = read?; // dict of tensors
for a in weights.arrays_mut
write?; // and out again, any format
# Ok::
What it does
| Formats | .npy .npz .pkl/.pickle .pt/.pth .joblib .h5/.hdf5 |
| Python types | dict list tuple set str bytes int (arbitrary precision) float complex bool None |
| Arrays | np.ndarray and torch.Tensor → ndarray::ArrayD, both directions |
| Tables | pandas / polars frames via Frame, with a feature-gated polars bridge |
| Element types | bool, i8–i64, u8–u64, f16, bf16, f32, f64, f128, x87 80-bit, complex64/128/256, and 8-bit floats (e4m3, e5m2) |
| Streaming | chunked reading and writing for files larger than RAM |
| Casting | pick the stored width on the way in and on the way out |
Choosing numeric types
Both directions are configurable, which is the point of the *_as options:
use *;
// "I know these should be f32, whatever the file says."
let opts = new.float_as.int_as;
let v = read_with?;
// And symmetrically on the way out, clamping instead of failing.
let out = new
.float_as
.cast_policy;
write_with?;
# Ok::
CastPolicy::Strict is the default and refuses any conversion that would lose
information; Saturate clamps to the target range; Wrap truncates like a C
cast. Note that f8 here means an eight-bit float, unlike numpy's f8,
which means an eight-byte one.
Streaming a file that does not fit in memory
use *;
let mut r = read_lazy?;
while let Some = r.next_chunk?
let mut w = write_lazy?;
# let block = F32;
for _ in 0..10_000
w.finish?; // fails if fewer elements arrived than were declared
# Ok::
Writing is validated
Shapes and types are checked before anything reaches the disk, because numpy
distinguishes a scalar 2 from [2] from [[2]]:
use *;
# let value = Array;
// Declare the exact shape numpy should see.
write_with?;
// And a string is not an array, so this is an error, not a corrupt file:
assert!;
# Ok::
Safety
Pickle is an executable format, and pyferrite does not execute it. The
virtual machine resolves an allow-list of numpy, torch and collections
constructors; anything else becomes an inert PyObject that records the
class name and its arguments without calling anything. Modules such as os,
sys and subprocess are refused outright. Allocation is bounded by
ReadOptions::max_alloc, so a hostile header cannot exhaust memory.
Dataframes
Frame is the neutral table type. A numpy record array becomes one on read; a
pickled pandas.DataFrame is reassembled from its BlockManager by
interop::pandas::to_frame; and with the polars-interop feature, Frame
converts to and from a polars::DataFrame.
# use *;
let v = read?;
if let Some = to_frame?
# Ok::
Documentation
Full per-function, per-parameter documentation with worked examples:
Feature flags
| Flag | Default | Effect |
|---|---|---|
deflate |
yes | compressed .npz, gzip/zlib .joblib, deflate-filtered HDF5 |
polars-interop |
no | converts Frame to and from a polars::DataFrame |
Turning deflate off removes the only non-trivial dependency; uncompressed
files still work.
Status
Version 0.0.1 is the first public release: complete enough to move real models between Python and Rust, but the API may still change before 0.1.0.
The minimum supported Rust version is 1.70 for the default features and for
--no-default-features. On a toolchain that old, resolve dependencies with
CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback cargo update (or Cargo's
MSRV-aware resolver) so that transitive crates are not newer than the
compiler. The polars-interop feature follows the requirements of polars 0.35
itself and needs a more recent compiler.
Licence
MIT, see LICENSE.
Authors, Bryansk State Engineering Technological University (BGITU), Bryansk, Russia:
- Andrey O. Ferubko, postgraduate student, Department of Information Technologies, ferubko1999@yandex.ru
- Oleg D. Kazakov, Candidate of Economic Sciences, Associate Professor, Head of the Department of Information Technologies (scientific supervisor), kazakov@bgitu.ru
Авторы: Ферубко Андрей Олегович, аспирант кафедры информационных технологий, и Казаков Олег Дмитриевич, канд. экон. наук, доцент, заведующий кафедрой информационных технологий (научный руководитель), Брянский государственный инженерно-технологический университет (БГИТУ).