ferray_numpy_interop/
lib.rs

1//! # ferray-numpy-interop
2//!
3//! A companion crate providing owning conversions between ferray arrays and
4//! external array ecosystems:
5//!
6//! - **NumPy** (via PyO3) — feature `"python"`
7//! - **Apache Arrow** — feature `"arrow"`
8//! - **Polars** — feature `"polars"`
9//!
10//! All three backends are feature-gated and disabled by default. Enable them
11//! in your `Cargo.toml`:
12//!
13//! ```toml
14//! [dependencies.ferray-numpy-interop]
15//! version = "0.1"
16//! features = ["arrow"]  # or "python", "polars"
17//! ```
18//!
19//! ## Memory semantics
20//!
21//! Every conversion in this crate currently **copies** the data buffer.
22//! The previous documentation claimed "zero-copy where possible", but in
23//! practice all six conversion paths (NumPy / Arrow / Polars × both
24//! directions) allocate a new buffer and memcpy the elements:
25//!
26//! | Path                     | Reason                                       |
27//! |--------------------------|----------------------------------------------|
28//! | `NumPy → ferray`         | `PyReadonlyArray::iter().cloned().collect()` |
29//! | `ferray → NumPy`         | `Array::to_vec_flat()` then `from_vec`       |
30//! | `Arrow ↔ ferray`         | `PrimitiveArray::values()` cloned into `Vec` |
31//! | `Polars ↔ ferray`        | `ChunkedArray` → `Vec<T>` via per-chunk copy |
32//!
33//! True zero-copy ferray↔NumPy would require ferray arrays to share the
34//! raw buffer with a Python-owned `PyArray` (refcount handshake plus
35//! pinning), which is a significant design change. Zero-copy to Arrow
36//! would require ferray arrays to expose their backing buffer as an
37//! `arrow::buffer::Buffer` with a compatible `Drop` hook. Both are
38//! tracked as potential follow-ups; for now the crate provides a
39//! correct, allocation-aware API that clearly acknowledges the copy.
40//!
41//! The copies are usually still *cheap enough* for interop boundaries —
42//! they are a single `memcpy` per conversion, not per element — but
43//! callers on hot paths should prefer to stay inside one ecosystem.
44//!
45//! ## Design principles
46//!
47//! 1. **Safety first** — every conversion validates dtypes and memory
48//!    layout before returning. No silent reinterpretation of memory.
49//! 2. **Honest about allocation** — see the table above. The docstrings
50//!    on individual functions say "copy" explicitly.
51//! 3. **Explicit errors** — dtype mismatches, null values, and
52//!    unsupported types produce clear
53//!    [`FerrayError`](ferray_core::FerrayError) messages.
54
55pub mod dtype_map;
56
57#[cfg(feature = "python")]
58pub mod numpy_conv;
59
60#[cfg(feature = "arrow")]
61pub mod arrow_conv;
62
63#[cfg(feature = "polars")]
64pub mod polars_conv;
65
66// Re-export the main conversion traits at crate root for ergonomics.
67
68#[cfg(feature = "arrow")]
69pub use arrow_conv::{
70    FromArrow, FromArrowBool, ToArrow, ToArrowBool, array2_from_arrow_columns,
71    array2_to_arrow_columns, arrayd_from_arrow_flat, arrayd_to_arrow_flat,
72};
73
74#[cfg(feature = "polars")]
75pub use polars_conv::{
76    FromPolars, FromPolarsBool, ToPolars, ToPolarsBool, array2_from_polars_dataframe,
77    array2_to_polars_dataframe,
78};
79
80#[cfg(feature = "python")]
81pub use numpy_conv::{AsFerray, IntoNumPy};
ferray_numpy_interop/lib.rs

ferray_numpy_interop/
lib.rs
