ndarrow 0.0.4

Zero-copy bridge between Apache Arrow and ndarray
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130

# ndarrow

[![Crates.io](https://img.shields.io/crates/v/ndarrow.svg)](https://crates.io/crates/ndarrow)
[![Documentation](https://docs.rs/ndarrow/badge.svg)](https://docs.rs/ndarrow)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Build Status](https://img.shields.io/github/actions/workflow/status/georgeleepatterson/ndarrow/ci.yml?branch=main)](https://github.com/georgeleepatterson/ndarrow/actions)
[![Coverage](https://codecov.io/gh/georgeleepatterson/ndarrow/branch/main/graph/badge.svg)](https://codecov.io/gh/georgeleepatterson/ndarrow)

A zero-copy bridge between [Apache Arrow](https://arrow.apache.org/) and
[ndarray](https://docs.rs/ndarray). Convert Arrow arrays to ndarray views and back without
allocation overhead.

> ndarrow is under active development. Public APIs will change until v1. Pin your version.

## Install

```toml
[dependencies]
ndarrow = "0.0.4"
```

## What It Does

ndarrow lets you move data between Arrow and ndarray with zero allocations on the bridge path:

- **Arrow to ndarray**: borrow Arrow buffers as ndarray views (O(1), no copy)
- **ndarray to Arrow**: transfer owned ndarray buffers into Arrow arrays (O(1), ownership move)

This enables Arrow-native systems (DataFusion, Flight, IPC) to leverage ndarray-native
numerical libraries (like [nabled](https://crates.io/crates/nabled)) without paying for
serialization or conversion.

## Quick Example

```rust
use arrow_array::{Float64Array, FixedSizeListArray};
use ndarrow::{AsNdarray, IntoArrow};
use ndarray::Array1;

// Arrow -> ndarray (zero-copy view)
let arrow_array = Float64Array::from(vec![1.0, 2.0, 3.0, 4.0]);
let view = arrow_array.as_ndarray()?;  // ArrayView1<f64>, no allocation
assert_eq!(view[0], 1.0);

// ndarray -> Arrow (zero-copy ownership transfer)
let result = Array1::from_vec(vec![5.0, 6.0, 7.0, 8.0]);
let arrow_result = result.into_arrow()?;  // PrimitiveArray<Float64>, no allocation
```

## Type Mappings

| Arrow Type | ndarray Type | Direction | Copy? |
|---|---|---|---|
| `PrimitiveArray<T>` | `ArrayView1<T>` | Arrow -> ndarray | Zero-copy |
| `FixedSizeList<T>(D)` | `ArrayView2<T>` (M, D) | Arrow -> ndarray | Zero-copy |
| `arrow.fixed_shape_tensor` | `ArrayViewD<T>` | Arrow -> ndarray | Zero-copy |
| `arrow.variable_shape_tensor` | `VariableShapeTensorBatchView<T>` (`row() -> ArrayViewD<T>`) | Arrow -> ndarray | Zero-copy |
| `ndarrow.csr_matrix_batch` | `CsrMatrixBatchView<T>` (`row() -> CsrView<T>`) | Arrow -> ndarray | Zero-copy |
| `FixedSizeList<ndarrow.complex32>(D)` | `ArrayView2<Complex32>` | Arrow -> ndarray | Zero-copy |
| `FixedSizeList<ndarrow.complex64>(D)` | `ArrayView2<Complex64>` | Arrow -> ndarray | Zero-copy |
| `arrow.fixed_shape_tensor<ndarrow.complex32>` | `ArrayViewD<Complex32>` | Arrow -> ndarray | Zero-copy |
| `arrow.fixed_shape_tensor<ndarrow.complex64>` | `ArrayViewD<Complex64>` | Arrow -> ndarray | Zero-copy |
| `ndarrow.csr_matrix` | `CsrView<T>` | Arrow -> ndarray | Zero-copy |
| `Array1<T>` | `PrimitiveArray<T>` | ndarray -> Arrow | Zero-copy |
| `Array2<T>` (M, N) | `FixedSizeList<T>(N)` | ndarray -> Arrow | Zero-copy* |
| `ArrayD<T>` | `arrow.fixed_shape_tensor` | ndarray -> Arrow | Zero-copy* |
| `Array2<Complex32>` | `FixedSizeList<ndarrow.complex32>(N)` | ndarray -> Arrow | Zero-copy* |
| `Array2<Complex64>` | `FixedSizeList<ndarrow.complex64>(N)` | ndarray -> Arrow | Zero-copy* |
| `ArrayD<Complex32>` | `arrow.fixed_shape_tensor<ndarrow.complex32>` | ndarray -> Arrow | Zero-copy* |
| `ArrayD<Complex64>` | `arrow.fixed_shape_tensor<ndarrow.complex64>` | ndarray -> Arrow | Zero-copy* |

\* Zero-copy if standard (C-contiguous) layout. Allocates a layout copy otherwise.

## Supported Element Types

- `f32`, `f64` (first-class)
- Additional types via the `NdarrowElement` trait

## Null Handling

Null semantics are explicit at the call site:

```rust
// Validated: returns Err if nulls present (O(1) check)
let view = array.as_ndarray()?;

// Unchecked: caller guarantees no nulls (zero cost)
let view = unsafe { array.as_ndarray_unchecked() };

// Masked: returns view + validity bitmap (zero allocation)
let (view, mask) = array.as_ndarray_masked();
```

For numerical object payloads such as `FixedSizeList<T>(D) -> ArrayView2<T>`, the masked path is
row-oriented: it may return an outer row-validity bitmap, but actual inner component nulls are
still rejected on validated and masked ingress. Ragged tensor and batched CSR carriers now expose
column-level batch views through `variable_shape_tensor_batch_view` and `csr_matrix_batch_view`;
those views preserve the outer validity bitmap and provide `row()` / `iter()` accessors, while
`variable_shape_tensor_iter_masked` and `csr_matrix_batch_iter_masked` remain convenience wrappers.

## Extension Types

ndarrow uses canonical Arrow extension types where they exist:

- `arrow.fixed_shape_tensor` — fixed-shape multi-dimensional data
- `arrow.variable_shape_tensor` — variable-shape data (e.g., multi-vectors)

And defines its own for gaps:

- `ndarrow.csr_matrix` — CSR sparse matrix representation
- `ndarrow.complex32` — complex32 scalar element representation
- `ndarrow.complex64` — complex64 scalar element representation

## Performance Guarantee

Bridge conversions are O(1) regardless of array size. The bridge creates views (pointer +
shape) or transfers buffer ownership. It never touches the data. See
[docs/PERFORMANCE_CONTRACTS.md](docs/PERFORMANCE_CONTRACTS.md) for the full allocation contract.

## Quality Gates

```bash
just checks
```

## License

Licensed under the Apache License, Version 2.0.

See [LICENSE](LICENSE) for details.