libjay 0.1.0

Independent, modern implementations of the J and APL array languages: parallel and vectorized, embeddable from Rust, Python, and C
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119

# libjay

Independent, embeddable implementations of the [J](https://www.jsoftware.com/)
and APL array languages, as one dependency-light Rust crate. **Not a
DataFrame library** and not a framework: the relationship to your code is the
one PCRE has — a small language inside a string literal, compiled once, run
many times, with the heavy machinery hidden inside.

The Cargo package is `libjay`; the library is `jay` (so the linker artifact
is not `liblibjay`), matching the C `-ljay` and the Python `import jay`.

```sh
cargo add libjay
```

## Compile once, run many times

```rust
use jay::{compile, Array, Data, Dialect, Lang};

let program = compile(Lang::J, "(+/ % #) {x}", &Dialect::default())?;

let x = Array::new(vec![5], Data::F64(vec![3.0, 1.0, 4.0, 1.0, 5.0].into()));
let value = program.run(&[x], &mut |s| print!("{s}"))?;   // Some(2.8) — the mean
```

`{name}` holes become parameters. `program.params` reports them in the order
`run` expects; arguments are positional. `run` returns `Option<Array>` —
`None` when the last sentence yields no value (an assignment, or `echo`/`⎕←`).
The closure is the output sink for `echo` and `⎕←`; stdout is the sandbox
default and no other I/O is open.

`Dialect` carries the host's settings — today APL's `⎕IO`
(`Dialect { index_origin: Some(0) }`); J's index origin is 0 and is not
configurable. `Lang::Apl` selects the other frontend, with its own semantics:
J reduces along the leading axis, APL along the trailing one.

Errors carry a span into the source. `Error::render(source)` renders a
compile error, `Program::render_error` a run error, both the way the CLI
would:

```
length error: arguments do not agree: left shape 3, right shape 2
  1 2 3 + 1 2
  ^^^^^^^^^^^
note: frames first differ at axis 0: 3 vs 2
```

`jay::fmt::format_array(&value, &program.fmt)` renders a result the way its
language displays it.

## What is inside

Both frontends lower to one language-agnostic IR: an `Expr` tree over a
`Verb` combinator tree (Prim / Rank / Reduce / Fork / Hook / Atop /
Windowed). One generic rank-and-agreement engine executes everything, so
APL's `+/` is simply `Rank(Reduce(+), 1)` and no J assumption reaches the
runtime. A compile-time pass fuses chains of elementwise primitives into one
blockwise kernel, absorbing a trailing full-rank reduction; anything it will
not fuse falls back to the subtree it replaced, so results and error messages
cannot change. Dense arrays of bool / i64 / f64 / characters, row-major.

## Threads

Execution is parallel by default — part of what a compiled expression is,
not a switch the caller flips. Elementwise passes, pure rank cells and
leading-axis reductions split above 65,536 element operations.

A `Program` is immutable, holds no data, and is `Send + Sync`: share one
across threads, or wrap it in an `Arc` and run it concurrently. The pool is
the crate's own, not rayon's global one, so an embedding host that also uses
rayon keeps its pool intact; work started inside a rayon worker stays in that
worker's pool. `LIBJAY_THREADS` sets the size (read once, at first use),
otherwise it is the machine's available parallelism.

An associative float reduction may be regrouped across chunks, which
reorders the rounding; everything else is bit-identical to the sequential
path.

## Devices

Where a program runs is separate from what it is bound to, and the CPU is
one of the answers:

```rust
let device = jay::Device::default_gpu();          // None where there is none
let value = match &device {
    Some(d) => program.run_on(d, &args, &mut sink)?,
    None => program.run(&args, &mut sink)?,
};
```

`jay::device::available()` lists the adapters. What reaches a GPU is the
fused elementwise kernels, generated as WGSL at run time and dispatched
through wgpu (Metal, Vulkan, DX12); everything else, and any chain the
device will not take, runs on the CPU, and `Program::explain_on` names each
placement and its reason. The backend is in every build and dormant without
an adapter — there is no feature flag and no second artifact.

libjay computes floats in f64 and will not lose that quietly: on an adapter
without `SHADER_F64` — Metal has none — an f64 chain stays on the CPU unless
the caller asks for `Precision::F32`. `Device::upload` returns an array that
carries its own device allocation, so repeated runs over it upload nothing.

## More

- [Language coverage](https://github.com/amyodov/libjay/blob/main/docs/coverage.md)
  — what each frontend understands today, and the deliberate divergences from
  the reference implementations.
- [Benchmarks](https://github.com/amyodov/libjay/blob/main/bench/README.md) —
  against Polars, numba and numpy.
- [Embedding from C](https://github.com/amyodov/libjay/blob/main/docs/embedding.md)
  — the `libjay-capi` crate builds `libjay.so`/`.dylib` plus `libjay.a` and a
  hand-written `jay.h`.
- [Python surface](https://github.com/amyodov/libjay/blob/main/python/README.md)
  of the same engine.
- [Source and issues](https://github.com/amyodov/libjay).

MIT licensed.