libjay 0.2.0

Independent, modern implementations of the J and APL array languages: parallel and vectorized, embeddable from Rust, Python, and C
Documentation

libjay

Independent, embeddable implementations of the J and APL array languages, as one dependency-light Rust crate. Not a DataFrame library and not a framework: the relationship to your code is the one PCRE has — a small language inside a string literal, compiled once, run many times, with the heavy machinery hidden inside.

The Cargo package is libjay; the library is jay (so the linker artifact is not liblibjay), matching the C -ljay and the Python import jay.

cargo add libjay

Compile once, run many times

use jay::{compile, Array, Data, Dialect, Lang};

let program = compile(Lang::J, "(+/ % #) {x}", &Dialect::default())?;

let x = Array::new(vec![5], Data::F64(vec![3.0, 1.0, 4.0, 1.0, 5.0].into()));
let value = program.run(&[x], &mut |s| print!("{s}"))?;   // Some(2.8) — the mean

let mut lines = ["3 1 4"].into_iter().map(str::to_string);
let echoed = jay::compile(Lang::Apl, "⍞", &Dialect::default())?
    .run_io(&[], &mut |s| print!("{s}"), &mut || lines.next())?;   // reads one line

{name} holes become parameters. program.params reports them in the order run expects; arguments are positional. run returns Option<Array> — None when the last sentence yields no value (an assignment, or echo/⎕←). The closure is the output sink for echo, ⎕← and ⍞←; stdout is the sandbox default and no other I/O is open. run_io is the same call with an input source too — a closure answering one line per call and None at the end — which is what APL's ⍞ and ⎕ and J's 1!:1 ]1 read; run itself has none, and an expression that reads says so rather than reading anything.

Dialect carries the host's settings — today APL's ⎕IO (Dialect { index_origin: Some(0) }); J's index origin is 0 and is not configurable. Lang::Apl selects the other frontend, with its own semantics: J reduces along the leading axis, APL along the trailing one.

Errors carry a span into the source. Error::render(source) renders a compile error, Program::render_error a run error, both the way the CLI would:

length error: arguments do not agree: left shape 3, right shape 2
  1 2 3 + 1 2
  ^^^^^^^^^^^
note: frames first differ at axis 0: 3 vs 2

jay::fmt::format_array(&value, &program.fmt) renders a result the way its language displays it.

What is inside

Both frontends lower to one language-agnostic IR: an Expr tree over a Verb combinator tree (Prim / Rank / Reduce / Fork / Hook / Atop / Windowed). One generic rank-and-agreement engine executes everything, so APL's +/ is simply Rank(Reduce(+), 1) and no J assumption reaches the runtime. A compile-time pass fuses chains of elementwise primitives into one blockwise kernel, absorbing a trailing full-rank reduction; anything it will not fuse falls back to the subtree it replaced, so results and error messages cannot change. Dense arrays of bool / i64 / f64 / characters. Array carries a private row-major/column-major layout flag — every array libjay builds for itself is row-major, a table crosses from its columns without a copy, and transpose (|:, ⍉) is exactly the flag flipped, moving no elements.

Threads

Execution is parallel by default — part of what a compiled expression is, not a switch the caller flips. Elementwise passes, pure rank cells and leading-axis reductions split above 65,536 element operations.

A Program is immutable, holds no data, and is Send + Sync: share one across threads, or wrap it in an Arc and run it concurrently. The pool is the crate's own, not rayon's global one, so an embedding host that also uses rayon keeps its pool intact; work started inside a rayon worker stays in that worker's pool. LIBJAY_THREADS sets the size (read once, at first use), otherwise it is the machine's available parallelism.

An associative float reduction may be regrouped across chunks, which reorders the rounding; everything else is bit-identical to the sequential path.

Devices

Where a program runs is separate from what it is bound to, and the CPU is one of the answers:

let device = jay::Device::default_gpu();          // None where there is none
let value = match &device {
    Some(d) => program.run_on(d, &args, &mut sink)?,
    None => program.run(&args, &mut sink)?,
};

jay::device::available() lists the adapters. What reaches a GPU is the fused elementwise kernels, generated as WGSL at run time and dispatched through wgpu (Metal, Vulkan, DX12); everything else, and any chain the device will not take, runs on the CPU, and Program::explain_on names each placement and its reason. The backend is in every build and dormant without an adapter — there is no feature flag and no second artifact.

libjay computes floats in f64 and will not lose that quietly: on an adapter without SHADER_F64 — Metal has none — an f64 chain stays on the CPU unless the caller asks for Precision::F32. Device::upload returns an array that carries its own device allocation, so repeated runs over it upload nothing.

More

MIT licensed.