# arb — SPEC
**arb** is a standalone, original language on **fusevm/JIT** for **visualizing and modifying Unix pipelines**: drop it in a pipe and it spawns a **dynamic TUI (ratatui) or served web page (zgui components)** built from a declarative spec. It is a **jq/xpath/css/yq superset** — every leg machine-checked against its own reference tool, with the per-leg containment measured in §8 — an interactive **megafilter/map** over the live passthrough, its own **Tcl/Tk-flavored DSL**, and a **preset library / package manager** so users share dashboards — *a TUI for every pipeline*. (LSP/DAP stdio frontends ship. Akka-style actors ship: `actor NAME(state) { on MSG { … } }` + a `via NAME * N` pipeline op that fans the stream across a supervised worker pool in parallel — see §15.)
Original language (stryke's class), **not a port**. MIT, standalone crate, lean (rubyrs-scale, not stryke-scale).
---
## 0. Positioning
- **The synthesis + ecosystem**, not any single leg. Prior art per leg: Tcl'88, Tk'88, Expect'90 (spawn/react), dasel (unified query), ratatui (TUI), Streamlit/Textual-serve (served web UI), filt (interactive pipe grep — *single filter box, filter-only, TUI-only; not comparable*). No tool is a pipe-native, dual-target (terminal+web), component-generating UI language with a shareable dashboard registry.
- **Not a server-only thing**: terminal-invoked, pipe-driven. Web target spawns a local UI host (like `textual serve`), not a daemon.
## 1. Invocation
```sh
cmd | arb dash.arb # TUI (ratatui)
cmd | arb -t web dash.arb # served zgui page + WS live update
cmd | arb -p http # preset (== import http)
cmd | arb # zero-config: sniff input/upstream cmd → auto preset
cmd | arb -e 'text .t <- in' # inline
producer | arb dash.arb | consumer # mid-pipe: controls shape downstream output
arb -l # list presets/packages
arb save dash.arb as api # register a user preset
```
**fzf drop-in.** `arb --fzf` resolves its options the way the `fzf` binary does —
`$FZF_DEFAULT_OPTS_FILE`, then `$FZF_DEFAULT_OPTS`, then the command line, later
winning — and honors the presentation set (`--layout`/`--reverse`, `--border`,
`--info`, `--color`, `--pointer`, `--marker`, `--ellipsis`, `--scrollbar`,
`--scroll-off`, `--cycle`, `--tac`, `--tiebreak`, `--bind`, `--min-height`,
`--ansi`) on top of the matching and I/O set (`--exact`, `--no-sort`/`--sort`,
`-q`/`--query`, `-i`/`--ignore-case`, `+i`/`--no-ignore-case`, `--smart-case`,
`--multi`, `--prompt`, `--header`, `--header-lines`, `--height`, `--preview`,
`-f`/`--filter`, `-d`/`--delimiter`, `-n`/`--nth`, `--with-nth`,
`--print-query`, `--expect`, `--with-shell`). Child processes (a `--preview`
command) start with `$SHELL -c` like fzf's, not `sh -c`. A repeated option takes
its LAST value, as fzf does — `$FZF_DEFAULT_OPTS` and the call site both setting `--prompt` is the
normal case, not an error. The whole fzf 0.74 option surface parses, including
every `--no-X` negation, which cancels an earlier `--X` rather than being
dropped — `--preview CMD --no-preview` leaves no preview — and an
optional-value option consumes the next word only under fzf's own rule for it
(a word that isn't a flag for `--border`, a number for `--gap`). Matching
itself is fzf's algorithm, ported from `src/algo/algo.go` into `src/algo.rs`
(`FuzzyMatchV2` + `ExactMatchNaive`), so rankings are identical line for line —
verifiable with `--filter`, which prints the ranked matches without a UI.
`--height 100%` is fzf's full-screen form and runs on the alternate screen, so
quitting restores the terminal untouched; a smaller `--height` draws below the
cursor and scrolls the terminal to make room, sized like fzf's (percentages get
the `--min-height=10+` floor); arb places that viewport itself rather than
asking crossterm, whose cursor query writes to stdout — arb's data channel — and
never returns when stdout is a pipe. The picker it draws — layout, separator,
pointer/marker glyphs, palette, scrollbar, ellipsis, scroll margin, ranking —
matches what `fzf` draws for the same environment, so `ZPWR_FZF='arb --fzf'`
changes nothing on screen. Flags with no arb analog are accepted and ignored, and
a `--bind` naming an unimplemented action is skipped rather than half-applied.
## 2. Lexical (Tcl-flavored, NOT Tcl)
```
verb arg arg { block } # command + args; {} is a verbatim block
# comment (no $, no [cmd] subst, no expr{})
; # optional stmt separator (newline also separates)
.a.b.c # widget path (dot hierarchy, Tk-style)
```
## 3. Values
```
42 3.14 "s" 's' true false nil
1s 500ms 2m # durations
1kb 4mb # sizes
/re/ /re/i # regex
[1,2,3] # list
{a:1, b:2} # map
```
## 4. Variables (Python/Swift-lite)
```
max = 100 # immutable
var n = 0 # mutable
n = n + 1
```
## 5. Functions & lambdas
```
fn pct(v, m): v / m * 100 # single-expr body
fn norm(xs) { hi = max(xs); map(x => x/hi, xs) } # block; last expr returns
dbl = x => x * 2 # lambda
add = (a, b) => a + b
pct(50, 200) # call
```
## 6. Operators
```
+ - * / % arithmetic
== != < <= > >= comparison
and or not logical
+ string concat
x matches /re/ regex test
x in [..] membership
a..b range
( … ) grouping — a full expression, including a comparison, a
logical, or a `?:` (`(x > 1) and (x < 5)`, `not (a or b)`)
|> value pipe: xs |> filter(even) |> sum
```
Every value is an f64, so a predicate has a NUMBER for its value: `1` when it
holds and `0` when it does not, for `==`/`<`/`and`/`or`/`not`/`in` alike. That
matters wherever an expression's value is printed rather than tested —
`map x > 1 or x > 2` emits `1`, never a count of how many sides held.
A number literal may carry an exponent (`1e17`, `2e+2`, `1.5e-6`, `1E3`), which
is also how arb prints one back.
### How a computed number prints
A computed value renders the way `jq -r` renders a computed double: the shortest
digits that round-trip and, among those, the ones NEAREST the value, positional
up to the point where jq switches, then exponential with a signed
two-digit-minimum exponent.
```
1e15 * 1 -> 1000000000000000 1e16 * 1 -> 1e+16
1.5e16 * 1 -> 15000000000000000 1.5e17 * 1 -> 1.5e+17
0.0001 * 1 -> 0.0001 0.00001 * 1 -> 1e-05
1e18 / 3 -> 333333333333333300 2e-9 * 2 -> 4e-09
```
The switch is on the DIGIT COUNT, not the magnitude — `1e16` is exponential and
`1.5e16`, one significant digit longer, is not.
Shortest and shortest-AND-CLOSEST are different rules, and the difference shows.
Where neighbouring doubles sit further apart than one unit in the last decimal
place, several decimals of the shortest length parse back to the same double, and
only one of them is nearest. `191510495617760.12 * 1` prints those digits because
they are the nearest; `…13` round-trips just as well and is what stopping at the
first round-tripping candidate gives. The ties bunch around 1e14..1e17, seldom
enough that a pool spread evenly across the exponent range almost never lands on
one.
A result with no finite value follows jq as well: an infinity CLAMPS to ±DBL_MAX
(`1e308 * 2` -> `1.7976931348623157e+308`) and a NaN prints as `null`
(`1e308 * 2 * 0`). Neither has a spelling in JSON.
One deliberate difference from jq remains:
- **There is no literal preservation.** A number jq never computes on is held as
a DECIMAL rather than a double and reprinted from that, so `.n` on
`{"n":1e17}` gives `1E+17` — and unary `-.` preserves it too, printing `-1E-7`
where the computed path gives `-1e-07`. It is not the source text being echoed:
`1e17`, `1E17` and `1e+17` all come back as `1E+17`, decNumber's canonical
form.
Matching it is not a formatter change. That decimal holds values no double can
— `{"n":1e400}` reprints as `1E+400`, and a 30-digit integer keeps all 30
digits, where an f64 overflows and rounds respectively. Following jq here would
mean carrying an arbitrary-precision decimal beside every number, which is
exactly the "every value is an f64" rule above. So arb prints the double, and
`scripts/expr_paths.sh` reports the difference rather than hiding it.
### `%` takes a remainder, it does not truncate
`%` is the f64 remainder of the two operands: `5.5 % 3` is `2.5`. jq truncates
both operands to integers first and answers `2`. The two agree on every integer
input, which is why both harnesses scored parity for `%` until their corpora grew
a fractional value. The probes for it are red, on purpose, and stay that way
rather than being reworded to match either side.
For arb's own expressions (`map x % 3`) the remainder is the settled rule: every
value here is an f64, and truncating the operands would be the one place the
language stopped believing that.
What is NOT settled is the jq context. `%` also reaches the jq front-end — a
leading `.` routes the whole body there — and lowers to the same fusevm op, so
`out { in.json; .n % 3 }` on `{"n":5.5}` answers `2.5` where jq answers `2`. That
front-end promises jq's answers, and arb already settles this exact kind of
collision by CONTEXT rather than by picking one meaning globally: `flatten` and
`to_entries` answer as jq while the native `entries` keeps its line-per-key
shape, under distinct spellings. Whether `%` should join them is open;
`scripts/jq_parity.sh` probes it so the difference is reported either way.
## 7. Pipe & sources
```
in stdin (lines)
in.json in.xml in.html in.yaml in.toml in.csv parsed stream
out.yaml out.json out.props out.xml out.csv out.tsv [INDENT]
re-serialize the stream (yq's -o= / -I) ✅
spawn ps aux arb launches a process; its stdout is the stream ✅
spawn { tail -f a.log; grep err } block form → one `sh -c` string ✅
! vmstat 1 every 1s re-run a command on a timer ✅ (headless runs it once)
< file.log read a file as the stream ✅ (quote absolute paths: `< "/var/log/x"`)
out { … } downstream emission (to | next), shaped by controls
spawn -pty CMD run CMD on a pseudo-terminal (acts interactive) ✅
send "text" write to a `spawn -pty` child's stdin (Expect) ✅
```
`spawn CMD` (or `spawn { CMD }`) makes a spec self-sourcing: arb runs CMD via
`sh -c` and feeds its stdout into the stream in place of stdin (fire-and-forget;
the child dies with arb). `< FILE` reads a file as the stream instead (folded to
`cat -- FILE`). `! CMD every Ns` re-runs CMD on a timer, feeding each run's
output into the stream — in the TUI/served dashboard it loops in the background;
headless (piped onward) it runs CMD **once**, because a reducer/emit over an
endless timer source could never terminate. At most one stream source may be
declared (`spawn`/`< FILE`/`! …`); a CLI `--run 'PROD | _ | CONS'` producer wins
over all of them. **`spawn -pty CMD`** runs the spawn command on a pseudo-terminal
so it behaves as if interactive, and keeps a writer to its stdin so a **`send
"text"`** action (a bind/expect action, like `set`/`beep`/`exec`) can drive it —
Expect-style automation, e.g. `expect { /password:/ send "hunter2\n" }`. The
`send` write happens in the TUI (headless falls back to a plain pipe). The
`sel` selection widget (§9) exposes a widget's highlighted row as `.<path>.sel`
for a `send`/`where`/`tell` to consume.
## 8. Query — jq/xpath/css/yq superset (uniform over all formats)
```
field NAME key (jq .name); field a b c = a.b.c; field N = Nth ws column
fields N M … project/reorder whitespace columns (1-based): fields 1 3 -> cols 1 and 3
each iterate (jq [])
find TAG recursive descent (xpath //)
attr NAME attribute (xpath @, css)
sel { CSS } CSS selector (html)
where(PRED) filter (jq select)
pick a b c project object to keys (jq {a,b,c}); keeps listed order
b64 base64-encode each line
b64d base64-decode each line (invalid passes through)
hex lowercase hex-encode each line (byte-wise)
unhex hex-decode each line to UTF-8 (invalid passes through)
urlenc percent-encode each line
urldec percent-decode each line
extract /re/ first regex match per line (capture group 1 if any); no-match dropped
split DELIM explode each line by DELIM into one line per part
substr A B character substring [A,B) 0-based, clamped
chars explode each line into one char per line
title title-case each line
repeat N repeat each line's content N times
set K V set json object key K to string V
del K remove json object key K (jq del)
rename OLD NEW rename json object key OLD to NEW
default K V set json object key K to V only if absent
merge merge all json objects into one (later keys win)
floor floor each numeric line
ceil ceil each numeric line
clamp LO HI clamp each numeric line into [LO,HI]
abs absolute value of each numeric line
round round each numeric line to the nearest integer
commafy thousands-group each numeric line (1234567 -> 1,234,567)
bytes humanize a byte count, 1024-based (1536 -> 1.5 KB)
duration humanize seconds as the two largest units (3661 -> 1h 1m)
delta consecutive differences of the numeric series (n -> n-1) — a counter's rate-of-change
cumsum running (cumulative) total of the numeric series
sma N trailing simple moving average, window N (length-preserving; smooths a noisy series)
ewma A exponentially-weighted moving average, smoothing factor A in (0,1] (s0=x0)
median median of numeric lines (reducer)
stddev population standard deviation (reducer)
percentile N Nth percentile 0-100, nearest-rank (reducer); p50 p90 p95 p99 are sugar
range max minus min of numeric lines (reducer)
product product of numeric lines (reducer)
bins N bucket numeric lines into N equal-width bins -> (label -> count) pairs
apply .name splice an `input .name` widget's live value in as a sub-pipeline (megafilter/map)
sort_by F stable-sort json records by field F (numeric when all values parse, else lexical; non-objects last)
unique_by F keep first JSON record per distinct value of field F (dedup by F)
count_by F count json records grouped by field F (value -> count, count desc)
min_by F return the JSON record whose numeric field F is smallest
max_by F emit the record with the largest numeric field F (reducer)
has KEY keep only JSON-object lines that contain key KEY
entries jq to_entries: emit {"key":k,"value":v} per key of each JSON object line
flatten flatten a JSON array to its leaves, ALL levels (jq flatten), one leaf per line
add jq add: sum a numeric JSON-array line, concat a string array, [] -> ""
over N keep numeric lines strictly greater than N (drops non-numeric)
under N keep numeric lines strictly less than N
between A B keep numeric lines x with A <= x <= B (inclusive), drop the rest
enumerate prefix each line with its 1-based index and a tab
words split each line on whitespace into one word per line (flatten)
dedup collapse adjacent duplicate lines to one (classic uniq)
tailn N keep the last N lines (complement of take)
pad N right-pad each line with spaces to a minimum width N (no truncation)
lpad N left-pad each line with spaces to minimum width N
grepf FIELD /re/ keep lines whose FIELD (json key or 1-based ws column) matches /re/
flip reverse the characters of each line (Unicode scalar reversal)
names vals native line-per-key / line-per-value (jq's `keys` and
`values` keep their own spellings and their own shapes)
map(FN) jq map: `[.[] | FN]` — returns one ARRAY line, scoped per input
count sum min max avg tally aggregates
sort sort_by(FN) group_by(FN) uniq
over N under N numeric threshold
index N slice A B positional
```
| jq/xpath/css | arb |
|---|---|
| `.users[].name` | `field users; each; field name` |
| `.items[] \| select(.price>10)` | `field items; each; where(price>10)` |
| `//a/@href` | `find a; attr href` |
| `div.card h2` | `sel {div.card h2}` |
**Literal front-ends.** The right column is arb's native form, but inside a
`source { … }` / `out { … }` body you may write the **jq** or **xpath** literal
directly — it compiles to the same ops:
```
out { in.json; .users[] | select(.age >= 18) | .name } # jq: path, iterate, filter
out { in.json; map(.price); add } # jq map() -> one array, then a native verb
out { in.html; //a/@href } # xpath: descendant + attribute
out { in.html; //div[@class]//span/text() } # xpath: predicate + child + text()
```
**The whole jq language.** arb implements jq, not a subset of it: identity `.`,
key/path `.foo.bar`, iterate `.[]`/`.foo[]`, index `.[N]` (and negative `.[-1]`),
slice `.[a:b]`, pipe `|`, the comma operator `.a, .b`, object and array
construction (`{a: .a}`, `[.a, .b]`), parentheses, `if`/`then`/`elif`/`else`,
`try`/`catch` and the `?` suppressor, the `//` alternative, `..` recursive
descent, `as` bindings including array/object destructuring and the `?//`
alternative, `reduce` and `foreach`, `label`/`break`, `def` with filter and value
parameters (and recursion), `$ENV`/`$__loc__`, string interpolation `\(…)` and
every `@format`, the whole assignment family (`=`, `|=`, `+=`, `-=`, `*=`, `/=`,
`%=`, `//=`), path expressions (`path`, `paths`, `getpath`, `setpath`,
`delpaths`, `del`, `pick`, `to_entries`/`from_entries`/`with_entries`), the regex
family (`test`/`match`/`capture`/`scan`/`split`/`splits`/`sub`/`gsub`), the
stream builtins (`tostream`/`fromstream`/`truncate_stream`, `input`/`inputs`,
`limit`/`first`/`last`/`nth`/`until`/`while`/`repeat`/`isempty`), and the
date and libm surfaces.
That is not a list to be taken on trust. `scripts/jq_parity.sh` carries a
`superset_probe` that compares jq's own `builtins` output against arb's and fails
on any `name/arity` jq defines and arb does not; it currently reports none
missing. On its first run it reported 44, which is how `JOIN`, `bsearch`, `skip`,
`toboolean`, `trimstr`, `format` and the libm names from `acosh` to `yn` came to
be implemented. Every other construct is byte-diffed against `jq -rc` on the same
input, and a construct applied to the WRONG TYPE must be refused by both engines
— with jq's refusal checked, so a refusal arb invented alone is never scored as
parity.
### Containment is measured per leg
One containment probe per reference, each enumerating that reference's own
surface and reporting what arb is missing. Current run: **825 probes, 825 pass,
0 diverged, no allowlist.** The round-4 jq additions (`repeat`, positional
`sub`/`gsub` generators, code-point `indices`, the `s`/`p` regex flags, UTC
`strftime` zones, string repetition, and jq's fixed refusal wordings) were
measured on a machine without `yq`: 728 pass, 0 diverged, 159 yq probes skipped
(703 pass / 24 diverged on the same corpus before the fixes). The round-5 additions
(value parameters that yield several values run once per combination in jq's
order, the translator's `.[]`/`map`/`to_entries`/`flatten`/`add` keeping key
order and number literals, literal-preserving negation, empty regex matches after
a match, `{$name: P}` patterns, and jq's refusals for `unique`/`min`/`max`/
`*_by`/`reverse`/`input`/non-path calls, and `fromjson` as jq's own `jv_parse.c`
reader — what it accepts and its positioned refusals, and jq 1.8's decNumber
number model: two literals compare as decimals, a computed `-0` keeps its sign,
`have_decnum` is true) measured 759 pass, 0 diverged, 159 skipped (729 pass /
30 diverged on the same corpus before the fixes). The round-6 additions (`as`
binding jq 1.8's whole expression, a `def` after a pipe, literal- and
negation-led programs, a trailing comma in an object, quoted subscripts that
generate or interpolate, `strptime`/`mktime`/`gmtime`/`strftime %s` as jq's C,
`in.json` reading JSON texts rather than lines, and `input_line_number` over the
whole stream) measured 772 pass, 0 diverged, 159 skipped (760 pass / 12 diverged
on the same corpus before the fixes). The round-7 additions (array assignment's
NaN / too-large index refusals, `contains` comparing jv kinds only at the top
level, `f_match`'s argument wordings, `%` with NaN or a `-1` divisor, object-key
slices as `parse_slice` reads them, `escape_string`'s `\0` for NUL in
`@csv`/`@tsv`/`@html`/`@sh`, `range`/`limit`/`skip` as the RANGE opcode and
`builtin.jq` define them, and `nearbyint` rounding half to even) measured 797
pass, 0 diverged, 159 skipped (774 pass / 23 diverged on the same corpus before
the fixes). The round-8 additions (`try`/`catch` bodies that are negations,
`{$v: e}` keying by the value of `$v`, an empty array pattern refused, `builtins`
without `_` internals, `delpaths` as `jv_dels`, `?` after an index suppressing only
that step, jq's `path_intact` tracking through `label`/`reduce`/`foreach`/`try` with
its "near attempt to …" refusals, `jv_dump_string_trunc` byte truncation, input
lines read by the `jv_parse.c` port (`NaN`, `Infinity`, `.5`, BOM), `implode` of
NaN, `jv_get`/`jv_set` refusals, `length` keeping a number literal, `f_match`'s
empty-capture key order, and `_strindices` wordings) measured 816 pass, 0
diverged, 159 skipped (797 pass / 19 diverged on the same corpus before the
fixes). The seeded differential fuzz rounds that followed (they are a `cargo test`
of their own, not part of that script's corpus) moved the regex family onto
Oniguruma — the engine jq links, in its Perl_NG syntax, so look-around,
backreferences, `\K`, `\h`, the modifier letters and the engine's own refusal
text are jq's — and followed jq where its bytecode shows through: `path_intact`
tracking in destructuring patterns, `and`/`or` and the right-hand side of an
arithmetic assignment, `walk`/`map_values`/`from_entries`/`sub`/`gsub`/`splits`
as jq's own definitions, a parsed number never being `jv_identical` to one a
builtin computed from it, an update stopping at its first output, `?//`
catching what is raised downstream of it (a `break` included), `reduce`
re-entered over a `null` input, and `break $l` as the error jq compiles it to
(`$l | error`, the label value being `{"__jq": n}`): a `try`/`?` between a
`break` and its label catches it with that object, a handler that re-raises it
still ends the label, and `n` counts every label the run entered, builtin.jq's
own included, and input string bodies decoded as `jv_parse.c` decodes them: a
high surrogate with no low one after it, a non-hex `\u` body and a raw control
byte make the line non-JSON (so it keeps the text reading) instead of a repaired
value jq never produces. fixes). The 825 above is the last run WITH `yq`; it was not re-measured since.
| leg | reference | containment | status |
|---|---|---|---|
| jq | `jq` 1.8.2 `builtins` | every name present | **superset** |
| xpath | XPath 1.0 (W3C REC) via `xmllint` | **48 of 48 enumerated constructs present** | **superset** of the enumerated surface |
| css | Selectors 3/4 via `scraper` | 24 selection probes match the equivalent XPath | matches the reference on every probe |
| yq | mikefarah/yq v4.53.6 | every enumerated operator present | **superset** of the enumerated surface |
**XPath.** The subset this section used to document — `//`, `/`, `@`, `[@a]`,
`[@a='v']`, `[contains(@a,'v')]` — was XPath-SHAPED syntax compiled to a CSS
selector, and the probe found 46 of 48 constructs missing. `src/xpath.rs` is a
real engine now: `xpath_syntax.rs` lexes and parses the grammar of the W3C
Recommendation, `xpath_eval.rs` evaluates it over the parsed document. All 13
axes in explicit `axis::test` syntax, the node tests `node()`/`text()`/
`comment()`/`processing-instruction()` plus name and `*` wildcard tests,
predicates with correct proximity positions (§2.4 — and a predicate on a REVERSE
axis counts along that axis, so `preceding-sibling::p[1]` is the NEAREST
preceding sibling), the 27-function core library (§4), and the comparison,
arithmetic and union operators with §3.4's existential node-set semantics.
**59 `xp_probe`s** byte-diff the engine against `xmllint --html --xpath`.
**The four silent wrong answers are gone.** They were the reason this leg was
the priority: the promise below — that anything outside the subset is a hard
error, "never silently reinterpreted" — did not hold for them.
`[@a='x' or @a='y']` and a chained predicate `[@a='x'][@b='y']` both exited 0
with an EMPTY selection where XPath selects nodes, and a ROOTED path
(`/li/text()`, `/div/h2/text()`) exited 0 with a NON-EMPTY node set where XPath
selects nothing, because a leading `/step` was compiled as a descendant match.
A real engine removes the whole class by construction; `tests/xpath.rs` pins
each of the four against xmllint's own answer.
**The data model (§5) is reconciled with html5ever explicitly.** A doctype is
not one of §5's seven node types, so `count(//node())` must not see it. And
html5ever implements the HTML5 tree-construction algorithm, which ALWAYS creates
a `head` and a table `tbody`, where libxml2 creates each only when the source
has one — measured: `xmllint --html --xpath 'count(//head)'` answers 0 for
`<a>X</a><p>P</p>` and 1 for markup with a real `<head>`, and `count(//tbody)`
answers 0 for `<table><tr>…` and 1 when `<tbody>` is written. Both are hidden
from the data model AND from the serialization, so `//*`, `//node()` and every
positional predicate over them agree with the reference.
**Two stated boundaries, neither silent.** At arb's COMMAND position an
expression must carry a character only XPath uses (`/`, `@`, `::`, `(`, `[`,
`|`): a bare NCName is a legal relative location path, so `bogus` would parse as
`child::bogus`, select nothing and exit 0 — the silent answer this engine exists
to remove, reintroduced through the front door. It stays an `unknown verb`
diagnostic, and the step is spelled `//bogus`, `./bogus` or `child::bogus`.
And `contains` is the one core-function spelling jq already answers, so at
command position it stays jq's; XPath's two-argument form is reachable inside a
predicate, where it is probed. Both are the `keys`/`names` rule again — a
language does not get to claim a spelling another meaning already owns.
**CSS.** `sel` hands its selector to `scraper`. Both gaps this section used to
list are closed and probed. A `Selector::parse` failure is a hard error at BUILD
time naming the selector, where it used to map to "no matches" — so a MALFORMED
selector was indistinguishable from one that legitimately selects nothing, the
same silent-answer failure ruled out above (5 `css_bad` probes). And an
attribute value may be written with either quote: arb's command lexer turns
`"…"` into a string argument, and the reconstruction re-quotes it, so
`a[href="/x"]` no longer reaches the parser as `a[href=/x]`.
**yq.** `src/xpath.rs`'s problem was a missing engine; this leg's was a missing
VALUE MODEL. `yq_superset_probe` found **61 operators missing**, and among them
every node-metadata accessor — `anchor`, `alias`, `tag`, `style`, the three
comment positions, `key`, `is_key`, `path`, `parent`, `line`, `column`, `kind`.
Those are not builtins anyone forgot. They read metadata a YAML node carries and
a jq value has no slot for, which is the reason yq exists over jq at all, and
they cannot be added one at a time.
So the metadata rides ALONGSIDE the value, in the shape `JqVal::Num(f64,
Option<literal>)` already used for a number's source text — same problem, same
answer, already in the tree. `JqVal::Node` boxes a value with its
`crate::ynode::NodeMeta`, `bare()` unwraps at every operation whose answer is
about the value, and a node with nothing to record is not boxed at all. The
load-bearing property is that the JSON reader cannot construct the variant: only
`crate::yaml` can, so a JSON program reaches every answer through exactly the
arms it reached them through before, and **the jq leg is untouched by
construction rather than by inspection**.
Two things the parser does not hand over are recovered from the SPANS it does. A
`#` inside a scalar's span is content (`pw: "a#b"`, a comment-looking line inside
a `|` block) and one outside every span opens a comment; an anchor NAME is read
backwards from its node, over the whitespace and optional tag between them, and
only for a node the parser already reported as anchored. Comment ATTACHMENT
follows six rules measured against `yq v4.53.6` rather than taken from its
documentation — the head comment lands on the KEY node, the line comment on the
VALUE node, and a block followed by a blank line becomes the PREVIOUS entry's
foot.
What the probes check, all byte-diffed and none normalized:
| probe | what it asserts |
|---|---|
| `yq_superset_probe` | every enumerated yq operator exists — **61 missing before, 0 now** |
| `yq_probe` (111) | each operator's ANSWER equals `yq -o=json -I=0`'s on the same node |
| `yq_rt_probe` (14) | `in.yaml; out.yaml` returns the SOURCE FILE byte for byte |
| `yq_norm_probe` (3) | a shape yq does not return either: arb normalizes exactly as yq does |
| `yq_write_probe` (8) | a metadata assignment produces the document yq produces |
| `yq_fmt_probe` (12) | `out.props` is byte-identical to `yq -o=props` |
The round trip is the strongest of the six, and it is asserted against the source
file rather than against yq's output — which is stricter, not looser. `yq '.'` is
not idempotent on every shape: it re-folds a `>` block onto one line, escapes a
non-BMP character as a `"\U0001F680"` sequence, collapses a multi-line flow
collection, reorders `!!map &am` to `&am !!map`, and drops a trailing `...`.
Requiring arb to match yq there would require arb to reproduce yq's own
infidelities. "The file comes back" is the property the claim names, and it
implies matching yq everywhere yq does return the file.
The three shapes yq will not return either are not dropped for that — a shape
nobody checks is the only bad outcome. `yq_norm_probe` asserts them against the
REFERENCE instead, byte for byte and nothing normalized, so arb has to normalize
the same way yq does rather than merely being allowed to differ from the source.
Asserting the weaker of two true properties beats asserting neither.
The fixtures are one per feature the model has to carry — comments in every
position (and on sequence items versus mappings), anchors with aliases and merge
keys, deeply nested anchors, aliases inside merge keys, all six scalar styles,
flow versus block, tags on scalars and on collections, empty values and nulls,
non-ASCII, number spellings, explicit `---`/`...` markers, and a multi-document
stream — so a failure names which one broke.
The reference is asked with `--yaml-fix-merge-anchor-to-spec`. `<<` has two
readings and yq ships both: its default lets a merged key override an explicit
one, and the flag follows the YAML spec. yq's own warning calls the default
"isn't to the yaml spec". arb implements the spec rule, so asking the reference in
its other mode would report a divergence whose content is "arb is right" — which
is not what a divergence should mean. The two modes differ only on merge keys.
`out.FORMAT [INDENT]` is yq's `-o=`/`-I` in arb's spelling (`out.yaml`,
`out.json`, `out.props`, `out.xml`, `out.csv`, `out.tsv`). The DEFAULT rendering
is unchanged: one compact JSON line per document, which is what every existing
pipeline expects.
**What the yq leg does not claim.** ONE spelling is genuinely out, and it is not
a grammar problem. yq's `ref` (`.a ref $x | $x = 5`) binds a MUTABLE HANDLE to a
node, so a later assignment through `$x` edits the document. Nothing in the value
model is mutable — values are `Rc`-shared and copied on write, which is what makes
`reduce`/`foreach`/path updates affordable — so honouring the spelling would mean
a second, mutable value model, and `with(p; f)` already provides the capability.
The other two spellings the docs used to list here are ACCEPTED now: `.a anchor =
"x"` and `.[] as $item ireduce (0; . + $item)` both occupy grammar positions jq
leaves empty, so claiming them cost the jq leg nothing. arb takes yq's spelling
and its own for both.
One behaviour still differs by construction: a node's `path`/`key`/`parent` are
recorded at READ time, so a node relocated by `map`/`pick`/`+` reports where it
was read from, where yq's real parent pointers would follow it. Closing that means
a reference cycle through `Rc` for the whole document, which is a worse trade than
the accessor is worth. (`filename` used to be listed here too; it answers the real
path now when the spec names one with `< FILE`, and `-` for a pipe, which is what
yq answers for a piped document.)
Four YAML inputs are answered where yq ERRORS rather than matched — `.inf`,
`-.inf`, `.nan` and an integer past `i64` — which is the same direction as the
>2^53 deviation above: arb answers, the reference refuses. Being more capable than
the reference is not a containment failure, and none of the four is "fixed" to
match.
**What the node model costs.** Measured on a 20k-record, 2.3MB document against
the same data as JSON, both on the same debug binary: JSON parses in 0.131s and
the same document as YAML in 1.907s, while adding a filter over the result moves
neither (0.183s and 1.904s). So running a query over boxed values costs the same
as over bare ones — `bare()` is free in practice — and the metadata is a tax on
the READ, once, not on every operation. Three suspects were measured inside that
read: formatting a throwaway rendering per scalar was a fifth of it and is gone,
and the per-node path vector and the empty-`Rc` count each moved the total by less
than the run-to-run noise. What remains is saphyr's scanner plus one `Rc<YNode>`
per YAML node.
**What the numbers do not claim.** Containment is measured over the surface each
probe ENUMERATES — for xpath, the 13 axes, 27 core functions and node tests of
the 1999 Recommendation, not XPath 2.0+.
Two engines sit behind that. A jq literal that maps onto arb's line-stream ops
(a path, an iterate, a `select`, a `map`) is translated to them, which is what
keeps arb's own promises about a non-JSON line. Everything else compiles to a
program on arb's jq engine, whose value model is jq's: object keys keep INSERTION
order (`keys_unsorted` and `to_entries` expose it), and a number keeps the source
LITERAL it was read with until arithmetic touches it, printed in decNumber's
canonical form — so `1.50` stays `1.50`, `1e2` prints as `1E+2`, and
`12345678901234567890` round-trips. Two literals compare as the decimals they
spell, whatever their spelling (`3.000000000000000000000001 == 3` is false, and
`sort`/`unique`/`group_by` keep the two apart); a number arithmetic has touched
compares as the double it is, on either side. A literal below the smallest
double is `0.0` carrying the literal's sign, so `-1e-400 * 0` is `-0`.
Names resolve at COMPILE time, as jq's do. A word that is neither an arb verb nor
a jq builtin is still arb's `unknown verb`, not a jq runtime failure — a typo'd
arb verb is far likelier than a jq program, and that diagnostic is what points at
the real mistake.
Supported xpath: **XPath 1.0**, as a real engine rather than a subset — all 13
axes, the four node tests plus name and `*` wildcard tests, predicates with
correct proximity positions, the 27-function core library, and the comparison,
arithmetic and union operators. A standalone `@attr` step is arb's line-stream
continuation (`//a; @href`) rather than XPath's `attribute::` axis from the
document node, which is the one place the pipeline shape shows through: an
expression that is a RELATIVE location path is evaluated per line so it composes
with the step before it, and every other expression is one question about the
document, answered once. A leading `@` belongs to jq instead when it names
one of jq's formats (`@base64`, `@csv`, …) or is followed by a string, which
makes it a format string whatever the name: `@foo "lit"` answers `lit` as jq
does, and `@foo "\(.)"` fails as jq's `foo is not a valid format`. Over
JSON VALUES — after `in.json`/`in.yaml`/`in.toml`/`in.csv`/`in.tsv`, with no
`sel`/`find`/xpath step since — every leading `@` is jq's, so a bare `@foo` (or
`@base32`, which jq 1.8 does not have) raises jq's error instead of selecting no
attribute from a JSON line; after `in.html`, `in`, or an HTML step `@href` is
still the attribute continuation. A body command that opens with a
double-quoted string is jq's too even when jq refuses it: XPath spells a string
literal the same way, so `"\(1;2)"` is jq's syntax error, not the XPath literal
text `\(1;2)`. **Anything
that is not well-formed XPath is a hard error** (`xpath: …`) anchored to the offending verb, raised at BUILD time before
any input is read — never silently reinterpreted.
Where a jq builtin shares a spelling with an arb NATIVE verb (`sort`, `min`,
`max`, `floor`, `abs`, `length`, `add`, `flatten`, `first`, `last`, `split`,
`join`, `del`, `index`, `contains`, `range`, `match`, `repeat`, …) the BARE word
is the native verb and the CALL spelling is jq's — `sort_by v` is arb's,
`sort_by(.v)` is jq's, and `. | sort` reaches jq's. That is the context rule
below, not a jq leak.
**`#` inside `sel { … }` is an ID, not a comment.** `#` opens a comment to
end-of-line wherever a command is expected, and `sel`'s brace used to be re-lexed
as commands — so `sel { #main }`, a spelling the docs advertise, lexed to an
EMPTY block and reported `sel: expected a CSS selector` (exit 1) instead of
selecting the element. `sel` is the one brace in the language whose contents are
not commands (they are a CSS selector), so it is re-lexed with the comment rule
off and `sel { #main }`, `sel { #main p }` and the selector list
`sel { #main p, #two }` all select what xmllint's equivalent XPath selects. Every
other brace holds commands and keeps `#` as a comment.
**`keys` is jq's, and the native verb is `names`.** A native verb is matched
BEFORE the jq fall-through, so a native verb named `keys` shadowed jq's builtin:
bare `keys` printed a line per key where jq prints one sorted array. The context
rule cannot save a spelling jq itself defines — `. | keys` reached jq's answer,
but the bare word could not — so the native line-per-key verb is spelled `names`
and `keys` in every spelling is jq's. `stdlib/json.arb` runs `names; tally` and
its in-language test (`arb --test stdlib/json.arb`) pins the shape. **The jq leg
of the superset now has ZERO divergences**, machine-checked every run.
**A YAML number keeps its LITERAL, the same way a JSON number does.** It did
not: `ratio: 1.50` printed `1.5` where `yq -o=json` prints `1.50`, while the
JSON reader beside it printed `1.50` for the same text. serde's data model has
nowhere to put a number's source text — a plain scalar reaches the visitor as an
`f64` with the text already discarded — so the YAML reader now composes from the
parser's EVENT stream (`src/yaml.rs`), where a scalar arrives as its raw text,
and builds numbers through the same `num_from_literal` the JSON reader uses. One
number representation, one renderer, both formats.
An INTEGER is the other rule and is not an oversight: it renders as its VALUE,
never its text, because that is what `yq -o=json` prints (`007` -> `7`,
`0xFF` -> `255`, `1_000` -> `1000`). Both rules are probed.
**With that, the jq leg reports ZERO divergences.** So does every other leg: the
whole run is 825 probes, 825 pass, 0 diverged, no allowlist. The script exits
with the divergence count, so a clean run is an exit status and not a summary
line anyone has to read.
**One deviation runs the other way.** For an integer above 2^53, jq's own
arithmetic loses up to an ULP: `jq` answers `true` to
`(-516424571754902561 + 0) == -516424571754902500` when the correctly-rounded
double is `…600`. arb reads every number with Rust's correctly-rounded parser and
prints the shortest decimal that round-trips, so it answers `…600` and differs
from the reference by being right. It is not "fixed" to match, and the generated
sweep in `tests/jqlang.rs` states the tolerance explicitly: the computed path may
differ from jq ONLY above 2^53, and byte-matches everywhere else.
**jq expression bodies are jq VALUES.** A `select(…)` predicate, a `map(…)` body
and a bare arithmetic stage evaluate over JSON values, not over arb's f64
expression language (§6):
* a comparison yields a boolean — `map(. > 1)` is `[false,true,true]`, not `[0,1,1]`;
* `select` uses jq truthiness, where only `false` and `null` are falsy, so
`select(.a)` keeps `0`, `""`, `[]` and `{}`;
* `==` compares type as well as value (`1` is not `"1"`), and an absent key is
`null`, so `select(.b == null)` is how "field missing" is spelled;
* `+` is overloaded per type — number add, string concat, array concat, object
merge, and `null` as the identity on either side — with `-` on arrays (set
difference), `*` on objects (recursive merge) and on a string (repetition),
and `/` on two strings (split);
* `%` truncates both operands to integers first, as jq's does. Arb's own
`map x % 3` keeps §6's f64 remainder; the jq context follows jq, which is the
same context gating that separates jq `to_entries` from native `entries`;
* nested field paths work inside a body (`select(.a.b > 1)`, `map(.a.b)`), and a
path may continue THROUGH an iterate (`.users[].name`, `.a[].b.c`);
* a top-level JSON STRING renders RAW, the way `jq -r` prints one: a line reading
`"hello"` is `hello`, and `"a\"b"` is `a"b`. That holds for the filters that
emit their input line unchanged — identity `.`, `select(…)` and `values` — as
well as for the ones that already rendered (`.a`, `.[]`, a slice). Other types
keep the source line, which is also what jq does with a number it never computed
(`1.50` stays `1.50`);
* a TYPE mismatch is a hard error, not an answer. `null | .[]`, `3 | .a`,
`"hello" | .[1]`, `true | length`, `{"a":1} | . + 3`, `.n / 0` and `.n % 0` all
refuse with `jq: …` on stderr and a non-zero exit, matching what `jq` itself
does on the same input (exit 5, as jq uses);
* an uncaught error ends its INPUT, not the stream, as jq's main loop does: the
values that input produced before the error are printed, the error goes to
stderr, the next line still runs, and the exit status is the LAST line's (5
when it raised, else 0). A non-string error value is reported as jq words it,
`jq: (not a string): {"a":1}`. `halt` and `halt_error` end the whole run with
their own status, and `halt_error` writes its message unprefixed — a string
raw with no newline, any other value as JSON plus a newline, `null` not at all.
The TUI, REPL, `serve` and `--test` still show the first error as a refusal;
* `in.json` reads JSON TEXTS, as jq's parser does, not one document per line: a
pretty-printed document spanning lines is ONE input (rendered compactly), and
several values on one line (`1 2 3`, `{"a":1}{"a":2}`) are one input each. A
line holding exactly one value passes through untouched. A line that is no
part of a document — or an opened document that never balances, or balances
but does not parse — keeps the text reading below, line by line.
`input_line_number` counts lines, so a multi-line document reports the line it
ends on.
Those type rules apply to a line that PARSES as JSON. arb's stream is TEXT and
`jq` has no reading of a non-JSON line at all (it refuses the whole input), so
arb keeps its line-stream behaviour there rather than inventing one: a path
yields `null`, an iterate/slice passes the line through, and an EXPRESSION sees
the line as jq's string — `. * 2` over a line reading `abc` is `abcabc`.
A compare may test strings as well as numbers — `select(.status == "ok")`, and the
ordered forms too (`select(.s < "abd")`), which follow jq's total order
(`null < false < true < numbers < strings < arrays < objects`) — and a key that is
not a bare identifier is reachable through the bracket form, `.["a b"]`. A
subscript keeps its type: `.["0"]` is an object key and `.[0]` is an array index,
so `[1,2] | .["0"]` refuses rather than reading the first element. Path results
follow jq on absent data: an explicit null, a missing key and an out-of-range
index all render as `null`, not as an empty line. (Native `field NAME` is
unchanged — it still yields "" on a miss and still falls back to logfmt, because
it runs over plain-text streams as well as JSON.)
**Where a jq literal and a native verb share a spelling.** arb's pipeline is a
LINE stream, so a jq filter returning one array is emitted as one compact array
line, while the native verb of the same family emits a line per element. The two
spellings are kept distinct wherever they differ:
| spelling | shape | note |
|---|---|---|
| `to_entries` (jq) | one `[{"key":…,"value":…},…]` line | matches jq |
| `entries` (native) | one line per key | SPEC §8, unchanged |
| `keys` (jq, any spelling) | one `["a","b"]` line | matches jq |
| `names` (native) | one line per key | SPEC §8, the renamed native verb |
| `. \| add` (jq) | folds from `null`, so `[]` is `null` | matches jq |
| `add` (native verb) | `[] -> ""`, string-joins a mixed array | SPEC §8, unchanged |
| `. \| length` (jq) | refuses a boolean | matches jq |
| `length` (native verb) | falls back to the raw line's char count | runs over plain text too |
The rule is CONTEXT, not spelling: a body command that begins with a jq literal
(`.`, `select(`, `map(`, `has(`) is handed to the jq front-end whole, and inside
it every builtin answers as jq does. A bare alphanumeric word is looked up in the
native verb table first, and only a word with no native verb falls through to jq.
That gating is what separates jq `to_entries` from native `entries`. A shared
word FOLLOWED by a jq operator (`|`, `,`, `as`, `and`/`or`, arithmetic or a
comparison) is not bare: none of those verbs (`length`, `add`, `first`, `floor`,
`max`, …) takes an argument, so `length | . + 1` and `length as $l | …` are jq
programs. `sort -n` keeps its flags.
**The context rule has one boundary, and `keys` was over it.** The native table is
matched BEFORE the fall-through, so a native verb spelled exactly like a jq
builtin does not merely win the bare word — it SHADOWS jq's builtin, and the
superset contract breaks on that spelling. Every other collision in the table
above is safe because the two spellings DIFFER (`entries` vs `to_entries`, `vals`
vs `values`, `sort_by v` vs `sort_by(.v)`), and jq's own name stays reachable.
`keys` was the single spelling where they did not differ: jq's canonical short
name and arb's native verb were the same word, so no context could route both.
It is resolved by giving the native verb a name of its own. **`names`** is the
line-per-key verb; **`keys` in every spelling is jq's** sorted array. Three
things decide the direction, and they point the same way:
* jq owns the name. `keys` is jq's canonical spelling and has no shorter form,
while arb's native verbs already prefer the SHORT spelling of a shared idea
(`vals` beside jq's `values`, `entries` beside jq's `to_entries`). arb had no
shorter spelling to retreat to, so it took a distinct one instead.
* Nothing is lost. `names` is the same op it always was — line per key, and a
non-object line passes through untouched, which jq's `keys` refuses to do. Both
behaviours are now reachable, under names that do not collide.
* The one shipped preset that depended on it moved with it. `stdlib/json.arb`
runs `names; tally` and its in-language test (`arb --test stdlib/json.arb`)
pins the line-per-key shape, so the rename is covered by a test that fails if
it regresses.
With that, `scripts/jq_parity.sh` reports **zero jq divergences**: every probe in
the jq leg byte-matches `jq -rc` on the same input.
### In-language unit tests (`arb --test`)
A spec can carry its own tests: a `test "NAME" { … }` block feeds sample lines
through a query pipeline and asserts the output. `arb --test spec.arb` runs every
block headlessly and exits 0 (all passed) / 1 (any failed), with
[TAP](https://testanything.org/) output — so a dashboard's transforms are
regression-tested in CI, in the same language they're written in.
```
test "keeps 5xx, drops others" {
given "200 ok" "503 down" "404 x" "500 err" # input lines (one per arg)
run { in; match /5\d\d/ } # the pipeline (reuses source/out grammar)
want "503 down" "500 err" # expected output lines
}
test "counts errors" {
given "e" "e" "ok" "e"
run { in; match /e/; count } # a reducer → one scalar line
want "3"
}
test "jq path" { given "{\"u\":{\"name\":\"bob\"}}"; run { in.json; .u.name }; want "bob" }
```
`run { … }` is the ordinary `source`/`out` body — native verbs and the jq/xpath
literal front-ends are all testable. The output is flattened for comparison: a
scalar renders as one line, `tally`/`count_by` pairs as `key\tvalue`. Test blocks
are ignored by every mode except `--test` (they don't render a widget).
## 9. Widgets ("Tk" register)
```
text .t -label L tail .t -label L table .t -cols "a,b,c"
list .t gauge .t -label L -max N spark .t
bars .t -label L histo .t chart .t
linegauge .t -max N scatter .t sparkline .t # bars / scatter
map .t -res high calendar .t # world map / month calendar
logview .t heatmap .t treemap .t # log / grid / rects
gantt .t diff .t # time spans / +- coloring
logo .t clear .t rule .t # splash / spacer / divider
select .s -prompt P -header H input .i -placeholder P # interactive
sel .ps # per-widget selection list -> .ps.sel (§14)
tabs .t -tabs {a b} block .t -title T -border frame .f
.t configure -max 200 # reconfigure (merge opts into a declared widget)
```
Any widget takes `-color NAME` (green/red/yellow/orange/magenta/blue/white/gray,
default cyan) to tint its border and accent — same color in the TUI and web.
`select` is an interactive fuzzy picker (fzf as a one-widget spec; `source`
projects the candidate display, `search` derives a separate match key); `input`
is a live field whose value drives `apply`/`bind`/`out`. The chart family:
`linegauge` (a compact one-line `gauge`), `scatter` (braille scatter of a numeric
series), `sparkline` (a block-bar sparkline, the fixed-height counterpart to
`spark`'s braille line), `map` (a braille world map plotting `lon lat` points from
the stream — geo scatter), and `calendar` (a month grid highlighting days that
appear as `YYYY-MM-DD` in the stream). Scrollable list widgets (`tail`/`list`/
`block`/`frame`) draw a scrollbar when content overflows. `sel` is an in-dashboard
selection list over its own `source` — Up/Down (or a click) move a cursor and its
highlighted row is published as the control value `.<path>.sel`, readable from
`where`/`apply`/`tell`/`send` (the per-widget-named-source selection accessor).
arb wires the full ratatui data-widget set — `Paragraph`, `List`, `Table`,
`Tabs`, `Gauge`, `LineGauge`, `BarChart`, `Sparkline`, `Chart`, `Canvas`
(scatter/map), `Calendar`, `Scrollbar`, `Block` — leaving only the non-data
utilities (`Clear`, `RatatuiLogo`) unused.
## 10. Layout (auto by default)
```
# no grid → widgets auto-tile (vertical flow). Only add geometry to override.
layout horizontal # auto-tile in a row instead of a column
grid .a -row 0 -col 0 -span 2 # -span = colspan; -rowspan/-colspan explicit
grid .b -row 1 -col 0
# Proportional tracks (Tk `grid`-style, ratatui Constraints under the hood):
rows "1 2 1" # 3 rows; the middle is 2× tall (weights)
cols "20c * 2*" # col 0 fixed 20 cells, col 1 weight 1, col 2 weight 2
gap 1 # 1 blank cell between every row/column
```
A **track** is `N` (a proportional weight → `Fill`; the common case), `Nc` (a fixed cell count → `Length`), `N%` (a percentage of the
axis → `Percentage`), or `N*` / `*` (a proportional weight → `Fill`; bare `*` is
weight 1). `rows`/`cols` size the grid's tracks (unset = equal weights); a shorter
spec sizes the leading tracks and the rest fill. `gap N` inserts blank cells
between tracks. Without any `grid` cell, widgets auto-tile in the `layout`
direction (`vertical` default, `horizontal` for a row), sized by the flow-axis
track spec when given. The **served web dashboard uses a responsive CSS grid** —
`rows`/`cols`/`gap`/`layout` shape the terminal TUI.
## 11. Binding
```
source .t { in.json; each; where(is5xx); count; every 1s } # stream → widget via query
.g <- cpu_pct # reactive: widget follows a signal
.t <- now() every 1s # sampled bind
```
## 12. Interactive controls — megafilter/map the passthrough
Controls render AND feed `out`. A control's path used as a value = its current state.
```
filter .q # text box → .q string
facet .lv -field level # facets → .lv selected set
slider .th -field lat -min 0 -max 5s # → .th value
check .on -label live select .k -opts {a b c}
out {
where(match(.q)) # filter by text box
where(level in .lv) # filter by facet selection
where(lat < .th) # filter by slider
map(x => pick(x, {ts msg})) # MAP the passthrough, not just filter
}
```
## 13. Expect — stream reactions
A matching stream line fires an action (space-form args, per §2 — not paren calls):
```
expect /5\d\d/ { alert "5xx"; flash .log red } # regex → a block of actions
expect /panic|OOM/ beep # …or a single action
expect /down/ exec "notify-send arb"
timeout 5s alert "stream idle" # fire when no new line for 5s (Ns/Nms/Nm)
# actions: set .name V | quit | beep | alert MSG | flash .w COLOR | exec CMD
# | send "text" (to a spawn -pty child) | { … } (a block, run in order)
```
The block form groups several clauses under one `expect`:
```
expect {
/panic|OOM/ { alert "crash"; flash .log red } # each clause: /re/ ACTION
/5\d\d/ beep
/shutdown/ quit
}
```
`spawn CMD` (SPEC §7) launches a process whose stdout feeds the stream;
`spawn -pty CMD` runs it on a pseudo-terminal and a `send "text"` action writes
to its stdin, so an `expect { /re/ send "…" }` clause automates it (Expect).
## 14. Events — bind (Tk)
```
bind C-q quit # a control key → an action (any §13 action)
bind <Enter> quit # Tk named keys: <Enter> <Esc> <Tab> <Key-x>
bind <Click> beep # any mouse press → an action
bind <Resize> { alert resized } # terminal size change → an action
bind C-r { alert reloaded; beep } # block form
```
**Mouse** (SGR reporting, enabled on the TUI's `/dev/tty`): **left-click** a
control to focus/toggle it (checkbox, facet option), click-drag a `slider`,
click a `tabs` label to select it, click an fzf row to move the cursor
(**double-click** a row to pick it, like Enter). **Right-click** a control
resets it to its empty/default (slider→min, checkbox→off, text/facet→cleared).
**Middle-click** focuses without acting. The **wheel** scrolls: over a
scrollable widget (`tail`, `list`, `text`, `table`, `block`, `frame`) it banks
older rows and walks back toward the live tail; elsewhere it moves the focused
`facet` cursor; in fzf mode it moves the selection. The raw button byte also
carries **Shift/Alt/Ctrl** modifier bits (`mouse_shift`/`mouse_alt`/`mouse_ctrl`).
Everything is decoded from the raw tty byte stream and hit-tested against the
rendered widget rects. To copy text, hold **Shift** and drag for your terminal's
native char-precise selection (arb captures the mouse for its widgets).
**Selection accessor** — a `sel` widget (a per-widget selection list over its own
`source`) publishes its highlighted row as the control value `.<path>.sel`, so a
`ps aux | arb` dashboard with `sel .ps` + `source .ps { in }` exposes `.ps.sel` =
the current process row. Move the cursor with Up/Down (or click a row); the value
updates live and is readable anywhere a control is — `where match(.ps.sel)`,
`apply`, `tell w job(.ps.sel)`, or a `send` to a PTY child. ⬜ OSC-52 whole-widget
copy is deferred (line-granular, terminal-gated — Shift+drag is the better path).
## 15. Actors — Akka-style concurrency
An **actor** is a named behavior with a single scalar `state` and one handler per
message. Handlers run arb expressions (the same fusevm-lowered core as
`map`/`where`/`calc`) over `state`, the message parameters, and any locals they
assign; `reply EXPR` sends a value back to an `ask`/`via` caller.
```
actor worker(state) {
on job(x) {
state = state + 1 # per-worker running count
reply x * x # value sent back to ask / via
}
on reset { state = 0 }
}
```
Handlers are separated like every other verb — a newline or `;` (a `{ … }` block
does **not** end a command, so two handlers on one line is an error).
**Runtime** — one OS thread per actor, each blocking on an `mpsc` mailbox:
| Operation | Meaning |
| --- | --- |
| *spawn* | start an actor with an initial `state`; returns a mailbox ref |
| *send* (tell) | post a message, fire-and-forget |
| *ask* | post a message with a one-shot reply channel, block for `reply` |
| *pool* | a supervised round-robin pool of N copies; a worker whose thread dies is respawned on the next dispatch |
**Pipeline fan-out (`via`)** is the stream-facing consumer — it fans the passthrough
across a pool in parallel, each line's scalar becoming a `job(x)` ask whose reply
is the output line, order preserved:
```
source .out { in; via worker * 8 } # 8-worker pool
out { in; via worker } # default: one worker per hardware thread
```
A pure-map handler (`reply x * x`) is deterministic regardless of pool width. A
handler that mutates `state` is partitioned across workers, so with `N > 1` a line
sees only the state of the worker that handled it (documented non-determinism —
use `via worker * 1` for a single sequential accumulator).
**Session refs + event-driven messaging** — spawn long-lived actors/pools in the
spec and drive them from key/stream/timer events:
```
actor worker(state) { on job(x) { state = state + 1; reply x * x } }
spawn w = worker(0) # a session actor ref
pool p = worker * 8 # a supervised session pool
supervise p { on crash { stop } } # fail-stop (default is restart = respawn)
input .out # a control widget to show a reply
bind C-t tell w job(5) # tell: fire-and-forget on a keypress
bind C-a ask .out w job(.th) # ask: reply written into `.out`
expect /error/ tell w job(1) # drive an actor from a stream match
```
| Form | Meaning |
| --- | --- |
| `spawn NAME = ACTOR(init)` | a single session actor, initial `state` = `init` |
| `pool NAME = ACTOR * N` | a supervised N-worker pool (`* N` optional → one per thread) |
| `supervise NAME { on crash { restart \| stop } }` | crash policy; `restart` (default) respawns a dead worker, `stop` is fail-stop |
| `tell REF MSG(args)` | action: post a message, fire-and-forget |
| `ask .CTRL REF MSG(args)` | action: ask, write the reply into control `.CTRL` |
`args` are arb expressions evaluated when the action fires, with live control
substitution (`.th` → that input's current value). `spawn NAME = …` is
disambiguated from the `spawn CMD` process source by the `=`. The message tell is
`tell` (not `send`) because `send` is the Expect PTY action (§14). Session refs
run only in the interactive TUI; the served web target renders widgets but does
not fire bind/expect actions.
The runtime lives in `src/actor.rs`: declaration + handler compiler + one
`mpsc`-mailbox thread per actor (`spawn`/`send`/`ask`/`pool`, plus `Session` for
the named session refs). `via` is `QueryOp::Via`, evaluated by `actor::run_via`
(rayon fan-out over the pool). `heavy(x)`-style user functions are not part of
arb's expression grammar — handler bodies use arb expressions directly.
## 16. Targets
```
target tui # default
target web -port 8080 # served page + WS
theme neon-noir # one of 31 built-in palettes (arb --list-themes)
theme custom 201 231 93 219 57 53 # a custom 6-index (256-color) palette
set refresh 250 # ms redraw throttle
```
**Themes** — `theme NAME` sets the active color palette from the 31 built-ins
(the storageshower HUD palettes shared with the sibling `iftoprs`/`htoprs` apps:
`neon-sprawl`, `acid-rain`, `neon-noir`, `blade-runner`, `night-city`,
`megacorp`, `zaibatsu`, `iftopcolor`, … — `arb --list-themes` prints them with
swatches). Each is a 6-color palette `(primary, accent, alt, mid, dim, bg)` of
256-color terminal indices; `theme custom c1 c2 c3 c4 c5 c6` supplies your own.
With a theme active the whole TUI recolors from the palette as one system — a
widget with no `-color` takes a slot chosen by its kind (value gauges → accent,
bars → alt, series/plots → mid, text/containers → primary) so a dashboard is
multi-colored like the iftop/htop HUD rather than monochrome, and a themed fzf
picker maps the palette onto fzf's own colour slots (rows → primary, matches →
accent, cursor bar → bg, counter/header/separator → dim). `arb --fzf` starts from
**fzf's palette**, not the arb theme, unless a theme is asked for explicitly
(`--theme NAME`, a spec `theme` directive, or the live `Ctrl-T` chooser) — the
drop-in has to look like `fzf`. An explicit `-color <slot>` (`accent`/`primary`/`alt`/`mid`/`dim`/
`bg`) resolves through the palette too. The fixed semantic names (`-color green`
/`red`/…) remain theme-independent explicit overrides.
**A theme is always active by default** (matching the sibling `iftoprs`/`htoprs`
apps, which default to `neon-sprawl`), so every dashboard — including the stdlib
presets, which set no `-color` — is themed out of the box. Resolution precedence:
`--theme NAME` (per-run) → the spec's own `theme` directive → the `~/.arb/config.toml`
`[ui] theme` global default → the baked `neon-sprawl`. Set the global default with
**`arb --set-theme NAME`** (persists to the config), preview all 31 with
`arb --list-themes`, and opt out to the classic cyan look with `theme off` (or
`--theme off`). At runtime, **`Ctrl-T`** cycles the theme live in every mode
(dashboard, form, and the fzf picker) and saves the choice to `~/.arb` — a
control key, because a bare letter would be swallowed by the megafilter / a text
input. **`Ctrl-G`** toggles a help overlay of the global keys plus the spec's
`bind`s. The served web dashboard keeps zgui-core's own colorscheme picker.
Ships today as `arb --serve --port N`: a std-only HTTP server renders the same
spec as a live browser dashboard, pushing widget data over a WebSocket (hand-rolled
handshake, no dependency) with automatic fallback to polling `/data`. The page is
built from **[`zgui-core`](https://github.com/MenkeTechnologies/zgui-core)** — the
shared cyberpunk web-component toolkit — vendored as a git submodule at
`lib/zgui-core` and bundled into the binary at build time (`build.rs` →
`include_str!`). The page mounts `ZGui.appShell` (splash, filter bar, ⌘K palette,
settings/colorscheme) and renders each widget with the matching component:
`gauge`→`ZGui.gauge`, `chart`→`ZGui.chart`, `spark`→`ZGui.sparkline`,
`bars`/`histo`→`ZGui.statBars`, `table`→`ZGui.dataTable`, containers/log →
`ZGui.card`+`ZGui.logView`, fed live from `/data`. `input` widgets render as
editable fields that `POST /set?name=..&value=..` on change; the server holds a
live input store and re-resolves each widget's pipeline against it every frame,
so a typed field reshapes the browser dashboard just like the TUI megafilter.
## 17. Modules & presets (presets = stdlib script imports)
```
import http # stdlib or user module by name
import "./mylib.arb" # file
import gauges as g # namespaced
# resolution: local → ~/.arb/lib/NAME → installed pkg (~/.arb/pkg/NAME) → stdlib
```
- **Dashboard module (preset):** top-level widget/source/layout stmts. `arb -p http` == `import http`.
- **Component module (library):** exports `fn`s that build widget-groups: `g.cpu(.c)`.
Both are just `.arb` files. Compose: `import http; gauge .mine …; grid .mine -row 2`.
## 18. Package manager (ported from stryke `[ffi.exports]`/`load_cdylib` + znative ABI)
`arb.toml`:
```toml
[package]
name = "arb-k8s"
version = "0.1.0"
license = "MIT"
[deps]
arb-http = "0.2"
[exports] # ← stryke.toml [ffi.exports]
modules = ["pods", "nodes"] # .arb dashboards + components
[exports.native] # ← znative / load_cdylib
widgets = ["flamegraph"] # cdylib: new widgets / formats / actors
formats = ["protobuf"]
```
Kinds: **script packages** (pure `.arb`) and **native packages** (`cargo add arb-native`, ship cdylib, stable versioned ABI).
```sh
arb install arb-k8s arb add arb-http arb publish arb search k8s arb update
```
Distribution: native → crates.io (like fusevm/znative); script → git index + GitHub repos (like the stryke-* family).
**Ships today** (std-only, `git` subprocess as transport — no in-process TLS):
`arb update` clones/pulls the index repo into `~/.arb/registry`; `arb search Q`
greps its `index.json`; `arb install NAME` / `arb add NAME` `git clone`s the
package into `~/.arb/pkg/NAME` and validates its `arb.toml` + entry module before
keeping it (rolled back on failure); `arb uninstall NAME`. A package's `[deps]`
are resolved recursively from the same index, with each dep's version-constraint
**checked** against the index version (`semver`), a visited-set cycle guard,
skip-already-installed, and full rollback of the run if any dep fails or a
constraint is unsatisfiable. The module resolver reads `~/.arb/pkg` as the §17
`pkg` tier, so `import NAME` finds an installed package. **`arb publish
[GIT_URL]`** registers the package for real: it validates it, fast-forward-pulls
the index clone, upserts the package's `{repo, version, desc}` entry into
`index.json`, commits, and pushes to the index remote (default
[`github.com/MenkeTechnologies/arb-registry`](https://github.com/MenkeTechnologies/arb-registry)).
With write access the entry lands directly; when the push fails the commit stays
local and arb prints git's reason with the fork+PR flow -- it never falsely
claims a push succeeded, nor assumes every failure is a denied one.
`GIT_URL` defaults to the package repo's `origin` remote. A package declaring
`[exports.native]` is rejected (native/cdylib loading isn't built — never
installed with an inert native half). Full multi-version semver *resolution*
(one index ref per name today, like the crates.io index tip) and native/cdylib
packages remain future work.
## 19. Ecosystem — "a TUI for every pipeline"
Community publishes `arb-<tool>` packages. `cmd | arb` sniffs the upstream command (or data shape) → resolves the matching package → renders. Every common pipeline (docker/kubectl/psql/nginx/git/systemctl/…) gets a shared, installable dashboard. No registry of shareable pipeline TUIs exists today — this is the ecosystem leg.
**Ships today** — zero-config **data-shape sniffing**: `cmd | arb` (no spec, piped) peeks the first stream lines (via a non-blocking `poll`, so it never delays startup or hangs on an idle producer) and auto-selects the matching stdlib preset — JSON object streams → `json`/`logs`/`nginx`, tool headers → `docker`/`top`/`k8s`, git-log → `git`, CSV/TSV → `table` — replaying the peeked lines so nothing is lost, and falling back to the plain tail on no match. The **upstream-command** leg (identifying the producer process by argv) is deferred: the data-shape leg dominates it cross-platform (macOS pipe-peer matching needs fragile FFI), and covers every motivating producer via its header/shape.
## 20. Architecture (fusevm frontend, original — mechanics ported, semantics fresh)
Deps (lean): `fusevm` (`jit`, `jit-disk-cache`, `aot`, `ffi`), `ratatui`+`crossterm`, `clap`, `regex`, `onig` (Oniguruma, for jq's regex family), `rayon`; the served web dashboard is **std-only** (hand-rolled HTTP + RFC 6455 WebSocket, no async runtime) and renders with the vendored `zgui-core` toolkit (git submodule `lib/zgui-core`, bundled by `build.rs`); REPL: `reedline`+`nu-ansi-term`+`libc`+`toml`; parsers: `serde_json` + `saphyr-parser` (YAML, `src/yaml.rs`) + `toml` + `scraper` (HTML/CSS) + `base64`/`percent-encoding`.
Actual tree:
```
src/lexer.rs Tcl-flavored reader
src/parser.rs command + block grammar → AST
src/ast.rs AST types (Command / Arg)
src/spec.rs spec interpreter: widgets, source/out pipelines, query-verb
parse, import resolution, preset library
src/query.rs jq/xpath/css/yq engine (pipeline eval over every format)
src/jq.rs jq-literal front-end: translates `source`/`out` body stages to query ops
src/jqlang.rs complete jq language engine (parse_json, programs, builtins)
src/jqval.rs jq value expressions (`select(…)`/`map(…)` bodies, arithmetic stages)
src/jvparse.rs jq 1.8 JSON text parser port (fallback reader)
src/jsondocs.rs `in.json`: one item per JSON document
src/xpath.rs XPath front-end; xpath_syntax.rs (lexer/parser) + xpath_eval.rs (evaluator)
src/yaml.rs YAML reader (event stream → jq value model); ynode.rs node metadata + writer; yqfmt.rs yq encoders/decoders
src/algo.rs fzf matching algorithm port; pattern.rs fzf extended-query port; fzf.rs fzf option compatibility
src/sniff.rs zero-config data-shape sniffing
src/testrun.rs in-language test runner (`--test`)
src/tiers.rs fusevm execution-tier probe (`--tiers`)
src/pty.rs `spawn -pty` pseudo-terminal source
src/rust_ffi.rs inline `rust { … }` FFI blocks
src/hosted.rs running arb inside a host process (no `exit()`)
src/err.rs `SpecError` with source spans
src/actor.rs actor system (§15): actor/on/reply parse + handler compiler, mpsc-mailbox threads (spawn/send/ask/pool), `via` parallel stream fan-out
src/theme.rs 31 built-in color palettes (storageshower, shared with iftoprs/htoprs) + custom palette; theme-aware color resolution
src/expr.rs expression layer: fn/lambdas/operators → fusevm::Chunk on the VM
src/stream.rs stdin ring buffer + stream stats
src/tui.rs ratatui backend: render, event loop, fzf mode
src/serve.rs live web server + WebSocket push; renders via zgui-core (appShell + components)
src/web.rs static HTML snapshot export (--html)
build.rs bundle lib/zgui-core/webui/*.js + all.css -> one JS/CSS asset, embedded in serve.rs
lib/zgui-core/ git submodule: the shared cyberpunk web-component toolkit (window.ZGui.*)
src/repl.rs interactive REPL (--repl)
src/pkg.rs registry client (install/search/update/publish) over a git index
src/lsp.rs Language Server over stdio (--lsp): diagnostics/symbols/hover/completion/signatureHelp/definition/references/highlight/rename/folding/formatting/semanticTokens
src/dap.rs Debug Adapter over stdio (--dap): step the stream, regex breakpoints, inspect the paused line/stats/controls
src/cache.rs rkyv script cache (~/.arb/scripts.rkyv): outer zero-copy rkyv shard, inner bincode AST blob, FxHash+schema key — skips lex+parse for a seen spec
src/banner.rs startup/help art
src/main.rs binary entry point; calls `cli::run_argv`
src/cli.rs CLI (clap) + dispatch + fzf option handling + `--filter`
src/lib.rs crate root
```
The compute core (expressions, `calc`, `where`) lowers to a `fusevm::Chunk` and
runs on the VM; declarative widget/layout construction is plain Rust and needs no
VM. Language design (lexer/parser/ast/interp/semantics) is arb-original.
A chunk can execute on either of two fusevm tiers — the interpreter, or native
code from the Cranelift block JIT once the chunk has been invoked past
`block_threshold` (default 1). `scripts/expr_paths.sh` runs the expression
corpus through both, pinned via `FUSEVM_JIT_BLOCK_THRESHOLD`, because a
construct the tiers disagree about answers differently for the first row of a
stream than for the rest. One such disagreement is open and is fusevm's, not
arb's: `Op::Div` by a zero divisor yields `Value::Undef` (prints `0`) on the
interpreter and IEEE infinity in compiled code, so `map x / 0` over a stream
prints `0` and then `1.7976931348623157e+308` (the infinity, clamped on output
as above), and `where x / 0 > 1` keeps a different subset of identical lines. arb
does not paper over it locally — the op is shared with every sibling frontend.
The clamp does not hide it either: `Value::Undef` never reaches the non-finite
branch, so clamping moves only the compiled side and the two tiers still differ.
All SPEC modules now have code (script-package registry + the actor system
included; native/cdylib packages remain future work).
## 21. Milestones
Status: ✅ shipped · 🟡 partial · ⬜ planned · ❌ out of scope.
0. ✅ **Walking skeleton** — `echo hi | arb -e 'text .t <- in'`: lex→parse→lower→fusevm→one ratatui widget from stdin.
1. ✅ Core widgets + auto-layout + `source`/query basics.
2. ✅ Presets/imports + stdlib (logs/http/json/table/top/metrics) + module namespacing `import X as Y` (prefixes widget paths, `apply`, control refs, `set`/`flash` targets).
3. ✅ Interactive controls + `out` passthrough shaping (megafilter/map): `input`/`apply`, the `filter`/`facet`/`slider`/`check` control widgets (interactive in both the TUI and the served web dashboard, incl. dynamic `-field` facet candidates), and control-path predicates — numeric `where lat < .th`, string `where match(.q)`, and set `where level in .lv`.
4. ✅ Expect reactions + events/bind — `expect /re/ ACTION` and the multi-clause `expect { /re/ ACTION; … }` block, `bind C-<key> ACTION` with actions `set`/`quit`/`beep`/`alert`/`flash`/`exec` and `{ … }` block form; Tk named keys `<Enter>`/`<Esc>`/`<Tab>`/`<Key-x>`; `timeout Ns ACTION` idle reactions; `spawn CMD` process input source, `spawn -pty CMD` + the `send "text"` action (Expect-style automation of a PTY child). The `sel` selection widget publishes a widget's highlighted row as `.<path>.sel` for `where`/`apply`/`tell`/`send` to consume (§9).
5. ✅ Web target — `arb --serve` HTTP + WebSocket live dashboard rendered with the `zgui-core` component toolkit (appShell + per-widget components); `arb --html` static export.
6. ✅ Actors — Akka-style message-passing (§15): `actor NAME(state) { on MSG(p) { … reply EXPR } }` declarations compiled to `ActorDef`; a real runtime of one `mpsc`-mailbox OS thread per actor with *spawn* / *send* (tell) / *ask* (await reply) / supervised round-robin *pool* (respawns a dead worker); handler bodies run arb expressions (fusevm) over `state` + params + locals. Two surfaces: the `via NAME * N` pipeline op fans the stream across a pool in parallel (rayon), order preserved; and the session-ref surface — top-level `spawn NAME = ACTOR(init)` / `pool NAME = ACTOR * N` bindings, a `supervise NAME { on crash { restart \| stop } }` crash policy, and the `tell REF MSG(args)` / `ask .CTRL REF MSG(args)` bind/expect actions that drive them (interactive TUI).
7. ✅ Package manager — local preset library (`--save`/`--install`/`--uninstall`/`--installed`) + a networked registry over a git index hosted on GitHub (`arb update`/`search`/`install`/`add`/`uninstall`/`publish`, `~/.arb/pkg` resolver tier, transitive `[deps]` with semver constraint-checking). `arb publish` upserts the package's entry into the index and pushes it (default registry `github.com/MenkeTechnologies/arb-registry`). *(native/cdylib packages + multi-version semver resolution: ⬜)*
8. ✅ LSP/DAP — `arb --lsp` ships a full server: diagnostics (real source ranges, UTF-16 columns), `documentSymbol`, `hover`, `completion` (CORPUS verbs + dot-context `.path` names + widget `-flags`), `signatureHelp`, `definition`/`references`/`documentHighlight`/`rename` over widget `.path` names, `foldingRange`, `formatting`, and `semanticTokens/full`. `arb --dap` is a real steppable debugger over the stream model: each incoming line is a step, breakpoints are regex predicates (a `SourceBreakpoint.condition`, or unconditional = single-step), function breakpoints (`setFunctionBreakpoints`) name a query VERB and stop when the paused line reaches that stage — a name arb has no verb for comes back `verified: false` rather than being silently accepted — the stack trace is the query-pipeline stages, scopes expose the matched line + stream stats + control values, and `evaluate` runs arb's real expression evaluator against the paused line. The `program` (spec) and `input` (data file) come from the `launch` request since DAP owns stdio; `stepIn`/`stepOut` collapse to `next` (a stream has no call nesting). Diagnostics anchor to the offending verb even when nested inside a `source`/`out` body (not the enclosing directive). *(per-token argument precision — squiggle the `/re/` itself, not its verb — ⬜)*