# Evaluator measurements, 2026-10-14
This report compares two fixed evaluator revisions on a set of local workloads.
It records both the small-query regressions and the gains from sharing larger
values. It is a historical comparison, not a benchmark of the current release.
For the execution model and advice on measuring your application, see
[execution and costs](computation.md).
## Measurement method
The 2026-10-14 experiment compares the original evaluator at
`948ded4ff5157f84894129be7fdc4a4fe8cbc338` (tree
`a0ba5ad12ca4c4c0df9490b30b07dfa74bc463ab`) with the prepared evaluator at
`1d4767277d3cf7eadec4e27079aa9a570ebd9af2`. The tables call these variants
"original" and "prepared". They describe those fixed revisions, rather than
tracking the current branch. Both host guides were exercised against their
actual core pin `a278c32cace3200d496a3c0649f4cf069996e2ad`, whose product source
matches the measured prepared evaluator. Zincite source is
`dd426038ece5825fdbad80517d11a978870ea681`; Zirium source is
`fb3d0c516d0dc01b85d2a9d3070107debac6739f`.
Rust 1.98.1 release builds ran on an Apple M1 Max with 10 cores, 64 GiB and
arm64 macOS 26.6.2. One serial shuffled schedule, seed 29, contains 34 cases,
two variants per case, one separate warmup child and five measured child batches.
The 408 records contain 68 warmups, 295 measured successes and 45 measured
refusals, with no target execution failures or unfinished records. Statistics
are median (minimum–maximum) across five batches, normalized per request.
They are not thousands of independent samples. No corpus or solver was run.
Each compared pair uses matching System allocator observation with atomic
allocation, live-byte and high-water counters. The sorting driver additionally
records requested bytes. These counters add overhead in both the original and
prepared variants. These are library timings with instrumentation enabled. The
full ratio cannot be attributed to evaluator code alone. Additional
preparation/copy/frame probes run in separate prepared-source variants. Their
overhead is retained separately and is not subtracted from phase times.
The original workload cases time parsing, public preflight and evaluation, and
JSON conversion, serialization and cleanup separately. Startup, fixed fixtures
and later AST/output teardown are excluded. Binding trees are built before
timing. Exact-child `wait4` peak RSS is reported in macOS bytes converted to
MiB; it includes startup and fixtures and is not phase-specific live heap. The
small/projection fixture includes an unused pure binding. RSS alone establishes
neither reclamation nor absence of leaks.
### Original workloads
The original workloads exercise small reports, projection and repeated reads of
a saved binding. All seven completed pairs have identical outputs. The
projection retains sixteen outer names and produces three integer fields per
row. Bindings contain 128-byte strings and are read eight times. Allowances
remain 1,000,000 work, 100,000 results, 100,000 stream items and nesting 64.
| small, 1 | 1.825 (1.817–1.841) | 4.631 (4.587–4.664) | 2.538 |
| projection, 10 | 79.043 (78.869–80.790) | 49.567 (48.938–49.845) | 0.627 |
| projection, 100 | 648.237 (645.087–673.704) | 306.550 (302.408–322.208) | 0.473 |
| projection, 1000 | 6465.854 (6403.521–6561.333) | 3007.396 (2927.479–3111.521) | 0.465 |
| binding, 8 | 7.929 (7.882–8.063) | 9.356 (9.201–9.821) | 1.180 |
| binding, 64 | 35.375 (34.985–35.871) | 11.152 (11.064–11.330) | 0.315 |
| binding, 256 | 131.221 (130.600–132.433) | 17.927 (17.799–18.107) | 0.137 |
The prepared evaluator takes 2.538 times as long for the small report and 1.180
times as long for the smallest binding case. Larger inputs benefit from sharing,
but those gains do not remove the small-query overhead. The measured small
report is `{"answer": 7 * 6, "count": length([1, 2, 3])}`; the smallest binding
case reads eight 128-byte strings eight times. All paired cases keep the same
allowances.
| small, 1 | 2.239 / 0.584 | 2.300 / 0.582 | 2.297 (2.297–2.359) | 2.719 (2.703–2.734) |
| projection, 10 | 2.847 / 6.212 | 2.834 / 6.132 | 2.578 (2.562–2.609) | 2.953 (2.938–2.969) |
| projection, 100 | 4.606 / 59.700 | 3.913 / 58.658 | 2.766 (2.719–2.797) | 3.141 (3.109–3.156) |
| projection, 1000 | 15.396 / 647.688 | 14.021 / 615.396 | 4.438 (4.219–4.453) | 4.797 (4.750–4.891) |
| binding, 8 | n/a / 0.610 | n/a / 0.569 | 2.141 (2.141–2.156) | 2.453 (2.438–2.484) |
| binding, 64 | n/a / 0.690 | n/a / 0.564 | 2.188 (2.172–2.188) | 2.484 (2.453–2.516) |
| binding, 256 | n/a / 0.857 | n/a / 0.588 | 2.281 (2.266–2.312) | 2.547 (2.531–2.562) |
The raw records retain the spread for every phase. The companion 8,192-string
binding request refuses on the original work allowance and completes on the
prepared runtime with value `[8192,8192,8192,8192,8192,8192,8192,8192]`,
spending 49,556 work units. Sharing changes cumulative spending near old
thresholds. This status difference is excluded from successful paired ratios; a
refusal duration is not successful throughput.
### Recursion and repeated arguments
These typed-builder fixtures are constructed before timing. Evaluation includes
fresh Budget setup and JSON result-summary construction. Parse and render timers
are unavailable for these cases; recorded zeros must not be read as free parsing
or rendering. Function cases retain base-033 allowances: 100,000,000 work,
10,000,000 results, nesting 64, stream items 100,000; tail/report active depth is
4, and non-tail active depth is 64. Branching/infinite controls instead retain
their 20,000-work refusal allowance.
| tail, 100 | 297.029 (292.010–299.017) | 2.266 (2.266–2.281) |
| tail, 1000 | 2869.033 (2853.142–2883.892) | 2.172 (2.156–2.172) |
| tail, 10000 | 29498.625 (29356.084–30351.917) | 2.156 (2.141–2.156) |
| tail, 50000 | 147004.000 (146690.834–147537.791) | 2.156 (2.141–2.188) |
| unchanged, 100 | 297.454 (289.121–301.921) | 2.750 (2.641–2.766) |
| unchanged, 1000 | 2531.438 (2509.396–2617.271) | 2.578 (2.578–2.578) |
| unchanged, 10000 | 25239.834 (24953.791–25424.417) | 2.562 (2.562–2.594) |
| depth, 8 | 36.758 (35.846–39.842) | 2.266 (2.250–2.281) |
| depth, 32 | 110.658 (104.842–111.167) | 2.297 (2.281–2.297) |
| native, 100 | 771.959 (735.000–813.083) | 4.016 (4.000–4.047) |
| native, 1000 | 3963.458 (3901.208–4079.750) | 4.047 (4.000–4.078) |
| native, 10000 | 35814.292 (35609.875–36185.458) | 4.031 (4.000–4.047) |
The accumulator returns `n*(n+1)/2`. The unchanged-argument fixture passes a
record with 1,024 strings of 256 bytes plus a scalar size field through every
tail call, reading that size through a lexical local. Across all measured
lengths, temporary probes record one body preparation per request, sixteen
prepared nodes for the accumulator and fifteen for the report. Both copy zero
string payload bytes and zero outer-environment entries. The report has one
whole-record admission per request. Repeated reads still spend work.
Accumulator live peaks are two frames including the entry, eight continuations,
six runtime values and three local slots at 100 through 50,000 steps. Report
peaks are two/eight/five/three at 100 through 10,000 steps. Interpreter gauges
return to zero after success and refusal. Heap high-water above retained fixtures
is at most 4,384 bytes for the accumulator and 52,080 bytes for the report.
A batch's previous result-summary overlaps the next request by 656 bytes;
peaks are raw maxima, never divided by batch size. Cumulative allocation counts
still grow with executed work. These observations establish bounded retained
state for these cases, not constant total allocation or constant time.
Non-tail n=8 and n=32 peak at ten and thirty-four frames including entry,
with heap peaks 14,624 and 51,680 bytes. At n=128 the explicit active-depth
limit refuses with 65 total frames, 325 continuations and a 100,660-byte heap
peak. The public error category is `Other`, message
`active function depth limit exceeded`, with sixteen retained call locations
and forty-eight omitted. A later scalar operation on the same spent budget
still succeeds. Branching and infinite tail recursion separately refuse with
`WorkLimit`; a wrong result type refuses with `expected integer`. None are
included in the successful recursion table.
The native fixture retains the same public API checks used in the earlier
function experiment: nine requests, twelve body preparations and sixty-four
prepared nodes, independent of tail length. It exercises native projection,
typed native arguments, two contract refusals, native failure, bounded host
reentry and borrowed-source identity. The native timer includes the complete
fixture and assertions. Its fixed 40,104-byte heap high-water and zero
post-fixture live allocated bytes hold at 100, 1,000 and 10,000 steps. Six
copied string bytes belong to the two intentional wrong-default literals.
Live-state peaks are dominated by bounded reentry and all gauges return to zero.
This gives direct release evidence beyond RSS.
### Wide reports and stable sorting
Narrow rows contain id and integer score; wide rows add sixteen 256-byte strings
and a 283-byte common-prefix string score in the final field. Scores follow
`(id*37+11)%97`. Every result id sequence is checked against independently
stable-sorted ids; descending ties and first-input extrema are checked too.
The full `evaluate_value` path includes preflight, preparation and owned output;
fixture construction and result verification are outside timing. Allowances
remain those of base-036: 20,000,000,000 work, 10,000,000 results, nesting 64 and
100,000 stream items. These larger fixed allowances describe a separate wide
workload; they are not substituted into the original baseline comparisons.
| narrow_sort, 128 | 154.875 (148.167–167.167) | 2.188 (2.172–2.188) |
| narrow_sort, 1024 | 936.042 (891.959–983.542) | 3.266 (3.266–3.391) |
| narrow_sort, 8192 | 7164.625 (6953.459–7242.750) | 11.750 (11.750–12.109) |
| wide_input, 128 | 938.334 (907.125–958.417) | 4.406 (4.406–4.422) |
| wide_input, 1024 | 7150.000 (7020.625–7276.833) | 20.406 (20.391–20.500) |
| wide_sort, 128 | 1295.834 (1290.458–1324.666) | 4.406 (4.406–4.422) |
| wide_sort, 1024 | 11046.125 (10981.917–11309.708) | 20.500 (20.422–20.547) |
| wide_descending, 1024 | 11178.334 (10796.459–11289.542) | 20.453 (20.422–20.547) |
| wide_min, 1024 | 6411.334 (6278.333–6742.833) | 14.312 (14.297–14.359) |
| wide_max, 1024 | 6388.792 (6340.167–6455.000) | 14.344 (14.297–14.344) |
At 1,024 wide rows, requested heap high-water above the fixture is 8,013,748
bytes for the input-copy control, 8,055,476 for sorting and 3,111,936 for either
extremum. Sorting's required owned output remains a substantial cost. The
prepared evaluator's key probes record exactly one extraction per row. The
temporary comparison probe was attached to a different comparator and yielded no
usable count; zero in that raw field does not mean zero sorting comparisons. The
prepared driver's copy-node fields in the adapted sorting driver are unavailable
placeholders. No claim uses either field.
The historical base-036 run remains separately attributed to its saved source:
90 samples at 128, 1,024 and 8,192 rows. At 8,192 rows it observed 97,768
ascending or 97,744 descending comparisons and 8,191 extrema comparisons. The
prepared evaluator retains those ordering algorithms unchanged. Those old
observations are not relabeled as samples from the prepared evaluator or
combined with its statistics.
<span id="reproduce-and-interpret"></span>
## Evidence and limitations
The source snapshots, benchmark drivers, build logs and raw samples are kept in
a private development bundle. It also records source and binary hashes, the
exact configuration and the shuffled schedule. That bundle is not included in
this repository, so this page alone is not a complete reproduction package. The
source revisions and measurement boundaries above identify what was compared.
Earlier experiments remain separate records.
The original collector expected a `DepthLimit` category for the n=128 control;
the API reports `Other` with its explicit depth message. All twelve raw captures
(two warmups and ten measured refusals) and their original `invalid_output`
diagnosis are retained. Derived analysis validates the exact category, message,
bounded trace and successful follow-up before classifying them as refusals.
It changes no duration and performs no recollection. Original collector and
corrected analyzer versions are both retained.
Five batches on one machine provide a bounded local comparison, not a workload
survey or portable latency prediction. The allocator observer, different fixture
shapes, selected allowances and omitted host setup/rendering limit
extrapolation. The experiment has no per-phase live-memory measure and no
clean-allocator production timing. Temporary instrumentation is absent from
product source. The comparison records small-query regressions at the measured
revisions. Its bounded tail-call storage and ownership results do not establish
that every program is efficient.