1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
;;; tensor_bench.lisp — the Phase 3.1/3.3 tensor training benchmark.
;;; Timings, not golden output — do NOT add to run_tests.sh.
;;;
;;; Re-records the numbers ROADMAP 3.1 and 3.3 quote, all from ONE machine state:
;;; 8x16->8 x1000 SGD steps — interpreted graph-grad vs compiled graph-compile-grad
;;; 64x256->64 x100 SGD steps — same two paths
;;; Run benchmarks/tensor_torch_bench.py for the 1-thread float64 PyTorch side;
;;; it uses the same shapes, step counts, LR and inits, and the two sides should
;;; land on the SAME final loss (that agreement is what proves the loops match).
;;;
;;; Why absolutes drift: these are wall-clock on whatever machine/thermal state
;;; you run them on. The RATIO is the durable claim — see ROADMAP 3.1.
;;;
;;; JIT marshalling, cut in two passes. Each A/B'd on one machine state with
;;; this file, medians of 3; every step verified bit-identical to graph-grad
;;; across 5 shapes / 37,353 values.
;;;
;;; v0.43.0 — the generated kernel stopped copying: it borrows its input tensors
;;; (was `.to_vec()` per input, ~295 KB/call at 64x256->64) and borrows on the
;;; SumTo same-shape identity (was a clone, ~164 KB/call).
;;; 64x256->64 x100 JIT 143.3 ms -> 130.6 ms (-8.9%)
;;; 8x16->8 x1000 JIT 6.18 ms -> 5.78 ms (-6.5%)
;;; Narrowed the gap to the interpreter but did NOT flip it (~5% slower still).
;;;
;;; v0.44.0 — per-argument-pointer ABI: the caller stopped copying too (was one
;;; flat buffer in + one out, ~885 KB/call at 64x256->64).
;;; 64x256->64 x100 JIT 127.5 ms -> 114.6 ms (-10.1%)
;;; 8x16->8 x1000 JIT ~5.78 ms -> ~5.75 ms (flat — little to marshal)
;;; THIS FLIPPED IT: at 64x256->64 the JIT now BEATS graph-grad (~1.14x), where
;;; it had been ~1.12x slower. Cumulative 143.3 -> 114.6 ms (-20%).
;;;
;;; graph-grad (interpreted) is untouched by both — the medium interpreted row
;;; is noisy (~117-134 ms run to run), so compare JIT numbers, which are stable.
;;;
;;; Loss = mean((relu(xW+b) - t)^2). The graph IR has no capture, so everything
;;; the loss needs is a param or a literal; the mean divisor is passed as `nn`.
;;; Inputs are integer-derived /8 — exactly representable, so Rusty and torch
;;; start from bit-identical data.
;; One SGD run. `g` is whatever computes (loss gx gW gb gnn) — graph-grad itself
;; (interpreted: rebuilds+optimizes the graph every call) or a compiled
;; NativeGrad from graph-compile-grad. Identical loop either way.
;; median of 3 without a general sort: total minus the extremes
;; median of REPS timed runs, so a single thermal blip can't set the record