tak-cli 0.0.2

[!CAUTION]

🚧 GO AWAY 🚧

This is a half-baked experiment. Do not use it. Do not depend on it. Do not package it.

It is vibe-coded slop that exists so one person can find out whether an idea is any good. It is not a product, it has no support, no stability guarantees, and no roadmap you should trust. Interfaces will change without warning. It may be renamed, rewritten from scratch, or deleted outright. If it eats your afternoon, that is on you — you were warned in a very large box at the top of the README.

Please do not file issues or open pull requests. They will most likely be closed unread. There is nothing here to contribute to yet.

If you actually want to benchmark something

Use one of these instead. They are real, finished, maintained software written by people who care about your use case:

Seriously. Go use hyperfine.

Still reading?

Fine. Here is the idea, so you can decide it's uninteresting and leave.

Premise: wall-clock time on a shared CI runner has a noise floor of roughly 10–20%, which is the same size as the regressions people want to catch. Every threshold-based CI benchmark therefore either cries wolf or sees nothing, and eventually gets muted.

Measured on a 32-core Linux box, six mise startup commands, comparing coefficient of variation across metrics:

Under 3.2× CPU oversubscription, wall-clock medians moved about 150%. Instruction counts moved 0.035%. So: gate CI on instruction counts, report wall clock without gating.

Re-measured through tak itself, benchmarking /bin/ls /usr in a container, quiet versus 32 spinning cores:

tests/counters.rs asserts that determinism on every CI run, and skips cleanly where valgrind is unavailable.

Corollary that cost me an afternoon to learn: syscall count and max RSS are not deterministic — they move with thread scheduling. Only instruction counts support a tight gate.

Second premise: the data belongs in the repository. tak stores results in refs/notes/tak, one JSON object per line, merged with git's cat_sort_uniq strategy so concurrent CI writers never conflict. No database, no account, no service.

This has a genuinely nice property. The notes tree is keyed by commit SHA as path names, so it does not reference the annotated commits at all — which means a single shallow fetch of one ref returns the entire history without cloning the repository:

$ git fetch --depth 1 origin '+refs/notes/tak:refs/notes/tak'
fetch of ONLY refs/notes/tak --depth 1:  36ms
refs present:                            1
project commit objects fetched:          0
notes cover:                             100 commits
on-disk size:                            124K

100 commits × 6 benchmarks packed to 124K. Ten thousand commits projects to roughly 6 MB.

Status

Measuring and storing work. Nothing reads the data back yet.

• tak run — wall clock (min / p50 / mean / max), no shell

• instruction counts via cachegrind, degrading to timing-only where absent

• refs/notes/tak storage — run --record, push, history, init

• concurrent CI writers merge via cat_sort_uniq

• tak backfill — benchmark published release binaries to bootstrap history

• asset selection via crates/asset-picker, extracted from mise

• tak.toml — declare benchmarks so CI and a laptop measure the same thing

• tak compare <ref> — interleaved A/B against another revision

• PR reporting

• change-point detection instead of thresholds

Releases are automated with release-plz and publish to crates.io via trusted publishing — see RELEASING.md. The crate is tak-cli; the binary is tak.

Declaring benchmarks

tak run with no command runs whatever tak.toml declares, searching upward from the working directory. The point is that CI and a laptop measure the same thing — a command line in a workflow file drifts from the one people run locally, and the numbers stop being comparable without anyone noticing.

[bench.startup]
cmd = ["./target/release/mycli", "--version"]

[bench.help]
cmd = "mycli --help"    # split on whitespace; there is no shell
runs = 10

tak run --bench startup runs one of them. A command after -- always wins, so ad-hoc measurement never depends on repository state.

Counters on macOS and Windows

There is no usable cachegrind on Apple Silicon and none on Windows, so tak run there records timing only and says so. Run it in a container to get the deterministic metric on any host:

docker build -f docker/Dockerfile -t tak .
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/w" -w /w \
  tak run --bench startup -- ./mycli --help

The build context is the repository root, not docker/, because the build stage copies Cargo.toml and src/. And --user is not decoration: tak spawns the thing you are measuring, so the benchmark subject inherits the container's privileges over the mounted working tree.

The CI gate lives on the Linux job regardless, so this only matters locally.

Prior art this steals from

• hyperfine — the UX everyone knows. Its own 2.0 roadmap asks for most of what's above.
• poop — hardware counters, no shell, first command as reference.
• mise — crates/asset-picker is mise's release-asset selection logic extracted verbatim, tests and all. Picking the wrong asset yields a binary that runs and reports numbers that look real, so a naive heuristic here is worse than none.
• git-appraise — one JSON object per line in git notes, merged with cat_sort_uniq. The storage design is lifted from here wholesale.
• git-perf — independently arrived at git-notes storage for performance data. Validates the approach; does not measure anything itself.
• Chronologer — benchmarking backwards across git history. Best idea in the field, sitting unused.
• Nyrkiö / Hunter — E-divisive change point detection, proven at DataStax, TigerBeetle, Confluent.

License

MIT