Fastmash summarizes, groups and reshapes tab-separated text and quoted CSV from the command line. It accepts the command language of GNU datamash, so most existing scripts can switch by changing one word.
$ printf 'site\treading\nwest\t2\neast\t4\nwest\t6\n' | fastmash -H -s -g site sum reading mean reading
GroupBy(site) sum(reading) mean(reading)
east 4 4
west 8 4
Why Fastmash
- Fast. On representative real-world jobs Fastmash runs up to four times faster than GNU datamash 1.9, for example quantiles on genome annotations and grouped summaries per gene. Every release publishes its full results, including the jobs where GNU datamash is still faster: see the benchmarks.
- The datamash commands you already use. Over 70 operations and modes,
datamash's options and field selectors, grouping, headers,
crosstab,transposeand more. - Useful table workflows. Read and write quoted CSV, name result columns, calculate weighted means, select the highest or lowest complete records, compare two datasets and inspect or validate table health.
- The same answer everywhere. Arithmetic is implemented in software, so the same Fastmash version gives identical results on every supported machine, whatever its CPU or math library.
- Refuses rather than guesses. Where GNU datamash 1.9 can crash, truncate or print undefined values, Fastmash reports a clear error or a refusal (exit status 77). Memory and disk use grow with the job and are checked.
Install
Fastmash runs on Linux x86-64 (glibc) and on Windows through WSL2.
Download a prebuilt binary from the releases page, or build it with Cargo:
See the installation guide for
requirements, verification and removal. Installing Fastmash does not replace or
alter an installed datamash.
Quick start
# Sum and mean of columns 1 and 2
|
# 4 3
# Group by column 1 after sorting, then count and take the median of column 2
# Quoted CSV, named fields and a stable output label
# Weighted mean and the highest five complete records
# Inspect a table, or compare numerical summaries of two exports
# Per-row transformations
--csv reads and writes strict quoted CSV; --csv-in and --csv-out select
each direction separately. -t, retains literal comma splitting for existing
datamash commands. The user guide covers every
operation, option and mode, including their format and resource limits.
Compatibility
Fastmash targets the command language and results of GNU datamash 1.9. Where it differs, the difference is intentional and listed in Differences from GNU datamash. The most visible ones:
- Fastmash's numerical results come from its own portable arithmetic. The final digit or the sign of a NaN can occasionally differ from a local GNU build.
- Inputs that crash GNU datamash 1.9 or give it undefined behavior are errors or refusals in Fastmash.
--sort-cmdand the explicitlinemode are not provided.
Quoted CSV, custom result names, weighted mean, Top-N selection, dataset
comparison and table health are Fastmash extensions. Help and readable health
reports use terminal color automatically; --no-color suppresses it. Calculation
results, CSV, health TSV and version output always keep plain data bytes.
Status
Fastmash is at 0.1, a preview release: usable today on Linux x86-64 and WSL2, with the differences above. Before 1.0, minor releases may still change behavior; every change is listed in the changelog. macOS on Apple Silicon is a later target. Windows users run the Linux version through WSL2; native Windows is outside the current product scope. See the roadmap.
Documentation
- User guide: operations, grouping, headers, numbers and output, errors and exit statuses
- Architecture: how Fastmash is built
- Testing: how Fastmash is verified
- Glossary: the project's vocabulary
Contributing
Bug reports, compatibility findings and benchmarks from real workloads are especially welcome. Read CONTRIBUTING.md to get started, and SECURITY.md to report a vulnerability.
License
Licensed under either of Apache License, Version 2.0 or MIT license, at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Fastmash by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Fastmash is an independent implementation and is not affiliated with the GNU Project. It contains no GNU datamash code; its behavior was matched by studying GNU datamash's documentation, source code and observed output.