Summarize, group and transform fields from standard input, or compare two files.
Usage: fastmash [OPTIONS] OPERATION FIELDS [OPERATION FIELDS ...]
fastmash [OPTIONS] top:N FIELD | bottom:N FIELD
fastmash [OPTIONS] compare BEFORE AFTER [RANKING] OPERATION FIELDS [...]
fastmash [OPTIONS] health [CONTROLS]
fastmash [OPTIONS] MODE [FIELDS]
Fastmash supports Linux, Windows through WSL2 and macOS 15 or later.
Examples:
Sum the first field (prints 60):
printf '10\n20\n30\n' | fastmash sum 1
Combine scattered groups and sum the second field (prints a<TAB>1, b<TAB>5):
printf 'b\t2\na\t1\nb\t3\n' | LC_ALL=C fastmash -s -g 1 sum 2
Fields and grouping:
Fields are numbered from 1. Use lists (1,3), ranges (2-4), or input-header
names with --header-in or -H. Names are exact and case-sensitive.
Backslash escapes one byte in a name. The first duplicate name wins.
Paired operations use LEFT:RIGHT, such as pcov 1:2 or wmean price:units.
Weighted mean example (prints 7):
printf '10\t2\n5\t3\n' | fastmash wmean 1:2
Lists of pairs are allowed, but ranges cannot be combined with pairs.
Operations and results follow request order. Operation names ignore case.
-g FIELDS summarizes adjacent records with equal keys.
Add -s when equal keys are scattered through the input.
Grouping also accepts groupby, grouping or gb before its keys and operations.
Options:
-g, --group=FIELDS Summarize adjacent groups with these keys
-s, --sort Sort stably by keys before grouping or duplicate removal
-i, --ignore-case Ignore ASCII case in grouping, sorting and text summaries
--header-in Use the first input record as field names
--header-out Request result labels
-H, --headers Same as --header-in --header-out
--result-name=INDEX:NAME Replace a result label, counting results from 1
--csv-in Read quoted CSV for calculations, selection or comparison
--csv-out Write quoted CSV for calculations, selection or comparison
--csv Same as --csv-in --csv-out
-t BYTE, --field-separator=BYTE
Split ordinary text at one literal byte
-W, --whitespace Split ordinary text on runs of spaces and tabs
--output-delimiter=BYTE Separate text output fields with this byte
-c BYTE, --collapse-delimiter=BYTE
Separate unique/collapse list values (default comma)
-z, --zero-terminated End records with NUL instead of newline
-C, --skip-comments Skip text lines starting with # or ; after spaces/tabs
--narm Skip whole-field NA, N/A and NaN tokens, ignoring case
--vnlog Read annotated whitespace records with named fields
-f, --full Copy a representative record before calculated results
--no-strict Allow ragged transpose/reverse input
-F TEXT, --filler=TEXT Fill missing transpose/crosstab cells (default N/A)
--format=FORMAT Format numbers with one %e/%f/%g/%a conversion
-R, --round=N Display N fractional places (1..50)
-S VALUE, --seed VALUE Set rand's reproducible seed
--color=auto|always|never
Set terminal presentation (default auto)
--no-color Same as --color=never
-h, --help Show this help without reading input
-V, --version Show the package version without reading input
-- End option processing
Input and output:
Input defaults to tab-separated fields and newline-terminated records.
Use -t, for literal comma splitting, or --csv-in for quoted CSV.
Separators take one literal byte. Backslash escapes such as '\t' are not
expanded. Omit -t for tab-separated input.
--csv reads and writes CSV. These switches do not infer headers.
Example with a header and a named field (prints sum(revenue), then 35):
printf 'region,revenue\nwest,20\neast,15\n' | fastmash --csv -H sum revenue
To name that result total, add --result-name=1:total.
CSV supports calculations, Top-N selection and comparison, including their
supported grouped workflows. Table health and other modes use ordinary text.
CSV input conflicts with -t, -W and -C. Either CSV direction conflicts with
-z and --vnlog. CSV output also conflicts with --output-delimiter.
Delimited-text output to files or pipes uses the input separator by default.
With -W, text output uses tabs. CSV output uses commas and newline endings.
Result names require --header-out or -H. INDEX counts expanded results,
excluding keys and --full copies. Selection copies fields without result names.
--format and --round change displayed numbers, not calculations.
Sum first, then display two decimal places (prints 3.60):
printf '1.2\n2.4\n' | LC_ALL=C fastmash --round=2 sum 1
Use --format=%.0f to display no decimal places.
--narm does not trim spaces or supply missing fields. For wmean it omits a
whole pair after validating its nonmissing partners. Other paired statistics
remove missing values independently on each side.
Details: https://fastmash.io/guide/input
https://fastmash.io/guide/csv-and-result-names
Operations:
count Count values, including text and empty fields
countunique Count distinct text values
unique, uniq Sorted distinct text values
collapse All text values in input order, including duplicates
first, last First or last accepted value
rand Random accepted value
min, max Minimum or maximum numeric value
absmin Value with the smallest magnitude, keeping its sign
absmax Value with the largest magnitude, keeping its sign
range Maximum minus minimum
sum Sum of values
mean Arithmetic mean
wmean Weighted mean of a VALUE:WEIGHT field pair
geomean Geometric mean
harmmean Harmonic mean
ms Mean of squared values
rms Square root of the mean of squared values
median Middle sorted value, averaging the middle pair for even samples
mode Value in the longest sorted run (first run wins ties)
antimode Value in the shortest sorted run (first run wins ties)
q1, q3 First or third quartile using type-7 interpolation
iqr Third quartile minus first quartile
perc[:N] Percentile (default 95, integer 1..100, type-7 interpolation)
trimmean[:P] Mean after trimming fraction P from each end (default 0, 0..0.5)
pvar, svar Population or sample variance
pstdev Population standard deviation
sstdev Sample standard deviation
madraw Median absolute deviation without scaling
mad Median absolute deviation scaled by 1.4826
pskew, sskew Population or sample skewness
pkurt, skurt Population or sample excess kurtosis
jarque Jarque-Bera normality p-value
dpo D'Agostino-Pearson normality p-value
pcov, scov Population or sample covariance of a LEFT:RIGHT pair
ppearson Population Pearson correlation of a LEFT:RIGHT pair
spearson Sample Pearson correlation, not Spearman rank correlation
dotprod Dot product of a LEFT:RIGHT pair
Weights for wmean must be finite and nonnegative. Each nonempty calculation
needs positive retained weight. Results can depend on contribution order.
Sample requirements, missing values and numerical limits:
https://fastmash.io/guide/operations
https://fastmash.io/guide/weighted-mean
Per-row numeric operations: getnum[:TYPE] FIELD extracts the first number.
Types: n (digits), i (signed integer), h (hex), o (octal), p (digits/dot,
default), d (signed digits/dot). No match or failed conversion gives zero.
round, floor, ceil, trunc and frac FIELD transform each selected number.
round uses nearest integer with ties away from zero. frac keeps the sign.
Round each value to an integer (prints 1, then 2):
printf '1.2\n2.4\n' | LC_ALL=C fastmash round 1
Use --round=2 sum 1 instead to sum first and display two decimal places.
Binning: bin[:WIDTH] FIELD returns the lower bucket boundary (default 100).
strbin[:COUNT] FIELD assigns field bytes to buckets 0..COUNT-1 (default 10).
cut/echo FIELD prints the selected fields of each record.
base64 FIELD encodes field bytes using padded standard Base64.
debase64 FIELD strictly decodes Base64.
md5, sha1, sha224, sha256, sha384 and sha512 FIELD produce hexadecimal checksums.
MD5 and SHA-1 are compatibility checksums, not security mechanisms.
dirname, basename, extname and barename FIELD transform paths without file access.
Per-row operations can be combined with each other, but not with aggregates or
grouping keys. Details: https://fastmash.io/guide/modes
Field modes: reverse reverses the fields of each record.
nop/noop consumes input without output, or copies records with --full.
Use --no-strict to accept changing widths with reverse.
Table modes: check [N lines] [N fields] checks widths and optional counts.
transpose exchanges rows and columns. Widths must match unless --no-strict is
used, with --filler for missing cells. Headers are ordinary records.
rmdup/dedup FIELD keeps the first record for each distinct key. Use -s to sort.
Only one duplicate-removal key is supported.
crosstab/ct KEY1,KEY2 [OP FIELD] makes a pivot table (default operation count).
Use -s to combine scattered keys. --filler supplies missing combinations.
Details: https://fastmash.io/guide/modes
https://fastmash.io/guide/grouping
Top-N selection: top:N FIELD keeps the highest N complete records.
bottom:N FIELD keeps the lowest. N is a positive integer.
FIELD is one position or escaped header name. Named fields need --header-in or -H.
Results follow numeric rank order, with earlier input records winning ties.
With -g, each adjacent group has its own cutoff. Add -s for scattered keys.
--narm omits records with whole-field NA/N/A/NaN ranking values.
Other NaNs are errors. --header-out or -H requests a copied-field header.
At most N candidates are kept per group or dataset. N limits records, not bytes.
Example (prints 30, then 20):
printf '10\n30\n20\n' | fastmash top:2 1
Details: https://fastmash.io/guide/top-n-records
Dataset comparison: compare BEFORE AFTER applies numerical aggregates to two
files. Use - for standard input on one side. -H reads headers and labels results.
Named fields bind independently, allowing different field orders.
-g aligns keys across each entire dataset, including scattered occurrences.
Reports include before, after, signed difference and percentage change, with
added or removed keys marked explicitly. Text-result aggregates are unsupported.
Before operations, rank RESULT [absolute|percent] orders keys by change magnitude.
RESULT is a 1-based expanded result index. The default measure is absolute.
Optional limit N keeps at most N ranked keys. Unrankable keys still appear.
Both input scans and all calculations finish before report output.
For two files whose headers include region and revenue, rank the five largest
percentage changes in each region's total revenue:
fastmash -H -g region compare before.tsv after.tsv rank 1 percent limit 5 sum revenue
Details: https://fastmash.io/guide/dataset-comparison
Table health: health inspects widths, headers, missing values and mixed types.
Inspection leaves input unchanged and reports inferred findings as advice.
FIELD is one position or a unique input-header name.
type FIELD integer|number|text supplies an expected type.
required FIELD rejects absent and empty values.
nonmissing FIELD also rejects NA/N/A/NaN tokens.
width N declares the field count for data and any input header.
validate requires at least one supplied rule and returns status 1 for violations.
Inferred findings alone do not fail validation.
examples N keeps N examples per finding (default 3, zero hides them).
tsv selects the versioned report format for scripts.
Controls may be combined, for example:
fastmash --header-in health type price number nonmissing id validate < data.tsv
Details and report schema: https://fastmash.io/guide/table-health
Runtime and numerical limits:
Sorting runs inside Fastmash and needs no other program. Large sorts can
spill to TMPDIR (default /tmp), which must be writable when spilling.
FASTMASH_SORT_MEMORY_BYTES sets the sort-buffer target, not total memory.
Operations needing every value keep samples in memory, even when sorting spills.
Records, fields, groups and samples grow as needed, subject to available memory.
One Fastmash version gives the same numerical results on supported machines
for the same input and explicit settings. Last digits can differ from GNU.
Details: https://fastmash.io/guide/grouping#large-inputs
https://fastmash.io/guide/output
Sorting locales:
C, POSIX and C.UTF-8 use byte order. Verified language locales use alphabetic
ordering, with documented differences from GNU. Unknown locale names use C.
Language-locale keys must be valid UTF-8 without NUL bytes.
Supported glibc language locales:
aa af agr ak am anp ayc az bem ber bg bhb bi bs byn ce chr crh
cy de dv en eo es eu ff fil fr fy ga gbm gd gez gl gv ha he hif hne hr hsb ht hy
ia id ig is it ks kv kw lb lg li lij ln lv mag mfe mg mhr miq mjw mk mn mnw ms
mt nds nhn niu nl nr nso om os pa pap pl pt quz rif ro ru rw sat scn sd sgs sid
sm so sq sr ss ssy st su sw ta tcy the ti tig tk tn to tok tpi ts unm ur uz ve
wa wae wal wo xh yo yuw zgh zu, except az_IR, ber_DZ, crh_UA, en_CA, fr_CA,
pa_IN, sd_IN and sr_RS.
Locale rules are built in. You do not need installed locale data.
Set LC_ALL=C.UTF-8 for decimal points and byte ordering.
Details: https://fastmash.io/guide/output#locales
https://fastmash.io/guide/differences
Exit statuses:
0 Success
1 Invalid command or input, or an input/output failure
77 Unsupported feature or locale, or a checked resource or numerical limit
70 Internal numerical failure. Please report it
A failed command may leave partial output. Always check the exit status.
Details: https://fastmash.io/guide/errors
More information:
User guide: https://fastmash.io/guide/
Installation: https://fastmash.io/install
Migration from GNU datamash: https://fastmash.io/guide/migrating