fastmash 0.7.0

Fast command-line statistics on text data, compatible with GNU datamash
Summarize, group and transform fields from standard input, or compare two files.

Usage: fastmash [OPTIONS] OPERATION FIELDS [OPERATION FIELDS ...]
       fastmash [OPTIONS] top:N FIELD | bottom:N FIELD
       fastmash [OPTIONS] compare BEFORE AFTER [RANKING] OPERATION FIELDS [...]
       fastmash [OPTIONS] health [CONTROLS]
       fastmash [OPTIONS] MODE [FIELDS]

Fastmash supports Linux, Windows through WSL2 and macOS 15 or later.

Examples:
  Sum the first field (prints 60):
  printf '10\n20\n30\n' | fastmash sum 1
  Combine scattered groups and sum the second field (prints a<TAB>1, b<TAB>5):
  printf 'b\t2\na\t1\nb\t3\n' | LC_ALL=C fastmash -s -g 1 sum 2

Fields and grouping:
  Fields are numbered from 1. Use lists (1,3), ranges (2-4), or input-header
  names with --header-in or -H. Names are exact and case-sensitive.
  Backslash escapes one byte in a name. The first duplicate name wins.
  Paired operations use LEFT:RIGHT, such as pcov 1:2 or wmean price:units.
  Weighted mean example (prints 7):
    printf '10\t2\n5\t3\n' | fastmash wmean 1:2
  Lists of pairs are allowed, but ranges cannot be combined with pairs.
  Operations and results follow request order. Operation names ignore case.
  -g FIELDS summarizes adjacent records with equal keys.
  Add -s when equal keys are scattered through the input.
Grouping also accepts groupby, grouping or gb before its keys and operations.

Options:
  -g, --group=FIELDS      Summarize adjacent groups with these keys
  -s, --sort              Sort stably by keys before grouping or duplicate removal
  -i, --ignore-case       Ignore ASCII case in grouping, sorting and text summaries
  --header-in             Use the first input record as field names
  --header-out            Request result labels
  -H, --headers           Same as --header-in --header-out
  --result-name=INDEX:NAME Replace a result label, counting results from 1
  --csv-in                Read quoted CSV for calculations, selection or comparison
  --csv-out               Write quoted CSV for calculations, selection or comparison
  --csv                   Same as --csv-in --csv-out
  -t BYTE, --field-separator=BYTE
                          Split ordinary text at one literal byte
  -W, --whitespace        Split ordinary text on runs of spaces and tabs
  --output-delimiter=BYTE Separate text output fields with this byte
  -c BYTE, --collapse-delimiter=BYTE
                          Separate unique/collapse list values (default comma)
  -z, --zero-terminated   End records with NUL instead of newline
  -C, --skip-comments     Skip text lines starting with # or ; after spaces/tabs
  --narm                  Skip whole-field NA, N/A and NaN tokens, ignoring case
  --vnlog                 Read annotated whitespace records with named fields
  -f, --full              Copy a representative record before calculated results
  --no-strict             Allow ragged transpose/reverse input
  -F TEXT, --filler=TEXT   Fill missing transpose/crosstab cells (default N/A)
  --format=FORMAT         Format numbers with one %e/%f/%g/%a conversion
  -R, --round=N           Display N fractional places (1..50)
  -S VALUE, --seed VALUE  Set rand's reproducible seed
  --color=auto|always|never
                          Set terminal presentation (default auto)
  --no-color              Same as --color=never
  -h, --help              Show this help without reading input
  -V, --version           Show the package version without reading input
  --                      End option processing

Input and output:
  Input defaults to tab-separated fields and newline-terminated records.
  Use -t, for literal comma splitting, or --csv-in for quoted CSV.
  Separators take one literal byte. Backslash escapes such as '\t' are not
  expanded. Omit -t for tab-separated input.
  --csv reads and writes CSV. These switches do not infer headers.
  Example with a header and a named field (prints sum(revenue), then 35):
    printf 'region,revenue\nwest,20\neast,15\n' | fastmash --csv -H sum revenue
  To name that result total, add --result-name=1:total.
  CSV supports calculations, Top-N selection and comparison, including their
  supported grouped workflows. Table health and other modes use ordinary text.
  CSV input conflicts with -t, -W and -C. Either CSV direction conflicts with
  -z and --vnlog. CSV output also conflicts with --output-delimiter.
  Delimited-text output to files or pipes uses the input separator by default.
  With -W, text output uses tabs. CSV output uses commas and newline endings.
  Result names require --header-out or -H. INDEX counts expanded results,
  excluding keys and --full copies. Selection copies fields without result names.
  --format and --round change displayed numbers, not calculations.
  Sum first, then display two decimal places (prints 3.60):
    printf '1.2\n2.4\n' | LC_ALL=C fastmash --round=2 sum 1
  Use --format=%.0f to display no decimal places.
  --narm does not trim spaces or supply missing fields. For wmean it omits a
  whole pair after validating its nonmissing partners. Other paired statistics
  remove missing values independently on each side.
  Details: https://fastmash.io/guide/input
           https://fastmash.io/guide/csv-and-result-names

Operations:
  count          Count values, including text and empty fields
  countunique    Count distinct text values
  unique, uniq   Sorted distinct text values
  collapse       All text values in input order, including duplicates
  first, last    First or last accepted value
  rand           Random accepted value
  min, max       Minimum or maximum numeric value
  absmin         Value with the smallest magnitude, keeping its sign
  absmax         Value with the largest magnitude, keeping its sign
  range          Maximum minus minimum
  sum            Sum of values
  mean           Arithmetic mean
  wmean          Weighted mean of a VALUE:WEIGHT field pair
  geomean        Geometric mean
  harmmean       Harmonic mean
  ms             Mean of squared values
  rms            Square root of the mean of squared values
  median         Middle sorted value, averaging the middle pair for even samples
  mode           Value in the longest sorted run (first run wins ties)
  antimode       Value in the shortest sorted run (first run wins ties)
  q1, q3         First or third quartile using type-7 interpolation
  iqr            Third quartile minus first quartile
  perc[:N]       Percentile (default 95, integer 1..100, type-7 interpolation)
  trimmean[:P]   Mean after trimming fraction P from each end (default 0, 0..0.5)
  pvar, svar     Population or sample variance
  pstdev         Population standard deviation
  sstdev         Sample standard deviation
  madraw         Median absolute deviation without scaling
  mad            Median absolute deviation scaled by 1.4826
  pskew, sskew   Population or sample skewness
  pkurt, skurt   Population or sample excess kurtosis
  jarque         Jarque-Bera normality p-value
  dpo            D'Agostino-Pearson normality p-value
  pcov, scov     Population or sample covariance of a LEFT:RIGHT pair
  ppearson       Population Pearson correlation of a LEFT:RIGHT pair
  spearson       Sample Pearson correlation, not Spearman rank correlation
  dotprod        Dot product of a LEFT:RIGHT pair

Weights for wmean must be finite and nonnegative. Each nonempty calculation
needs positive retained weight. Results can depend on contribution order.
Sample requirements, missing values and numerical limits:
https://fastmash.io/guide/operations
https://fastmash.io/guide/weighted-mean

Per-row numeric operations: getnum[:TYPE] FIELD extracts the first number.
Types: n (digits), i (signed integer), h (hex), o (octal), p (digits/dot,
default), d (signed digits/dot). No match or failed conversion gives zero.
round, floor, ceil, trunc and frac FIELD transform each selected number.
round uses nearest integer with ties away from zero. frac keeps the sign.
Round each value to an integer (prints 1, then 2):
  printf '1.2\n2.4\n' | LC_ALL=C fastmash round 1
Use --round=2 sum 1 instead to sum first and display two decimal places.

Binning: bin[:WIDTH] FIELD returns the lower bucket boundary (default 100).
strbin[:COUNT] FIELD assigns field bytes to buckets 0..COUNT-1 (default 10).

cut/echo FIELD prints the selected fields of each record.
base64 FIELD encodes field bytes using padded standard Base64.
debase64 FIELD strictly decodes Base64.
md5, sha1, sha224, sha256, sha384 and sha512 FIELD produce hexadecimal checksums.
MD5 and SHA-1 are compatibility checksums, not security mechanisms.
dirname, basename, extname and barename FIELD transform paths without file access.

Per-row operations can be combined with each other, but not with aggregates or
grouping keys. Details: https://fastmash.io/guide/modes

Field modes: reverse reverses the fields of each record.
nop/noop consumes input without output, or copies records with --full.
Use --no-strict to accept changing widths with reverse.

Table modes: check [N lines] [N fields] checks widths and optional counts.
transpose exchanges rows and columns. Widths must match unless --no-strict is
used, with --filler for missing cells. Headers are ordinary records.
rmdup/dedup FIELD keeps the first record for each distinct key. Use -s to sort.
Only one duplicate-removal key is supported.
crosstab/ct KEY1,KEY2 [OP FIELD] makes a pivot table (default operation count).
Use -s to combine scattered keys. --filler supplies missing combinations.
Details: https://fastmash.io/guide/modes
         https://fastmash.io/guide/grouping

Top-N selection: top:N FIELD keeps the highest N complete records.
bottom:N FIELD keeps the lowest. N is a positive integer.
FIELD is one position or escaped header name. Named fields need --header-in or -H.
Results follow numeric rank order, with earlier input records winning ties.
With -g, each adjacent group has its own cutoff. Add -s for scattered keys.
--narm omits records with whole-field NA/N/A/NaN ranking values.
Other NaNs are errors. --header-out or -H requests a copied-field header.
At most N candidates are kept per group or dataset. N limits records, not bytes.
Example (prints 30, then 20):
  printf '10\n30\n20\n' | fastmash top:2 1
Details: https://fastmash.io/guide/top-n-records

Dataset comparison: compare BEFORE AFTER applies numerical aggregates to two
files. Use - for standard input on one side. -H reads headers and labels results.
Named fields bind independently, allowing different field orders.
-g aligns keys across each entire dataset, including scattered occurrences.
Reports include before, after, signed difference and percentage change, with
added or removed keys marked explicitly. Text-result aggregates are unsupported.
Before operations, rank RESULT [absolute|percent] orders keys by change magnitude.
RESULT is a 1-based expanded result index. The default measure is absolute.
Optional limit N keeps at most N ranked keys. Unrankable keys still appear.
Both input scans and all calculations finish before report output.
For two files whose headers include region and revenue, rank the five largest
percentage changes in each region's total revenue:
  fastmash -H -g region compare before.tsv after.tsv rank 1 percent limit 5 sum revenue
Details: https://fastmash.io/guide/dataset-comparison

Table health: health inspects widths, headers, missing values and mixed types.
Inspection leaves input unchanged and reports inferred findings as advice.
FIELD is one position or a unique input-header name.
type FIELD integer|number|text supplies an expected type.
required FIELD rejects absent and empty values.
nonmissing FIELD also rejects NA/N/A/NaN tokens.
width N declares the field count for data and any input header.
validate requires at least one supplied rule and returns status 1 for violations.
Inferred findings alone do not fail validation.
examples N keeps N examples per finding (default 3, zero hides them).
tsv selects the versioned report format for scripts.
Controls may be combined, for example:
  fastmash --header-in health type price number nonmissing id validate < data.tsv
Details and report schema: https://fastmash.io/guide/table-health

Runtime and numerical limits:
  Sorting runs inside Fastmash and needs no other program. Large sorts can
  spill to TMPDIR (default /tmp), which must be writable when spilling.
  FASTMASH_SORT_MEMORY_BYTES sets the sort-buffer target, not total memory.
  Operations needing every value keep samples in memory, even when sorting spills.
  Records, fields, groups and samples grow as needed, subject to available memory.
  One Fastmash version gives the same numerical results on supported machines
  for the same input and explicit settings. Last digits can differ from GNU.
  Details: https://fastmash.io/guide/grouping#large-inputs
           https://fastmash.io/guide/output

Sorting locales:
  C, POSIX and C.UTF-8 use byte order. Verified language locales use alphabetic
  ordering, with documented differences from GNU. Unknown locale names use C.
  Language-locale keys must be valid UTF-8 without NUL bytes.
  Supported glibc language locales:
  aa af agr ak am anp ayc az bem ber bg bhb bi bs byn ce chr crh
  cy de dv en eo es eu ff fil fr fy ga gbm gd gez gl gv ha he hif hne hr hsb ht hy
  ia id ig is it ks kv kw lb lg li lij ln lv mag mfe mg mhr miq mjw mk mn mnw ms
  mt nds nhn niu nl nr nso om os pa pap pl pt quz rif ro ru rw sat scn sd sgs sid
  sm so sq sr ss ssy st su sw ta tcy the ti tig tk tn to tok tpi ts unm ur uz ve
  wa wae wal wo xh yo yuw zgh zu, except az_IR, ber_DZ, crh_UA, en_CA, fr_CA,
  pa_IN, sd_IN and sr_RS.
  Locale rules are built in. You do not need installed locale data.
  Set LC_ALL=C.UTF-8 for decimal points and byte ordering.
  Details: https://fastmash.io/guide/output#locales
           https://fastmash.io/guide/differences

Exit statuses:
  0   Success
  1   Invalid command or input, or an input/output failure
  77  Unsupported feature or locale, or a checked resource or numerical limit
  70  Internal numerical failure. Please report it
  A failed command may leave partial output. Always check the exit status.
  Details: https://fastmash.io/guide/errors

More information:
  User guide: https://fastmash.io/guide/
  Installation: https://fastmash.io/install
  Migration from GNU datamash: https://fastmash.io/guide/migrating