utf8-norm 1.0.1

utf8-norm, validate and normalize UTF-8 Unicode data

Version 1.0.1 licensed GPLv3. (C) 2019 Leonora Tindall nora@nora.codes Fast command line Unicode normalization, supporting stream safety transformations as well as NFC, NFD, NFKD, and NFKC. Exits with failure if the incoming stream is not valid UTF-8.

Usage: utf8-norm [--nfc | --nfd | --nfkc | --nfkd] [--stream-safe] [--crlf]

<infile> (default stdin) - file from which to read bytes.
<outfile> (default stdout) - file to which to write normalized Unicode.
-w, --crlf  - write CRLF (Windows) instead of LF only (Unix) at the end of lines.
-d, --nfd   - write NFD (canonical decomposition).
-D, --nfkd  - write NFKD (compatibility decomposition).
-c, --nfc   - write NFC (canonical composition computed from NFD). This is the default.
-C, --nfkc  - write NFKC (canonical composition computed from NFC).
-s, --stream-safe   - write stream-safe bytes (Conjoining Grapheme Joiners, UAX15-D4).
-V, --version - output version information and exit

utf8-norm was created at Rust Belt Rust 2019 in Dayton, OH. Thanks to @j41manning for her excellent talk regarding Unicode handling in Rust.

Natively install as cargo install utf8-norm or from your distribution's package manager.