comex
Strip the comments out of source code, or throw the code away and keep only the comments. You don't have to tell it what language the file is.
Given profile.js:
// Look up a user's profile.
export
Take the comments out:
$ comex profile.js
export async function profile(id) {
const url = `https://api.example.com/users/#${id}`;
return fetch(url);
}
Or keep nothing else:
$ comex -c profile.js
Look up a user's profile.
'#' here is a fragment
One round trip is enough:
the endpoint already embeds the account.
Note what survived: the // inside the URL and the # fragment inside the template
string. Getting that right is most of the work.
Why not just use the extension?
Often there isn't one. Text shows up on stdin, or as a blob from somebody's API, or in a
file called Dockerfile or Makefile or .env or install-sh. An extension that does
exist can still be wrong. So comex reads the bytes and works out the comment syntax for
itself, then tells you what it decided:
$ comex -d profile.js
c (confidence 0.48) - // and /* */ - C, C++, C#, Java, JS, TS, Go, Rust, Swift, PHP
scores: c=41 sql=12
When it can't tell, it says unknown and hands your file back untouched.
Install
$ cargo install comex # CLI
$ cargo add comex # library
No dependencies, and no regex anywhere in it.
CLI
$ comex -c -m src/*.rs | grep -i todo # audit the TODOs in a tree
$ comex -k app.py # strip, keeping line numbers stable
$ comex -i -s hash deploy.sh # rewrite in place, skip detection
$ cat weird_blob | comex -d # identify before you act
Give it no FILE, or -, and it reads stdin.
comex [OPTIONS] [FILE]...
MODES:
(default) print the code with every comment removed
-c, --comments print only the comments
-d, --detect print the detected syntax and confidence, change nothing
OPTIONS:
-s, --syntax <NAME> force a comment syntax instead of detecting it
-i, --in-place rewrite each FILE instead of writing to stdout
-k, --keep-lines preserve line numbering (leave blank lines behind)
-b, --keep-blank keep lines that held nothing but a comment
-m, --markers keep comment markers when printing comments
-D, --docstrings treat leading triple-quoted strings as comments
-S, --no-shebang allow a leading `#!` line to be treated as a comment
-n, --no-trim do not trim whitespace left behind by a removed comment
--stats print per-file counts to stderr
-l, --list list the supported comment syntaxes
Library
let src = "int x = 1; // the answer\n/* gone */\nint y = 2;\n";
assert_eq!;
assert_eq!;
If you already know the language, say so and skip detection. If you don't trust it, ask what it decided before you act:
use ;
// Skip detection when you already know the language.
let out = strip_comments_with;
assert_eq!;
// Or inspect what detection thinks before acting on it.
let d = detect;
println!;
Need something finer? comex::comments(src) hands back the comment spans as byte ranges
and gets out of your way:
for span in comments
Full API on docs.rs.
How it works
comex has no idea what JavaScript is. What it knows about is // and /* */, which
happen to cover C, C++, C#, Java, JavaScript, TypeScript, Go, Rust, Swift, Kotlin, Scala,
Dart and PHP. # covers Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile and
PowerShell. Dozens of languages, a handful of comment syntaxes. That ratio is the only
reason any of this works without extensions.
Detection is a scoring pass. It counts markers at the starts of lines, block delimiters that actually balance, how lines end, shebangs, preprocessor directives, a few keyword shapes. Each syntax family then scores itself against that pile of evidence under two rules:
- No markers, no score. Structure alone never licenses comment removal. A shell script full of braces and semicolons is not C.
- Bonuses cannot outvote markers. Context can at most double a family's marker score, so a pile of keywords can never overwhelm a single opposing marker.
Nothing clears the bar, nothing gets touched.
Scanning is one pass of a hand-written byte state machine. A 256-entry lookup table means most bytes cost a single probe before they're skipped. No regex, nothing to backtrack into. Roughly 70-250 MiB/s on a laptop, detection included.
String literals are where the difficulty lives, as the URL in the example above
suggests. comex handles raw strings (r#"..."#), character literals, Rust lifetimes
(&'a T), Matlab transposes, doubled-quote escapes ('it''s'), Lua long brackets, Ruby
=begin, Python docstrings, and the bodies of HTML <script> and <style> tags. An
unterminated quote stops at the end of its line, so one stray apostrophe in a shell script
can't eat the rest of the file.
Safety
Leaving a comment behind is a cosmetic failure. Deleting a line of code is not. Every ambiguous call in comex resolves toward the first one.
The rules are tuned against a sweep of ~10,000 real files: the macOS SDK headers, the Python standard library, a pile of npm packages. Two invariants get checked on every one.
- Deletion only. The output is always a subsequence of the input. Bytes can disappear. Nothing else happens to them.
- Stability. Stripping twice gives you what stripping once gave you, so a stripped file never comes back as a different language.
Point it at your own code and see:
$ cargo run --release --example sweep -- ~/your/repo
files 2753
bytes 23.18 MiB
throughput 75 MiB/s
no invariant violations
Supported syntaxes
| Name | Markers | Covers |
|---|---|---|
c |
// /* */ |
C, C++, C#, Java, JS, TS, Go, Rust, Swift, Kotlin, PHP, CSS |
hash |
# |
Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile, PowerShell |
sql |
-- /* */ |
SQL dialects |
lua |
-- --[[ ]] |
Lua |
haskell |
-- {- -} |
Haskell, Elm, PureScript |
lisp |
; #| |# |
Lisp, Scheme, Clojure, Emacs Lisp, assembly |
markup |
<!-- --> |
HTML, XML, SVG, Markdown, Vue, Svelte (plus embedded JS/CSS) |
ml |
(* *) |
OCaml, F#, Pascal, Mathematica |
percent |
% |
TeX/LaTeX, Erlang, Prolog, Matlab/Octave |
vb |
' REM |
Visual Basic, VBA, VBScript |
batch |
REM :: |
Windows batch |
fortran |
! |
modern Fortran |
ini |
; # |
INI, desktop entries, gitconfig |
unknown |
none | nothing detected; input passes through unchanged |
Pass any of these to -s to skip detection entirely.
Limits
All deliberate, and all of them err toward leaving your file alone.
- One ambiguous marker with nothing else to go on stays
unknown. Pass-sif you know better. - PHP files that use both
#and//lose the#comments, because the C family wins the scoring. Those comments are deprecated in PHP anyway. - JavaScript regex literals aren't fully parsed. A
/followed by an unbalanced quote can hide a trailing comment on that line. It can't delete code. - Some data formats use two comment-ish markers at once. There are test-data files out
there that comment with
--and use#as data, and comex can pick the wrong one.-dwill show you a low confidence when that happens. x=1#cin Python isn't a comment here, because there's no space before the#. YAML, TOML, shell and Ruby all want that boundary, andurl: https://x/#tophas to survive.
License
MIT © Alec Menke