comex 1.0.0

Fast, dependency-free comment extraction and removal for source files, with content-based syntax detection (no file extensions required).
Documentation
  • Coverage
  • 100%
    86 out of 86 items documented1 out of 11 items with examples
  • Size
  • Source code size: 143.2 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 825.6 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 2s Average build duration of successful builds.
  • all releases: 2s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • AlecMenke/comex
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • AlecMenke

comex

crates.io docs.rs CI license

Strip the comments out of source code, or throw the code away and keep only the comments. You don't have to tell it what language the file is.

Given profile.js:

// Look up a user's profile.
export async function profile(id) {
  const url = `https://api.example.com/users/#${id}`;  // '#' here is a fragment
  /* One round trip is enough:
     the endpoint already embeds the account. */
  return fetch(url);
}

Take the comments out:

$ comex profile.js
export async function profile(id) {
  const url = `https://api.example.com/users/#${id}`;
  return fetch(url);
}

Or keep nothing else:

$ comex -c profile.js
Look up a user's profile.

'#' here is a fragment
One round trip is enough:
the endpoint already embeds the account.

Note what survived: the // inside the URL and the # fragment inside the template string. Getting that right is most of the work.

Why not just use the extension?

Often there isn't one. Text shows up on stdin, or as a blob from somebody's API, or in a file called Dockerfile or Makefile or .env or install-sh. An extension that does exist can still be wrong. So comex reads the bytes and works out the comment syntax for itself, then tells you what it decided:

$ comex -d profile.js
c (confidence 0.48) - // and /* */ - C, C++, C#, Java, JS, TS, Go, Rust, Swift, PHP
scores: c=41 sql=12

When it can't tell, it says unknown and hands your file back untouched.

Install

$ cargo install comex          # CLI
$ cargo add comex              # library

No dependencies, and no regex anywhere in it.

CLI

$ comex -c -m src/*.rs | grep -i todo     # audit the TODOs in a tree
$ comex -k app.py                         # strip, keeping line numbers stable
$ comex -i -s hash deploy.sh              # rewrite in place, skip detection
$ cat weird_blob | comex -d               # identify before you act

Give it no FILE, or -, and it reads stdin.

comex [OPTIONS] [FILE]...

MODES:
    (default)             print the code with every comment removed
    -c, --comments        print only the comments
    -d, --detect          print the detected syntax and confidence, change nothing

OPTIONS:
    -s, --syntax <NAME>   force a comment syntax instead of detecting it
    -i, --in-place        rewrite each FILE instead of writing to stdout
    -k, --keep-lines      preserve line numbering (leave blank lines behind)
    -b, --keep-blank      keep lines that held nothing but a comment
    -m, --markers         keep comment markers when printing comments
    -D, --docstrings      treat leading triple-quoted strings as comments
    -S, --no-shebang      allow a leading `#!` line to be treated as a comment
    -n, --no-trim         do not trim whitespace left behind by a removed comment
        --stats           print per-file counts to stderr
    -l, --list            list the supported comment syntaxes

Library

let src = "int x = 1; // the answer\n/* gone */\nint y = 2;\n";

assert_eq!(comex::strip_comments(src),   "int x = 1;\nint y = 2;\n");
assert_eq!(comex::extract_comments(src), "the answer\ngone\n");

If you already know the language, say so and skip detection. If you don't trust it, ask what it decided before you act:

use comex::{Dialect, Options};

// Skip detection when you already know the language.
let out = comex::strip_comments_with("x = 1 # hi\n", &Dialect::Hash.syntax(), &Options::default());
assert_eq!(out, "x = 1\n");

// Or inspect what detection thinks before acting on it.
let d = comex::detect(src);
println!("{} ({:.2})", d.dialect.name(), d.confidence);

Need something finer? comex::comments(src) hands back the comment spans as byte ranges and gets out of your way:

for span in comex::comments(src) {
    println!("{:?} at {}..{}: {}", span.kind, span.start, span.end, span.inner(src));
}

Full API on docs.rs.

How it works

comex has no idea what JavaScript is. What it knows about is // and /* */, which happen to cover C, C++, C#, Java, JavaScript, TypeScript, Go, Rust, Swift, Kotlin, Scala, Dart and PHP. # covers Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile and PowerShell. Dozens of languages, a handful of comment syntaxes. That ratio is the only reason any of this works without extensions.

Detection is a scoring pass. It counts markers at the starts of lines, block delimiters that actually balance, how lines end, shebangs, preprocessor directives, a few keyword shapes. Each syntax family then scores itself against that pile of evidence under two rules:

  1. No markers, no score. Structure alone never licenses comment removal. A shell script full of braces and semicolons is not C.
  2. Bonuses cannot outvote markers. Context can at most double a family's marker score, so a pile of keywords can never overwhelm a single opposing marker.

Nothing clears the bar, nothing gets touched.

Scanning is one pass of a hand-written byte state machine. A 256-entry lookup table means most bytes cost a single probe before they're skipped. No regex, nothing to backtrack into. Roughly 70-250 MiB/s on a laptop, detection included.

String literals are where the difficulty lives, as the URL in the example above suggests. comex handles raw strings (r#"..."#), character literals, Rust lifetimes (&'a T), Matlab transposes, doubled-quote escapes ('it''s'), Lua long brackets, Ruby =begin, Python docstrings, and the bodies of HTML <script> and <style> tags. An unterminated quote stops at the end of its line, so one stray apostrophe in a shell script can't eat the rest of the file.

Safety

Leaving a comment behind is a cosmetic failure. Deleting a line of code is not. Every ambiguous call in comex resolves toward the first one.

The rules are tuned against a sweep of ~10,000 real files: the macOS SDK headers, the Python standard library, a pile of npm packages. Two invariants get checked on every one.

  • Deletion only. The output is always a subsequence of the input. Bytes can disappear. Nothing else happens to them.
  • Stability. Stripping twice gives you what stripping once gave you, so a stripped file never comes back as a different language.

Point it at your own code and see:

$ cargo run --release --example sweep -- ~/your/repo
files    2753
bytes    23.18 MiB
throughput 75 MiB/s
no invariant violations

Supported syntaxes

Name Markers Covers
c // /* */ C, C++, C#, Java, JS, TS, Go, Rust, Swift, Kotlin, PHP, CSS
hash # Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile, PowerShell
sql -- /* */ SQL dialects
lua -- --[[ ]] Lua
haskell -- {- -} Haskell, Elm, PureScript
lisp ; #| |# Lisp, Scheme, Clojure, Emacs Lisp, assembly
markup <!-- --> HTML, XML, SVG, Markdown, Vue, Svelte (plus embedded JS/CSS)
ml (* *) OCaml, F#, Pascal, Mathematica
percent % TeX/LaTeX, Erlang, Prolog, Matlab/Octave
vb ' REM Visual Basic, VBA, VBScript
batch REM :: Windows batch
fortran ! modern Fortran
ini ; # INI, desktop entries, gitconfig
unknown none nothing detected; input passes through unchanged

Pass any of these to -s to skip detection entirely.

Limits

All deliberate, and all of them err toward leaving your file alone.

  • One ambiguous marker with nothing else to go on stays unknown. Pass -s if you know better.
  • PHP files that use both # and // lose the # comments, because the C family wins the scoring. Those comments are deprecated in PHP anyway.
  • JavaScript regex literals aren't fully parsed. A / followed by an unbalanced quote can hide a trailing comment on that line. It can't delete code.
  • Some data formats use two comment-ish markers at once. There are test-data files out there that comment with -- and use # as data, and comex can pick the wrong one. -d will show you a low confidence when that happens.
  • x=1#c in Python isn't a comment here, because there's no space before the #. YAML, TOML, shell and Ruby all want that boundary, and url: https://x/#top has to survive.

License

MIT © Alec Menke