# comex
[](https://crates.io/crates/comex)
[](https://docs.rs/comex)
[](https://github.com/AlecMenke/comex/actions/workflows/ci.yml)
[](LICENSE)
Strip the comments out of source code, or throw the code away and keep only the comments.
You don't have to tell it what language the file is.
Given `profile.js`:
```js
// Look up a user's profile.
export async function profile(id) {
const url = `https://api.example.com/users/#${id}`; // '#' here is a fragment
/* One round trip is enough:
the endpoint already embeds the account. */
return fetch(url);
}
```
Take the comments out:
```console
$ comex profile.js
export async function profile(id) {
const url = `https://api.example.com/users/#${id}`;
return fetch(url);
}
```
Or keep nothing else:
```console
$ comex -c profile.js
Look up a user's profile.
'#' here is a fragment
One round trip is enough:
the endpoint already embeds the account.
```
Note what survived: the `//` inside the URL and the `#` fragment inside the template
string. Getting that right is most of the work.
## Why not just use the extension?
Often there isn't one. Text shows up on stdin, or as a blob from somebody's API, or in a
file called `Dockerfile` or `Makefile` or `.env` or `install-sh`. An extension that does
exist can still be wrong. So comex reads the bytes and works out the comment syntax for
itself, then tells you what it decided:
```console
$ comex -d profile.js
c (confidence 0.48) - // and /* */ - C, C++, C#, Java, JS, TS, Go, Rust, Swift, PHP
scores: c=41 sql=12
```
When it can't tell, it says `unknown` and hands your file back untouched.
## Install
```console
$ cargo install comex # CLI
$ cargo add comex # library
```
No dependencies, and no regex anywhere in it.
## CLI
```console
$ comex -c -m src/*.rs | grep -i todo # audit the TODOs in a tree
$ comex -k app.py # strip, keeping line numbers stable
$ comex -i -s hash deploy.sh # rewrite in place, skip detection
$ cat weird_blob | comex -d # identify before you act
```
Give it no FILE, or `-`, and it reads stdin.
```
comex [OPTIONS] [FILE]...
MODES:
(default) print the code with every comment removed
-c, --comments print only the comments
-d, --detect print the detected syntax and confidence, change nothing
OPTIONS:
-s, --syntax <NAME> force a comment syntax instead of detecting it
-i, --in-place rewrite each FILE instead of writing to stdout
-k, --keep-lines preserve line numbering (leave blank lines behind)
-b, --keep-blank keep lines that held nothing but a comment
-m, --markers keep comment markers when printing comments
-D, --docstrings treat leading triple-quoted strings as comments
-S, --no-shebang allow a leading `#!` line to be treated as a comment
-n, --no-trim do not trim whitespace left behind by a removed comment
--stats print per-file counts to stderr
-l, --list list the supported comment syntaxes
```
## Library
```rust
let src = "int x = 1; // the answer\n/* gone */\nint y = 2;\n";
assert_eq!(comex::strip_comments(src), "int x = 1;\nint y = 2;\n");
assert_eq!(comex::extract_comments(src), "the answer\ngone\n");
```
If you already know the language, say so and skip detection. If you don't trust it, ask
what it decided before you act:
```rust
use comex::{Dialect, Options};
// Skip detection when you already know the language.
let out = comex::strip_comments_with("x = 1 # hi\n", &Dialect::Hash.syntax(), &Options::default());
assert_eq!(out, "x = 1\n");
// Or inspect what detection thinks before acting on it.
let d = comex::detect(src);
println!("{} ({:.2})", d.dialect.name(), d.confidence);
```
Need something finer? `comex::comments(src)` hands back the comment spans as byte ranges
and gets out of your way:
```rust
for span in comex::comments(src) {
println!("{:?} at {}..{}: {}", span.kind, span.start, span.end, span.inner(src));
}
```
Full API on [docs.rs](https://docs.rs/comex).
## How it works
comex has no idea what JavaScript is. What it knows about is `//` and `/* */`, which
happen to cover C, C++, C#, Java, JavaScript, TypeScript, Go, Rust, Swift, Kotlin, Scala,
Dart and PHP. `#` covers Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile and
PowerShell. Dozens of languages, a handful of comment syntaxes. That ratio is the only
reason any of this works without extensions.
**Detection** is a scoring pass. It counts markers at the starts of lines, block
delimiters that actually balance, how lines end, shebangs, preprocessor directives, a few
keyword shapes. Each syntax family then scores itself against that pile of evidence under
two rules:
1. *No markers, no score.* Structure alone never licenses comment removal. A shell script
full of braces and semicolons is not C.
2. *Bonuses cannot outvote markers.* Context can at most double a family's marker score,
so a pile of keywords can never overwhelm a single opposing marker.
Nothing clears the bar, nothing gets touched.
**Scanning** is one pass of a hand-written byte state machine. A 256-entry lookup table
means most bytes cost a single probe before they're skipped. No regex, nothing to
backtrack into. Roughly 70-250 MiB/s on a laptop, detection included.
**String literals** are where the difficulty lives, as the URL in the example above
suggests. comex handles raw strings (`r#"..."#`), character literals, Rust lifetimes
(`&'a T`), Matlab transposes, doubled-quote escapes (`'it''s'`), Lua long brackets, Ruby
`=begin`, Python docstrings, and the bodies of HTML `<script>` and `<style>` tags. An
unterminated quote stops at the end of its line, so one stray apostrophe in a shell script
can't eat the rest of the file.
## Safety
Leaving a comment behind is a cosmetic failure. Deleting a line of code is not. Every
ambiguous call in comex resolves toward the first one.
The rules are tuned against a sweep of ~10,000 real files: the macOS SDK headers, the
Python standard library, a pile of npm packages. Two invariants get checked on every one.
- **Deletion only.** The output is always a subsequence of the input. Bytes can
disappear. Nothing else happens to them.
- **Stability.** Stripping twice gives you what stripping once gave you, so a stripped
file never comes back as a different language.
Point it at your own code and see:
```console
$ cargo run --release --example sweep -- ~/your/repo
files 2753
bytes 23.18 MiB
throughput 75 MiB/s
no invariant violations
```
## Supported syntaxes
| `c` | `//` `/* */` | C, C++, C#, Java, JS, TS, Go, Rust, Swift, Kotlin, PHP, CSS |
| `hash` | `#` | Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile, PowerShell |
| `sql` | `--` `/* */` | SQL dialects |
| `lua` | `--` `--[[ ]]` | Lua |
| `haskell` | `--` `{- -}` | Haskell, Elm, PureScript |
| `lisp` | `;` `#\| \|#` | Lisp, Scheme, Clojure, Emacs Lisp, assembly |
| `markup` | `<!-- -->` | HTML, XML, SVG, Markdown, Vue, Svelte (plus embedded JS/CSS) |
| `ml` | `(* *)` | OCaml, F#, Pascal, Mathematica |
| `percent` | `%` | TeX/LaTeX, Erlang, Prolog, Matlab/Octave |
| `vb` | `'` `REM` | Visual Basic, VBA, VBScript |
| `batch` | `REM` `::` | Windows batch |
| `fortran` | `!` | modern Fortran |
| `ini` | `;` `#` | INI, desktop entries, gitconfig |
| `unknown` | none | nothing detected; input passes through unchanged |
Pass any of these to `-s` to skip detection entirely.
## Limits
All deliberate, and all of them err toward leaving your file alone.
- One ambiguous marker with nothing else to go on stays `unknown`. Pass `-s` if you know
better.
- PHP files that use both `#` and `//` lose the `#` comments, because the C family wins
the scoring. Those comments are deprecated in PHP anyway.
- JavaScript regex literals aren't fully parsed. A `/` followed by an unbalanced quote can
hide a trailing comment on that line. It can't delete code.
- Some data formats use two comment-ish markers at once. There are test-data files out
there that comment with `--` and use `#` as data, and comex can pick the wrong one.
`-d` will show you a low confidence when that happens.
- `x=1#c` in Python isn't a comment here, because there's no space before the `#`. YAML,
TOML, shell and Ruby all want that boundary, and `url: https://x/#top` has to survive.
## License
MIT © Alec Menke