comex 1.0.0

Fast, dependency-free comment extraction and removal for source files, with content-based syntax detection (no file extensions required).
Documentation
# comex

[![crates.io](https://img.shields.io/crates/v/comex.svg)](https://crates.io/crates/comex)
[![docs.rs](https://img.shields.io/docsrs/comex)](https://docs.rs/comex)
[![CI](https://github.com/AlecMenke/comex/actions/workflows/ci.yml/badge.svg)](https://github.com/AlecMenke/comex/actions/workflows/ci.yml)
[![license](https://img.shields.io/crates/l/comex.svg)](LICENSE)

Strip the comments out of source code, or throw the code away and keep only the comments.
You don't have to tell it what language the file is.

Given `profile.js`:

```js
// Look up a user's profile.
export async function profile(id) {
  const url = `https://api.example.com/users/#${id}`;  // '#' here is a fragment
  /* One round trip is enough:
     the endpoint already embeds the account. */
  return fetch(url);
}
```

Take the comments out:

```console
$ comex profile.js
export async function profile(id) {
  const url = `https://api.example.com/users/#${id}`;
  return fetch(url);
}
```

Or keep nothing else:

```console
$ comex -c profile.js
Look up a user's profile.

'#' here is a fragment
One round trip is enough:
the endpoint already embeds the account.
```

Note what survived: the `//` inside the URL and the `#` fragment inside the template
string. Getting that right is most of the work.

## Why not just use the extension?

Often there isn't one. Text shows up on stdin, or as a blob from somebody's API, or in a
file called `Dockerfile` or `Makefile` or `.env` or `install-sh`. An extension that does
exist can still be wrong. So comex reads the bytes and works out the comment syntax for
itself, then tells you what it decided:

```console
$ comex -d profile.js
c (confidence 0.48) - // and /* */ - C, C++, C#, Java, JS, TS, Go, Rust, Swift, PHP
scores: c=41 sql=12
```

When it can't tell, it says `unknown` and hands your file back untouched.

## Install

```console
$ cargo install comex          # CLI
$ cargo add comex              # library
```

No dependencies, and no regex anywhere in it.

## CLI

```console
$ comex -c -m src/*.rs | grep -i todo     # audit the TODOs in a tree
$ comex -k app.py                         # strip, keeping line numbers stable
$ comex -i -s hash deploy.sh              # rewrite in place, skip detection
$ cat weird_blob | comex -d               # identify before you act
```

Give it no FILE, or `-`, and it reads stdin.

```
comex [OPTIONS] [FILE]...

MODES:
    (default)             print the code with every comment removed
    -c, --comments        print only the comments
    -d, --detect          print the detected syntax and confidence, change nothing

OPTIONS:
    -s, --syntax <NAME>   force a comment syntax instead of detecting it
    -i, --in-place        rewrite each FILE instead of writing to stdout
    -k, --keep-lines      preserve line numbering (leave blank lines behind)
    -b, --keep-blank      keep lines that held nothing but a comment
    -m, --markers         keep comment markers when printing comments
    -D, --docstrings      treat leading triple-quoted strings as comments
    -S, --no-shebang      allow a leading `#!` line to be treated as a comment
    -n, --no-trim         do not trim whitespace left behind by a removed comment
        --stats           print per-file counts to stderr
    -l, --list            list the supported comment syntaxes
```

## Library

```rust
let src = "int x = 1; // the answer\n/* gone */\nint y = 2;\n";

assert_eq!(comex::strip_comments(src),   "int x = 1;\nint y = 2;\n");
assert_eq!(comex::extract_comments(src), "the answer\ngone\n");
```

If you already know the language, say so and skip detection. If you don't trust it, ask
what it decided before you act:

```rust
use comex::{Dialect, Options};

// Skip detection when you already know the language.
let out = comex::strip_comments_with("x = 1 # hi\n", &Dialect::Hash.syntax(), &Options::default());
assert_eq!(out, "x = 1\n");

// Or inspect what detection thinks before acting on it.
let d = comex::detect(src);
println!("{} ({:.2})", d.dialect.name(), d.confidence);
```

Need something finer? `comex::comments(src)` hands back the comment spans as byte ranges
and gets out of your way:

```rust
for span in comex::comments(src) {
    println!("{:?} at {}..{}: {}", span.kind, span.start, span.end, span.inner(src));
}
```

Full API on [docs.rs](https://docs.rs/comex).

## How it works

comex has no idea what JavaScript is. What it knows about is `//` and `/* */`, which
happen to cover C, C++, C#, Java, JavaScript, TypeScript, Go, Rust, Swift, Kotlin, Scala,
Dart and PHP. `#` covers Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile and
PowerShell. Dozens of languages, a handful of comment syntaxes. That ratio is the only
reason any of this works without extensions.

**Detection** is a scoring pass. It counts markers at the starts of lines, block
delimiters that actually balance, how lines end, shebangs, preprocessor directives, a few
keyword shapes. Each syntax family then scores itself against that pile of evidence under
two rules:

1. *No markers, no score.* Structure alone never licenses comment removal. A shell script
   full of braces and semicolons is not C.
2. *Bonuses cannot outvote markers.* Context can at most double a family's marker score,
   so a pile of keywords can never overwhelm a single opposing marker.

Nothing clears the bar, nothing gets touched.

**Scanning** is one pass of a hand-written byte state machine. A 256-entry lookup table
means most bytes cost a single probe before they're skipped. No regex, nothing to
backtrack into. Roughly 70-250 MiB/s on a laptop, detection included.

**String literals** are where the difficulty lives, as the URL in the example above
suggests. comex handles raw strings (`r#"..."#`), character literals, Rust lifetimes
(`&'a T`), Matlab transposes, doubled-quote escapes (`'it''s'`), Lua long brackets, Ruby
`=begin`, Python docstrings, and the bodies of HTML `<script>` and `<style>` tags. An
unterminated quote stops at the end of its line, so one stray apostrophe in a shell script
can't eat the rest of the file.

## Safety

Leaving a comment behind is a cosmetic failure. Deleting a line of code is not. Every
ambiguous call in comex resolves toward the first one.

The rules are tuned against a sweep of ~10,000 real files: the macOS SDK headers, the
Python standard library, a pile of npm packages. Two invariants get checked on every one.

- **Deletion only.** The output is always a subsequence of the input. Bytes can
  disappear. Nothing else happens to them.
- **Stability.** Stripping twice gives you what stripping once gave you, so a stripped
  file never comes back as a different language.

Point it at your own code and see:

```console
$ cargo run --release --example sweep -- ~/your/repo
files    2753
bytes    23.18 MiB
throughput 75 MiB/s
no invariant violations
```

## Supported syntaxes

| Name      | Markers            | Covers |
|-----------|--------------------|--------|
| `c`       | `//`  `/* */`      | C, C++, C#, Java, JS, TS, Go, Rust, Swift, Kotlin, PHP, CSS |
| `hash`    | `#`                | Python, Ruby, shell, Perl, YAML, TOML, R, Make, Dockerfile, PowerShell |
| `sql`     | `--`  `/* */`      | SQL dialects |
| `lua`     | `--`  `--[[ ]]`    | Lua |
| `haskell` | `--`  `{- -}`      | Haskell, Elm, PureScript |
| `lisp`    | `;`   `#\| \|#`    | Lisp, Scheme, Clojure, Emacs Lisp, assembly |
| `markup`  | `<!-- -->`         | HTML, XML, SVG, Markdown, Vue, Svelte (plus embedded JS/CSS) |
| `ml`      | `(* *)`            | OCaml, F#, Pascal, Mathematica |
| `percent` | `%`                | TeX/LaTeX, Erlang, Prolog, Matlab/Octave |
| `vb`      | `'`   `REM`        | Visual Basic, VBA, VBScript |
| `batch`   | `REM` `::`         | Windows batch |
| `fortran` | `!`                | modern Fortran |
| `ini`     | `;`   `#`          | INI, desktop entries, gitconfig |
| `unknown` | none               | nothing detected; input passes through unchanged |

Pass any of these to `-s` to skip detection entirely.

## Limits

All deliberate, and all of them err toward leaving your file alone.

- One ambiguous marker with nothing else to go on stays `unknown`. Pass `-s` if you know
  better.
- PHP files that use both `#` and `//` lose the `#` comments, because the C family wins
  the scoring. Those comments are deprecated in PHP anyway.
- JavaScript regex literals aren't fully parsed. A `/` followed by an unbalanced quote can
  hide a trailing comment on that line. It can't delete code.
- Some data formats use two comment-ish markers at once. There are test-data files out
  there that comment with `--` and use `#` as data, and comex can pick the wrong one.
  `-d` will show you a low confidence when that happens.
- `x=1#c` in Python isn't a comment here, because there's no space before the `#`. YAML,
  TOML, shell and Ruby all want that boundary, and `url: https://x/#top` has to survive.

## License

MIT © Alec Menke