Skip to main content

Crate comex

Crate comex 

Source
Expand description

comex strips the comments out of source code, or strips everything but the comments, without being told what language it is looking at.

let src = "int x = 1; // the answer\n/* gone */\nint y = 2;\n";
assert_eq!(comex::strip_comments(src), "int x = 1;\nint y = 2;\n");
assert_eq!(comex::extract_comments(src), "the answer\ngone\n");

§How it works

Languages are grouped by comment syntax, not by name: // + /* */ covers C, C++, C#, Java, JavaScript, TypeScript, Go, Rust, Swift, Kotlin and PHP; # covers Python, Ruby, shell, Perl, YAML, TOML and Make. detect scores those families against structural evidence in the text — line-leading markers, balanced block delimiters, shebangs, preprocessor lines, how lines end — and picks a winner, or Dialect::Unknown when the evidence is too thin. scan then makes one linear pass with a hand-written state machine: no regex, no backtracking, and a lookup table so uninteresting bytes cost one probe each.

String literals are tracked, which is the whole difficulty: "https://example.com" contains // and "#hashtag" contains #, and neither is a comment. Raw strings, character literals, Rust lifetimes, doubled-quote escapes and Lua long brackets are all handled, and an unterminated quote is confined to its own line so that one odd apostrophe cannot swallow a file.

§Safety bias

Ambiguity resolves toward leaving the input alone. Deleting a real line of code is much worse than missing a comment, so weak evidence yields Dialect::Unknown and an untouched file. When you do know the language, skip detection entirely:

use comex::{Dialect, Options};
let out = comex::strip_comments_with("x = 1 # hi\n", &Dialect::Hash.syntax(), &Options::default());
assert_eq!(out, "x = 1\n");

Structs§

BlockRule
A comment delimited by an opening and closing marker, possibly spanning lines.
Detection
The outcome of detect.
LineRule
A comment that runs from a marker to the end of the line.
Options
Output settings for stripping and extraction.
ScanConfig
Knobs for scan.
Span
A classified byte range of the input.
StringRule
A string or character literal. Strings are not output, they are protected: a # inside "..." must not be mistaken for a comment.
Syntax
The lexical rules used to scan a file.

Enums§

Dialect
The comment-syntax families comex understands.
SpanKind
What a Span covers.

Constants§

THRESHOLD
Minimum winning score. Below this the input is treated as having no comment syntax.

Functions§

comments
The comment spans of src, detecting the syntax from its content.
comments_with
The comment spans of src under an explicit syntax.
detect
Identify the comment syntax of src from its content alone.
extract_comments
Keep only the comments in src, detecting the syntax from its content.
extract_comments_with
Keep only the comments in src using an explicit syntax.
scan
Classify src under syn, returning spans in source order and without overlap.
strip_comments
Remove every comment from src, detecting the syntax from its content.
strip_comments_with
Remove every comment from src using an explicit syntax.