Crate decomment 

Remove comments from source text.

One entry point. strip takes text and an Options value and returns the text with the selected comments removed. It never fails and never panics. The scanner is a single pass over the bytes with an explicit state for code, line comment, block comment, each string form, and regex literal, so a comment marker inside a string, a template, or a regex literal is left alone.

use decomment::{strip, Language, Options};

let source = "let url = \"http://example.com\"; // not the url\n";
let result = strip(source, Options::new(Language::Rust));
assert_eq!(result.output, "let url = \"http://example.com\"; \n");
assert_eq!(result.unterminated, None);

Limitations

• Thirteen languages, listed in Language. Shell, Ruby, PHP, Perl, YAML, and Lua are out, because heredocs, interpolation, and embedded sub-languages need more than a byte scanner. A half lexer for them would damage heredoc bodies.
• Comments inside an embedded HTML <script> or <style> element are left alone. Extract the element and call again with Language::JavaScript or Language::Css.
• Telling a regex literal from a division needs the preceding token. Two shapes are read as division, if (x) /re/.test(y) and a regex right after a block close. Getting either right needs a parser. Both fail in the direction that keeps code.
• The mirror of that limit. A / after an operator opens a regex candidate, so in text that is not valid JavaScript the candidate can run past a later // and leave that line comment in place. Nothing is deleted.
• Stripping is not idempotent. Removing a block comment can put a / next to a * that was not next to it before.

use decomment::{strip, Language, Options};

let once = strip("//* c */* b", Options::new(Language::Css)).output;
assert_eq!(once, "/* b");
let twice = strip(&once, Options::new(Language::Css)).output;
assert_eq!(twice, "");

Guarantees

The output is always a byte subsequence of the input. The scanner deletes and never inserts, reorders, or rewrites. Every byte inside a string, a template, a raw string, or a regex literal reaches the output. With Options::preserve_newlines the output holds exactly as many \n bytes and as many \r bytes as the input.

use decomment::{strip, Language, Options};

let mut options = Options::new(Language::JavaScript);
options.preserve_newlines = true;
let result = strip("a /* one\r\ntwo */ b", options);
assert_eq!(result.output, "a \r\n b");

Behavior worth stating

Each reading below is the one that keeps code. A scanner that reads the same bytes differently deletes something a compiler would have kept.

A regex literal is scanned as a literal, so a // inside it is not a line comment.

use decomment::{strip, Language, Options};

let source = "var re = /a//b/; var x = 1;";
assert_eq!(strip(source, Options::new(Language::JavaScript)).output, source);

/**/ is an empty block comment. The closer search starts at the byte after the two byte opener, so /*/ is unterminated and /***/ has body *.

use decomment::{strip, Language, Options};

assert_eq!(strip("a/**/b", Options::new(Language::JavaScript)).output, "ab");

Block comment nesting is a per-language fact. Rust and Swift nest. C, C++, C#, CSS, Go, HTML, Java, JavaScript, SQL, and TypeScript do not.

use decomment::{strip, Language, Options};

let source = "/* a /* b */ c */ var x = 1;";
let js = strip(source, Options::new(Language::JavaScript)).output;
assert_eq!(js, " c */ var x = 1;");
let rust = strip(source, Options::new(Language::Rust)).output;
assert_eq!(rust, " var x = 1;");

A quote always opens a string, whatever byte sits before it. A tagged template is string data, and text that is not valid source is still read with the quoted region as data.

use decomment::{strip, Language, Options};

let js = Options::new(Language::JavaScript);
assert_eq!(strip("tag`raw // text`; // real", js).output, "tag`raw // text`; ");
assert_eq!(strip("a\"raw // text\"; // real", js).output, "a\"raw // text\"; ");

An unterminated block comment is deleted to end of input, which is what a compiler does with one, and the deletion is reported.

use decomment::{strip, Language, Options, UnterminatedKind};

let result = strip("var x = 1; /* cut\nvar y = 2;", Options::new(Language::JavaScript));
assert_eq!(result.output, "var x = 1; ");
assert_eq!(result.unterminated.unwrap().kind, UnterminatedKind::BlockComment);
assert_eq!(result.unterminated.unwrap().offset, 11);

Code inside a ${} substitution is code, so a comment there is removed. The template text around it is string data and is left alone.

use decomment::{strip, Language, Options};

let js = Options::new(Language::JavaScript);
assert_eq!(strip("`a ${ b /* c */ } d`", js).output, "`a ${ b  } d`");

A line terminator after a block comment belongs to the code, not to the comment, so it stays.

use decomment::{strip, Language, Options};

let js = Options::new(Language::JavaScript);
assert_eq!(strip("x;\n/* b */\ny;\n", js).output, "x;\n\ny;\n");

Every option composes with every other option. No flag turns another off.

use decomment::{strip, Language, Mode, Options};

let mut options = Options::new(Language::JavaScript);
options.mode = Mode::First;
options.preserve_newlines = true;
let result = strip("a /* 1\n */ b /* 2 */ c", options);
assert_eq!(result.output, "a \n b /* 2 */ c");

Modes

Mode::All is not Mode::Line followed by Mode::Block. Under All, /* // */ strips to nothing and Line leaves it alone. Under All, // /* strips to nothing and Block leaves it alone. There is no composition law between the modes.

use decomment::{strip, Language, Mode, Options};

let mut options = Options::new(Language::JavaScript);
assert_eq!(strip("/* // */", options).output, "");
options.mode = Mode::Line;
assert_eq!(strip("/* // */", options).output, "/* // */");

Structs

Options

Settings for one crate::strip call.

Stripped

The result of a crate::strip call.

Unterminated

A construct that was still open when the input ran out.

Enums

Language

Comment and literal syntax to scan with.

Mode

Which comments to remove.

UnterminatedKind

Which construct was still open at end of input.

Functions

strip

Strip comments from source.