Module token 

Token and friends, ported from include/hermes/Parser/JSLexer.h (Token, RegExpLiteral, StoredComment, StoredToken).

Unlike the C++, which holds UniqueString * pointers and SMLoc pointers into the buffer, the Rust port is offset-based: locations are SMRange (buffer id + byte offsets) and interned values are AtomBytes handles into the AtomTable. A Token carries its kind, source range, and (depending on the kind) a numeric value, an interned identifier/string-literal/raw value, or a RegExpLiteral — the complete Token surface, matching the C++.

The full set of value getters/setters (numeric/identifier/string/template/regexp/bigint/jsx) is part of the faithful Token surface; the public accessors are part of the lexer’s API (consumed by the parser), and the pub(crate) setters are all used by the scanners or exercised by tests.

Structs

RegExpLiteral

Port of JSLexer.h’s RegExpLiteral: an interned body and flags.

StoredComment

Represents a comment stored while lexing the file. Port of StoredComment.

StoredToken

Stored token when lexing. Port of StoredToken.

Token

Encapsulates the information contained in the current token. We only ever create one of these, but it is cleaner to keep the data in a separate class. Port of Token.

Enums

CommentKind

The kind of a stored comment. Port of StoredComment::Kind.