Struct JSLexer 

pub struct JSLexer<'a> { /* private fields */ }

The Hermes JavaScript lexer. Port of hermes::parser::JSLexer.

The lexer borrows the SourceErrorManager (for diagnostics) and the AtomTable interner, and owns a Cursor over a clone of the source buffer.

Implementations

impl<'a> JSLexer<'a>

pub fn dump_token(&self, out: &mut String)

Format the current token like the C++ js-lexer-dump line (without the trailing newline): "<start> <end> <nl> <KIND>[ <field> ...]". Phase-1b-i emits the ident= field for identifiers / private identifiers / reserved words; the other literal fields land in later phases.

impl<'a> JSLexer<'a>

pub fn advance_in_jsx_child(&mut self) -> &Token

Advance to the next token while scanning a JSX child. Port of JSLexer::advanceInJSXChild (JSLexer.cpp:749-809). Emits l_brace / less for { / <, eof at end of input, and otherwise accumulates a single jsx_text token (with HTML entities decoded into the value and kept verbatim in the raw) up to the next { / < / EOF.

impl<'a> JSLexer<'a>

pub fn lookahead1<const REQUIRE_NO_NEWLINE: bool>( &mut self, expected: Option<TokenKind>, ) -> Option<TokenKind>

Look ahead one token, restoring the lexer state afterwards (unless the next token matches expected, in which case the lookahead is consumed). Port of lookahead1 (JSLexer.cpp:1038-1095).

The C++ template <bool RequireNoNewLine> is the const generic REQUIRE_NO_NEWLINE.

pub fn lookahead2<const REQUIRE_NO_NEWLINE: bool>( &mut self, expected_ident: AtomBytes, ) -> Option<TokenKind>

Look ahead two tokens: if the next token is expected_ident, return the kind of the token after it; otherwise None. ALWAYS restores the lexer state. Port of lookahead2 (JSLexer.cpp:1100-1154).

The C++ template <bool RequireNoNewLine> is the const generic REQUIRE_NO_NEWLINE; the C++ single make_scope_exit that always restores becomes a computed result followed by an unconditional restore.

pub fn is_let_followed_by_decl_start(&mut self) -> bool

\return true if the let keyword (the current token) is followed by the start of a declaration. Port of isLetFollowedByDeclStart (JSLexer.cpp:134-176).

pub fn is_using_followed_by_identifier(&mut self) -> bool

\return true if the using keyword (the current token) is followed by an identifier with no intervening line terminator. Port of isUsingFollowedByIdentifier (JSLexer.cpp:178-204).

DEVIATION: the C++ takes the Keywords &kw and asserts the current token is kw.identUsing; we keep that as a debug_assert against the interned identifier bytes.

pub fn is_await_using_followed_by_identifier( &mut self, ident_using: AtomBytes, ) -> bool

\return true if await (the current token) is followed by using and then an identifier, with no intervening line terminators. Port of isAwaitUsingFollowedByIdentifier (JSLexer.cpp:206-253).

DEVIATION: the C++ takes Keywords &kw; the kw.identUsing atom is passed in as ident_using.

impl<'a> JSLexer<'a>

pub fn set_prev_token_end_loc(&mut self, loc: SMLoc)

Set the end location of the previous token. Port of setPrevTokenEndLoc.

pub fn save_point(&self) -> SavePoint

Store state of the lexer and allow rescanning from that point. Port of JSLexer::SavePoint::SavePoint (JSLexer.h:778-794). Can only save state when the current token is a punctuator, identifier, or rw_extends.

DEVIATION: the C++ SavePoint is an RAII-style object holding a JSLexer *. Rust cannot hold a &mut JSLexer across an advance call (which also needs &mut JSLexer), so we model it as a plain value snapshot whose restore(&mut JSLexer) re-applies the saved state.

pub fn is_current_token_a_directive(&mut self) -> bool

Check whether the current token is a directive, in other words is it a string literal without escapes or new line continuations, followed by either new line, semicolon or right brace. This doesn’t move the input pointer, so the optional semicolon, brace or the new line will be consumed normally by the next advance call. Port of isCurrentTokenADirective (JSLexer.cpp:911-1021).

\return true if the token can be interpreted as a directive.

pub fn rescan_rbrace_in_template_literal(&mut self) -> &Token

Rescan the } token as a TemplateMiddle or TemplateTail. Should be called in the middle of parsing a template literal. Port of rescanRBraceInTemplateLiteral (JSLexer.cpp:1023-1035).

impl<'a> JSLexer<'a>

pub fn new( buf_id: SourceId, sm: &'a mut SourceErrorManager, strtab: &'a AtomTable, grammar_context: GrammarContext, ) -> JSLexer<'a>

Construct a lexer over the buffer identified by buf_id in sm. Port of JSLexer::JSLexer + initializeWithBufferId. The reserved-word pre-interning (initializeReservedIdentifiers) is performed here so that res_word_ident is a cheap lookup during identifier scanning.

pub fn new_with_convert_surrogates( buf_id: SourceId, sm: &'a mut SourceErrorManager, strtab: &'a AtomTable, _grammar_context: GrammarContext, convert_surrogates: bool, ) -> JSLexer<'a>

Like new, but with control over the convert_surrogates option. When convert_surrogates is set, get_string_literal re-encodes the internal WTF-8 string form into valid UTF-8 (combining surrogate pairs and replacing unpaired surrogates with U+FFFD). Port of the JSLexer constructor’s convertSurrogates parameter.

pub fn get_string_literal(&self, bytes: &[u8]) -> AtomBytes

Intern a string-literal value, applying the convert_surrogates re-encoding when the option is set. Port of getStringLiteral (JSLexer.h:689-694).

pub fn token(&self) -> &Token

\return the current token.

pub fn is_strict_mode(&self) -> bool

\return whether the lexer is in strict mode. Port of isStrictMode.

pub fn set_strict_mode(&mut self, strict_mode: bool)

Set strict mode (affects future-reserved-word recognition). Port of setStrictMode.

pub fn is_new_line_before_current_token(&self) -> bool

\return whether a line terminator preceded the current token.

pub fn set_store_comments(&mut self, store_comments: bool)

Set whether comments should be stored instead of skipped. Port of setStoreComments.

pub fn get_store_tokens(&self) -> bool

\return whether tokens are being stored. Port of getStoreTokens.

pub fn set_store_tokens(&mut self, store_tokens: bool)

Set whether every token should be stored as it is lexed. Port of setStoreTokens.

pub fn store_current_token(&mut self)

Unconditionally store the current token in the token storage. Port of storeCurrentToken (JSLexer.h:548-551).

pub fn get_stored_comments(&self) -> &[StoredComment]

\return any stored comments to this point. Port of getStoredComments.

pub fn move_stored_comments(&mut self) -> Vec<StoredComment>

\return any stored comments to this point, moving them out of storage in the lexer and clearing the storage. Port of moveStoredComments.

pub fn get_stored_tokens(&self) -> &[StoredToken]

\return any stored tokens to this point. Port of getStoredTokens.

pub fn get_source_url(&self) -> Option<&str>

\return the source URL from the magic comment, or None if there was no magic comment. Port of getSourceURL.

pub fn get_source_mapping_url(&self) -> Option<&str>

\return the source mapping URL from the magic comment, or None if there was no magic comment. Port of getSourceMappingURL.

pub fn prev_token_end(&self) -> SMLoc

\return the end location of the previous token.

pub fn get_cur_loc(&self) -> SMLoc

\return the current char pointer location. Port of getCurLoc (JSLexer.h:567-569), which returns SMLoc::getFromPointer(curCharPtr_); the offset-based equivalent is the cursor’s current location.

pub fn get_buffer_id(&self) -> SourceId

\return the source buffer id we’re currently parsing. Port of getBufferId (JSLexer.h:704-706).

pub fn get_source_mgr(&self) -> &SourceErrorManager

\return the SourceErrorManager. Port of getSourceMgr (JSLexer.h:516).

pub fn get_source_mgr_mut(&mut self) -> &mut SourceErrorManager

\return a mutable reference to the SourceErrorManager so the parser can report errors through the lexer. Mirrors the non-const getSourceMgr() overload in JSLexer.h:516 (C++ returns a non-const reference).

pub fn get_string_table(&self) -> &AtomTable

\return the string interner. Port of getStringTable (JSLexer.h:523). (The C++ getAllocator has no Rust analog — the port uses the global allocator and AtomTable’s own interning, so there is no bump allocator.)

pub fn buffer_bytes(&self) -> &[u8]

\return the logical bytes of the buffer (the source text without the trailing NUL sentinel). Pointer->offset adaptation of getBufferStart/ getBufferEnd (JSLexer.h:709-716): C++ returns bufferStart_/ bufferEnd_ pointers; the offset-based equivalent is the buffer byte slice, with get_buffer_start == 0 and get_buffer_end == its length.

pub fn get_buffer_start(&self) -> u32

\return the start offset of the buffer (always 0). Pointer->offset adaptation of getBufferStart (JSLexer.h:708-711).

pub fn get_buffer_end(&self) -> u32

\return the end offset of the buffer (the logical byte length, excluding the trailing NUL sentinel). Pointer->offset adaptation of getBufferEnd (JSLexer.h:713-716).

pub fn check_following_character(&self, c: u8) -> bool

For certain identifier-like syntactic forms, like Flow’s renders? number, we need to check that the ? comes immediately after renders with no whitespace. Port of Token::checkFollowingCharacter (JSLexer.h:229-234), relocated from Token to JSLexer: our offset-based Token has no buffer reference, so the check reads the buffer byte at the current token’s end offset.

\return true iff the character directly after the current token matches c. The next byte could be the EOF NUL or the start of a UTF-8 sequence, but it is always present (the buffer is NUL-terminated, so the end offset is always in-bounds).

pub fn token_input_str(&self) -> &[u8]

\return the source text [start, end) of the current token. Port of Token::inputStr (JSLexer.h:136-140), relocated from Token to JSLexer: our offset-based Token has no buffer reference, so the slice is taken from the cursor’s buffer.

pub fn get_identifier(&self, name: &[u8]) -> AtomBytes

Intern an identifier. Port of getIdentifier(StringRef) (JSLexer.h:685-687).

pub fn convert_cur_token_to_ident_op(&mut self, kind: TokenKind)

Convert the current token to an identifier-operator token. Port of convertCurTokenToIdentOp (JSLexer.h:827-831). \pre the current token is an identifier which is an IDENT_OP operator.

pub fn force_eof(&mut self)

Force an EOF at the next token. Port of forceEOF.

pub fn seek(&mut self, loc: SMLoc)

Move the lexer to the specified spot. Any future advance calls will start from this position (the current token is not updated until such a call). Port of seek (JSLexer.h:699-701).

pub fn advance(&mut self, grammar_context: GrammarContext) -> &Token

Advance to the next token and return it. Port of JSLexer::advance (JSLexer.cpp:255-745).