Struct Lexer

pub struct Lexer<'a, 'i> { /* private fields */ }

The Lexer struct is responsible for tokenizing input source code into discrete tokens based on PHP language syntax. It is designed to work with PHP code from version 7.0 up to 8.4.

The lexer reads through the provided input and processes it accordingly.

It identifies PHP-specific tokens, including operators, keywords, comments, strings, and other syntax elements, and produces a sequence of Token objects that are used in further stages of compilation or interpretation.

The lexer is designed to be used in a streaming fashion, where it reads the input source code in chunks and produces tokens incrementally. This allows for efficient processing of large source files and minimizes memory usage.

Implementations

impl<'a, 'i> Lexer<'a, 'i>

pub fn new(interner: &'i ThreadedInterner, input: Input<'a>) -> Lexer<'a, 'i>

Creates a new Lexer instance.

Parameters

• interner: The interner to use for string interning.
• input: The input source code to tokenize.

Returns

A new Lexer instance that reads from the provided byte slice.

pub fn scripting( interner: &'i ThreadedInterner, input: Input<'a>, ) -> Lexer<'a, 'i>

Creates a new Lexer instance for parsing a script block.

Parameters

• interner: The interner to use for string interning.
• input: The input source code to tokenize.

Returns

A new Lexer instance that reads from the provided byte slice.

pub fn has_reached_eof(&self) -> bool

Check if the lexer has reached the end of the input.

If this method returns true, the lexer will not produce any more tokens.

pub fn get_position(&self) -> Position

Get the current position of the lexer in the input source code.

pub fn advance(&mut self) -> Option<Result<Token, SyntaxError>>

Tokenizes the next input from the source code.

This method reads from the input and produces the next Token based on the current [LexerMode]. It handles various lexical elements such as inline text, script code, strings with interpolation, comments, and different PHP-specific constructs.

Returns

• Some(Ok(Token)) if a token was successfully parsed.
• Some(Err(SyntaxError)) if a syntax error occurred while parsing the next token.
• None if the end of the input has been reached.

Examples

use mago_interner::ThreadedInterner;
use mago_syntax::lexer::Lexer;
use mago_source::SourceIdentifier;
use mago_syntax_core::input::Input;

let interner = ThreadedInterner::new();

let source = SourceIdentifier::dummy();
let input = Input::new(source, b"<?php echo 'Hello, World!'; ?>");

let mut lexer = Lexer::new(&interner, input);

while let Some(result) = lexer.advance() {
    match result {
        Ok(token) => println!("Token: {:?}", token),
        Err(error) => eprintln!("Syntax error: {:?}", error),
    }
}

Notes

• It efficiently handles tokenization by consuming input based on patterns specific to PHP syntax.
• The lexer supports complex features like string interpolation and different numeric formats.

Errors

Returns Some(Err(SyntaxError)) in cases such as:

• Unrecognized tokens that do not match any known PHP syntax.
• Unexpected tokens in a given context, such as an unexpected end of string.

Panics

This method should not panic under normal operation. If it does, it indicates a bug in the lexer implementation.

See Also

• Token: Represents a lexical token with its kind, value, and span.
• SyntaxError: Represents errors that can occur during lexing.

Trait Implementations

impl<'a, 'i> Debug for Lexer<'a, 'i>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.