Crate ferric_lexer 

Lexer Module

The lexer’s purpose is to tokenize the input. Tokenization is the process of converting a sequence of characters into a sequence of tokens. These tokens are then used by the parser to generate an Abstract Syntax Tree (AST).

The lexer uses a combination of enumerations and macros to identify and categorize the different parts of the input.

Path: src/lexer/mod.rs

Structs

Lexer

The Lexer struct holds the input string and provides the functionality to tokenize it.

Enums

BuiltInFunction

Represents built-in functions provided by the language.

Keyword

Represents reserved keywords in the language.

Operator

Represents operators in the language.

Token

Represents the different types of tokens that can be identified by the lexer.