Module phile::lexer

This module contains type definitions and functions for breaking up unstructured source text into lexemes and tokens. It also provides types for mapping tokens to their original location within the source code in a human-readable manner.

Structs

Location	Represents the location of a single extended grapheme cluster in the sources fed to the lexer.
Range	A half-open range representing a source span.
Token	Represents a lexeme and its associated type and location information as an abstract token.

Enums

TokenKind

Describes the type of a single token or lexeme.

Traits

Ranged

This trait is to be implemented by entities that correspond to some range in the source. This is used for generating location information in user-visible error messages.

Functions

lex	Given an array of source strings, returns an array of tokens extracted from those strings, or an error if there is a syntactic (more precisely, lexical) error in any of the source strings.