lexer_rs/

lib.rs

//a Documentation
#![warn(missing_docs)]
// #![warn(missing_doc_code_examples)]
/*!

# Lexer library

This library provides a generic mechanism for parsing data into
streams of tokens.

This is commonly used in human-readable language compilers and
interpreters, to convert from a text stream into values that can then
be parsed according to the grammar of that language.§

A simple example would be for a calculator that operates on a stream
of numbers and mathematical symbols; the first step of processing that
the calculator must do is to convert the text stream into abstract
tokens such as 'the number 73' and 'the plus sign'. Once the
calculator has such tokens it can piece them together into a real
expression that it can then evaluate.

## Basic concept

The basic concept of a lexer is to convert a stream of (e.g.) [char]
into a stream of 'Token' - which will be specific to the lexer. The
lexer starts at the beginning of the text, and moves through consuming
characters into tokens.

## Lexer implementations

A lexer is not difficult to implement, and there are many alternative
approaches to doing so. A very simple approach for a [String] would be
to have a loop that matches the start of the string with possible
token values (perhaps using a regular expression), and on finding a
match it can 'trim' the front of the String, yield the token, and then
loop again.

This library provides an implementation option that gives the ability
to provide good error messages when things go wrong; it provides a
trait that allows abstraction of the lexer from the consumer (so that
one can get streams of tokens from a String, a BufRead, etc.); it
provides the infrastructure for any lexer using a simple mechanism for
parsing tokens.

# Positions in files

The crate provides some mechanisms for tracking the position of
parsing within a stream, so that error messages can be appropriately
crafted for the end user.

Tracking the position as a minimum is following the byte offset within
the file; additionally the line number and column number can also be
tracked. The [UserPosn] trait provides for this.

As Rust utilizes UTF8 encoded strings, not all byte offsets correspond
to actual [char]s in a stream, and the column separation between two
characters is not the difference between their byte offsets. The
[PosnInCharStream] adds to the [UserPosn] trait to manage this.

The bare minimum for a lexer handling UTF8-encoded strings does not
require tracking of lines and columns; only the byte offset tracking
*has* to be used; using a [usize] as the [PosnInCharStream]
implementation provides for this (as the *byte offset* within a [str].

The [Lexer] trait thus has an associated stream position type (its
'State'): this must be lightweight as it is moved around and copied
frequently, and must be static.

# Tokens

The token type that the [Lexer] trait produces from its parsing is
supplied by the client; this is normally a simple enumeration.

The parsing is managed by the [Lexer] with the client providing a
slice of matching functions; each matching function is applied in
turn, and the first that returns an Ok of a Some of a token yields the
token and advances the parsing state. The parsers can generate an
error if they detect a real error in the stream (not just a mismatch
to their token type).

# Error reporting

With the file position handling used within the [Lexer] it is possible
to display contextual error information - so if the whole text is
retained by the [Lexer] then an error can be displayed with the text
from the source with the error point/region highlighted.

Support for this is provided by the [FmtContext] trait, which is
implemented particularly for [LexerOfString].

!*/

//a Imports
mod char_stream;
mod fmt_context;
mod lexer;
mod posn_and_span;

//a Exports
pub use char_stream::CharStream;
pub use fmt_context::FmtContext;

pub use posn_and_span::LineColumn;
pub use posn_and_span::StreamCharPos;
pub use posn_and_span::StreamCharSpan;
pub use posn_and_span::{PosnInCharStream, UserPosn};

pub use crate::lexer::LexerOfStr;
pub use crate::lexer::LexerOfString;
pub use crate::lexer::ParserIterator;
pub use crate::lexer::SimpleParseError;
pub use crate::lexer::{BoxDynLexerParseFn, Lexer, LexerError, LexerParseFn, LexerParseResult};