Crate langbox

A simple framework to build compilers and interpreters

This crate requires a nightly compiler because of the try_trait_v2 feature.

Usage

use langbox::*;

enum JsonTokenKind {
    // ...
}

// Make the reader an uninhabited type because we don't construct any objects of it
enum JsonTokenReader {}

impl TokenReader for JsonTokenReader {
    type TokenKind = JsonTokenKind;
     
    fn read_token(text: &str) -> ReadTokenResult<Self::TokenKind> {
        // ...
    }
}

struct JsonValue {
    // ...
}

fn jvalue() -> impl Parser<JsonTokenKind, JsonValue, String> {
    // ...
}

fn main() {
    // FileServer manages loading files for us.
    // It ensures the same physical file is never loaded twice.
    let mut file_server = FileServer::new();

    let file_path = "path/to/json/file.json";
    let file_id = file_server.register_file(file_path).unwrap();

    // After we have loaded a file we can tokenize it using a Lexer.
    type JLexer<'a> = Lexer<'a, JsonTokenReader, whitespace_mode::Remove>;
    let lexer = JLexer::new(file_id, &file_server);
    let tokens = lexer.collect::<Vec<_>>();
    let stream = TokenStream::new(&tokens);

    // Finally after all files have been tokenized we can parse the token stream.
    match jvalue().run(stream) {
        ParseResult::Match { value, .. } => {
            // `value` contains the parsed JSON value
        }
        ParseResult::NoMatch => { /* empty input */ }
        ParseResult::Err(_) => panic!("malformed JSON input"),
    }
}

Modules

whitespace_mode

Defines how a lexer processes whitespace

Macros

choice

Defines a parser as a choice of other parsers

parse_fn

Defines a parser using a closure function.

parser

Defines a parser using combinator DSL

sequence

Defines a parser as a sequence of other parsers

Structs

FileId

Uniquely identifies a file on the host computer

FileServer

Platform independant wrapper for reading unique files

Lexer

Transforms a source file into a token stream

ReadTokenResult

The result of reading a token

SourceFile

A file containing source code

SourceRef

A reference to source text

TextPosition

Defines a position in the text of a source file

TextSpan

Defines one contiguous region in the text of a source file

Token

A syntax token

TokenStream

A stream of syntax tokens

Enums

ParseResult

The result of running a parser

RegisterFileError

An error that can occur when registering a file

RegisterMemoryFileError

An error that can occur when registering an in-memory file

Traits

Parser

Transforms tokens from a token stream into structured data.

TextSpanCollection

A collection of text spans

TokenReader

Reads tokens from text input

TupleParser

A parser returning a tuple

WhitespaceMode

Defines how a lexer processes whitespace

Functions

always

Always matches.

between

Matches three parsers in sequence, returning only the second result.

eof

Matches the end of the token stream.