Crate wolf_lexer 

The wolfpack lexer: source text to tokens (spec chapter 1).

The lexer is written by hand. It never stops at an error: every problem becomes a diagnostic, the offending text becomes an TokenKind::Error token, and lexing continues. It never panics on any input.

Structs

Comment

An ordinary comment. It has no meaning for the language, but is kept for wolf fmt.

InvalidUtf8

A file whose bytes are not valid UTF-8 (spec chapter 1, rule 1.1).

Lexed

The result of lexing one file.

Token

A token: its kind and where it is.

Enums

FPart

A part of an interpolated string.

TokenKind

The kind of a token, with the value of literals.

Functions

decode

Decodes the bytes of a source file. A byte order mark is kept; the lexer skips it.

dump

Renders tokens one per line, for snapshot tests and wolf check --emit tokens.

lex

Lexes the text of one file.