Crate basic_lexer

basic_lexer is a basic lexical scanner designed for use as the first stage of compiler construction, and produces tokens required by a parser. It was originally intended to support the parallel project RustLr, which is a LR-style parser generator, although each project is independent of the other. It is not intended to be the best possible lexical scanner as it does not build a DFA from regular expressions. It does not count white-spaces so it would not be appropriate for scanning python-like syntax. However, as this author did not find a satisfactory solution that suited all his needs, he was compelled to create his own. It is a component of a compiler that can be improved upon modularly.

The structures Str_tokenizer and File_tokenizer both implement Iterator and return Tokens. File_tokenizer is more full-featured and recognizes multi-line string literals and multi-line comments, with the option of keeping the comments as special tokens. The File_tokenizer is also capable of distinguishing keywords (such as “if”, “else”, “while”) from other alphanumeric tokens.

Example:

  let mut fscan = File_tokenizer::new("test.c");
  fscan.add_keywords("while return");
  fscan.set_line_comment("//");
  fscan.set_keep_comments(false);
  fscan.set_keep_newline(true);
  while let Some(token) = fscan.next() {
    println!("{:?}, line {}",&token,fscan.line_number());
  }

The supplied main.rs and file test.c contain additional usage examples.

Structs

File_tokenizer

a Token Iterator on a given file

Str_tokenizer

A Token Iterator on a given &str.

Enums

Token

Tokens are returned by the iterators Str_tokenizer and File_tokenizer.