Struct Parser 

pub struct Parser {
    pub parse_comments: bool,
    /* private fields */
}

Provide methods to implement a recursive descent parser.

This struct wraps a stream (an instance of Read) and provides two primitives.

• You can “peek” at upcoming characters in the stream.
• You can “consume” characters from the stream.

These primitives are used to construct a series of more complex methods, but essentially all you can do is look at what is coming up in the stream, and then consume and discard characters from the stream.

Whitespace

Aside from methods specifically to consume whitespace, there are special versions of some “consume” methods that consume whitespace following a match. These methods end with _ws and consume any trailing whitespace following a successful match.

Whitespace includes only those characters that satisfy the char::is_whitespace() predicate. This includes Unicode whitespace characters. See Self::consume_ws_only for details.

Comments

This is able to parse comments, using an embedded CommentParser instance. The default comment parser handles C and C++-style comments.

If you want to configure the comment parser, you can borrow a mutable reference to it using Self::borrow_comment_parser().

use trivet::parsers::comments::CommentParser;
use trivet::Parser;
use trivet::parse_from_string;

let mut parser = parse_from_string("# This is a comment.");
let mut compar = parser.borrow_comment_parser();
compar.enable_c = false;
compar.enable_cpp = false;
compar.enable_python = true;
parser.consume_ws();
assert!(parser.is_at_eof());

If your language does not support comments, you can disable them completely and possibly gain a bit of performance by setting the Self::parse_comments flag to false.

Parsing Strings

String parsing is provided by a StringParser instance. This instance can be obtained and configured using Self::borrow_string_parser.

Strings are parsed on-demand by the Self::parse_string method. This takes an optional terminator character.

use trivet::Parser;
use trivet::errors::ParseResult;
use trivet::parse_from_string;

// We will use «..» quotes here.
let mut parser = parse_from_string(r#"«This is\b\bwas\u{a}\x09some text»"#);
// Consume the opening quote mark.
parser.consume();
// Now consume until the closing mark.
let text = parser.parse_string_until_delimiter('\u{BB}')?;
assert_eq!(text, "This is\x08\x08was\n\tsome text");

Parsing Numbers

Number parsing is provided by a NumberParser instance. This instance can be obtained and configured using Self::borrow_number_parser.

Numbers are parsed on-demand by the Self::parse_i128, Self::parse_u128, and Self::parse_f64 methods and their _ws counterparts.

Parsing Keywords

Keyword parsing is provided by a KeywordParser instance. This instance can be obtained and configured using Self::borrow_keyword_parser.

Keywords are parsed on-demand by the Self::parse_keyword method and its _ws counterpart.

Method Names

There are many methods. Most method names have the following form:

[peek/consume][type][whitespace]

[peek/consume]

• The peek methods look ahead at the character source. Their return is either the characters themselves or a Boolean indicating a match or failure to match.
• The consume methods discard characters from the character source. Their return is either nothing (for unconditional consumes) or a Boolean if something was conditionally consumed.
• The peek_and_consume method both match a string and consume that string when it is matched, and only when it is matched.

[type]

• Methods that work with a single character have no additional specification.
• Methods that work with a specific number of characters are suffixed with _n. These do not test characters, but return or consume them unconditionally.
• Methods that work with a vector of chars have _chars in the name.
• Methods that work with a string have _str in the name. These methods are absent from ParserCore.

[whitespace]

• Methods that consume trailing whitespace (possibly including comments) are suffixed with _ws. These methods are absent from ParserCore.

Some method name combinations do not make sense, and are not present. For instance, peek_ws. The following is a quick reference to the implemented methods. Methods indicated with an * are in Parser, but not in ParserCore.

Method	Input	Return	Only in Parser
Self::peek		char
Self::peek_n	usize	String
Self::peek_n_vec	usize	Vec<char>
Self::peek_offset	usize	char
Self::peek_chars	Vec<char>	bool
Self::peek_chars_greedy (see below)	Vec<char>	bool	*
Self::peek_str	&str	bool	*
Self::peek_str_greedy (see below)	&str	bool	*
Self::consume
Self::consume_n	usize
Self::consume_ws		bool	*
Self::peek_and_consume	char	bool
Self::peek_and_consume_ws	char	bool	*
Self::peek_and_consume_chars	Vec<char>	bool
Self::peek_and_consume_chars_ws	Vec<char>	bool	*
Self::peek_and_consume_str	&str	bool	*
Self::peek_and_consume_str_ws	&str	bool	*
Self::take_while	closure	String
Self::take_while_unless	closure	String
Self::take_until	String	String
Self::take_until_greedy (see below)	String	String	*
Self::consume_while	closure	bool
Self::consume_until	String	bool

The additional method Self::consume_ws_only consumes whitespace and is present in ParserCore, but does not consume comments even if there is a comment parser installed.

The special _greedy methods exist to handle a special case. Suppose the input stream contains """"""" and you want to match on three consecutive quotation marks. The Self::peek_str method will match on the first three quotation marks. The Self::peek_str_greedy method will match on the last three quotation marks. These methods are not in ParserCore.

Example

To use this, make an instance with Self::new(), giving a stream and a name for the stream. The name given will be used when constructing Loc instances. If the name is the empty string, then Console locations are created; otherwise File locations are created.

The following shows how to create a parser around the standard input.

use std::io;
use trivet::Parser;
use trivet::parse_from_stdin;

// Make a new parser around the standard input.
let mut parser = parse_from_stdin();
// Invoke methods to parse...

Use Self::peek() to peek at the next character in the stream, and use Self::consume() to discard that character. Note that Self::peek() returns a result, and may be Ok(None) if at the end of stream. The following uses these two methods to ignore whitespace (though there actually is a method just for that).

use trivet::Parser;
use trivet::parse_from_string;

// Make a new parser around the standard input.
let mut parser = parse_from_string("\t\tx");

// While the next character is whitespace, consume it.
while parser.peek().is_whitespace() {
    parser.consume();
}

// The stream should now be pointing to the first
// non-whitespace character.
assert_eq!(parser.peek(), 'x');

This would be a very tedious way to parse anything, so there are other methods to help. In particular, Self::consume_ws() will do the job just outlined above.

More interestingly you can look to see if the next thing in the stream is a given string with Self::peek_str() and Self::peek_str_greedy(). Usually once you match an upcoming string, you probably want to consume it. To both check for the string and consume it if you find it, use Self::peek_and_consume() (for single characters) or Self::peek_and_consume_str() (for strings).

You may also want to consume a sequence of characters, such as all the digits or letters in a stream. The method Self::take_while() will do that. It takes a predicate on characters (usually a lambda) and consumes characters while the predicate is true. All the characters are then returned. The following uses this to obtain the next decimal integer from the stream.

use std::io;
use trivet::Parser;
use trivet::parse_from_string;

// Make a new parser around the standard input.
let mut parser = parse_from_string("-14x");

// Look for a minus sign.
let minus = parser.peek_and_consume('-');

// Read the next integer from the stream.
let digits = parser.take_while(|ch| ch.is_ascii_digit());
let number: i64 = (if minus { -1 } else { 1 })
    * digits.parse().unwrap_or(0);
assert_eq!(number, -14i64);

// The stream now points to the first non-digit character.
assert_eq!(parser.peek(), 'x');

Fields

parse_comments: bool

If true (default) then the Self::consume_ws method and other _ws methods look for and consume comments. If false, then they consume whitespace only.

Implementations

impl Parser

pub fn new(name: &str, decoder: Decode) -> Self

Create a new parser using the given decoder as the source of characters. A name is given that will be used when creating Loc instances.

pub fn borrow_core(&mut self) -> &mut ParserCore

Borrow the parser core. This returns a mutable borrow of the internal parser core. This is probably only needed if you want to run or test an external comment parser.

Examples found in repository

examples/numbers.rs (line 26)

pub fn main() {
    println!("Enter a number to see how Trivet parses it.  Enter a blank line to stop.");
    let numpar = NumberParser::new();
    print!("> ");
    let _ = stdout().flush();
    for line in stdin().lock().lines() {
        if line.is_err() {
            break;
        }
        let line = line.unwrap();

        // Consume any whitespace.
        let mut parser = parse_from_string(&line);
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }

        // Try to read a number as pieces.
        let parts = numpar.get_parts(parser.borrow_core());
        println!("{:?}", parts);
        if parts.is_apparent_float() {
            let value: f64 = parts.into();
            println!("f64: {}", value);
        } else {
            let value: i128 = parts.into();
            println!("i128: {}", value);
        }

        // Go around again.
        print!("> ");
        let _ = stdout().flush();
    }
}

examples/book_json_parser.rs (line 138)

pub fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.parse_comments = false;
    let numpar = parser.borrow_number_parser();
    numpar.settings.permit_binary = false;
    numpar.settings.permit_hexadecimal = false;
    numpar.settings.permit_octal = false;
    numpar.settings.permit_underscores = false;
    numpar.settings.decimal_only_floats = true;
    numpar.settings.permit_plus = false;
    numpar.settings.permit_leading_zero = false;
    numpar.settings.permit_empty_whole = false;
    numpar.settings.permit_empty_fraction = false;
    let strpar = parser.borrow_string_parser();
    strpar.set(trivet::strings::StringStandard::JSON);
    let _ = parser
        .borrow_core()
        .replace_whitespace_test(Box::new(|ch| [' ', '\n', '\r', '\t'].contains(&ch)));
    parser.consume_ws();
    let result = parse_value_ws(&mut parser);
    match result {
        Err(error) => {
            println!("ERROR: {}", error);
            std::process::exit(1);
        }
        Ok(json) => {
            // If there is any trailing stuff that is not whitespace, then this is not a valid
            // JSON file.
            if parser.is_at_eof() {
                // Print the JSON value.
                println!("{:?}", json);
                std::process::exit(0)
            } else {
                println!("Found unexpected trailing characters after JSON value.");
                std::process::exit(1);
            }
        }
    }
}

pub fn borrow_comment_parser(&mut self) -> &mut CommentParser

Borrow the comment parser. This returns a mutable borrow of the internal comment parser that will allow you to configure the parser.

Examples found in repository

examples/book_primitives_lua.rs (line 11)

fn main() {
    // Text to parse.  Note that the comment ends on line 5 at column 12, with
    // the first non-comment position at column 13.
    let mut parser = trivet::parse_from_string(
        r#"
        --[[
            I am a long form
            Lua comment.
        --]]"#,
    );
    parser.borrow_comment_parser().enable_c = false;
    parser.borrow_comment_parser().enable_cpp = false;
    parser.borrow_comment_parser().custom = Box::new(|parser: &mut trivet::ParserCore| -> bool {
        if parser.peek_and_consume_chars(&['-', '-', '[', '[']) {
            parser.take_until("--]]");
            true
        } else if parser.peek_and_consume_chars(&['-', '-']) {
            parser.take_while(|ch| ch != '\n');
            true
        } else {
            false
        }
    });
    parser.borrow_comment_parser().enable_custom = true;
    parser.consume_ws();
    assert_eq!(parser.loc().to_string(), "<string>:5:13");
    assert!(parser.is_at_eof());
}

pub fn replace_comment_parser(&mut self, compar: CommentParser) -> CommentParser

Replace the internal comment parser with the given instance of a comment parser. The prior comment parser is returned.

pub fn borrow_number_parser(&mut self) -> &mut NumberParser

Borrow the number parser. This returns a mutable borrow of the internal number parser that will allow you to configure the parser.

Examples found in repository

examples/book_json_parser.rs (line 125)

pub fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.parse_comments = false;
    let numpar = parser.borrow_number_parser();
    numpar.settings.permit_binary = false;
    numpar.settings.permit_hexadecimal = false;
    numpar.settings.permit_octal = false;
    numpar.settings.permit_underscores = false;
    numpar.settings.decimal_only_floats = true;
    numpar.settings.permit_plus = false;
    numpar.settings.permit_leading_zero = false;
    numpar.settings.permit_empty_whole = false;
    numpar.settings.permit_empty_fraction = false;
    let strpar = parser.borrow_string_parser();
    strpar.set(trivet::strings::StringStandard::JSON);
    let _ = parser
        .borrow_core()
        .replace_whitespace_test(Box::new(|ch| [' ', '\n', '\r', '\t'].contains(&ch)));
    parser.consume_ws();
    let result = parse_value_ws(&mut parser);
    match result {
        Err(error) => {
            println!("ERROR: {}", error);
            std::process::exit(1);
        }
        Ok(json) => {
            // If there is any trailing stuff that is not whitespace, then this is not a valid
            // JSON file.
            if parser.is_at_eof() {
                // Print the JSON value.
                println!("{:?}", json);
                std::process::exit(0)
            } else {
                println!("Found unexpected trailing characters after JSON value.");
                std::process::exit(1);
            }
        }
    }
}

pub fn replace_number_parser(&mut self, numpar: NumberParser) -> NumberParser

Replace the internal string parser with the given instance of a string parser. The prior string parser is returned.

pub fn borrow_string_parser(&mut self) -> &mut StringParser

Borrow the string parser. This returns a mutable borrow of the internal string parser that will allow you to configure the parser.

Examples found in repository

examples/book_json_parser.rs (line 135)

pub fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.parse_comments = false;
    let numpar = parser.borrow_number_parser();
    numpar.settings.permit_binary = false;
    numpar.settings.permit_hexadecimal = false;
    numpar.settings.permit_octal = false;
    numpar.settings.permit_underscores = false;
    numpar.settings.decimal_only_floats = true;
    numpar.settings.permit_plus = false;
    numpar.settings.permit_leading_zero = false;
    numpar.settings.permit_empty_whole = false;
    numpar.settings.permit_empty_fraction = false;
    let strpar = parser.borrow_string_parser();
    strpar.set(trivet::strings::StringStandard::JSON);
    let _ = parser
        .borrow_core()
        .replace_whitespace_test(Box::new(|ch| [' ', '\n', '\r', '\t'].contains(&ch)));
    parser.consume_ws();
    let result = parse_value_ws(&mut parser);
    match result {
        Err(error) => {
            println!("ERROR: {}", error);
            std::process::exit(1);
        }
        Ok(json) => {
            // If there is any trailing stuff that is not whitespace, then this is not a valid
            // JSON file.
            if parser.is_at_eof() {
                // Print the JSON value.
                println!("{:?}", json);
                std::process::exit(0)
            } else {
                println!("Found unexpected trailing characters after JSON value.");
                std::process::exit(1);
            }
        }
    }
}

pub fn replace_string_parser(&mut self, strpar: StringParser) -> StringParser

Replace the internal string parser with the given instance of a string parser. The prior string parser is returned.

pub fn borrow_keyword_parser(&mut self) -> &mut KeywordParser

Borrow the keyword parser. This returns a mutable borrow of the internal keyword parser that will allow you to configure the parser.

Examples found in repository

examples/book_keywords_permit.rs (line 18)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_permit = true;
    parser.borrow_keyword_parser().permit = Box::new(keyword);
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_transform.rs (line 3)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_transform = true;
    parser.borrow_keyword_parser().transform =
        Box::new(|ch| if [':', '.'].contains(&ch) { '_' } else { ch });
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

pub fn replace_keyword_parser(&mut self, strpar: KeywordParser) -> KeywordParser

Replace the internal keyword parser with the given instance of a keyword parser. The prior keyword parser is returned.

pub fn parse_string_match_delimiter(&mut self) -> ParseResult<String>

Parse the next string. The parse is assumed to be pointing to the opening delimiter, and the string is terminated by the next occurrence of the delimiter.

pub fn parse_string_until_delimiter( &mut self, delimiter: char, ) -> ParseResult<String>

Parse the next string. The opening delimiter, if any, should have already been consumed. Provide the closing delimiter to look for.

pub fn parse_string_match_delimiter_ws(&mut self) -> ParseResult<String>

Parse the next string. The parse is assumed to be pointing to the opening delimiter, and the string is terminated by the next occurrence of the delimiter. Consume any whitespace after the closing delimiter.

Examples found in repository

examples/book_string_simple.rs (line 9)

fn main() -> trivet::errors::ParseResult<()> {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        // Save the location for later use in error messages.
        let loc = parser.loc();
        // Parse strings only for the supported delimiters.
        let string = match parser.peek() {
            // The initial and final delimiters match.
            '"' | '\'' => parser.parse_string_match_delimiter_ws()?,
            // Here the initial and final delimiters do not match.
            '«' => {
                parser.consume();
                parser.parse_string_until_delimiter_ws('»')?
            }
            // This is not a legal opening delimiter.
            ch => {
                return Err(trivet::errors::syntax_error(
                    loc,
                    &format!("Invalid delimiter: '{}'", ch),
                ));
            }
        };
        println!("  --> {:?}", string);
    }
    Ok(())
}

examples/book_json_parser.rs (line 25)

pub fn parse_value_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    match parser.peek() {
        // Check for an object.
        '{' => parse_object_ws(parser),

        // Check for an array.
        '[' => parse_array_ws(parser),

        // Check for a string.
        '"' => Ok(JSON::String(parser.parse_string_match_delimiter_ws()?)),

        // Check for a number.
        ch if ch == '-' || ch.is_ascii_digit() => Ok(JSON::Number(parser.parse_f64_decimal_ws()?)),

        // Check for Boolean true.
        't' if parser.peek_and_consume_str_ws("true") => Ok(JSON::Boolean(true)),

        // Check for Boolean false.
        'f' if parser.peek_and_consume_str_ws("false") => Ok(JSON::Boolean(false)),

        // Check for null.
        'n' if parser.peek_and_consume_str_ws("null") => Ok(JSON::Null),

        // Error.
        ch => Err(trivet::errors::unexpected_text_error(
            parser.loc(),
            "the start of a JSON value",
            &ch.to_string(),
        )),
    }
}

/// Parse a JSON object.  On entry the parser is assumed to be pointing to the first
/// character of the object (a curly brace `{`).  Trailing whitespace is consumed.
fn parse_object_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    parser.consume(); // Consume opening brace.
    parser.consume_ws();
    let mut first = true;
    let mut map = HashMap::new();
    while !parser.peek_and_consume_ws('}') {
        // If not the first element, then we expect to have a comma here.  JSON requires
        // a comma between members of an object.
        if first {
            first = false;
        } else if !parser.peek_and_consume_ws(',') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "There must be a comma (,) between members of an object.",
            ));
        }

        // Look for the quotation mark for a string.
        if parser.peek() != '"' {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be quoted strings.",
            ));
        }
        let name = parser.parse_string_match_delimiter_ws()?;

        // Look for the colon.
        if !parser.peek_and_consume_ws(':') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be followed by a colon (:).",
            ));
        }

        // Get the value.
        let value = parse_value_ws(parser)?;

        // Add the name, value pair to the map.
        map.insert(name, value);
    }
    Ok(JSON::Object(map))
}

examples/book_cookbook_things.rs (line 18)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    loop {
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }
        if parser.peek_and_consume_str_ws("string") {
            parser.expect(':')?;
            parser.consume_ws();
            let ch = parser.peek();
            if ch == '"' || ch == '\'' {
                let string = parser.parse_string_match_delimiter_ws()?;
                println!("Got string: {}", StringEncoder::new().encode(&string));
            }
        } else if parser.peek_and_consume_str_ws("integer") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_u128_ws()?;
            println!("Got integer: {}", number);
        } else if parser.peek_and_consume_str_ws("float") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_f64_ws()?;
            println!("Got float: {}", number);
        } else if parser.peek_and_consume_str_ws("json") {
            parser.expect(':')?;
            parser.consume_ws();
            let json = JSONParser::new().parse_value_unstandard_ws(&mut parser)?;
            println!("Got JSON:\n{}", json);
        } else {
            return Err(syntax_error(
                parser.loc(),
                "Unexpected element found in stream",
            ));
        }
    }
    Ok(())
}

pub fn parse_string(&self, string: &str) -> ParseResult<String>

Use the embedded string parse to parse the provided string. This reads from the provided string; nothing is consumed during this process. The primary use case for this is matching a more complex string environment and then using the embedded string parser to decode the result of the match.

pub fn parse_string_until_delimiter_ws( &mut self, terminator: char, ) -> ParseResult<String>

Parse the next string. The opening delimiter, if any, should have already been consumed. Provide the closing delimiter to look for. Consume any trailing whitespace after the closing delimiter.

Examples found in repository

examples/book_string_simple.rs (line 13)

fn main() -> trivet::errors::ParseResult<()> {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        // Save the location for later use in error messages.
        let loc = parser.loc();
        // Parse strings only for the supported delimiters.
        let string = match parser.peek() {
            // The initial and final delimiters match.
            '"' | '\'' => parser.parse_string_match_delimiter_ws()?,
            // Here the initial and final delimiters do not match.
            '«' => {
                parser.consume();
                parser.parse_string_until_delimiter_ws('»')?
            }
            // This is not a legal opening delimiter.
            ch => {
                return Err(trivet::errors::syntax_error(
                    loc,
                    &format!("Invalid delimiter: '{}'", ch),
                ));
            }
        };
        println!("  --> {:?}", string);
    }
    Ok(())
}

pub fn parse_u128(&mut self) -> ParseResult<u128>

Parse an integer. The parser should be on the first character of the integer.

pub fn parse_i128(&mut self) -> ParseResult<i128>

Parse an integer. The parser should be on the first character of the integer.

pub fn parse_u64(&mut self) -> ParseResult<u64>

Parse an integer. The parser should be on the first character of the integer.

pub fn parse_i64(&mut self) -> ParseResult<i64>

Parse an integer. The parser should be on the first character of the integer.

pub fn parse_f64(&mut self) -> ParseResult<f64>

Parse a floating point number. The parser should be on the first character of the number. Radix specifiers are honored, if configured.

pub fn parse_f64_decimal(&mut self) -> ParseResult<f64>

Parse a floating point number. The parser should be on the first character of the number. Radix specifiers are not allowed; the number is assumed to be decimal.

pub fn parse_u128_ws(&mut self) -> ParseResult<u128>

Parse an integer. The parser should be on the first character of the integer. Any trailing whitespace (and possibly comments) is consumed.

Examples found in repository

examples/book_cookbook_things.rs (line 24)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    loop {
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }
        if parser.peek_and_consume_str_ws("string") {
            parser.expect(':')?;
            parser.consume_ws();
            let ch = parser.peek();
            if ch == '"' || ch == '\'' {
                let string = parser.parse_string_match_delimiter_ws()?;
                println!("Got string: {}", StringEncoder::new().encode(&string));
            }
        } else if parser.peek_and_consume_str_ws("integer") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_u128_ws()?;
            println!("Got integer: {}", number);
        } else if parser.peek_and_consume_str_ws("float") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_f64_ws()?;
            println!("Got float: {}", number);
        } else if parser.peek_and_consume_str_ws("json") {
            parser.expect(':')?;
            parser.consume_ws();
            let json = JSONParser::new().parse_value_unstandard_ws(&mut parser)?;
            println!("Got JSON:\n{}", json);
        } else {
            return Err(syntax_error(
                parser.loc(),
                "Unexpected element found in stream",
            ));
        }
    }
    Ok(())
}

pub fn parse_i128_ws(&mut self) -> ParseResult<i128>

Parse an integer. The parser should be on the first character of the integer. Any trailing whitespace (and possibly comments) is consumed.

pub fn parse_u64_ws(&mut self) -> ParseResult<u64>

Parse an integer. The parser should be on the first character of the integer. Any trailing whitespace (and possibly comments) is consumed.

pub fn parse_i64_ws(&mut self) -> ParseResult<i64>

Parse an integer. The parser should be on the first character of the integer. Any trailing whitespace (and possibly comments) is consumed.

pub fn parse_f64_ws(&mut self) -> ParseResult<f64>

Parse a floating point number. The parser should be on the first character of the number. Radix specifiers are honored, if configured. Trailing whitespace is consumed.

Examples found in repository

examples/book_math_parser.rs (line 17)

pub fn parse_number_ws(parser: &mut Parser) -> ParseResult<f64> {
    parser.parse_f64_ws()
}

examples/book_numbers_float.rs (line 4)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        match parser.parse_f64_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_cookbook_things.rs (line 29)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    loop {
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }
        if parser.peek_and_consume_str_ws("string") {
            parser.expect(':')?;
            parser.consume_ws();
            let ch = parser.peek();
            if ch == '"' || ch == '\'' {
                let string = parser.parse_string_match_delimiter_ws()?;
                println!("Got string: {}", StringEncoder::new().encode(&string));
            }
        } else if parser.peek_and_consume_str_ws("integer") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_u128_ws()?;
            println!("Got integer: {}", number);
        } else if parser.peek_and_consume_str_ws("float") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_f64_ws()?;
            println!("Got float: {}", number);
        } else if parser.peek_and_consume_str_ws("json") {
            parser.expect(':')?;
            parser.consume_ws();
            let json = JSONParser::new().parse_value_unstandard_ws(&mut parser)?;
            println!("Got JSON:\n{}", json);
        } else {
            return Err(syntax_error(
                parser.loc(),
                "Unexpected element found in stream",
            ));
        }
    }
    Ok(())
}

pub fn parse_f64_decimal_ws(&mut self) -> ParseResult<f64>

Parse a floating point number. The parser should be on the first character of the number. Radix specifiers are not allowed; the number is assumed to be decimal. Trailing whitespace is consumed.

Examples found in repository

examples/book_json_parser.rs (line 28)

pub fn parse_value_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    match parser.peek() {
        // Check for an object.
        '{' => parse_object_ws(parser),

        // Check for an array.
        '[' => parse_array_ws(parser),

        // Check for a string.
        '"' => Ok(JSON::String(parser.parse_string_match_delimiter_ws()?)),

        // Check for a number.
        ch if ch == '-' || ch.is_ascii_digit() => Ok(JSON::Number(parser.parse_f64_decimal_ws()?)),

        // Check for Boolean true.
        't' if parser.peek_and_consume_str_ws("true") => Ok(JSON::Boolean(true)),

        // Check for Boolean false.
        'f' if parser.peek_and_consume_str_ws("false") => Ok(JSON::Boolean(false)),

        // Check for null.
        'n' if parser.peek_and_consume_str_ws("null") => Ok(JSON::Null),

        // Error.
        ch => Err(trivet::errors::unexpected_text_error(
            parser.loc(),
            "the start of a JSON value",
            &ch.to_string(),
        )),
    }
}

pub fn parse_keyword(&mut self) -> ParseResult<String>

Parse a keyword from the stream using the keyword parser. The parser should be on the first character of the keyword.

pub fn parse_keyword_ws(&mut self) -> ParseResult<String>

Parse a keyword from the stream using the keyword parser. The parser should be on the first character of the keyword. Trailing whitespace is consumed.

Examples found in repository

examples/book_keywords.rs (line 4)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_permit.rs (line 21)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_permit = true;
    parser.borrow_keyword_parser().permit = Box::new(keyword);
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_transform.rs (line 7)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_transform = true;
    parser.borrow_keyword_parser().transform =
        Box::new(|ch| if [':', '.'].contains(&ch) { '_' } else { ch });
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

pub fn loc(&self) -> Loc

Get the current location in the parse. This will return either a console (if the name is the empty string) or a file location (if the name was not the empty string).

Examples found in repository

examples/integers.rs (line 8)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> ParseResult<u64> {
    let result = parser.take_while(|ch| ch.is_ascii_digit());
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(error(parser.loc(), err)),
    }
}

examples/book_building_parse_integer_2.rs (line 5)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<u64> {
    let result = parser.take_while(|ch| ch.is_ascii_digit());
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(trivet::errors::error(parser.loc(), err)),
    }
}

examples/book_building_sequence.rs (line 5)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<u64> {
    let result = parser.take_while(|ch| ch.is_ascii_digit());
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(trivet::errors::error(parser.loc(), err)),
    }
}

fn parse_identifier(parser: &mut trivet::Parser) -> String {
    let mut result = parser.take_while(|ch| ch.is_alphabetic());
    result += parser.take_while(|ch| ch.is_alphanumeric()).as_str();
    result
}

#[derive(Debug)]
enum Thing {
    Number(trivet::Loc, u64),
    Id(trivet::Loc, String),
}

fn parse_sequence_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<Vec<Thing>> {
    let mut things = Vec::new();
    let _ = parser.consume_ws();
    while !parser.is_at_eof() {
        match parser.peek() {
            ch if ch.is_ascii_digit() => {
                things.push(Thing::Number(parser.loc(), parse_unsigned_integer(parser)?));
            }
            ch if ch.is_alphabetic() => {
                things.push(Thing::Id(parser.loc(), parse_identifier(parser)));
            }
            ch => {
                return Err(trivet::errors::unexpected_text_error(
                    parser.loc(),
                    "number or identifier",
                    &ch.to_string(),
                ));
            }
        }
        let _ = parser.consume_ws();
    }
    Ok(things)
}

examples/book_building_parse_integer_1.rs (line 9)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<u64> {
    let mut result = String::new();
    while parser.peek().is_ascii_digit() {
        result.push(parser.peek());
        parser.consume();
    }
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(trivet::errors::error(parser.loc(), err)),
    }
}

examples/book_math_parser.rs (line 34)

pub fn parse_primitive_ws(parser: &mut Parser) -> ParseResult<f64> {
    // Look for leading minus sign.
    let is_neg = parser.peek_and_consume('-');

    // Look for a parenthesized expression.  Here is where our bottom-up approach
    // otherwise breaks.
    let value = if parser.peek_and_consume('(') {
        let value = parse_top_expr_ws(parser)?;
        if !parser.peek_and_consume_ws(')') {
            return Err(syntax_error(parser.loc(), "Missing closing parenthesis."));
        }
        value
    } else {
        // Note that we have already parsed the negative sign from the number above, so
        // we wil get back a nonnegative value here.  That's fine, since we will fix it
        // at the end.
        parse_number_ws(parser)?
    };
    Ok(if is_neg { -value } else { value })
}

examples/book_string_simple.rs (line 5)

fn main() -> trivet::errors::ParseResult<()> {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        // Save the location for later use in error messages.
        let loc = parser.loc();
        // Parse strings only for the supported delimiters.
        let string = match parser.peek() {
            // The initial and final delimiters match.
            '"' | '\'' => parser.parse_string_match_delimiter_ws()?,
            // Here the initial and final delimiters do not match.
            '«' => {
                parser.consume();
                parser.parse_string_until_delimiter_ws('»')?
            }
            // This is not a legal opening delimiter.
            ch => {
                return Err(trivet::errors::syntax_error(
                    loc,
                    &format!("Invalid delimiter: '{}'", ch),
                ));
            }
        };
        println!("  --> {:?}", string);
    }
    Ok(())
}

Additional examples can be found in:

• examples/book_primitives_lua.rs
• examples/book_json_parser.rs
• examples/book_cookbook_things.rs

pub fn is_at_eof(&self) -> bool

Determine if the parser has reached the end of the stream. If this is true, then no further characters are available from this parser.

Examples found in repository

examples/book_math_parser.rs (line 108)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    parser.consume_ws();
    while !parser.is_at_eof() {
        let number = parse_top_expr_ws(&mut parser)?;
        println!("  {}", number);
    }
    Ok(())
}

examples/book_numbers_float.rs (line 3)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        match parser.parse_f64_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/passthrough.rs (line 8)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    while !parser.is_at_eof() {
        // Read some bytes and then write them out.
        let value = parser.peek_n(57);
        // We can't use value.len() because that is in bytes, not characters.
        for _ in value.chars() {
            parser.consume();
        }
        print!("{}", value);
    }
    Ok(())
}

examples/book_keywords.rs (line 3)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_permit.rs (line 20)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_permit = true;
    parser.borrow_keyword_parser().permit = Box::new(keyword);
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_transform.rs (line 6)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_transform = true;
    parser.borrow_keyword_parser().transform =
        Box::new(|ch| if [':', '.'].contains(&ch) { '_' } else { ch });
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

Additional examples can be found in:

• examples/book_building_sequence.rs
• examples/numbers.rs
• examples/book_string_simple.rs
• examples/book_primitives_lua.rs
• examples/json_validator.rs
• examples/book_cookbook_things.rs
• examples/book_json_parser.rs

pub fn peek(&mut self) -> char

Peek at the next character in the stream. In order to be as fast as is reasonable, no stream checking is done. If the stream is at the end, then you should get null characters, but you should not rely on that, since the null is also a valid character in a file. Instead, be sure to check Parser::is_at_eof.

If this method is invoked too many times without any characters being consumed, then it will panic to indicate that parsing has stalled. See PEEK_LIMIT.

Examples found in repository

examples/book_building_parse_integer_1.rs (line 3)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<u64> {
    let mut result = String::new();
    while parser.peek().is_ascii_digit() {
        result.push(parser.peek());
        parser.consume();
    }
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(trivet::errors::error(parser.loc(), err)),
    }
}

examples/book_building_sequence.rs (line 25)

fn parse_sequence_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<Vec<Thing>> {
    let mut things = Vec::new();
    let _ = parser.consume_ws();
    while !parser.is_at_eof() {
        match parser.peek() {
            ch if ch.is_ascii_digit() => {
                things.push(Thing::Number(parser.loc(), parse_unsigned_integer(parser)?));
            }
            ch if ch.is_alphabetic() => {
                things.push(Thing::Id(parser.loc(), parse_identifier(parser)));
            }
            ch => {
                return Err(trivet::errors::unexpected_text_error(
                    parser.loc(),
                    "number or identifier",
                    &ch.to_string(),
                ));
            }
        }
        let _ = parser.consume_ws();
    }
    Ok(things)
}

examples/book_string_simple.rs (line 7)

fn main() -> trivet::errors::ParseResult<()> {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        // Save the location for later use in error messages.
        let loc = parser.loc();
        // Parse strings only for the supported delimiters.
        let string = match parser.peek() {
            // The initial and final delimiters match.
            '"' | '\'' => parser.parse_string_match_delimiter_ws()?,
            // Here the initial and final delimiters do not match.
            '«' => {
                parser.consume();
                parser.parse_string_until_delimiter_ws('»')?
            }
            // This is not a legal opening delimiter.
            ch => {
                return Err(trivet::errors::syntax_error(
                    loc,
                    &format!("Invalid delimiter: '{}'", ch),
                ));
            }
        };
        println!("  --> {:?}", string);
    }
    Ok(())
}

examples/book_json_parser.rs (line 17)

pub fn parse_value_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    match parser.peek() {
        // Check for an object.
        '{' => parse_object_ws(parser),

        // Check for an array.
        '[' => parse_array_ws(parser),

        // Check for a string.
        '"' => Ok(JSON::String(parser.parse_string_match_delimiter_ws()?)),

        // Check for a number.
        ch if ch == '-' || ch.is_ascii_digit() => Ok(JSON::Number(parser.parse_f64_decimal_ws()?)),

        // Check for Boolean true.
        't' if parser.peek_and_consume_str_ws("true") => Ok(JSON::Boolean(true)),

        // Check for Boolean false.
        'f' if parser.peek_and_consume_str_ws("false") => Ok(JSON::Boolean(false)),

        // Check for null.
        'n' if parser.peek_and_consume_str_ws("null") => Ok(JSON::Null),

        // Error.
        ch => Err(trivet::errors::unexpected_text_error(
            parser.loc(),
            "the start of a JSON value",
            &ch.to_string(),
        )),
    }
}

/// Parse a JSON object.  On entry the parser is assumed to be pointing to the first
/// character of the object (a curly brace `{`).  Trailing whitespace is consumed.
fn parse_object_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    parser.consume(); // Consume opening brace.
    parser.consume_ws();
    let mut first = true;
    let mut map = HashMap::new();
    while !parser.peek_and_consume_ws('}') {
        // If not the first element, then we expect to have a comma here.  JSON requires
        // a comma between members of an object.
        if first {
            first = false;
        } else if !parser.peek_and_consume_ws(',') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "There must be a comma (,) between members of an object.",
            ));
        }

        // Look for the quotation mark for a string.
        if parser.peek() != '"' {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be quoted strings.",
            ));
        }
        let name = parser.parse_string_match_delimiter_ws()?;

        // Look for the colon.
        if !parser.peek_and_consume_ws(':') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be followed by a colon (:).",
            ));
        }

        // Get the value.
        let value = parse_value_ws(parser)?;

        // Add the name, value pair to the map.
        map.insert(name, value);
    }
    Ok(JSON::Object(map))
}

examples/book_cookbook_things.rs (line 16)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    loop {
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }
        if parser.peek_and_consume_str_ws("string") {
            parser.expect(':')?;
            parser.consume_ws();
            let ch = parser.peek();
            if ch == '"' || ch == '\'' {
                let string = parser.parse_string_match_delimiter_ws()?;
                println!("Got string: {}", StringEncoder::new().encode(&string));
            }
        } else if parser.peek_and_consume_str_ws("integer") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_u128_ws()?;
            println!("Got integer: {}", number);
        } else if parser.peek_and_consume_str_ws("float") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_f64_ws()?;
            println!("Got float: {}", number);
        } else if parser.peek_and_consume_str_ws("json") {
            parser.expect(':')?;
            parser.consume_ws();
            let json = JSONParser::new().parse_value_unstandard_ws(&mut parser)?;
            println!("Got JSON:\n{}", json);
        } else {
            return Err(syntax_error(
                parser.loc(),
                "Unexpected element found in stream",
            ));
        }
    }
    Ok(())
}

pub fn consume(&mut self)

Consume the next character from the stream, if there is one. If not, then do nothing.

If this method is invoked too many times after reaching the end of file, then it will panic to indicate that parsing has stalled. See EOF_LIMIT.

Examples found in repository

examples/book_building_parse_integer_1.rs (line 5)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<u64> {
    let mut result = String::new();
    while parser.peek().is_ascii_digit() {
        result.push(parser.peek());
        parser.consume();
    }
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(trivet::errors::error(parser.loc(), err)),
    }
}

examples/passthrough.rs (line 13)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    while !parser.is_at_eof() {
        // Read some bytes and then write them out.
        let value = parser.peek_n(57);
        // We can't use value.len() because that is in bytes, not characters.
        for _ in value.chars() {
            parser.consume();
        }
        print!("{}", value);
    }
    Ok(())
}

examples/book_string_simple.rs (line 12)

fn main() -> trivet::errors::ParseResult<()> {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        // Save the location for later use in error messages.
        let loc = parser.loc();
        // Parse strings only for the supported delimiters.
        let string = match parser.peek() {
            // The initial and final delimiters match.
            '"' | '\'' => parser.parse_string_match_delimiter_ws()?,
            // Here the initial and final delimiters do not match.
            '«' => {
                parser.consume();
                parser.parse_string_until_delimiter_ws('»')?
            }
            // This is not a legal opening delimiter.
            ch => {
                return Err(trivet::errors::syntax_error(
                    loc,
                    &format!("Invalid delimiter: '{}'", ch),
                ));
            }
        };
        println!("  --> {:?}", string);
    }
    Ok(())
}

examples/book_json_parser.rs (line 51)

fn parse_object_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    parser.consume(); // Consume opening brace.
    parser.consume_ws();
    let mut first = true;
    let mut map = HashMap::new();
    while !parser.peek_and_consume_ws('}') {
        // If not the first element, then we expect to have a comma here.  JSON requires
        // a comma between members of an object.
        if first {
            first = false;
        } else if !parser.peek_and_consume_ws(',') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "There must be a comma (,) between members of an object.",
            ));
        }

        // Look for the quotation mark for a string.
        if parser.peek() != '"' {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be quoted strings.",
            ));
        }
        let name = parser.parse_string_match_delimiter_ws()?;

        // Look for the colon.
        if !parser.peek_and_consume_ws(':') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be followed by a colon (:).",
            ));
        }

        // Get the value.
        let value = parse_value_ws(parser)?;

        // Add the name, value pair to the map.
        map.insert(name, value);
    }
    Ok(JSON::Object(map))
}

/// Parse a JSON array.  On entry the parser is assumed to be pointing to the first
/// character of the array (a square bracket `[`).  Trailing whitespace is consumed.
fn parse_array_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    parser.consume(); // Consume opening bracket.
    parser.consume_ws_only();
    let mut first = true;
    let mut array = vec![];
    while !parser.peek_and_consume_ws(']') {
        // If not the first element, then we expect to have a comma here.  JSON requires
        // a comma between members of an array.
        if first {
            first = false;
        } else if !parser.peek_and_consume_ws(',') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "There must be a comma (,) between members of an array.",
            ));
        }

        // Get the value.
        let value = parse_value_ws(parser)?;

        // Add the value to the array.
        array.push(value);
    }
    Ok(JSON::Array(array))
}

pub fn peek_offset(&mut self, n: usize) -> char

Peek at an offset in the stream. That is, peek at a character at a given position. The position index is zero-based, with the next character to read (the result of a simple Self::peek) being at index zero.

If there are not enough characters in the stream, then null (\0) is returned. The distance is limited by the maximum lookahead; attempts to look past it will also return a null.

Note the distinction between this method and Self::peek_n; peek_n(1) method will return the character at position zero, so it is equivalent to peek() and to peek_offset(0).

pub fn peek_n(&mut self, n: usize) -> String

Peek at characters in the stream. If there are fewer than n characters in the stream, then fewer are returned. If the stream is exhausted, an empty string is returned.

If this method is invoked too many times without any characters being consumed, then it will panic to indicate that parsing has stalled. See PEEK_LIMIT.

Examples found in repository

examples/passthrough.rs (line 10)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    while !parser.is_at_eof() {
        // Read some bytes and then write them out.
        let value = parser.peek_n(57);
        // We can't use value.len() because that is in bytes, not characters.
        for _ in value.chars() {
            parser.consume();
        }
        print!("{}", value);
    }
    Ok(())
}

pub fn peek_n_vec(&mut self, n: usize) -> Vec<char>

Peek at characters in the stream. If there are fewer than n characters in the stream, then fewer are returned. If the stream is exhausted, an empty vector is returned.

If this method is invoked too many times without any characters being consumed, then it will panic to indicate that parsing has stalled. See PEEK_LIMIT.

This method is similar to Self::peek_n, but does not construct a string for the result, which can be better in some cases.

pub fn consume_n(&mut self, n: usize)

Consume a given number of characters from the stream. The end of file is not checked during this. If there are no characters to consume, nothing is done.

If this method is invoked too many times after reaching the end of file, it will panic to indicate that parsing has stalled. See EOF_LIMIT.

pub fn peek_chars(&mut self, chars: &[char]) -> bool

Check the next characters in the stream. If the next characters exactly match those given in the vector, then true is returned. Otherwise false is returned. Nothing is consumed.

pub fn peek_and_consume_chars(&mut self, chars: &[char]) -> bool

Check the next characters in the stream and, if the match, consume them and return true. Otherwise return false.

pub fn take_until(&mut self, token: &str) -> String

Consume characters until an end token is found. The characters consumed are returned without the end token.

pub fn take_while<T: Fn(char) -> bool>(&mut self, include: T) -> String

Consume characters so long as the test is true. Return the characters that satisfy the first test.

Examples found in repository

examples/integers.rs (line 5)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> ParseResult<u64> {
    let result = parser.take_while(|ch| ch.is_ascii_digit());
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(error(parser.loc(), err)),
    }
}

examples/book_building_parse_integer_2.rs (line 2)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<u64> {
    let result = parser.take_while(|ch| ch.is_ascii_digit());
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(trivet::errors::error(parser.loc(), err)),
    }
}

examples/book_building_sequence.rs (line 2)

fn parse_unsigned_integer(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<u64> {
    let result = parser.take_while(|ch| ch.is_ascii_digit());
    match result.parse::<u64>() {
        Ok(number) => Ok(number),
        Err(err) => Err(trivet::errors::error(parser.loc(), err)),
    }
}

fn parse_identifier(parser: &mut trivet::Parser) -> String {
    let mut result = parser.take_while(|ch| ch.is_alphabetic());
    result += parser.take_while(|ch| ch.is_alphanumeric()).as_str();
    result
}

examples/book_keywords.rs (line 8)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_permit.rs (line 25)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_permit = true;
    parser.borrow_keyword_parser().permit = Box::new(keyword);
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_transform.rs (line 11)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_transform = true;
    parser.borrow_keyword_parser().transform =
        Box::new(|ch| if [':', '.'].contains(&ch) { '_' } else { ch });
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

pub fn take_while_unless<T: Fn(char) -> bool, U: Fn(char) -> bool>( &mut self, include: T, exclude: U, ) -> String

Consume characters so long as either test is true. Return only those characters that satisfy the first test. The exclude predicate is checked first.

pub fn take<S, K>(&mut self, skip: S, stop: K) -> (Vec<char>, Option<char>)

where S: Fn(char) -> bool, K: Fn(char) -> bool,

Consume and return characters. This works as follows.

If the current character satisfies stop, then the parse is stopped and the result is returned, regardless of whether any other predicates match.

If the current character satisfies skip, then the character is skipped.

Other characters (those that do not match skip or stop) are collected and returned.

The returned pair contains all matched characters and the character that caused the stop, or None if parsing stopped because the end of stream was reached. Because the stopping character is returned, it is also consumed.

pub fn consume_while<T: Fn(char) -> bool>(&mut self, include: T) -> bool

Consume characters so long as the test is true. Returns true if any characters are consumed.

pub fn consume_until(&mut self, token: &str) -> bool

Consume characters until the given end token is found. Returns true if any characters are consumed. The end token is also consumed.

pub fn consume_ws_only(&mut self) -> bool

Consume all whitespace starting at the current position. The definition of whitespace used here is the same as the Unicode standard.

At the time of writing, the following is the definition of whitespace used.

0009..000D    ; White_Space # Cc   [5] <control-0009>..<control-000D>
0020          ; White_Space # Zs       SPACE
0085          ; White_Space # Cc       <control-0085>
00A0          ; White_Space # Zs       NO-BREAK SPACE
1680          ; White_Space # Zs       OGHAM SPACE MARK
2000..200A    ; White_Space # Zs  [11] EN QUAD..HAIR SPACE
2028          ; White_Space # Zl       LINE SEPARATOR
2029          ; White_Space # Zp       PARAGRAPH SEPARATOR
202F          ; White_Space # Zs       NARROW NO-BREAK SPACE
205F          ; White_Space # Zs       MEDIUM MATHEMATICAL SPACE
3000          ; White_Space # Zs       IDEOGRAPHIC SPACE

This method is slightly faster than the Self::consume_ws method as it does not look for comments.

Examples found in repository

examples/book_json_parser.rs (line 97)

fn parse_array_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    parser.consume(); // Consume opening bracket.
    parser.consume_ws_only();
    let mut first = true;
    let mut array = vec![];
    while !parser.peek_and_consume_ws(']') {
        // If not the first element, then we expect to have a comma here.  JSON requires
        // a comma between members of an array.
        if first {
            first = false;
        } else if !parser.peek_and_consume_ws(',') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "There must be a comma (,) between members of an array.",
            ));
        }

        // Get the value.
        let value = parse_value_ws(parser)?;

        // Add the value to the array.
        array.push(value);
    }
    Ok(JSON::Array(array))
}

pub fn peek_and_consume(&mut self, ch: char) -> bool

Peek at the next character in the stream. If it is the given character, consume it and return true. Otherwise return false.

Examples found in repository

examples/book_math_parser.rs (line 27)

pub fn parse_primitive_ws(parser: &mut Parser) -> ParseResult<f64> {
    // Look for leading minus sign.
    let is_neg = parser.peek_and_consume('-');

    // Look for a parenthesized expression.  Here is where our bottom-up approach
    // otherwise breaks.
    let value = if parser.peek_and_consume('(') {
        let value = parse_top_expr_ws(parser)?;
        if !parser.peek_and_consume_ws(')') {
            return Err(syntax_error(parser.loc(), "Missing closing parenthesis."));
        }
        value
    } else {
        // Note that we have already parsed the negative sign from the number above, so
        // we wil get back a nonnegative value here.  That's fine, since we will fix it
        // at the end.
        parse_number_ws(parser)?
    };
    Ok(if is_neg { -value } else { value })
}

pub fn expect(&mut self, ch: char) -> ParseResult<()>

Expect a character at this point in the stream and if it is found, consume it. If it nos not found, generate an error.

Examples found in repository

examples/book_cookbook_things.rs (line 14)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    loop {
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }
        if parser.peek_and_consume_str_ws("string") {
            parser.expect(':')?;
            parser.consume_ws();
            let ch = parser.peek();
            if ch == '"' || ch == '\'' {
                let string = parser.parse_string_match_delimiter_ws()?;
                println!("Got string: {}", StringEncoder::new().encode(&string));
            }
        } else if parser.peek_and_consume_str_ws("integer") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_u128_ws()?;
            println!("Got integer: {}", number);
        } else if parser.peek_and_consume_str_ws("float") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_f64_ws()?;
            println!("Got float: {}", number);
        } else if parser.peek_and_consume_str_ws("json") {
            parser.expect(':')?;
            parser.consume_ws();
            let json = JSONParser::new().parse_value_unstandard_ws(&mut parser)?;
            println!("Got JSON:\n{}", json);
        } else {
            return Err(syntax_error(
                parser.loc(),
                "Unexpected element found in stream",
            ));
        }
    }
    Ok(())
}

pub fn expect_chars(&mut self, ch: &[char]) -> ParseResult<()>

Expect a character sequence at this point in the stream and if it is found, consume it. If it is not found, generate an error.

pub fn peek_str(&mut self, value: &str) -> bool

Peek at the stream and determine if the next characters are the given string. True is returned if so, and false otherwise. Nothing is consumed.

pub fn peek_chars_greedy(&mut self, chars: &[char]) -> bool

Check the next characters in the stream. If the next characters exactly match those given in the vector, and do not also match one position more in the stream, then true is returned. Otherwise false is returned. Nothing is consumed.

pub fn peek_str_greedy(&mut self, value: &str) -> bool

Peek ahead and see if the provided string is present in the stream. This method returns false if the token is also at an offset of one, which may seem an odd choice, but allows matching end tokens where there are repeated characters, such as """.

This is a relatively costly operation.

pub fn peek_and_consume_ws(&mut self, ch: char) -> bool

Peek at the next character in the stream. If it is the given character, consume it and return true. Otherwise return false. Consume any trailing whitespace iff there is a match.

Examples found in repository

examples/book_math_parser.rs (line 33)

pub fn parse_primitive_ws(parser: &mut Parser) -> ParseResult<f64> {
    // Look for leading minus sign.
    let is_neg = parser.peek_and_consume('-');

    // Look for a parenthesized expression.  Here is where our bottom-up approach
    // otherwise breaks.
    let value = if parser.peek_and_consume('(') {
        let value = parse_top_expr_ws(parser)?;
        if !parser.peek_and_consume_ws(')') {
            return Err(syntax_error(parser.loc(), "Missing closing parenthesis."));
        }
        value
    } else {
        // Note that we have already parsed the negative sign from the number above, so
        // we wil get back a nonnegative value here.  That's fine, since we will fix it
        // at the end.
        parse_number_ws(parser)?
    };
    Ok(if is_neg { -value } else { value })
}

/// Parse an expression containing exponentiation or bitwise operators.
///
/// On entry the parser is assumed to be pointing at the first character.  On
/// exit any trailing whitespace is consumed.
///
pub fn parse_exp_expr_ws(parser: &mut Parser) -> ParseResult<f64> {
    // Parse an initial primitive.
    let mut left = parse_primitive_ws(parser)?;
    loop {
        // Look for an operator.
        left = if parser.peek_and_consume_str_ws("**") {
            let right = parse_primitive_ws(parser)?;
            left.powf(right)
        } else {
            break;
        }
    }
    Ok(left)
}

/// Parse multiplication, division, and remainder operations.
///
/// On entry the parser is assumed to be pointing at the first character.  On
/// exit any trailing whitespace is consumed.
///
pub fn parse_product_expr_ws(parser: &mut Parser) -> ParseResult<f64> {
    let mut left = parse_exp_expr_ws(parser)?;
    loop {
        left = if parser.peek_and_consume_ws('*') {
            left * parse_exp_expr_ws(parser)?
        } else if parser.peek_and_consume_ws('/') {
            left / parse_exp_expr_ws(parser)?
        } else {
            break;
        }
    }
    Ok(left)
}

/// Parse an expression containing arithmetic and bitwise operators.
///
/// On entry the parser is assumed to be pointing at the first character.  On
/// exit any trailing whitespace is consumed.
///
pub fn parse_top_expr_ws(parser: &mut Parser) -> ParseResult<f64> {
    let mut left = parse_product_expr_ws(parser)?;
    loop {
        left = if parser.peek_and_consume_ws('+') {
            left + parse_product_expr_ws(parser)?
        } else if parser.peek_and_consume_ws('-') {
            left - parse_product_expr_ws(parser)?
        } else {
            break;
        };
    }
    Ok(left)
}

examples/book_json_parser.rs (line 55)

fn parse_object_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    parser.consume(); // Consume opening brace.
    parser.consume_ws();
    let mut first = true;
    let mut map = HashMap::new();
    while !parser.peek_and_consume_ws('}') {
        // If not the first element, then we expect to have a comma here.  JSON requires
        // a comma between members of an object.
        if first {
            first = false;
        } else if !parser.peek_and_consume_ws(',') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "There must be a comma (,) between members of an object.",
            ));
        }

        // Look for the quotation mark for a string.
        if parser.peek() != '"' {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be quoted strings.",
            ));
        }
        let name = parser.parse_string_match_delimiter_ws()?;

        // Look for the colon.
        if !parser.peek_and_consume_ws(':') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "Names in a JSON object must be followed by a colon (:).",
            ));
        }

        // Get the value.
        let value = parse_value_ws(parser)?;

        // Add the name, value pair to the map.
        map.insert(name, value);
    }
    Ok(JSON::Object(map))
}

/// Parse a JSON array.  On entry the parser is assumed to be pointing to the first
/// character of the array (a square bracket `[`).  Trailing whitespace is consumed.
fn parse_array_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    parser.consume(); // Consume opening bracket.
    parser.consume_ws_only();
    let mut first = true;
    let mut array = vec![];
    while !parser.peek_and_consume_ws(']') {
        // If not the first element, then we expect to have a comma here.  JSON requires
        // a comma between members of an array.
        if first {
            first = false;
        } else if !parser.peek_and_consume_ws(',') {
            return Err(trivet::errors::syntax_error(
                parser.loc(),
                "There must be a comma (,) between members of an array.",
            ));
        }

        // Get the value.
        let value = parse_value_ws(parser)?;

        // Add the value to the array.
        array.push(value);
    }
    Ok(JSON::Array(array))
}

pub fn peek_and_consume_chars_ws(&mut self, chars: &[char]) -> bool

Peek at the stream and determine if the next characters are the given characters in order. If so, consume them and any trailing whitespace, and then return true. Otherwise consume nothing and return false.

pub fn peek_and_consume_str(&mut self, value: &str) -> bool

Peek at the stream and determine if the next characters are the given string. If so, consume them and return true. Otherwise return false.

pub fn peek_and_consume_str_ws(&mut self, value: &str) -> bool

Peek at the stream and determine if the next characters are the given string. If so, consume them and return true. Otherwise return false. Consume trailing whitespace.

Examples found in repository

examples/book_math_parser.rs (line 56)

pub fn parse_exp_expr_ws(parser: &mut Parser) -> ParseResult<f64> {
    // Parse an initial primitive.
    let mut left = parse_primitive_ws(parser)?;
    loop {
        // Look for an operator.
        left = if parser.peek_and_consume_str_ws("**") {
            let right = parse_primitive_ws(parser)?;
            left.powf(right)
        } else {
            break;
        }
    }
    Ok(left)
}

examples/book_json_parser.rs (line 31)

pub fn parse_value_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<JSON> {
    match parser.peek() {
        // Check for an object.
        '{' => parse_object_ws(parser),

        // Check for an array.
        '[' => parse_array_ws(parser),

        // Check for a string.
        '"' => Ok(JSON::String(parser.parse_string_match_delimiter_ws()?)),

        // Check for a number.
        ch if ch == '-' || ch.is_ascii_digit() => Ok(JSON::Number(parser.parse_f64_decimal_ws()?)),

        // Check for Boolean true.
        't' if parser.peek_and_consume_str_ws("true") => Ok(JSON::Boolean(true)),

        // Check for Boolean false.
        'f' if parser.peek_and_consume_str_ws("false") => Ok(JSON::Boolean(false)),

        // Check for null.
        'n' if parser.peek_and_consume_str_ws("null") => Ok(JSON::Null),

        // Error.
        ch => Err(trivet::errors::unexpected_text_error(
            parser.loc(),
            "the start of a JSON value",
            &ch.to_string(),
        )),
    }
}

examples/book_cookbook_things.rs (line 13)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    loop {
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }
        if parser.peek_and_consume_str_ws("string") {
            parser.expect(':')?;
            parser.consume_ws();
            let ch = parser.peek();
            if ch == '"' || ch == '\'' {
                let string = parser.parse_string_match_delimiter_ws()?;
                println!("Got string: {}", StringEncoder::new().encode(&string));
            }
        } else if parser.peek_and_consume_str_ws("integer") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_u128_ws()?;
            println!("Got integer: {}", number);
        } else if parser.peek_and_consume_str_ws("float") {
            parser.expect(':')?;
            parser.consume_ws();
            let number = parser.parse_f64_ws()?;
            println!("Got float: {}", number);
        } else if parser.peek_and_consume_str_ws("json") {
            parser.expect(':')?;
            parser.consume_ws();
            let json = JSONParser::new().parse_value_unstandard_ws(&mut parser)?;
            println!("Got JSON:\n{}", json);
        } else {
            return Err(syntax_error(
                parser.loc(),
                "Unexpected element found in stream",
            ));
        }
    }
    Ok(())
}

pub fn consume_ws(&mut self) -> bool

Consume whitespace. This is the method used by all the _ws method forms. It will also use the embedded comment parser to parse and discard comments.

Examples found in repository

examples/book_math_parser.rs (line 107)

pub fn main() -> ParseResult<()> {
    let mut parser = parse_from_stdin();
    parser.consume_ws();
    while !parser.is_at_eof() {
        let number = parse_top_expr_ws(&mut parser)?;
        println!("  {}", number);
    }
    Ok(())
}

examples/book_keywords.rs (line 9)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_permit.rs (line 26)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_permit = true;
    parser.borrow_keyword_parser().permit = Box::new(keyword);
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_keywords_transform.rs (line 12)

fn main() {
    let mut parser = trivet::parse_from_stdin();
    parser.borrow_keyword_parser().use_transform = true;
    parser.borrow_keyword_parser().transform =
        Box::new(|ch| if [':', '.'].contains(&ch) { '_' } else { ch });
    while !parser.is_at_eof() {
        match parser.parse_keyword_ws() {
            Err(err) => {
                println!("ERROR: {}", err);
                // Consume until whitespace, then resume trying.
                parser.take_while(|ch| !ch.is_whitespace());
                parser.consume_ws();
            }
            Ok(value) => {
                println!("  {}", value);
            }
        }
    }
}

examples/book_building_sequence.rs (line 23)

fn parse_sequence_ws(parser: &mut trivet::Parser) -> trivet::errors::ParseResult<Vec<Thing>> {
    let mut things = Vec::new();
    let _ = parser.consume_ws();
    while !parser.is_at_eof() {
        match parser.peek() {
            ch if ch.is_ascii_digit() => {
                things.push(Thing::Number(parser.loc(), parse_unsigned_integer(parser)?));
            }
            ch if ch.is_alphabetic() => {
                things.push(Thing::Id(parser.loc(), parse_identifier(parser)));
            }
            ch => {
                return Err(trivet::errors::unexpected_text_error(
                    parser.loc(),
                    "number or identifier",
                    &ch.to_string(),
                ));
            }
        }
        let _ = parser.consume_ws();
    }
    Ok(things)
}

examples/numbers.rs (line 20)

pub fn main() {
    println!("Enter a number to see how Trivet parses it.  Enter a blank line to stop.");
    let numpar = NumberParser::new();
    print!("> ");
    let _ = stdout().flush();
    for line in stdin().lock().lines() {
        if line.is_err() {
            break;
        }
        let line = line.unwrap();

        // Consume any whitespace.
        let mut parser = parse_from_string(&line);
        parser.consume_ws();
        if parser.is_at_eof() {
            break;
        }

        // Try to read a number as pieces.
        let parts = numpar.get_parts(parser.borrow_core());
        println!("{:?}", parts);
        if parts.is_apparent_float() {
            let value: f64 = parts.into();
            println!("f64: {}", value);
        } else {
            let value: i128 = parts.into();
            println!("i128: {}", value);
        }

        // Go around again.
        print!("> ");
        let _ = stdout().flush();
    }
}

Additional examples can be found in:

• examples/book_primitives_lua.rs
• examples/book_datetime_tester.rs
• examples/book_cookbook_things.rs
• examples/book_json_parser.rs

pub fn take_until_greedy( &mut self, chars: &[char], must_match: bool, ) -> ParseResult<String>

Consume characters until the given character sequence is found. This uses greedy matching (see Self::peek_chars_greedy) and returns the characters consumed. The end token is also consumed and not returned.

If the must_match flag is true, then the end token must be matched and consumed, otherwise an error is generated.