Struct LexerOfString

pub struct LexerOfString<P, T, E>
where
    P: PosnInCharStream,
{ /* private fields */ }

This provides a type that wraps an allocated String, and which tracks the lines within the string. It then provides a method to create a LexerOfStr that borrows the text, and which can the be used as a crate::Lexer.

This type also implements the FmtContext trait, which allows for pretty-printing the text between certain lines, to highlight certain characters or regions of the text.

Should the LexerOfStr return an error while parsing then the FmtContext implementation can be used to generate a very useful error message with the context of the error.

This also applies should the LexerOfStr output be used in a further grammar - the methods to pretty-print the text require just start and end positions of the position type, which must support PosnInCharStream.

The LexerOfString may be used more than once, replacing its text if necessary before a new parse is initiated. The String can also be retrieved, which is useful for droppign the LexerOfString but retaining the text for future parse or compilation stages

Implementations

impl<P, T, E> LexerOfString<P, T, E>

where P: PosnInCharStream, T: Debug + Clone, E: LexerError<P>,

pub fn set_text<S: Into<String>>(self, text: S) -> Self

Set the text

Examples found in repository

examples/calc.rs (line 158)

fn main() -> Result<(), String> {
    let args: Vec<String> = env::args().collect();
    if args.len() < 2 {
        return Err(format!("Usage: {} <expression>", args[0]));
    }
    let args_as_string = args[1..].join(" ");
    let c = CalcTokenParser::new();
    let l = LexerOfString::default().set_text(args_as_string);
    let ts = l.lexer();

    // let ts = TextStream::new(&args_as_string);

    println!("Parsing");
    let tokens = c.iter(&ts);
    for t in tokens {
        let t = {
            match t {
                Err(e) => {
                    println!();
                    let mut s = String::new();
                    l.fmt_context(&mut s, &e.pos, &e.pos).unwrap();
                    eprintln!("{}", s);
                    return Err(format!("{}", e));
                }
                Ok(t) => t,
            }
        };
        print!("{}", t);
    }
    println!();
    println!("Text parsed okay");
    Ok(())
}

examples/simple.rs (line 214)

fn main() -> Result<(), String> {
    let args: Vec<String> = env::args().collect();
    if args.len() < 2 {
        return Err(format!("Usage: {} <expression>", args[0]));
    }
    let args_as_string = args[1..].join(" ");

    let mut parsers = ParserVec::new();
    parsers.add_parser(|a, b, c| LexToken::parse_whitespace(a, b, c));
    parsers.add_parser(|a, b, c| LexToken::parse_comment_line(a, b, c));
    parsers.add_parser(|a, b, c| LexToken::parse_digits(a, b, c));
    parsers.add_parser(|a, b, c| LexToken::parse_char(a, b, c));

    let l = LexerOfString::default().set_text(args_as_string);
    let ts = l.lexer();
    let tokens = ts.iter(&parsers.parsers);

    println!("Parsing");
    for t in tokens {
        let t = {
            match t {
                Err(e) => {
                    println!();
                    let mut s = String::new();
                    l.fmt_context(&mut s, &e.pos, &e.pos).unwrap();
                    eprintln!("{}", s);
                    return Err(format!("{}", e));
                }
                Ok(t) => t,
            }
        };
        println!("{:?}", t);
    }
    println!();
    println!("Text parsed okay");
    Ok(())
}

pub fn take_text(&mut self) -> String

Take the text as a String out of the LexerOfString

pub fn text(&self) -> &str

Get a str reference to the text content

pub fn lexer(&self) -> LexerOfStr<'_, P, T, E>

Create a LexerOfStr that will parse the text

Examples found in repository

examples/calc.rs (line 159)

fn main() -> Result<(), String> {
    let args: Vec<String> = env::args().collect();
    if args.len() < 2 {
        return Err(format!("Usage: {} <expression>", args[0]));
    }
    let args_as_string = args[1..].join(" ");
    let c = CalcTokenParser::new();
    let l = LexerOfString::default().set_text(args_as_string);
    let ts = l.lexer();

    // let ts = TextStream::new(&args_as_string);

    println!("Parsing");
    let tokens = c.iter(&ts);
    for t in tokens {
        let t = {
            match t {
                Err(e) => {
                    println!();
                    let mut s = String::new();
                    l.fmt_context(&mut s, &e.pos, &e.pos).unwrap();
                    eprintln!("{}", s);
                    return Err(format!("{}", e));
                }
                Ok(t) => t,
            }
        };
        print!("{}", t);
    }
    println!();
    println!("Text parsed okay");
    Ok(())
}

examples/simple.rs (line 215)

fn main() -> Result<(), String> {
    let args: Vec<String> = env::args().collect();
    if args.len() < 2 {
        return Err(format!("Usage: {} <expression>", args[0]));
    }
    let args_as_string = args[1..].join(" ");

    let mut parsers = ParserVec::new();
    parsers.add_parser(|a, b, c| LexToken::parse_whitespace(a, b, c));
    parsers.add_parser(|a, b, c| LexToken::parse_comment_line(a, b, c));
    parsers.add_parser(|a, b, c| LexToken::parse_digits(a, b, c));
    parsers.add_parser(|a, b, c| LexToken::parse_char(a, b, c));

    let l = LexerOfString::default().set_text(args_as_string);
    let ts = l.lexer();
    let tokens = ts.iter(&parsers.parsers);

    println!("Parsing");
    for t in tokens {
        let t = {
            match t {
                Err(e) => {
                    println!();
                    let mut s = String::new();
                    l.fmt_context(&mut s, &e.pos, &e.pos).unwrap();
                    eprintln!("{}", s);
                    return Err(format!("{}", e));
                }
                Ok(t) => t,
            }
        };
        println!("{:?}", t);
    }
    println!();
    println!("Text parsed okay");
    Ok(())
}

Trait Implementations

impl<P, T: Debug, E: Debug> Debug for LexerOfString<P, T, E>

where P: PosnInCharStream + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<P, T, E> Default for LexerOfString<P, T, E>

where P: PosnInCharStream,

fn default() -> Self

Returns the “default value” for a type.

impl<P, T, E> FmtContext<P> for LexerOfString<P, T, E>

where P: PosnInCharStream, E: LexerError<P>,

fn line_length(&self, line: usize) -> usize

Return the length of the specified line

fn fmt_line(&self, f: &mut dyn Write, line: usize) -> Result

Format the line of text (potentially with coloring and so on).

fn fmt_context_single_line( &self, f: &mut dyn Write, start: &P, num_cols: usize, ) -> Result

where P: UserPosn,

Format a line of text with highlight on certain columns

fn fmt_context_multiple_lines( &self, f: &mut dyn Write, start: &P, end: &P, ) -> Result

where P: UserPosn,

Format multiple lines of text, highlighting certain lines

fn fmt_context(&self, fmt: &mut dyn Write, start: &P, end: &P) -> Result

where P: UserPosn,

Format text with highlighting between start and end