Struct yash_syntax::parser::lex::PlainLexer

pub struct PlainLexer<'a, 'b> { /* private fields */ }

Reference to Lexer with line continuation disabled.

This struct implements the RAII pattern for temporarily disabling line continuation. When you disable the line continuation of a lexer, you get an instance of PlainLexer. You can access the original lexer via the PlainLexer until you drop it, when the line continuation is automatically re-enabled.

Methods from Deref<Target = Lexer<'b>>

pub fn disable_line_continuation<'b>(&'b mut self) -> PlainLexer<'b, 'a>

Disables line continuation recognition onward.

By default, peek_char silently skips line continuation sequences. When line continuation is disabled, however, peek_char returns characters literally.

Call enable_line_continuation to switch line continuation recognition on.

This function will panic if line continuation has already been disabled.

pub async fn peek_char(&mut self) -> Result<Option<char>>

Peeks the next character.

If the end of input is reached, Ok(None) is returned. On error, Err(_) is returned.

If line continuation recognition is enabled, combinations of a backslash and a newline are silently skipped before returning the next character. Call disable_line_continuation to switch off line continuation recognition.

This function requires a mutable reference to self since it may need to read the next line if needed.

pub async fn location(&mut self) -> Result<&Location>

Returns the location of the next character.

If there is no more character (that is, it is the end of input), an imaginary location is returned that would be returned if a character existed.

This function requires a mutable reference to self since it needs to peek the next character.

pub fn consume_char(&mut self)

Consumes the next character.

This function must be called after peek_char has successfully returned the character. Consuming a character that has not yet been peeked would result in a panic!

pub fn index(&self) -> usize

Returns the position of the next character, counted from zero.

futures_executor::block_on(async {
    let mut lexer = Lexer::from_memory("abc", Source::Unknown);
    assert_eq!(lexer.index(), 0);
    let _ = lexer.peek_char().await;
    assert_eq!(lexer.index(), 0);
    lexer.consume_char();
    assert_eq!(lexer.index(), 1);
})

pub fn rewind(&mut self, index: usize)

Moves the current position back to the given index so that characters that have been consumed can be read again.

The given index must not be larger than the current index, or this function would panic.

futures_executor::block_on(async {
    let mut lexer = Lexer::from_memory("abc", Source::Unknown);
    let saved_index = lexer.index();
    assert_eq!(lexer.peek_char().await, Ok(Some('a')));
    lexer.consume_char();
    assert_eq!(lexer.peek_char().await, Ok(Some('b')));
    lexer.rewind(saved_index);
    assert_eq!(lexer.peek_char().await, Ok(Some('a')));
})

pub fn pending(&self) -> bool

Checks if there is any character that has been read from the input source but not yet consumed.

pub fn flush(&mut self)

Clears the internal buffer of the lexer.

Locations returned from location share a single code instance that is also retained by the lexer. The code grows long as the lexer reads more input. To prevent the code from getting too large, you can call this function that replaces the retained code with a new empty one. The new code’s start_line_number will be incremented by the number of lines in the previous.

pub fn reset(&mut self)

Clears an end-of-input or error status so that the lexer can resume parsing.

This function will be useful only in an interactive shell where the user can continue entering commands even after (s)he sends an end-of-input or is interrupted by a syntax error.

pub async fn consume_char_if<F>(&mut self, f: F) -> Result<Option<&SourceChar>> where
    F: FnMut(char) -> bool, 

Peeks the next character and, if the given decider function returns true for it, advances the position.

Returns the consumed character if the function returned true. Returns Ok(None) if it returned false or there is no more character.

pub fn source_string<I>(&self, i: I) -> String where
    I: SliceIndex<[SourceChar], Output = [SourceChar]>, 

Extracts a string from the source code.

This function returns the source code string for the range specified by the argument. The range must specify a valid index. If the index points to a character that have not yet read, this function will panic!.

Panics

If the argument index is out of bounds, i.e., pointing to an unread character.

pub fn location_range(&self, range: Range<usize>) -> Location

Returns a location for a given range of the source code.

All the characters in the range must have been consumed. If the range refers to an unconsumed character, this function will panic!

If the characters are from more than one Code fragment, the location will only cover the initial portion of the range sharing the same Code.

Panics

This function will panic if the range refers to an unconsumed character.

If the start index of the range is the end of input, it must have been peeked and the range must be empty, or the function will panic.

pub fn substitute_alias(&mut self, begin: usize, alias: &Rc<Alias>)

Performs alias substitution right before the current position.

This function must be called just after a word has been parsed that matches the name of the argument alias. No check is done in this function that there is a matching word before the current position. The characters starting from the begin index up to the current position are silently replaced with the alias value.

The resulting part of code will be characters with a Source::Alias origin.

After the substitution, the position will be set before the replaced string.

Panics

If the replaced part is empty, i.e., begin >= self.index().

pub fn is_after_blank_ending_alias(&self, index: usize) -> bool

Tests if the given index is after the replacement string of alias substitution that ends with a blank.

Panics

If index is larger than the currently read index.

pub async fn inner_program(&mut self) -> Result<String>

Parses an optional compound list that is the content of a command substitution.

This function consumes characters until a token that cannot be the beginning of an and-or list is found and returns the string that was consumed.

pub fn inner_program_boxed(
    &mut self
) -> Pin<Box<dyn Future<Output = Result<String>>>>

Like Lexer::inner_program, but returns the future in a pinned box.

pub async fn arithmetic_expansion(
    &mut self,
    start_index: usize
) -> Result<Option<TextUnit>>

Parses an arithmetic expansion.

The initial $ must have been consumed before calling this function. In this function, the next two characters are examined to see if they begin an arithmetic expansion. If the characters are ((, then the arithmetic expansion is parsed, in which case this function consumes up to the closing )) (inclusive). Otherwise, no characters are consumed and the return value is Ok(None).

The start_index parameter should be the index for the initial $. It is used to construct the result, but this function does not check if it actually points to the $.

pub async fn command_substitution(
    &mut self,
    start_index: usize
) -> Result<Option<TextUnit>>

Parses a command substitution of the form $(...).

The initial $ must have been consumed before calling this function. In this function, the next character is examined to see if it begins a command substitution. If it is (, the following characters are parsed as commands to find a matching ), which will be consumed before this function returns. Otherwise, no characters are consumed and the return value is Ok(None).

The start_index parameter should be the index for the initial $. It is used to construct the result, but this function does not check if it actually points to the $.

pub async fn line(&mut self) -> Result<String>

Reads a line literally.

This function recognizes no quotes or expansions. Starting from the current position, the line is read up to (but not including) the terminating newline.

pub async fn here_doc_content(&mut self, here_doc: &HereDoc) -> Result<()>

Parses the content of a here-document.

This function reads here-document content corresponding to the here-document operator represented by the argument and fills here_doc.content with the results. The argument does not have to be mutable because here_doc.content is a RefCell. Note that this function will panic if here_doc.content has been borrowed.

In case of an error, partial results may be left in here_doc.content.

pub async fn skip_if<F>(&mut self, f: F) -> Result<bool> where
    F: FnMut(char) -> bool, 

Skips a character if the given function returns true for it.

Returns Ok(true) if the character was skipped, Ok(false) if the function returned false, and Err(_) if an error occurred, respectively.

skip_if is a simpler version of consume_char_if.

pub async fn skip_blanks(&mut self) -> Result<()>

Skips blank characters until reaching a non-blank.

pub async fn skip_comment(&mut self) -> Result<()>

Skips a comment, if any.

A comment ends just before a newline. The newline is not part of the comment.

This function does not recognize line continuation inside the comment.

pub async fn skip_blanks_and_comment(&mut self) -> Result<()>

Skips blank characters and a comment, if any.

This function is the same as skip_blanks followed by skip_comment.

pub async fn operator(&mut self) -> Result<Option<Token>>

Parses an operator token.

pub async fn raw_param(
    &mut self,
    start_index: usize
) -> Result<Option<TextUnit>>

Parses a parameter expansion that is not enclosed in braces.

The initial $ must have been consumed before calling this function. This functions checks if the next character is a valid POSIXly-portable parameter name. If so, the name is consumed and returned. Otherwise, no characters are consumed and the return value is Ok(None).

The start_index parameter should be the index for the initial $. It is used to construct the result, but this function does not check if it actually points to the $.

pub async fn text<F, G>(
    &mut self,
    is_delimiter: F,
    is_escapable: G
) -> Result<Text> where
    F: FnMut(char) -> bool,
    G: FnMut(char) -> bool, 

Parses a text, i.e., a (possibly empty) sequence of TextUnits.

is_delimiter tests if an unquoted character is a delimiter. When is_delimiter returns true, the parser stops parsing and returns the text up to the delimiter.

is_escapable tests if a backslash can escape a character. When the parser founds an unquoted backslash, the next character is passed to is_escapable. If is_escapable returns true, the backslash is treated as a valid escape (TextUnit::Backslashed). Otherwise, it ia a literal (TextUnit::Literal).

is_escapable also affects escaping of double-quotes inside backquotes. See text_unit for details. Note that this function calls text_unit with WordContext::Text.

pub async fn text_with_parentheses<F, G>(
    &mut self,
    is_delimiter: F,
    is_escapable: G
) -> Result<Text> where
    F: FnMut(char) -> bool,
    G: FnMut(char) -> bool, 

Parses a text that may contain nested parentheses.

This function works similarly to text. However, if an unquoted ( is found in the text, all text units are parsed up to the next matching unquoted ). Inside the parentheses, the is_delimiter function is ignored and all non-special characters are parsed as literal word units. After finding the ), this function continues parsing to find a delimiter (as per is_delimiter) or another parentheses.

Nested parentheses are supported: the number of (s and )s must match. In other words, the final delimiter is recognized only outside outermost parentheses.

pub async fn token(&mut self) -> Result<Token>

Parses a token.

If there is no more token that can be parsed, the result is a token with an empty word and EndOfInput token identifier.

Trait Implementations

impl<'a, 'b> Debug for PlainLexer<'a, 'b>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a, 'b> Deref for PlainLexer<'a, 'b>

type Target = Lexer<'b>

The resulting type after dereferencing.

fn deref(&self) -> &Lexer<'b>

Dereferences the value.

impl<'a, 'b> DerefMut for PlainLexer<'a, 'b>

fn deref_mut(&mut self) -> &mut Lexer<'b>

Mutably dereferences the value.

impl Drop for PlainLexer<'_, '_>

fn drop(&mut self)

Executes the destructor for this type.