Crate shrub_parser

Expand description

A tokenizer and parsers for POSIX and bash shell syntax, with no dependency on a running shell.

shrub-parser is the bottom of shrub’s crate graph: shrub-core builds on it to run scripts, and front ends use it to inspect or highlight input. Use it directly to read shell scripts into a syntax tree without executing them.

§Quick start

Wrap the script in a reader, build a Parser, and call Parser::parse_program. The result is an ast::Program.

use std::io::Cursor;

use shrub_parser::{ParseError, Parser};

let mut parser = Parser::builder().build(Cursor::new("echo hello | wc -c\nls -l\n"));
let program = parser.parse_program()?;

assert_eq!(program.complete_commands.len(), 2);

Input that is not valid shell syntax returns a ParseError that says where parsing failed.

§Parser options

Parser::builder sets the options that change what the input means: extended globbing, POSIX mode, sh mode, and tilde expansion. Parser::new takes the same settings as a ParserOptions value. The defaults match bash.

§Tokenizing

tokenize_str splits a script into Tokens without building a tree. Each token is a word or an operator and records its SourceSpan. Words keep their quotes; unquote_str removes them.

use shrub_parser::{Token, TokenizerError, tokenize_str, unquote_str};

let tokens = tokenize_str("echo 'a b' > out")?;

assert_eq!(tokens.len(), 4);
assert!(matches!(tokens[2], Token::Operator(..)));
assert_eq!(unquote_str(tokens[1].to_str()), "a b");

tokenize_str_with_options takes TokenizerOptions, and uncached_tokenize_str skips the cache that tokenize_str keeps of recent inputs.

§Modules

ast: the syntax tree types.
word: parses a word into its parts, such as parameter and command expansions.
arithmetic: parses arithmetic expressions.
pattern: parses glob and extglob patterns.
test_command: parses [[ ... ]] and test expressions.
prompt: parses prompt strings such as PS1.
readline_binding: parses readline key binding syntax.

§Cargo features

Feature	Effect
`arbitrary`	Implements `arbitrary::Arbitrary` for the syntax tree and tokens, for fuzzing.
`debug-tracing`	Turns on the `peg` crate’s rule tracing.
`diagnostics`	Adds `PrettyError` and `ParseError::to_pretty_error`, which render a parse error with `miette`.
`serde`	Implements `Serialize` and `Deserialize` for the syntax tree and tokens.
`winnow-parser`	Adds the `winnow`-based parser, selected with `ParserImpl::Winnow`.

Modules§

arithmetic: Parser for shell arithmetic expressions.
ast: Defines the Abstract Syntax Tree (ast) for shell programs. Includes types and utilities for manipulating the AST.
pattern: Implements parsing for shell glob and extglob patterns.
prompt: Parser for shell prompt syntax (e.g., PS1).
readline_binding: Implements a parser for readline binding syntax.
test_command: Parser for shell test commands.
winnow_strwinnow-parser: String-based winnow parser
word: Parser for shell words, used in expansion and other contexts.

Structs§

ParseErrorLocation: Represents a parsing error with its location information
Parser: Implements parsing for shell programs.
ParserBuilder: Use builder syntax to set the inputs and finish with build().
ParserOptions: Options used to control the behavior of the parser.
PrettyErrordiagnostics: Represents an error that occurred while parsing tokens.
SourceInfo: Information about the source of tokens.
SourcePosition: Represents a position in source text.
SourcePositionOffset: Represents an offset in source text.
SourceSpan: Represents a span within source text.
TestCommandParseError: Represents an error that occurred while parsing a (non-extended) test command.
TokenizerOptions: Options controlling how the tokenizer operates.

Enums§

BindingParseError: Represents an error that occurred while parsing a key-binding specification.
ParseError: Represents an error that occurred while parsing tokens.
ParserImpl: Parser implementation to use
Token: Represents a token extracted from a shell script.
TokenizerError: Represents an error that occurred during tokenization.
WordParseError: Represents an error that occurred while parsing a word.

Functions§

parse_tokens: Parses a sequence of tokens into the abstract syntax tree (AST) of a shell program.
tokenize_str: Break the given input shell script string into tokens, returning the tokens.
tokenize_str_with_options: Break the given input shell script string into tokens, returning the tokens.
uncached_tokenize_str: Break the given input shell script string into tokens, returning the tokens. No caching is performed.
unquote_str: Returns s with its quoting removed, as the shell’s quote removal does.

Type Aliases§

TokenLocation: Compatibility alias for SourceSpan.