shrub_parser/lib.rs
1//! A tokenizer and parsers for POSIX and bash shell syntax, with no
2//! dependency on a running shell.
3//!
4//! `shrub-parser` is the bottom of shrub's crate graph: `shrub-core` builds on
5//! it to run scripts, and front ends use it to inspect or highlight input. Use
6//! it directly to read shell scripts into a syntax tree without executing them.
7//!
8//! ## Quick start
9//!
10//! Wrap the script in a reader, build a [`Parser`], and call
11//! [`Parser::parse_program`]. The result is an [`ast::Program`].
12//!
13//! ```
14//! use std::io::Cursor;
15//!
16//! use shrub_parser::{ParseError, Parser};
17//!
18//! let mut parser = Parser::builder().build(Cursor::new("echo hello | wc -c\nls -l\n"));
19//! let program = parser.parse_program()?;
20//!
21//! assert_eq!(program.complete_commands.len(), 2);
22//! # Ok::<(), ParseError>(())
23//! ```
24//!
25//! Input that is not valid shell syntax returns a [`ParseError`] that says
26//! where parsing failed.
27//!
28//! ## Parser options
29//!
30//! [`Parser::builder`] sets the options that change what the input means:
31//! extended globbing, POSIX mode, `sh` mode, and tilde expansion.
32//! [`Parser::new`] takes the same settings as a [`ParserOptions`] value. The
33//! defaults match bash.
34//!
35//! ## Tokenizing
36//!
37//! [`tokenize_str`] splits a script into [`Token`]s without building a tree.
38//! Each token is a word or an operator and records its [`SourceSpan`]. Words
39//! keep their quotes; [`unquote_str`] removes them.
40//!
41//! ```
42//! use shrub_parser::{Token, TokenizerError, tokenize_str, unquote_str};
43//!
44//! let tokens = tokenize_str("echo 'a b' > out")?;
45//!
46//! assert_eq!(tokens.len(), 4);
47//! assert!(matches!(tokens[2], Token::Operator(..)));
48//! assert_eq!(unquote_str(tokens[1].to_str()), "a b");
49//! # Ok::<(), TokenizerError>(())
50//! ```
51//!
52//! [`tokenize_str_with_options`] takes [`TokenizerOptions`], and
53//! [`uncached_tokenize_str`] skips the cache that [`tokenize_str`] keeps of
54//! recent inputs.
55//!
56//! ## Modules
57//!
58//! - [`ast`]: the syntax tree types.
59//! - [`word`]: parses a word into its parts, such as parameter and command
60//! expansions.
61//! - [`arithmetic`]: parses arithmetic expressions.
62//! - [`pattern`]: parses glob and extglob patterns.
63//! - [`test_command`]: parses `[[ ... ]]` and `test` expressions.
64//! - [`prompt`]: parses prompt strings such as `PS1`.
65//! - [`readline_binding`]: parses readline key binding syntax.
66//!
67//! ## Cargo features
68//!
69//! | Feature | Effect |
70//! |---|---|
71//! | `arbitrary` | Implements `arbitrary::Arbitrary` for the syntax tree and tokens, for fuzzing. |
72//! | `debug-tracing` | Turns on the `peg` crate's rule tracing. |
73//! | `diagnostics` | Adds `PrettyError` and `ParseError::to_pretty_error`, which render a parse error with `miette`. |
74//! | `serde` | Implements `Serialize` and `Deserialize` for the syntax tree and tokens. |
75//! | `winnow-parser` | Adds the `winnow`-based parser, selected with `ParserImpl::Winnow`. |
76
77#![cfg_attr(docsrs, feature(doc_cfg))]
78// TODO(unwrap): remove or scope this allow attribute
79#![allow(clippy::unwrap_used)]
80
81pub mod arithmetic;
82pub mod ast;
83pub mod pattern;
84pub mod prompt;
85pub mod readline_binding;
86pub mod test_command;
87pub mod word;
88
89mod error;
90mod parser;
91mod source;
92mod tokenizer;
93
94#[cfg(test)]
95mod snapshot_tests;
96
97#[cfg(feature = "diagnostics")]
98pub use error::miette::PrettyError;
99pub use error::{
100 BindingParseError, ParseError, ParseErrorLocation, TestCommandParseError, WordParseError,
101};
102#[cfg(feature = "winnow-parser")]
103pub use parser::winnow_str;
104pub use parser::{Parser, ParserBuilder, ParserImpl, ParserOptions, SourceInfo, parse_tokens};
105pub use source::{SourcePosition, SourcePositionOffset, SourceSpan};
106pub use tokenizer::{
107 Token, TokenLocation, TokenizerError, TokenizerOptions, tokenize_str,
108 tokenize_str_with_options, uncached_tokenize_str, unquote_str,
109};