Module shell_tokenizer 

POSIX Shell Argument Tokenizer

This module implements a tokenizer for splitting shell command lines into arguments, handling quoting and escape sequences according to POSIX shell rules.

This tokenizer produces correct results for valid inputs that only use single-quoted strings, double-quoted strings, and backslash escapes. If unsupported shell syntax is encountered (such as variable expansion, command substitution, globs, or other shell features), the tokenizer produces a best-effort result but populates the errors list in the result, indicating that the output should not be trusted.

Structs

Error

An error encountered during shell tokenization.

TokenizeResult

The result of tokenizing a shell command line.

Enums

ErrorKind

The kind of error encountered during shell tokenization.

Functions

tokenize

Tokenize a shell command line (as bytes) into arguments according to POSIX shell rules.

tokenize_str

Tokenize a shell command line string into arguments according to POSIX shell rules.