argtokens: Rust library to parse command-line arguments

argtokens extracts options and values out of command-line arguments.

Design

argtokens works like an iterator: you pull individual tokens from it (using next) and process them in sequence. If you want an option to take a value, you have to explicitly call next_value when encountering the option.

argtokens is cross-platform and supports no_std, no-alloc usage. It operates on an Argument trait that is implemented for &[u8], &[u16], &OsStr, and &str depending on the feature flags enabled.

argtokens requires the arguments to already be in list form (instead of a single string); it does not do shell argument splitting, unquoting, or unescaping.

The supported syntax is similar to what you often find on GNU and FreeBSD:

-abc is normally interpreted like -a -b -c. If you tell argtokens to extract a value after the -a, it is interpreted like -a bc instead.
Long options are supported, in both --opt val and --opt=val forms.
- and -- are interpreted as normal freestanding arguments, not as options. argtokens does not otherwise treat -- in a special way; you are free to either process it like any other freestanding argument or to change parsing behavior in response.
When given arg --opt, what you do when encountering the arg determines whether --opt is interpreted as an option (like GNU) or as a positional argument (like POSIX).
With a bit of extra code, it is possible to treat negative numbers (or other option-like arguments), such as the -2 in -2 + 3, as positional arguments instead of as options.

While argtokens limits option names to ASCII characters, option values and positional arguments can contain arbitrary characters which may or may not be valid in any encoding. For example, in --opt=val arg, opt can only consist of ASCII characters, while val and arg can contain any character sequence.

argtokens never panics nor aborts when encountering invalid input; you are always able to handle user errors.

argtokens aims to be suitable for use directly by applications or as the tokenizer/lexer of a higher-level argument parsing library.

Examples

The following example shows argtokens usage with std support enabled.

use std::ffi::OsString;

use argtokens::{ArgTokens, Token};

fn main() -> Result<(), &'static str> {
	let args: Vec<OsString> = std::env::args_os().skip(1).collect();
	let mut tokens = ArgTokens::new(args.iter().map(OsString::as_os_str));

	while let Some(token) = tokens.next() {
		match token {
			Token::Freestanding(arg) => println!("Arg: {arg:?}"),
			Token::Short(b'c') => println!("Mode: create"),
			Token::Short(b'x') => println!("Mode: extract"),
			Token::Short(b'f') | Token::Long(b"file") => {
				let value = tokens.next_value().ok_or("-f/--file requires value")?;
				println!("File: {value:?}");
			}
			_ => return Err("Error"),
		}
	}
	Ok(())
}

Refer to examples/realistic.rs in the source repository to see a much more complete example that you might use in a real application, with -- support and proper error handling.

The examples/no_std directory demonstrates no_std, no-alloc usage.

Feature flags

inline-next (default): Marks the ArgTokens::next and ArgTokens::next_with_check methods #[inline], which may affect code size and/or speed. This is most effective when you only call next or next_with_check from one place in the code (usually the main parser loop).
std (default): Allows argtokens to work on OsStr arguments. This also adds a few helpers to convert to and from OsStr.
str: Allows argtokens to work on str arguments. This is only useful in some very niche situations (e.g. WASI with no_std). In almost all cases you should be using the u8/u16 slice or OsStr argument type instead so that you can process non-Unicode arguments.

Alternatives

lexopt is another library that uses a similar approach to argtokens. There are differences in the API but the iterator-based architecture is very close.

clap uses a mapping-based architecture, making it very different from argtokens.

License & contributions

This project is licensed under the Mozilla Public License Version 2.0 (MPL-2.0).