Function mpdpopm::messages::tokenize

pub fn tokenize<'a>(buf: &'a mut [u8]) -> impl Iterator<Item = Result<&'a [u8]>>

Break buf up into individual tokens while removing MPD-style quoting.

When a client sends a command to mpdpopm, it will look like this on the wire:

sendmessage ${CHANNEL} “some-command "with space" simple "‘with single’ and \\"”

In other words, the MPD “sendmessage” command takes two parameters: the channel and the message. The recipient (i.e. us) is responsible for breaking up the message into its constituent parts (a command name & its arguments in our case).

The message will perforce be quoted according ot the MPD rules:

1. an un-quoted token may contain any printable ASCII character except space, tab, ’ & “

2. to include spaces, tabs, ’-s or “-s, the token must be enclosed in “-s, and any “-s or -s therein must be backslash escaped

When the messages is delivered to us, it has already been un-escaped; i.e. we will see the string:

some-command “with space” simple “‘with single’ and \”

This function will break that string up into individual tokens with one more level of escaping removed; i.e. it will return an iterator that will yield the four tokens:

1. some-command
2. with space
3. simple
4. ‘with single’ and

MPD has a nice implementation that modifies the string in place by copying subsequent characters on top of escape characters in the same buffer, inserting nulls in between the resulting tokens,and then working in terms of pointers to the resulting null-terminated strings.

Once I realized that I could split slices I saw how to implement an Iterator that do the same thing (an idiomatic interface to the tokenization backed by a zero-copy implementation). I was inspired by “My Favorite Rust Function Signature” https://www.brandonsmith.ninja/blog/favorite-rust-function.

NB. This method works in terms of a slice of u8 because we can’t index into Strings in Rust, and MPD deals only in terms of ASCII at any rate.