Function serde_token::tokenize

pub fn tokenize<'de, D, S>(deserializer: D, sink: S) -> Result<(), Error> where
    D: Deserializer<'de>,
    S: Sink<SinkItem = Token<'de>>, 

Transcodes a deserializer into a Sink of Tokens.

NOTE: currenty uses unsafe twice: for coercing an 'de lifetime on the deserialized and borrowed &[u8] or &str. I believe this is safe because the Tokenizer is only used here, and always with an associated Deserializer<'de>.