pub fn source_to_tokens(text: &str) -> Result<Vec<Token>, TokenizationError>
Expand description
Generate a Vec of Tokens from a str.
This function is complementary with tokens_to_source. The typical workflow is to use this function to generate a Vec of tokens, do something to modify it ( e.g., add/remove/replace tokens) then use tokens_to_source to turn it back into JSON5 source.
Unlike the tokenizing functions available in the crate::tokenize module, this function produces owned Token objects containing (among other fields) an owned String of the lexeme, rather than a crate::tokenize::Tokens struct.
§Examples
use json_five::{source_to_tokens, tokens_to_source};
use json_five::rt::tokenize::Token;
use json_five::tokenize::TokType::Whitespace;
let tokens = source_to_tokens(" {my: 'json5'} ").unwrap();
// remove all Whitespace tokens
let new_tokens:Vec<Token> = tokens.into_iter().filter(|tok| tok.tok_type != Whitespace).collect();
// turn tokens back into source
let new_source = tokens_to_source(&new_tokens);
assert_eq!(new_source, String::from("{my:'json5'}"))