Function source_to_tokens

Source
pub fn source_to_tokens(text: &str) -> Result<Vec<Token>, TokenizationError>
Expand description

Generate a Vec of Tokens from a str.

This function is complementary with tokens_to_source. The typical workflow is to use this function to generate a Vec of tokens, do something to modify it ( e.g., add/remove/replace tokens) then use tokens_to_source to turn it back into JSON5 source.

Unlike the tokenizing functions available in the crate::tokenize module, this function produces owned Token objects containing (among other fields) an owned String of the lexeme, rather than a crate::tokenize::Tokens struct.

§Examples

use json_five::{source_to_tokens, tokens_to_source};
use json_five::rt::tokenize::Token;
use json_five::tokenize::TokType::Whitespace;

let tokens = source_to_tokens("  {my: 'json5'}  ").unwrap();

// remove all Whitespace tokens
let new_tokens:Vec<Token> = tokens.into_iter().filter(|tok| tok.tok_type != Whitespace).collect();

// turn tokens back into source
let new_source = tokens_to_source(&new_tokens);
assert_eq!(new_source, String::from("{my:'json5'}"))