Function tokenize 

pub fn tokenize(content: &str) -> Result<Vec<Token>, TokenizationError>

Tokenizes the provided content string and produces a vector of tokens.

This function is part of the bracoxide crate and is used to tokenize a given string content. The tokenization process splits the string into meaningful units called tokens, which can be further processed or analyzed as needed.

Arguments

• content - The string to be tokenized.

Returns

• Result<Vec<Token>, TokenizationError> - A result that contains a vector of tokens if the tokenization is successful, or a TokenizationError if an error occurs during the tokenization process.

Errors

The function can return the following errors:

• TokenizationError::EmptyContent - If the content string is empty.
• TokenizationError::NoBraces - If the content string does not contain any braces.
• TokenizationError::FormatNotSupported - If the content string has an unsupported format, such as only an opening brace or closing brace without a corresponding pair.

Examples

use bracoxide::tokenizer::{Token, TokenizationError, tokenize};

let content = "{1, 2, 3}";
let tokens = tokenize(content);

match tokens {
    Ok(tokens) => {
        println!("Tokenization successful!");
        for token in tokens {
            println!("{:?}", token);
        }
    }
    Err(error) => {
        eprintln!("Tokenization failed: {:?}", error);
    }
}

In this example, the tokenize function from the bracoxide crate is used to tokenize the content string “{1, 2, 3}”. If the tokenization is successful, the resulting tokens are printed. Otherwise, the corresponding error is displayed.