Struct Tokenizer

pub struct Tokenizer<'a> { /* private fields */ }

Tokenizer for an R program.

Transforms an R program into an array of language tokens.

Implementations

impl<'a> Tokenizer<'a>

pub fn new(input: &'a str) -> Self

Returns a new tokenizer from an R program.

Arguments

• input - the R program to tokenize

Examples

use tergo_tokenizer::tokenizer::Tokenizer;

let r_program = r#"
a <- 7
print(a + 1)
"#;
let mut tokenizer = Tokenizer::new(r_program);
let tokens = tokenizer.tokenize();
println!("{tokens:?}");

pub fn tokenize(&mut self) -> Vec<CommentedToken<'_>>

Returns an array of tokens.

Examples

use tergo_tokenizer::tokenizer::Tokenizer;

let r_program = r#"
a <- 7
print(a + 1)
"#;
let mut tokenizer = Tokenizer::new(r_program);
let tokens = tokenizer.tokenize();
println!("{tokens:?}");