pub fn structural_tokenize(code: &str, lang: &str) -> Vec<StructuralToken>Expand description
Tokenize source into weighted structural tokens (motifs, keywords, literals, …).
pub fn structural_tokenize(code: &str, lang: &str) -> Vec<StructuralToken>Tokenize source into weighted structural tokens (motifs, keywords, literals, …).