Skip to main content

classify_token

Function classify_token 

Source
pub fn classify_token(kind: &str, is_named: bool, text: &str) -> TokenKind
Expand description

The shared C-family token classification.

Grammar kind names drive the mapping; anonymous (non-named) tokens are keywords when their kind is purely alphabetic and punctuation otherwise (operators, delimiters, and directive introducers like #include). Named leaves outside the known kinds — notably the opaque preproc_arg replacement text — classify as TokenKind::Unknown.