pub struct Kind(/* private fields */);Expand description
The kind of a token or node in a forged language’s syntax tree.
A schematic declares its vocabulary as text — rule names, keywords,
punctuation — so a forged language cannot have a Rust enum of its own.
Kind stands in for one: a small Copy value, assigned when the language is
forged, that compares as cheaply as an enum discriminant. Tokens and nodes
share the type, the model syntax_lang is built on.
Kinds are looked up by name with Language::kind
and named with Language::kind_name. Look a
kind up once, keep it, and compare against it while walking trees: the
comparison is a single integer compare.
| Name | Kind of |
|---|---|
a rule name, such as let_stmt | the node the rule builds |
a literal’s text, such as let or += | the keyword or symbol token |
a Pratt level’s node, or binary, prefix, postfix | an operator node |
IDENT, NUMBER, STRING, NEWLINE | the built-in token classes |
WHITESPACE, COMMENT | trivia tokens |
UNKNOWN | a character the lexer did not recognize (trivia) |
ERROR | a node wrapping tokens the parser skipped |
A kind belongs to the language that produced it. Comparing kinds from two
different languages is meaningless, and naming one language’s kind with
another language returns the wrong name or "<unknown>".
§Trivia
TokenKind::is_trivia answers without the language: whitespace, comments,
unrecognized characters, and — unless the schematic sets newlines = true —
line breaks are trivia. Trivia is kept in the tree, so it stays lossless,
but the parser never sees it.
§Examples
use lang_forge::Language;
use lang_forge::syntax_lang::TokenKind;
let lang = Language::from_lsf(
r#"
[language]
name = "sum"
[rules]
sum = "NUMBER ('+' NUMBER)*"
"#,
)?;
let plus = lang.kind("+").expect("the grammar uses '+'");
let space = lang.kind("WHITESPACE").expect("built in");
assert_eq!(lang.kind_name(plus), "+");
assert!(space.is_trivia());
assert!(!plus.is_trivia());
let parse = lang.parse("1 + 2");
let pluses = parse.tree().tokens().filter(|t| *t.kind() == plus).count();
assert_eq!(pluses, 1);