pub struct Kind(/* private fields */);Expand description
The kind of a token or node in a forged language’s syntax tree.
A schematic declares its vocabulary as text — rule names, keywords,
punctuation — so a forged language cannot have a Rust enum of its own.
Kind stands in for one: a small Copy value, assigned when the language is
forged, that compares as cheaply as an enum discriminant. Tokens and nodes
share the type, the model syntax_lang is built on.
Kinds are looked up by name with Language::kind
and named with Language::kind_name. Look a
kind up once, keep it, and compare against it while walking trees: the
comparison is a single integer compare.
| Name | Kind of |
|---|---|
a rule name, such as let_stmt | the node the rule builds |
a literal’s text, such as let or += | the keyword or symbol token |
a Pratt level’s node, or binary, prefix, postfix | an operator node |
IDENT, NUMBER, STRING, NEWLINE | the built-in token classes |
WHITESPACE, COMMENT | trivia tokens |
UNKNOWN | a character the lexer did not recognize (trivia) |
ERROR | a node wrapping tokens the parser skipped |
A format-2 sketch adds its own token classes, string classes and the kinds
generated for them, and the format-2 built-ins (DOC_COMMENT, SHEBANG,
…); docs/API.md has the full list.
A kind belongs to the language that produced it. Comparing kinds from two
different languages is meaningless, and naming one language’s kind with
another language returns the wrong name or "<unknown>".
§Index
Every kind has a dense index, its position in the
language’s kind table, and Language::kind_at
turns an index back into the kind. Adapters that need a plain integer (an
incremental reparser’s constants, a tree-sitter export, a language server)
use the pair. For a format-2 sketch the numbering is the deterministic one
of LSF2 §5.4; for a format-1 schematic it is an implementation detail that
may change between releases.
§Field labels
A format-2 grammar can label what a rule matches (cond:expr). The label
of the edge from a node to one of its children is carried by the child’s
kind: label returns it. Labels never take part in
comparison, ordering, or hashing — a labelled expr is still equal to
lang.kind("expr") — so code that only matches kinds is unaffected. See
Language::field_label.
§Trivia
TokenKind::is_trivia answers without the language: whitespace, comments,
unrecognized characters, and — unless the schematic sets newlines = true —
line breaks are trivia. Trivia is kept in the tree, so it stays lossless,
but the parser never sees it.
§Examples
use lang_forge::Language;
use lang_forge::syntax_lang::TokenKind;
let lang = Language::from_lsf(
r#"
[language]
name = "sum"
[rules]
sum = "NUMBER ('+' NUMBER)*"
"#,
)?;
let plus = lang.kind("+").expect("the grammar uses '+'");
let space = lang.kind("WHITESPACE").expect("built in");
assert_eq!(lang.kind_name(plus), "+");
assert!(space.is_trivia());
assert!(!plus.is_trivia());
assert_eq!(lang.kind_at(plus.index()), Some(plus));
let parse = lang.parse("1 + 2");
let pluses = parse.tree().tokens().filter(|t| *t.kind() == plus).count();
assert_eq!(pluses, 1);Implementations§
Source§impl Kind
impl Kind
Sourcepub const fn index(self) -> u16
pub const fn index(self) -> u16
The kind’s position in its language’s kind table.
Indexes are dense, from 0 to one less than
Language::kind_count;
Language::kind_at is the inverse.
§Examples
use lang_forge::Language;
let lang = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nitem = \"'go' NUMBER\"\n")?;
let go = lang.kind("go").expect("a keyword");
assert!(usize::from(go.index()) < lang.kind_count());
assert_eq!(lang.kind_at(go.index()), Some(go));Sourcepub const fn label(self) -> Option<u16>
pub const fn label(self) -> Option<u16>
The field label of the tree edge this kind sits on, if the grammar labelled it.
Only kinds read from a tree carry labels: a token or node a format-2
rule matched under name: gets that label’s id. Kinds returned by
Language::kind have none. Name a label with
Language::label_name.
§Examples
use lang_forge::Language;
let lang = Language::from_lsf(
"[sketch]\nformat = 2\n[language]\nname = \"pair\"\nversion = \"1.0.0\"\n\
[rules]\npair = \"key:IDENT '=' value:NUMBER\"\n",
)?;
let parse = lang.parse("x = 1");
let labels: Vec<Option<&str>> = parse
.tree()
.child_tokens()
.map(|t| t.kind().label().map(|l| lang.label_name(l).unwrap_or("?")))
.collect();
assert_eq!(labels, [Some("key"), None, None, None, Some("value")]);Sourcepub const fn unlabelled(self) -> Self
pub const fn unlabelled(self) -> Self
The same kind without a field label.
§Examples
use lang_forge::Language;
let lang = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nitem = \"NUMBER\"\n")?;
let number = lang.kind("NUMBER").expect("built in");
assert_eq!(number.unlabelled().label(), None);
assert_eq!(number.unlabelled(), number);