Crate syntax_lang 

syntax_lang

A lossless concrete syntax tree (CST) with trivia — the substrate a formatter, a language server, or a tree-sitter-style generator builds on. It owns no grammar: a language brings its own kind type, and syntax-lang supplies the tree shape, the builder that assembles it, and the traversals over it.

Model

A tree is Nodes all the way down to Token leaves. Each Node carries a kind, the Span of source it covers, and an ordered list of Element children — each child being a nested node or a leaf token. Node kinds and token kinds share one type K (the rowan model): a language defines a single enum with both composite variants (Expr, Root) and lexical ones (Ident, Plus, Whitespace), and implements TokenKind on it to mark trivia and the end marker.

Lossless means nothing is discarded. Trivia — whitespace and comments — is not filtered out; it rides as ordinary leaf tokens in source order. So the tree is a faithful record of the source: slicing the original text by a node’s span, or concatenating the node’s tokens, reproduces exactly the source that node came from. Tokens store spans rather than copies of the text, so the tree stays compact and the reconstruction is a zero-copy borrow.

Building a tree

A parser drives a Builder with three moves — open a node, push a token, close a node:

use syntax_lang::{Builder, Span, Token, TokenKind};

// One kind type for both nodes and tokens.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Kind {
    // nodes
    Root,
    Sum,
    // tokens
    Num,
    Plus,
    Space,
}

impl TokenKind for Kind {
    fn is_trivia(&self) -> bool {
        matches!(self, Kind::Space)
    }
}

// Build the CST for `1 + 2`, keeping the spaces as trivia tokens.
let mut b = Builder::new();
b.start_node(Kind::Root);
b.start_node(Kind::Sum);
b.token(Token::new(Kind::Num, Span::new(0, 1)));
b.token(Token::new(Kind::Space, Span::new(1, 2)));
b.token(Token::new(Kind::Plus, Span::new(2, 3)));
b.token(Token::new(Kind::Space, Span::new(3, 4)));
b.token(Token::new(Kind::Num, Span::new(4, 5)));
b.finish_node(); // close Sum
b.finish_node(); // close Root

let root = b.finish().expect("balanced");

// Lossless: the tree reproduces the source, trivia and all.
assert_eq!(root.text("1 + 2"), Some("1 + 2"));
assert_eq!(root.tokens().count(), 5);

// A formatter can now walk the significant tokens and skip the trivia.
let significant = root.tokens().filter(|t| !t.is_trivia()).count();
assert_eq!(significant, 3);

Walking a tree

Node::tokens yields every leaf in source order (the lossless stream); Node::descendants yields every node in pre-order; Node::children, Node::child_nodes, and Node::child_tokens step over the direct level. All of these are iterative — a tree tens of thousands of levels deep is walked, and freed, without overflowing the call stack.

Features

• std (default) — the standard library; without it the crate is no_std (it always needs alloc for the child vectors). Forwards to token-lang/std and span-lang/std.

Stability

The public surface is frozen as of 1.0.0 and follows Semantic Versioning: no breaking change before 2.0, additions arrive in minor releases, and the MSRV (Rust 1.85) only rises in a minor. The frozen surface is catalogued in docs/API.md.

Structs

Builder

Assembles a Node tree from a stream of start_node / token / finish_node calls — the shape a parser drives as it recognises the grammar.

Node

An interior node of a concrete syntax tree: a kind, the span of source it covers, and its ordered children.

Span

A half-open byte range start..end into a single source.

Spanned

A value together with the source Span it was parsed from.

Symbol

A small, copyable handle to a string held by an Interner.

Token

A single lexical token: a kind paired with the Span of source it covers.

Enums

BuildError

Why a Builder could not produce a tree.

Element

One child of a Node: either a nested node or a leaf token.

Traits

TokenKind

The classification a token kind exposes so that generic, language-agnostic code can reason about a token stream without knowing the concrete kind.