Crate lispexp 

lispexp — a pure-Rust reader (lexer + parser) for S-expression syntax across many Lisp dialects.

The crate is deliberately reader-only: it does not evaluate, expand macros, or interpret the numeric tower. It reads source text into data — the shape, positions, and reader-macro structure needed to statically analyze Lisp code — and accepts a superset of what any one implementation’s reader would, so it is a substrate for tools (linters, indexers, formatters), not a validator (ADR-0030). See docs/design.md and docs/adr/ for the design and the decisions behind it.

Quick start

use lispexp::{parse, Options};

let parsed = parse("(define (square x) (* x x))", &Options::scheme());
assert!(parsed.errors.is_empty());
assert_eq!(parsed.data[0].head_symbol(), Some("define"));
assert_eq!(parsed.data[0].items().unwrap().len(), 3);

The reader is fault-tolerant — a malformed form loses only itself and recovery resumes at the next top-level form (ADR-0004) — so always inspect Parsed::errors alongside Parsed::data. parsed.errors.is_empty() is a usable “structurally clean” check.

Choosing a dialect

There is one reader; a Dialect selects a preset of Options (ADR-0003). lispexp never infers a dialect across files — pick one per input, e.g. by file extension:

use lispexp::{Dialect, Options};

let options = match "core.clj".rsplit('.').next() {
    Some("clj" | "cljs" | "cljc" | "edn") => Options::clojure(),
    Some("scm" | "ss") => Options::scheme_superset(),
    Some("el") => Options::emacs_lisp(),
    _ => Options::for_dialect(Dialect::Scheme),
};

Presets are a starting point: adjust individual fields by assignment afterwards (the settings are orthogonal, ADR-0006).

Two layers

Both layers sit over the same Options (ADR-0015):

• parse — builds the Parsed datum tree. The common entry point.
• lex / Lexer — a linear token stream that tiles the input (every byte is covered), for consumers like a parinfer backend that need lexical state, not a tree. The tree drops comments and whitespace, so a trivia-sensitive tool reads those here and correlates by byte Span.

The lexer’s EOF contract: tokens always tile the input, and an unterminated construct at end-of-input is reported as one TokenKind::Unterminated token carrying the lexical state it was in, rather than an error or a truncated token stream.

Static-analysis utilities

Built on the tree, each opt-in and reader-only:

• walk — a pruning visitor that classifies each node as Class::Code or Class::Data, so a tool descends into code and skips quoted data; walk_regions refines Data into prunable Region::SealedData vs. porous Region::PorousData so a Skip never drops quasiquoted code, and code_nodes is a fixed-policy pre-order iterator over just the code nodes (ADR-0026).
• annotate — tags definition forms (name, arglist, docstring, body, method dispatch) across dialects, from a bundled per-dialect core plus a spec harvester that learns a project’s own def-macros (ADR-0019/0020, ADR-0031/0032).
• indent — harvests Emacs Lisp indent specs into a symbol → IndentSpec table (ADR-0022).
• detect — opt-in, content-aware dialect detection (extension registry + #lang/shebang/structural signals) that picks an Options for you; the reader itself stays passive (ADR-0034, ADR-0012).
• parse_form_at — reads exactly one top-level form at a byte offset, for incremental re-validation after an edit (ADR-0023).
• LineIndex — maps byte offsets to 1-based (line, byte-column) (ADR-0024).

Modules

annotate

Definition-form annotation (ADR-0019).

detect

Opt-in, content-aware dialect detection (ADR-0034).

indent

Indent specs: a first-class symbol → IndentSpec table (ADR-0022).

Structs

BlockComment

A block-comment delimiter pair (ADR-0007).

CharRoles

The per-dialect table of reader-macro prefix glyphs (ADR-0016).

CodeNodes

A pre-order iterator over the Class::Code nodes of a datum forest, created by code_nodes.

Datum

A single parsed unit of S-expression syntax, annotated with its source span and 1-based start line. Borrows &'a str slices from the source (ADR-0008).

FormAt

One top-level form read at or after a byte offset (ADR-0023).

Lexer

The lexer. Implements Iterator over Tokens.

LineIndex

A precomputed line/column index over a source string (ADR-0024).

Options

Reader/lexer configuration. Construct via a preset such as Options::scheme or Options::clojure, then adjust fields if needed.

ParseDialectError

The error returned by Dialect’s FromStr impl when the input names no known dialect.

ParseError

A non-fatal parse diagnostic. The reader is fault-tolerant (ADR-0004): it returns a partial tree plus a list of these, resynchronizing at the next top-level form.

Parsed

The result of reading a source string. Borrows the source (ADR-0008).

Span

A byte range into the source string: [start, end).

Token

One lexeme. Tokens tile the input — every byte belongs to exactly one token, whitespace and comments included (ADR-0015). Carries only a span; text is recovered by slicing the source.

Enums

CharSyntax

How character literals are introduced.

Class

Whether a subtree is executable code or inert data (ADR-0026).

DatumKind

The shape of a Datum.

Delim

Delimiter shape. The reader records shape; the consumer assigns meaning.

DelimRole

The role of a bracket pair [] or {} in a dialect.

Dialect

A named dialect. Presets are constructed via Options.

ErrorKind

A structured classification of a parse diagnostic (ADR-0023).

HashBracket

What #[ opens in a dialect. The single #[ dispatch has several mutually-exclusive meanings across dialects; this enum makes the choice explicit (like HashParen for #() instead of leaving it to the implicit order of competing flags. Distinct from the bare-[ delimiter role (Options::square), which still applies when this is HashBracket::None (e.g. Racket/Emacs #[...] hash-vectors).

HashParen

What #( means in a dialect.

Notation

Whether a reader-macro form appeared in shorthand or long-hand call form.

Prefix

The role of a reader-macro prefix. The glyph that triggers each is a per-dialect table (ADR-0016).

Region

A refinement of Class that splits Data by whether it is safe to prune (ADR-0026).

Terminator

The line-terminator kind at the end of a line (ADR-0024). Breaks are \n and \r\n only, so this is a closed set — matched exhaustively, like crate::Class.

TokenKind

The classification of a Token.

UnterminatedKind

The lexical state an TokenKind::Unterminated token was in when input ran out.

Walk

What the visitor callback asks the walker to do next.

Functions

code_nodes

Iterate the Class::Code nodes of data in pre-order — the read-only, fixed-policy counterpart to walk. It always prunes sealed data and descends porous quasiquote templates, so nested unquoted code is reached but quoted data is never yielded.

lex

Lex source under options, yielding a token stream that tiles the input. source must be at most u32::MAX bytes (Span stores u32 offsets).

parse

Parse source under options into a datum tree. Never panics — including on pathologically nested input: list/hash nesting deeper than a fixed depth cap (ADR-0028) stops descending, reports ErrorKind::DepthLimitExceeded once, and skips the too-deep subtree (ADR-0004), keeping prior siblings. source must be at most u32::MAX bytes (Span stores u32 offsets).

parse_form_at

Read exactly one top-level form at or after byte offset start (ADR-0023).

walk

Walk each top-level datum in data, invoking visit(datum, class) in pre-order. When the callback returns Walk::Skip, that datum’s children are pruned; Walk::Descend recurses; Walk::Stop aborts the whole walk. Top-level data start as Class::Code.

walk_regions

Like walk, but the callback receives a three-way Region instead of the binary Class, so pruning is safe: Region::is_prunable tells you whether Walk::Skip would lose code.