token-lang 1.0.0

Token type definitions - the shared seam between lexer and parser.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

# token-lang v0.2.0 — Core

The first release with domain logic: the token type and the seam a lexer and a
parser agree on. v0.1.0 was the scaffold; v0.2.0 is the crate.

## What is token-lang?

token-lang defines the token type that sits between a lexer and a parser. A
`Token<K>` is a *classified span* — a language-specific kind `K` paired with the
`Span` of source it covers — and the `TokenKind` trait gives generic code the small
set of questions a parser asks of any token: skip trivia, detect the end, read an
interned lexeme. It owns the token type and nothing else; the language supplies `K`,
so the crate stays reusable across every front-end built on it.

## The surface

- **`Token<K>`** — a kind paired with a `Span`. `new`, the `kind` / `span` /
  `into_kind` accessors, `map` (rewrite the kind, keep the span), and `as_ref`.
  `Copy` when `K` is; orders by span first so a stream sorts into source order;
  `Display` renders `kind @ start..end`. Converts to and from `span-lang`'s
  `Spanned<K>` and a `(K, Span)` tuple.
- **`TokenKind`** — the classification trait a language implements for its kind:
  `is_trivia`, `is_eof`, and `symbol` (the interned lexeme, as a `Symbol`). Every
  method has a default, so a kind with none of these needs only an empty `impl`.
  `Token` forwards all three when `K: TokenKind`.
- **Re-exports** — `Span`, `Spanned` from `span-lang` and `Symbol` from
  `intern-lang`, so a consumer names only `token-lang`.

This is where the family dependencies are wired: `span-lang` for the `where` of a
token, `intern-lang` for the interned `what`. Both are real, used dependencies, not
placeholders.

## Guarantees

The invariants are property-tested against direct field access and `span-lang`'s
`Spanned`:

- **Construction is lossless** — a token built from `(kind, span)` reads both back
  unchanged.
- **Spanned round-trip** — `Token` and `Spanned` hold the same two fields and
  convert both ways without loss.
- **Source order** — token ordering equals `(span, kind)` ordering, so sorting a
  stream sorts it by position.
- **`map` keeps the span** — only the kind changes.
- **Forwarders, not a second truth** — `Token::is_trivia` / `is_eof` / `symbol`
  return exactly what the kind returns.
- **Total and panic-free** — no constructor or accessor can fail, and a token adds
  no allocation beyond what `K` costs.

## Performance

The parser-side hot paths, measured with Criterion on the development machine
(Windows x86_64, Rust stable) over a 4,097-token stream:

| Operation (4,097 tokens) | Mean |
|---|---|
| Filter trivia + end markers | ~0.88 µs |
| Collect interned symbols | ~1.41 µs |
| Sort into source order | ~4.32 µs |

A `Token<K>` is `Copy` whenever `K` is and adds only an eight-byte span to the kind;
classification and the accessors are `#[inline]` and allocation-free.

## Compatibility

- **MSRV** Rust 1.85, edition 2024.
- **`no_std`** — the crate needs neither `std` nor `alloc`. The default `std`
  feature only forwards to the dependency builds; `--no-default-features` keeps the
  whole surface.
- **Breaking changes** — none against v0.1.0 (the scaffold had no surface).

## Verification

Run green on **Windows x86_64** against Rust stable and MSRV 1.85, default and all
features:

```bash
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
cargo build --no-default-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo +1.85 test --all-features
cargo bench --no-run
cargo deny check
```

Counts at this release: 13 unit + 7 property + 7 integration doctest-excluded tests
and 13 doctests on default features; the 3-test `serde` suite adds under
`--all-features`. The `--no-default-features` build is clean as `no_std`.

## What's next

The 0.x line finalizes the surface; `1.0.0` freezes it under SemVer. One decision
is open before the freeze — whether the `TokenKind::symbol` accessor stays on the
trait — recorded in [`dev/NOTES.md`](../../dev/NOTES.md).

## Installation

```toml
[dependencies]
token-lang = "0.2"

# optional: serde support for `Token`
token-lang = { version = "0.2", features = ["serde"] }
```

## Documentation

- [README](https://github.com/jamesgober/token-lang/blob/main/README.md)
- [API Reference](https://github.com/jamesgober/token-lang/blob/main/docs/API.md)
- [CHANGELOG](https://github.com/jamesgober/token-lang/blob/main/CHANGELOG.md)