# TEL Specification
## Context
TEL is the typed pattern language used by `tokmat` to extract structured fields from a tokenized,
classified input stream.
It is designed to work after tokenization and classification, not as a replacement for them.
## Mental model
Think of TEL as matching a sequence of token descriptors:
```text
Raw text -> token boundaries -> token types/classes -> TEL extraction
```
The crucial detail is that TEL matches the **class stream produced by the
model**, then uses the aligned raw-token stream to return captured fields. A
literal TEL segment therefore means "literal class-stream text", not
"reinterpret the original raw string after classification".
```text
Raw text: 11-47 OAK ST
Tokens: ["11", "-", "47", " ", "OAK", " ", "ST"]
Class stream: ["NUM", "-", "NUM", " ", "ALPHA", " ", "STREETTYPE"]
TEL literal: <!{{-}}?!>
```
If a different model maps the dash token to `DASH`, the class stream changes:
```text
Class stream: ["NUM", "DASH", "NUM", " ", "ALPHA", " ", "STREETTYPE"]
TEL class: <!DASH?!>
```
A TEL rule should answer:
- Which token shapes are required?
- Which semantic classes are required?
- Which parts should be captured?
- Which parts should be ignored or vanish from the result?
## Syntax
### Capturing groups
Capture a field by name:
```text
<<CITY>>
```
### Type modifiers
Apply token-shape constraints:
```text
<<CIVIC#>>
<<STREET@>>
<<UNIT@%>>
```
Meaning:
- `@` alpha-oriented token constraint
- `#` numeric token constraint
- `%` extended token forms
- `=` strict class handling
### Quantifiers
Control cardinality:
```text
<<NAME@+>>
<<SUFFIX@?>>
```
Meaning:
- `+` one or more
- `?` optional
### Greedy matching
Use `$` when a segment should absorb more of the eligible sequence:
```text
<<CITY@+$>>
```
### Explicit classes
Require a token to belong to a semantic class:
```text
<<TYPE::STREETTYPE>>
<<PROV::PROV>>
<<PC::PCODE>>
```
### Literal blocks
Match a literal phrase as a unit:
```text
{{PO BOX}}
{{#}}?
```
Literal blocks match exact text in the class stream. They are the right tool
when the model leaves a symbol, marker, or sentinel as a literal token class.
They are not a way to match a named class by its original raw spelling.
### Vanishing groups
Require a token or class to appear without returning it as a captured field:
```text
<!PROV!>
<!DASH?!>
<!@!>
<!@%?!>
<!DROP@!>
<!DROP::STREETTYPE!>
<!::STREETTYPE!>
<!@[ALPHA|ALPHA_EXTENDED]!>
<!+!>
<!,?!>
<!,{{?}}?!>
<!{{#}}?!>
<!{{1-1}}?!>
```
Vanishing groups are class-strict: `<!PROV!>` only matches a token whose class
is `PROV`, exactly like a `::PROV` capture. To skip one token regardless of its
class, use the shape meta-names `<!WORD!>` (word characters) or `<!WORDX!>`
(the model word definition).
Vanishing groups are capture-derived matchers: they use the same matching
semantics as a capture group, but they do not add a field to the output. This is
useful for symbols, marker tokens, prefixes, or structural classes that matter
to the match but should not appear in the captured result.
A body made only of identifier characters is treated as a class name. That keeps
`<!PROV!>` concise and class-strict. Capture-like bodies such as `<!DROP@!>`,
`<!DROP::STREETTYPE!>`, `<!@!>`, and `<!::STREETTYPE!>` use capture matching
semantics and discard the field name. Any other non-identifier body is treated
as literal class-stream text unless it is escaped through an embedded literal
block.
| `<!DASH!>` | Required `DASH` class | The model maps `-` to a class named `DASH` |
| `<!DASH?!>` | Optional `DASH` class | The dash may or may not be present as `DASH` |
| `<!@!>` | Required alpha token | You want to consume one alpha-like token without capture |
| `<!#!>` | Required numeric token | You want to consume one numeric token without capture |
| `<!@%?!>` | Optional alpha/extended token | You want the same type-modifier behavior as `<<FIELD@%?>>` without capture |
| `<!DROP@!>` | Required alpha token | You want a readable ignored label while dropping output |
| `<!DROP::STREETTYPE!>` | Required `STREETTYPE` class | You want capture-style explicit class syntax without output |
| `<!::STREETTYPE!>` | Required `STREETTYPE` class | You want anonymous explicit class syntax |
| `<!@[ALPHA|ALPHA_EXTENDED]!>` | Alpha class filter | You want capture-style class filtering without output |
| `<!+!>` | One or more word-like tokens | You want an anonymous multi-token skip without output |
| `<!,!>` | Required literal `,` | The model leaves comma as a literal token |
| `<!,?!>` | Optional literal `,` | The comma may or may not be present as literal text |
| `<!,{{?}}?!>` | Optional literal `,?` | The stream contains literal punctuation and `?` must be escaped |
| `<!-!>` | Required literal `-` | The model leaves `-` literal in the class stream |
| `<!-?!>` | Optional literal `-` | The literal dash may or may not be present |
| `<!{{-}}?!>` | Optional literal `-` | Backward-compatible literal-block spelling; not required for `-` |
| `<!{{#}}?!>` | Optional literal `#` | The model leaves a pound marker literal |
| `<!{{1-1}}?!>` | Optional literal `1-1` | The model leaves the whole sequence literal |
Do not mix the domains. `<!,?!>` does not consume a `COMMA` class,
`<!-?!>` does not consume a `DASH` class, and `<!{{1-1}}?!>` does not consume a
stream such as `NUM DASH NUM`. If `-` is part of `NUM_EXTENDED`,
`ALPHA_EXTENDED`, or `ALPHA_NUM_EXTENDED`, it is not a standalone class-stream
token and cannot be consumed by any vanishing literal. Capture the extended
token instead.
Because `@`, `#`, `%`, and `=` are type modifiers in vanishing groups, use
literal blocks to consume those characters literally:
```text
<!@!> # vanish one alpha token
<!{{@}}!> # vanish a literal @ token
<!#!> # vanish one numeric token
<!{{#}}!> # vanish a literal # token
```
## Word definition and model boundaries
The model's word definition decides where token boundaries occur before TEL
runs. By default, `tokmat` treats word characters, hyphen, and apostrophe as
word-like:
```text
\w\-'
```
A model can override this with `WORDDEFINITION.param`. That setting changes
what can appear in the class stream, which in turn changes which TEL rule is
appropriate.
```text
Default-style model:
Raw text: 11-47 OAK ST
Tokens: ["11-47", " ", "OAK", " ", "ST"]
Class stream: ["NUM_EXTENDED", " ", "ALPHA", " ", "STREETTYPE"]
No-hyphen word definition:
Raw text: 11-47 OAK ST
Tokens: ["11", "-", "47", " ", "OAK", " ", "ST"]
Class stream: ["NUM", "-", "NUM", " ", "ALPHA", " ", "STREETTYPE"]
No-hyphen model with DASH class:
Raw text: 11-47 OAK ST
Tokens: ["11", "-", "47", " ", "OAK", " ", "ST"]
Class stream: ["NUM", "DASH", "NUM", " ", "ALPHA", " ", "STREETTYPE"]
```
The TEL rule should match the stream you actually have:
```text
NUM_EXTENDED stream -> <<CIVIC::NUM_EXTENDED>> or class/filter logic
NUM - NUM stream -> <<UNIT::NUM>> <!{{-}}!> <<CIVIC::NUM>>
NUM DASH NUM stream -> <<UNIT::NUM>> <!DASH!> <<CIVIC::NUM>>
```
## Examples
### Street address
```text
<<CIVIC#>> <<STREET@+>> <<TYPE::STREETTYPE>>
```
Possible match:
```text
123 MAIN ST
```
Result:
```text
CIVIC = 123
STREET = MAIN
TYPE = ST
```
### PO Box
```text
{{PO BOX}} <<BOXNUM#>>
```
Possible match:
```text
PO BOX 99
```
Result:
```text
BOXNUM = 99
```
### City, province, postal code
```text
<<CITY@+$>> <<PROV::PROV>> <<PC::PCODE>>
```
Possible match:
```text
OTTAWA ON K1A0B1
```
Result:
```text
CITY = OTTAWA
PROV = ON
PC = K1A0B1
```
## Two-phase example
### Input
```text
123 MAIN ST
```
### Tokenization
```text
["123", " ", "MAIN", " ", "ST"]
```
### Type/class assignment
```text
["NUM", " ", "ALPHA", " ", "STREETTYPE"]
```
### TEL rule
```text
<<CIVIC#>> <<NAME@+>> <<TYPE::STREETTYPE>>
```
### Why this is metadata-driven
The last token is accepted because it belongs to the `STREETTYPE` class, not because the parser
has a hard-coded special case for the literal string `ST`.
That distinction is what makes TEL reusable across different token models.
## Practical guidance
- Use literal blocks for true fixed phrases.
- Use explicit semantic classes when the token meaning matters more than the literal spelling.
- Use class vanishing (`<!DASH?!>`) when a model emits a named class.
- Use literal vanishing (`<!-?!>`, `<!{{#}}?!>`) when a model leaves literal class-stream text.
- Use `+` for multi-token street or city names.
- Keep TEL focused on extraction semantics and let tokenization solve boundary issues first.
- Check the model word definition before deciding whether a hyphenated shape is one token, split
literal punctuation, or split named classes.
- If you reuse the same TEL rules at high volume, prefer compiling them once and reusing the
compiled form rather than recompiling under cache pressure.
## Source of truth
For the machine-oriented grammar, see `grammar/tel.ebnf`.
For executable edge cases, see the unit tests in `src/tel.rs` and `src/extractor.rs` covering
optional vanishing classes, optional vanishing literals, and model-dependent dash behavior.