token-align 0.1.0

Maps overlapping tokens and character positions between tokenized texts
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
# Changelog

## 0.1.0 - 2026-08-22

### Added

- Token overlap maps with inverse results for both input sequences.
- Character-position maps based on Unicode scalar indices.
- Default marker removal and Unicode 17.0.0 comparison rules.
- Pinned generated Unicode tables with the required Unicode License v3 notice.
- Caller-defined comparison forms through the `Normalizer` trait.
- Support for allocator-backed `no_std` programs.