Crate token_align 

Maps overlaps between token sequences and between character positions.

The default comparison form removes common subword markers and applies pinned Unicode compatibility decomposition. Use Exact or implement Normalizer when the input needs different comparison rules.

Structs

Exact

A comparison form that copies each token unchanged.

Fold

The default marker removal and Unicode comparison form.

Traits

Normalizer

Produces the comparison form of one token.

Functions

get_alignments

Aligns two token sequences with the default Fold comparison form.

get_alignments_with

Aligns two token sequences with a caller-provided comparison form.

get_charmap

Maps Unicode scalar positions in a to positions in b and back.

get_charmap_with

Maps Unicode scalar positions with a caller-provided comparison form.

Type Aliases

Alignment

One list of counterpart indices for every input token.

CharMap

One list of counterpart indices for every input character.