Crate tokenizations

Tokenizations alignment functions.

Functions

get_alignments

Returns the tokenizations alignments a2b (from a to b) and b2a (from b to a) based on the shortest edit script (SES).

get_charmap

Returns the character mappings c_a2b (from a to b) and c_b2a (from b to a) based on the shortest edit script (SES).

get_original_spansDeprecated

Type Aliases

Alignment

CharMap