Trait Normalizer

Source

pub trait Normalizer {
    // Required method
    fn normalize(&self, token: &str, out: &mut String);
}

Expand description

Produces the comparison form of one token.

out is empty when this method starts. Implementations append the complete comparison form and may leave it empty when the token has no counterpart. A caller can also use a closure with the same arguments.

Byte-level token schemes often encode a non-ASCII character as several visible characters. A normalizer for such a scheme can decode the token before appending its comparison form.

§Examples

This normalizer decodes one visible byte-level spelling before alignment.

use token_align::{get_alignments_with, Normalizer};

struct DecodeByteToken;

impl Normalizer for DecodeByteToken {
    fn normalize(&self, token: &str, out: &mut String) {
        if token == "Ã©" {
            out.push('é');
        } else {
            out.push_str(token);
        }
    }
}

let encoded = ["caf", "Ã©"];
let text = ["café"];
assert_eq!(
    get_alignments_with(&encoded, &text, &DecodeByteToken),
    (vec![vec![0], vec![0]], vec![vec![0, 1]])
);