Skip to main content

bytes_to_unicode

Function bytes_to_unicode 

Source
pub fn bytes_to_unicode() -> (HashMap<u8, char>, HashMap<char, u8>)
Expand description

Create byte to Unicode character mapping.

GPT-2 uses a specific mapping to avoid issues with certain bytes.