ZL_CustomTokenizeState_createAlphabetOutput

openzl_sys

Function ZL_CustomTokenizeState_createAlphabetOutput 

Source 
pub unsafe extern "C" fn ZL_CustomTokenizeState_createAlphabetOutput(
    ctx: *mut ZL_CustomTokenizeState,
    alphabetSize: usize,
) -> *mut c_void
Expand description

Creates the alphabet stream to store the tokenized alphabet. The width of each element in the alphabet must be the same width as the input stream.

@param alphabetSize The exact size of the alphabet.

@returns A pointer to write the alphabet into or NULL on error.