pub fn from_vec_token_sys(vec_sys: Vec<llama_token>) -> Vec<LlamaToken>Expand description
Converts a vector of llama_token to a vector of LlamaToken without memory allocation,
and consumes the original vector. This conversion is safe because LlamaToken is repr(transparent),
meaning it is just a wrapper around the raw llama_token type.
ยงSafety
This operation is safe because LlamaToken has a repr(transparent) attribute, ensuring that
the memory layout of LlamaToken is the same as that of the underlying llama_token type.