Module token 

Safe wrappers around llama_token_data and llama_token_data_array.

Modules

data

Safe wrapper around llama_token_data.

data_array

an rusty equivalent of llama_token_data_array.

Structs

LlamaToken

A safe wrapper for llama_token.

Functions

from_vec_token_sys

Converts a vector of llama_token to a vector of LlamaToken without memory allocation, and consumes the original vector. This conversion is safe because LlamaToken is repr(transparent), meaning it is just a wrapper around the raw llama_token type.

to_vec_token_sys

Converts a vector of LlamaToken to a vector of llama_token without memory allocation, and consumes the original vector. This conversion is safe because LlamaToken is repr(transparent), meaning it is just a wrapper around the raw llama_token type.