Skip to main content

tensor_to_tokens

ferrum_models::executor::common

Function tensor_to_tokens 

Source 
pub fn tensor_to_tokens(tensor: &TensorRef) -> Result<Vec<u32>>
Expand description

Extract token IDs from a TensorRef.