Struct RemoteTokenizer 

pub struct RemoteTokenizer { /* private fields */ }

A tokenizer a head without the model takes from a node serving it: what it says without encoding is the node’s own, kept here, and text is encoded by a node serving the model, once per request rather than once per token. Text it encoded before is answered here: a conversation sends its system prompt and earlier turns again with every turn.

Implementations

impl RemoteTokenizer

pub fn new( fleet: Arc<FleetManager>, model: String, export: TokenizerExportMsg, ) -> RemoteTokenizer

pub fn digest(&self) -> u64

The node’s tokenizer digest, which every node serving the model is held to.

Trait Implementations

impl Tokenizer for RemoteTokenizer

fn encode(&self, text: &str) -> Vec<u32>

fn token_bytes(&self, token: u32) -> Vec<u8>

fn vocab_size(&self) -> u32

fn special_tokens(&self) -> Vec<(String, u32)>

fn bos_token(&self) -> Option<u32>

fn eos_token(&self) -> u32

fn chat_template_jinja(&self) -> String

fn chat_template_named(&self, name: &str) -> Option<String>

fn encode_plain(&self, text: &str) -> Option<Vec<u32>>

fn encode_pieces(&self, pieces: &[(&str, bool)]) -> Option<Vec<u32>>