Skip to main content

CHARS_PER_TOKEN

Constant CHARS_PER_TOKEN 

Source
pub const CHARS_PER_TOKEN: usize = 4;
Expand description

Characters per token wherever borg estimates a token count: a transcript’s tool results here, a lesson line’s cost (borg whole.rs), the cost lines of borg status. Method: the usual rule of thumb for English text and code under today’s BPE tokenizers; an estimate, never a bound.