Struct ArithmeticEncoder 

pub struct ArithmeticEncoder { /* private fields */ }

Arithmetic encoder for near-optimal compression

Uses adaptive probability models to achieve compression rates close to the theoretical entropy limit.

Implementations

impl ArithmeticEncoder

pub fn new(alphabet_size: usize) -> Self

Create a new arithmetic encoder with uniform initialization

Arguments

• alphabet_size - Number of unique symbols

pub fn from_frequencies(frequencies: HashMap<u32, u64>) -> Self

Create encoder from existing frequencies

pub fn encode( &mut self, symbols: &[u32], adaptive: bool, ) -> TokenizerResult<Vec<u8>>

Encode symbols using arithmetic coding

Arguments

• symbols - Sequence of symbols to encode
• adaptive - Whether to use adaptive frequency updates

Returns

Compressed representation as bytes

pub fn frequencies(&self) -> &HashMap<u32, u64>

Get the codebook for inspection