Struct TokenizerConfig 

#[non_exhaustive]
pub struct TokenizerConfig {
    pub add_bos: bool,
    pub add_eos: bool,
    pub bos_token_id: u32,
    pub eos_token_id: u32,
    pub unk_token_id: u32,
    pub pad_token_id: u32,
    pub max_length: Option<usize>,
    pub byte_level_decode: bool,
}

Configuration knobs for an OxiTokenizer.

Marked #[non_exhaustive] so that new optional knobs can be added in future minor releases without breaking downstream code. Inside this crate struct literals with ..Default::default() continue to work.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

add_bos: bool

Whether to prepend a BOS (beginning-of-sequence) token.

add_eos: bool

Whether to append an EOS (end-of-sequence) token.

bos_token_id: u32

Token ID used for BOS.

eos_token_id: u32

Token ID used for EOS.

unk_token_id: u32

Token ID used for unknown tokens (fallback).

pad_token_id: u32

Token ID used for padding.

max_length: Option<usize>

Optional maximum output length (tokens are truncated, not padded).

byte_level_decode: bool

When true, the decoder applies the GPT-2 bytes ↔ unicode inverse map to every token string before emitting bytes (see crate::hf_format). When false, the legacy Ġ-stripping path is used (same behaviour as 0.1.x).

from_json_file / OxiTokenizer::from_hf_tokenizer_json set this to true automatically; hand-built configs default to false for backwards compatibility.

Trait Implementations

impl Clone for TokenizerConfig

fn clone(&self) -> TokenizerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerConfig

fn default() -> Self

Returns the “default value” for a type.