Struct SimpleTokenizerV1 

pub struct SimpleTokenizerV1 { /* private fields */ }

[Listing 2.3] Implementing a simple text tokenizer

Implementations

impl SimpleTokenizerV1

pub fn from_vocab(vocab: HashMap<&str, i32>) -> Self

Creates a new SimpleTokenizerV1 from a vocab.

use llms_from_scratch_rs::listings::ch02::SimpleTokenizerV1;
use std::collections::HashMap;

let vocab: HashMap<&str, i32> = HashMap::from([
    ("this", 1_i32),
    ("is", 2_i32),
    ("a", 3_i32),
    ("test", 4_i32)
]);
let tokenizer = SimpleTokenizerV1::from_vocab(vocab);

pub fn encode(&self, text: &str) -> Vec<i32>

Encode a text into its token ids.

pub fn decode(&self, ids: Vec<i32>) -> String

Decode token ids into its text.

Trait Implementations

impl Debug for SimpleTokenizerV1

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SimpleTokenizerV1

fn default() -> SimpleTokenizerV1

Returns the “default value” for a type.