Struct SimpleCharTokenizer 

pub struct SimpleCharTokenizer { /* private fields */ }

A character-level tokenizer that maps each unique character to a token ID.

Useful as a baseline or for character-level transformer models.

Example

use scirs2_text::tokenizer::{SimpleCharTokenizer, TransformerTokenizer};

let texts = &["abc", "bcd"];
let tok = SimpleCharTokenizer::build(texts);
let ids = tok.encode("abc");
assert_eq!(ids.len(), 3);
let decoded = tok.decode(&ids);
assert_eq!(decoded, "abc");

Implementations

impl SimpleCharTokenizer

pub fn build(texts: &[&str]) -> Self

Build a character tokenizer from training texts.

Token ID 0 is reserved for unknown characters (those not seen in training).

pub fn from_vocab(vocab: HashMap<char, u32>, unk_id: u32) -> Self

Create from an existing character vocabulary.

Trait Implementations

impl Clone for SimpleCharTokenizer

fn clone(&self) -> SimpleCharTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SimpleCharTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TransformerTokenizer for SimpleCharTokenizer

fn encode(&self, text: &str) -> Vec<u32>

Encode text into a sequence of token IDs.

fn decode(&self, ids: &[u32]) -> String

Decode a sequence of token IDs back into text.

fn vocab_size(&self) -> usize

Return the vocabulary size (number of distinct token IDs).