Struct SimpleWhitespaceTokenizer 

pub struct SimpleWhitespaceTokenizer { /* private fields */ }

A vocabulary-aware whitespace tokenizer.

Splits text on whitespace and maps each word to an integer token ID. Unknown words map to a reserved UNK ID.

Example

use scirs2_text::tokenizer::{SimpleWhitespaceTokenizer, TransformerTokenizer};

let texts = &["hello world", "hello there"];
let tok = SimpleWhitespaceTokenizer::build(texts, 100);
let ids = tok.encode("hello world");
let decoded = tok.decode(&ids);
assert_eq!(decoded, "hello world");

Implementations

impl SimpleWhitespaceTokenizer

pub fn build(texts: &[&str], max_vocab: usize) -> Self

Build a whitespace tokenizer from training texts.

Counts word frequencies and keeps the top max_vocab words. Token ID 0 is reserved for [UNK].

pub fn from_vocab(vocab: HashMap<String, u32>, unk_id: u32) -> Self

Create from an existing vocabulary.

Trait Implementations

impl Clone for SimpleWhitespaceTokenizer

fn clone(&self) -> SimpleWhitespaceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SimpleWhitespaceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TransformerTokenizer for SimpleWhitespaceTokenizer

fn encode(&self, text: &str) -> Vec<u32>

Encode text into a sequence of token IDs.

fn decode(&self, ids: &[u32]) -> String

Decode a sequence of token IDs back into text.

fn vocab_size(&self) -> usize

Return the vocabulary size (number of distinct token IDs).