Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Tokenizer for decoding captcha model output.

Uses CTC decoding to decode model logits into text. The model outputs 37 classes: blank (index 0) + 36 alphanumeric characters (0-9, a-z).

Implementations

impl Tokenizer

pub fn new() -> Self

pub const fn charset_len(&self) -> usize

Returns the number of characters in the charset.

pub fn decode<D: Dimension>(&self, logits: &ArrayView<'_, f32, D>) -> String

Decode model output logits to text using CTC greedy decoding. Collapses repeated characters and removes blanks.

Arguments

• logits - ArrayView with shape [batch, sequence_length, num_classes] or [sequence_length, num_classes]

Returns

Decoded text string

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.