Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

High-level tokenizer wrapper for DNN usage.

Provides a simple API to encode and decode tokens for LLMs. Models are loaded via Tokenizer::load().

using namespace cv::dnn;
Tokenizer tok = Tokenizer::load("/path/to/model/");
std::vector<int> ids = tok.encode("hello world");
std::string text = tok.decode(ids);

Implementations

impl Tokenizer

pub fn default() -> Result<Tokenizer>

Construct a tokenizer with a given method default BPE. For BPE method you normally call Tokenizer::load() to initialize model data.

pub fn load(model_config: &str) -> Result<Tokenizer>

Load a tokenizer from a model directory.

Expects the directory to contain:

• config.json with field model_type with value “gpt2” or “gpt4”.
• tokenizer.json produced by the corresponding model family.

The argument is a path prefix; this function concatenates file names directly (e.g. model_dir + “config.json”), so model_dir must end with an appropriate path separator.

Parameters

• model_config: Path to config.json for model.

Returns

A Tokenizer ready for use. Throws cv::Exception if files are missing or model_type is unsupported.

Trait Implementations

impl Boxed for Tokenizer

unsafe fn from_raw(ptr: <Tokenizer as OpenCVFromExtern>::ExternReceive) -> Self

Wrap the specified raw pointer

fn into_raw(self) -> <Tokenizer as OpenCVTypeExternContainer>::ExternSendMut

Return the underlying raw pointer while consuming this wrapper.

fn as_raw(&self) -> <Tokenizer as OpenCVTypeExternContainer>::ExternSend

Return the underlying raw pointer.

fn as_raw_mut( &mut self, ) -> <Tokenizer as OpenCVTypeExternContainer>::ExternSendMut

Return the underlying mutable raw pointer

impl Clone for Tokenizer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for Tokenizer

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for Tokenizer

impl TokenizerTrait for Tokenizer

fn as_raw_mut_Tokenizer(&mut self) -> *mut c_void

fn encode(&mut self, text: &str) -> Result<Vector<i32>>

Encode UTF-8 text to token ids (special tokens currently disabled).

fn decode(&mut self, tokens: &Vector<i32>) -> Result<String>

impl TokenizerTraitConst for Tokenizer

fn as_raw_Tokenizer(&self) -> *const c_void