Struct RbTokenizerBuilder 

pub struct RbTokenizerBuilder { /* private fields */ }

Builder for creating a Tokenizer with custom configuration.

The builder pattern allows for fluent configuration of tokenizer parameters. Uses RefCell for interior mutability since Magnus method! requires &self.

Trait Implementations

impl DataTypeFunctions for RbTokenizerBuilder

fn free(self: Box<Self>)

Called when the Ruby wrapper object is garbage collected.

fn mark(&self, marker: &Marker)

Called when Ruby marks this object as part of garbage collection.

fn size(&self) -> usize

Called by Ruby to establish the memory size of this data, to optimise when garbage collection happens.

fn compact(&self, compactor: &Compactor)

Called during garbage collection.

impl TypedData for RbTokenizerBuilder

fn class(ruby: &Ruby) -> RClass

Should return the class for the Ruby object wrapping the Rust type.

fn data_type() -> &'static DataType

Should return a static reference to a DataType with metadata about the wrapped type.

fn class_for(ruby: &Ruby, value: &Self) -> RClass

Used to customise the class wrapping a specific value of Self.