pub struct HfTokenizer { /* private fields */ }Expand description
Adapter over a locally configured Hugging Face tokenizer.
Implementations§
Source§impl HfTokenizer
impl HfTokenizer
Sourcepub fn new(
inner: Tokenizer,
truncation: Truncation,
add_special_tokens: bool,
limits: ResourceLimits,
) -> Result<Self>
pub fn new( inner: Tokenizer, truncation: Truncation, add_special_tokens: bool, limits: ResourceLimits, ) -> Result<Self>
Creates an adapter, disabling native padding/truncation so the specified RustTorch policies are the only length transformations.
Sourcepub fn from_file(
path: impl AsRef<Path>,
truncation: Truncation,
add_special_tokens: bool,
limits: ResourceLimits,
) -> Result<Self>
pub fn from_file( path: impl AsRef<Path>, truncation: Truncation, add_special_tokens: bool, limits: ResourceLimits, ) -> Result<Self>
Loads bounded local tokenizer JSON. No model or vocabulary is downloaded.
Sourcepub fn inner(&self) -> &Tokenizer
pub fn inner(&self) -> &Tokenizer
Read-only access to vocabulary and advanced tokenizer configuration.
Sourcepub fn vocabulary_size(&self) -> usize
pub fn vocabulary_size(&self) -> usize
Number of vocabulary entries, including added tokens.
Sourcepub fn encode(&self, text: &str) -> Result<EncodedSequence>
pub fn encode(&self, text: &str) -> Result<EncodedSequence>
Encodes one bounded UTF-8 sample with deterministic length policy.
Trait Implementations§
Auto Trait Implementations§
impl !Freeze for HfTokenizer
impl Forget for HfTokenizer
impl RefUnwindSafe for HfTokenizer
impl Send for HfTokenizer
impl Sync for HfTokenizer
impl Unpin for HfTokenizer
impl UnsafeUnpin for HfTokenizer
impl UnwindSafe for HfTokenizer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more