bert_tokenizer-0.1.3
- bert_tokenizer 0.1.3
- Permalink
- Docs.rs crate page
- Links
- Repository
- crates.io
- Source
- Owners
- yigit353
- Dependencies
- indexmap ^1.9.2 normal
  
  unicode-normalization ^0.1.22 normal
  
  unicode_categories ^0.1.1 normal
- Versions
- 32.26% of the crate is documented
Platform
Feature flags

Rust

Crate bert_tokenizer

Structs
Traits
Functions
Type Definitions

List of all items

Structs

BasicTokenizer
BasicTokenizerBuilder
FullTokenizer
FullTokenizerBuilder
WordPieceTokenizer
WordPieceTokenizerBuilder

Traits

Tokenizer

Functions

load_vocab

Type Definitions

Ids
InvVocab
Vocab