syntaxdot-tokenizers-0.5.0
- syntaxdot-tokenizers 0.5.0
- Permalink
- Docs.rs crate page
- MIT OR Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- danieldk
- Dependencies
- ndarray ^0.15 normal
  
  sentencepiece ^0.11 normal
  
  thiserror ^1 normal
  
  udgraph ^0.8 normal
  
  wordpieces ^0.5 normal
- Versions
- 56.25% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

syntaxdot_tokenizers0.5.0

Crate Items

Structs
Enums
Traits

List of all items

Structs

AlbertTokenizer
BertTokenizer
SentenceWithPieces
XlmRobertaTokenizer

Enums

TokenizerError

Traits

Tokenize