Docs.rs
syntaxdot-tokenizers-0.5.0
syntaxdot-tokenizers 0.5.0
Docs.rs crate page
MIT OR Apache-2.0
Links
Homepage
Documentation
Repository
crates.io
Source
Owners
danieldk
Dependencies
ndarray ^0.15
normal
sentencepiece ^0.11
normal
thiserror ^1
normal
udgraph ^0.8
normal
wordpieces ^0.5
normal
Versions
85.71%
of the crate is documented
Platform
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
☰
Crate syntaxdot_tokenizers
Version 0.5.0
All Items
Structs
Enums
Traits
?
Crate
syntaxdot_tokenizers
source
·
[
−
]
Structs
AlbertTokenizer
Tokenizer for ALBERT models.
BertTokenizer
BERT word piece tokenizer.
SentenceWithPieces
A sentence and its word pieces.
XlmRobertaTokenizer
Tokenizer for Roberta models.
Enums
TokenizerError
Traits
Tokenize
Trait for wordpiece tokenizers.