pub struct Bpe { /* private fields */ }Expand description
A byte-level BPE tokenizer.
Implementations§
Source§impl Bpe
impl Bpe
Sourcepub fn from_json(text: &str) -> Result<Self, NetworkError>
pub fn from_json(text: &str) -> Result<Self, NetworkError>
The same, from the file’s contents.
Sourcepub fn vocab_size(&self) -> usize
pub fn vocab_size(&self) -> usize
How many ids the vocabulary holds.
Sourcepub fn token_id(&self, token: &str) -> Option<u32>
pub fn token_id(&self, token: &str) -> Option<u32>
The id of a token written out in full, which is how a caller finds a
special token such as <|endoftext|>.
Sourcepub fn add_token(
&mut self,
token: &str,
count: usize,
) -> Result<u32, NetworkError>
pub fn add_token( &mut self, token: &str, count: usize, ) -> Result<u32, NetworkError>
Adds token to the vocabulary as a token matched whole, standing for
count ids in a row, and returns the first id. The ids follow the
vocabulary, which is where a textual inversion’s rows go in the
embedding table.
Spelled as diffusers does: the extra ids are token_1, token_2 and
so on, and writing token in a prompt stands for all of them.
Source§impl Bpe
impl Bpe
Sourcepub fn train_json<S: AsRef<str>>(
texts: impl IntoIterator<Item = S>,
vocab_size: usize,
specials: &[&str],
) -> Result<String, NetworkError>
pub fn train_json<S: AsRef<str>>( texts: impl IntoIterator<Item = S>, vocab_size: usize, specials: &[&str], ) -> Result<String, NetworkError>
Trains a byte-level BPE tokenizer on texts and returns it as a
tokenizer.json in the Hugging Face format, which Bpe::from_json
reads.
The vocabulary is specials first, then the 256 byte tokens, then
one token per merge until it holds vocab_size or no pair occurs
twice. Words are split with the GPT-2 rule, and a special token in the
text is a boundary that no merge crosses, as it is when encoding.
Ties between equally frequent pairs go to the lower pair of ids, so
the same corpus always trains the same file.
let corpus = ["the cat sat on the mat", "the dog sat on the log"];
let json = Bpe::train_json(corpus, 300, &["<|endoftext|>"])?;
let bpe = Bpe::from_json(&json)?;
let ids = bpe.encode("the cat<|endoftext|>")?;
assert_eq!(bpe.decode(&ids), "the cat<|endoftext|>");Sourcepub fn train_json_split_digits<S: AsRef<str>>(
texts: impl IntoIterator<Item = S>,
vocab_size: usize,
specials: &[&str],
) -> Result<String, NetworkError>
pub fn train_json_split_digits<S: AsRef<str>>( texts: impl IntoIterator<Item = S>, vocab_size: usize, specials: &[&str], ) -> Result<String, NetworkError>
Bpe::train_json with Qwen2’s split instead of GPT-2’s: every digit
is a word of its own, so no token spans two digits and numbers are
read one digit at a time.
The file carries the rule as a Split pre-tokenizer, so
Bpe::from_json and Hugging Face tokenizers both encode with it.
let corpus = ["port 8080", "port 8080", "port 8080"];
let bpe = Bpe::from_json(&Bpe::train_json_split_digits(corpus, 300, &[])?)?;
assert_eq!(bpe.encode(" 8080")?.len(), 5);Examples found in repository?
2fn main() {
3 let mut args = std::env::args().skip(1);
4 let path = args.next().unwrap();
5 let megabytes: usize = args.next().unwrap().parse().unwrap();
6 let mut bytes = Vec::new();
7 std::fs::File::open(&path).unwrap().take((megabytes << 20) as u64).read_to_end(&mut bytes).unwrap();
8 let text = String::from_utf8_lossy(&bytes);
9 let lines: Vec<&str> = text.split_inclusive('\n').collect();
10 let start = std::time::Instant::now();
11 let json = rusting_brain::Bpe::train_json_split_digits(&lines, 32_000, &["<|endoftext|>"]).unwrap();
12 println!("{megabytes} MB, {} lines: {:.1} s, json {} bytes", lines.len(), start.elapsed().as_secs_f64(), json.len());
13}Trait Implementations§
Auto Trait Implementations§
impl Freeze for Bpe
impl RefUnwindSafe for Bpe
impl Send for Bpe
impl Sync for Bpe
impl Unpin for Bpe
impl UnsafeUnpin for Bpe
impl UnwindSafe for Bpe
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more