Skip to main content

Bpe

Struct Bpe 

Source
pub struct Bpe { /* private fields */ }
Expand description

A byte-level BPE tokenizer.

Implementations§

Source§

impl Bpe

Source

pub fn from_file(path: impl AsRef<Path>) -> Result<Self, NetworkError>

Reads a tokenizer.json.

Source

pub fn from_json(text: &str) -> Result<Self, NetworkError>

The same, from the file’s contents.

Source

pub fn vocab_size(&self) -> usize

How many ids the vocabulary holds.

Source

pub fn token_id(&self, token: &str) -> Option<u32>

The id of a token written out in full, which is how a caller finds a special token such as <|endoftext|>.

Source

pub fn add_token( &mut self, token: &str, count: usize, ) -> Result<u32, NetworkError>

Adds token to the vocabulary as a token matched whole, standing for count ids in a row, and returns the first id. The ids follow the vocabulary, which is where a textual inversion’s rows go in the embedding table.

Spelled as diffusers does: the extra ids are token_1, token_2 and so on, and writing token in a prompt stands for all of them.

Source

pub fn encode(&self, text: &str) -> Result<Vec<u32>, NetworkError>

Tokenizes text.

Source

pub fn decode(&self, ids: &[u32]) -> String

Turns ids back into text, which is what a round-trip check needs.

Source§

impl Bpe

Source

pub fn train_json<S: AsRef<str>>( texts: impl IntoIterator<Item = S>, vocab_size: usize, specials: &[&str], ) -> Result<String, NetworkError>

Trains a byte-level BPE tokenizer on texts and returns it as a tokenizer.json in the Hugging Face format, which Bpe::from_json reads.

The vocabulary is specials first, then the 256 byte tokens, then one token per merge until it holds vocab_size or no pair occurs twice. Words are split with the GPT-2 rule, and a special token in the text is a boundary that no merge crosses, as it is when encoding. Ties between equally frequent pairs go to the lower pair of ids, so the same corpus always trains the same file.

let corpus = ["the cat sat on the mat", "the dog sat on the log"];
let json = Bpe::train_json(corpus, 300, &["<|endoftext|>"])?;
let bpe = Bpe::from_json(&json)?;
let ids = bpe.encode("the cat<|endoftext|>")?;
assert_eq!(bpe.decode(&ids), "the cat<|endoftext|>");
Source

pub fn train_json_split_digits<S: AsRef<str>>( texts: impl IntoIterator<Item = S>, vocab_size: usize, specials: &[&str], ) -> Result<String, NetworkError>

Bpe::train_json with Qwen2’s split instead of GPT-2’s: every digit is a word of its own, so no token spans two digits and numbers are read one digit at a time.

The file carries the rule as a Split pre-tokenizer, so Bpe::from_json and Hugging Face tokenizers both encode with it.

let corpus = ["port 8080", "port 8080", "port 8080"];
let bpe = Bpe::from_json(&Bpe::train_json_split_digits(corpus, 300, &[])?)?;
assert_eq!(bpe.encode(" 8080")?.len(), 5);
Examples found in repository?
examples/bench_bpe_train.rs (line 11)
2fn main() {
3    let mut args = std::env::args().skip(1);
4    let path = args.next().unwrap();
5    let megabytes: usize = args.next().unwrap().parse().unwrap();
6    let mut bytes = Vec::new();
7    std::fs::File::open(&path).unwrap().take((megabytes << 20) as u64).read_to_end(&mut bytes).unwrap();
8    let text = String::from_utf8_lossy(&bytes);
9    let lines: Vec<&str> = text.split_inclusive('\n').collect();
10    let start = std::time::Instant::now();
11    let json = rusting_brain::Bpe::train_json_split_digits(&lines, 32_000, &["<|endoftext|>"]).unwrap();
12    println!("{megabytes} MB, {} lines: {:.1} s, json {} bytes", lines.len(), start.elapsed().as_secs_f64(), json.len());
13}

Trait Implementations§

Source§

impl Clone for Bpe

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for Bpe

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Auto Trait Implementations§

§

impl Freeze for Bpe

§

impl RefUnwindSafe for Bpe

§

impl Send for Bpe

§

impl Sync for Bpe

§

impl Unpin for Bpe

§

impl UnsafeUnpin for Bpe

§

impl UnwindSafe for Bpe

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> Pointable for T

Source§

const ALIGN: usize

The alignment of pointer.
Source§

type Init = T

The type for initializers.
Source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
Source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
Source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
Source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V