ffbpe 0.1.9

Unicode-aware, streaming BPE training and tiktoken-compatible encoding
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# Pretokenizer and counters

## Pretokenizer

::: ffbpe.pretokenizer.PreTokenizer

## Unicode bigram counter

::: ffbpe._lib.BigramCounter

## Unicode bigram selection

::: ffbpe._lib.UnicodeBigramSelection

## Word counter

::: ffbpe._lib.WordCounter