ffbpe 0.1.10

Unicode-aware, streaming BPE training and tiktoken-compatible encoding
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# tiktoken compatibility

FFBPE includes a tiktoken-shaped [`Encoding`][ffbpe.tiktoken_compat.Encoding]
for code that expects common encoding and decoding methods.

```python
from ffbpe import Encoding

encoding = Encoding.from_files(
  "my-model",
  vocab_file="vocab.json",
  merges_file="merges.txt",
  special_tokens={"<|endoftext|>": 0},
)

ids = encoding.encode(
  "hello",
  disallowed_special=(),
)
assert encoding.decode(ids) == "hello"
```

The compatibility layer supports ordinary and batch encoding, special-token
checks, byte decoding, and local model files. It does not bundle OpenAI model
vocabularies or a network registry.

Use [`BpeEncoder`][ffbpe.encoder.BpeEncoder] directly for new FFBPE code unless
an existing integration benefits from the tiktoken-shaped surface.