eld_llm 0.0.1

An LLM built from scratch in Rust
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# BPE Public Benchmarks

## Compression ratio (chars / tokens)

| Model | WikiText-2 | Penn Treebank | BooksCorpus |
|---|---|---|---|
| GPT-2 (reported) | — | — | — |
| tiktoken (GPT-2 vocab) | — | — | — |
| Your BPE v1 | — | — | — |
| Your BPE v2 | — | — | — |

## Fertility score (avg tokens per word)

| Model | WikiText-2 | Penn Treebank | BooksCorpus |
|---|---|---|---|
| GPT-2 (reported) | — | — | — |
| tiktoken (GPT-2 vocab) | — | — | — |
| Your BPE v1 | — | — | — |
| Your BPE v2 | — | — | — |

## OOV rate (% unknown tokens)

| Model | WikiText-2 | Penn Treebank | BooksCorpus |
|---|---|---|---|
| GPT-2 (reported) | — | — | — |
| tiktoken (GPT-2 vocab) | — | — | — |
| Your BPE v1 | — | — | — |
| Your BPE v2 | — | — | — |

## Encode throughput (tokens / sec)

| Model | WikiText-2 | Penn Treebank | BooksCorpus |
|---|---|---|---|
| GPT-2 (reported) | — | — | — |
| tiktoken (GPT-2 vocab) | — | — | — |
| Your BPE v1 | — | — | — |
| Your BPE v2 | — | — | — |