rustify-ml 0.1.2

�

��i��	
��SrS
SK
Jr SSjrS SjrS!Sjr\S:XG
a3S
SKrSr\	"S	5
 \"\S
S9r
\	"S\"\
5
S
35
 Sr\	"S\S35
 \R"5r\"\5
Hr\"\SS
\
5rM \R"5\-
r\	"S\SS\S35
 \	"S\\-S-SS35
 \	"S\"\5
35
 \"\R)S5
5
r\	"S\S35
 \R"5r\"\5
Hr\"\5
rM
 \R"5\-
r\	"S\"\5
S\SS\S35
 \	"S\"\5
35
 gg)"u_

BPE (Byte-Pair Encoding) tokenizer — rustify-ml acceleration target.

This is a simplified but realistic BPE encode loop similar to what
tiktoken / HuggingFace tokenizers use internally. The inner while-loop
merge pass is O(n²) in the number of tokens and is the primary hotspot.

Expected Rust speedup: 10–50x via Vec<usize> + HashMap lookups.
�)
�annotationsc
�T
�[
URS
5
5
n0n[U
5
H	upEXCU'M SnU(ajSnSnU[U5
S-
:aKX'X'S-4nXS;a"SX5-nX�U'UR	US-5
 SnOUS-
nU[U5
S-
:aMKU(aMjU$)z�Encode text using BPE merge rules.

Args:
    text:   Input string to encode.
    merges: Ordered list of (a, b) merge pairs. Earlier = higher priority.

Returns:
    List of token IDs after all merges are applied.
�utf-8TFr�
�
)�list�encode�	enumerate�len�pop)	�text�merges�tokens�
merge_rank�rank�pair�changed�
i�new_ids	         �@d:\WindsurfProjects\rustify\rustify-ml\examples\bpe_tokenizer.py�
bpe_encoders����$�+�+�g�&�
'�F�.0�J���'�
���4��(��G�
���
�
��#�f�+�
�/�!��I�v�!�e�}�-�D��!��z�/�/��"�q�	��
�
�1�q�5�!����Q��
��#�f�+�
�/�!��'��M�c

��0n
[
[U5
S
-
5
H%nXXS
-4nU
RUS5S
-X'M' U
$)zDCount all adjacent pairs in a token list (used during BPE training).rr)�ranger�get)r�countsrrs    r�count_pairsr3sN��)+�F�
�3�v�;��?�
#�
��	�6�a�%�=�)���z�z�$�
�*�Q�.���$��Mrc
��
^	�[
URS
5
5
n/n[U
5
H�n[U5
m	T	(d U$[	T	U	4
SjS9nURU5
 S[
U5
-S-
nSn/nU[
U5
:aaU[
U5
S-
:a&X'X'S-4U:XaURU5
 US-
nOURX'5
 US-
nU[
U5
:aMaUnM� U$)zBTrain BPE: greedily merge the most frequent pair num_merges times.rc

�>
�T
U$)
N�)�
prs �r�<lambda>�build_vocab.<locals>.<lambda>Es	����
�r)
�keyrrr�)rr	rr�max�appendr)
r
�
num_mergesrr�
_�bestrr�
new_tokensrs
         @r�build_vocabr,<s����
�$�+�+�g�&�
'�F�$&�F�
�:�
�
��V�$���� �M��6�2�3���
�
�d���s�6�{�"�Q�&��
�
� "�
��#�f�+�o��3�v�;��?�"��	�6�a�%�=�'A�T�'I��!�!�&�)��Q��
��!�!�&�)�,��Q��
�
�#�f�+�o���#�&�Mr�__main__Na�the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. the quick brown fox jumps over the lazy dog. z!Training BPE vocab (50 merges)...�2)
r(z
  Learned z merge rulesi'z
Benchmarking bpe_encode xz...z  bpe_encode (50 chars): z.3fzs for z itersz  avg: i�z.4fzms per callz  output tokens: rz
Benchmarking count_pairs xz  count_pairs (z
 tokens): z  unique pairs: )r
�strr�list[tuple[int, int]]�return�	list[int])rr2r1zdict[tuple[int, int], int])r
r/r(�intr1r0)�__doc__�
__future__rrrr,�__name__�time�sample_text�printrr�n_iters�perf_counter�startrr)�result�elapsedrr	r�pairsr rr�<module>r@
s�
�


�
�
#�"
�J

�
�6�z���	=��

�
-�.�
���
4�F�	�J�s�6�{�m�<�
0�1��G�	�'��y��
4�5�����E�
�7�^�
��K���,�f�5������!�E�)�G�	�%�g�c�]�&��	��
H�I�	�G�G�g�%��,�S�1��
=�>�	��c�&�k�]�
+�,��+�$�$�W�-�
.�F�	�(��	��
5�6�����E�
�7�^�
��F�#������!�E�)�G�	�O�C��K�=�
�7�3�-�v�g�Y�f�
U�V�	��S��Z�L�
)�*�A
r