Skip to main content

Module tokenizer

sqlite_graphrag

Module tokenizer 

Source
Expand description

Tokenizer real do modelo de embeddings para contagem e chunking por tokens reais.

Functionsยง

count_passage_tokens
get_model_max_length
get_tokenizer
passage_token_offsets