Skip to main content

BGEM3_VOCAB_SIZE

llm_kernel::embedding::bgem3

Constant BGEM3_VOCAB_SIZE 

Source 
pub const BGEM3_VOCAB_SIZE: usize = 250_002;
Expand description

Vocabulary size of BGE-M3’s XLM-RoBERTa tokenizer — the dimension a pgvector sparsevec column must declare for these sparse vectors.