compute_tf_df_from_tokenized

Function compute_tf_df_from_tokenized 

Source
pub fn compute_tf_df_from_tokenized(
    tokenized_docs: &[Vec<String>],
    query_token_map: &QueryTokenMap,
) -> TfDfResult
Expand description

Computes term frequencies (TF) for each document, document frequencies (DF) for each term, and document lengths from pre-tokenized content.

Uses the query_token_map to convert string tokens to u8 indices for term frequencies.