Function create_ngrams 

pub fn create_ngrams(tokens: Vec<&str>, n: usize) -> HashMap<Vec<&str>, u32>

Creates n-grams from a list of tokens.

Given a list of tokens and the desired size n, this function generates n-grams, which are contiguous sequences of n tokens from the input list.

Arguments

• tokens - A vector of string slices representing individual tokens.
• n - The size of the n-grams to be created.

Returns

A HashMap where keys are n-grams (represented as vectors of string slices) and values are the counts of each n-gram in the input sequence.

Examples

use std::collections::HashMap;
use text_score::rouge::create_ngrams;

let tokens = vec!["this", "is", "an", "example"];
let n = 2;

let ngrams = create_ngrams(tokens, n);

// The result may look like: {"this is": 1, "is an": 1, "an example": 1}

Note

• The function uses a sliding window approach to iterate through the input tokens and create n-grams of the specified size n.
• The resulting n-grams are stored in a HashMap, where each key is an n-gram, and the corresponding value is the count of occurrences of that n-gram in the input sequence.