Struct NgramModel 

pub struct NgramModel { /* private fields */ }

N-gram language model

Implementations

impl NgramModel

pub fn new(n: usize, smoothing: SmoothingMethod) -> Self

Create a new n-gram model

Arguments

• n - Order of the model (1 for unigram, 2 for bigram, etc.)
• smoothing - Smoothing method to use

pub fn with_tokenizer(self, tokenizer: Box<dyn Tokenizer + Send + Sync>) -> Self

Set a custom tokenizer

pub fn train(&mut self, texts: &[&str]) -> Result<()>

Train the model on a corpus

pub fn probability(&self, context: &[&str], word: &str) -> Result<f64>

Calculate the probability of a word given its context

Arguments

• context - The previous n-1 words
• word - The word to predict

Returns

The probability P(word | context)

pub fn perplexity(&self, texts: &[&str]) -> Result<f64>

Calculate perplexity on a test corpus

Perplexity is a measure of how well the model predicts the test data. Lower perplexity indicates better performance.

pub fn generate( &self, max_length: usize, start_context: Option<&str>, ) -> Result<String>

Generate text using the language model

Arguments

• max_length - Maximum number of words to generate
• start_context - Optional starting context (must have n-1 words)

Returns

Generated text as a string

pub fn suggest_next( &self, context: &[&str], top_n: usize, ) -> Result<Vec<(String, f64)>>

Get the most likely next words given a context

Arguments

• context - The previous n-1 words
• top_n - Number of suggestions to return

Returns

Vector of (word, probability) pairs, sorted by probability (descending)

pub fn order(&self) -> usize

Get the n-gram order

pub fn vocabulary_size(&self) -> usize

Get the vocabulary size

Trait Implementations

impl Clone for NgramModel

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NgramModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.