Struct CommitMessageProcessor 

pub struct CommitMessageProcessor { /* private fields */ }

Commit message preprocessor that applies NLP transformations.

This processor applies a standard NLP pipeline:

1. Tokenization (word-level with punctuation handling)
2. Lowercasing
3. Stop words filtering (with custom software engineering stop words)
4. Stemming (Porter stemmer)

Examples

use organizational_intelligence_plugin::nlp::CommitMessageProcessor;

let processor = CommitMessageProcessor::new();
let message = "fix: race condition in mutex lock";
let tokens = processor.preprocess(message).unwrap();
assert!(tokens.contains(&"race".to_string()));
assert!(tokens.contains(&"condit".to_string())); // Stemmed

Implementations

impl CommitMessageProcessor

pub fn new() -> Self

Create a new commit message processor with default settings.

Uses:

• WordTokenizer for tokenization
• English stop words with custom software engineering adjustments
• Porter stemmer for normalization

Examples

use organizational_intelligence_plugin::nlp::CommitMessageProcessor;

let processor = CommitMessageProcessor::new();

pub fn with_custom_stop_words<I, S>(custom_stop_words: I) -> Self

where I: IntoIterator<Item = S>, S: AsRef<str>,

Create a processor with custom stop words.

Useful for domain-specific filtering (e.g., transpiler development).

Arguments

• custom_stop_words - Additional stop words to filter

Examples

use organizational_intelligence_plugin::nlp::CommitMessageProcessor;

let processor = CommitMessageProcessor::with_custom_stop_words(vec!["depyler", "internal"]);

pub fn preprocess(&self, message: &str) -> Result<Vec<String>>

Preprocess a commit message into normalized tokens.

Pipeline:

1. Tokenize into words
2. Lowercase
3. Filter stop words
4. Stem to root forms

Arguments

• message - Raw commit message

Returns

• Ok(Vec<String>) - Normalized tokens
• Err - If preprocessing fails

Examples

use organizational_intelligence_plugin::nlp::CommitMessageProcessor;

let processor = CommitMessageProcessor::new();
let tokens = processor.preprocess("fix: memory leak in parser").unwrap();
assert!(tokens.contains(&"memori".to_string())); // Stemmed "memory"
assert!(tokens.contains(&"leak".to_string()));
assert!(tokens.len() >= 2); // At least "memori" and "leak"

pub fn extract_ngrams(&self, tokens: &[String], n: usize) -> Result<Vec<String>>

Extract n-grams from a list of tokens.

N-grams are contiguous sequences of n tokens. Useful for detecting multi-word patterns like “null pointer” or “race condition”.

Arguments

• tokens - Input tokens
• n - Size of n-grams (1 = unigrams, 2 = bigrams, 3 = trigrams)

Returns

• Ok(Vec<String>) - N-grams joined with underscores
• Err - If n is 0 or greater than token count

Examples

use organizational_intelligence_plugin::nlp::CommitMessageProcessor;

let processor = CommitMessageProcessor::new();
let tokens: Vec<String> = vec![
    "fix".to_string(),
    "race".to_string(),
    "condition".to_string(),
    "mutex".to_string(),
];
let bigrams = processor.extract_ngrams(&tokens, 2).unwrap();
assert!(bigrams.contains(&"fix_race".to_string()));
assert!(bigrams.contains(&"race_condition".to_string()));

pub fn preprocess_with_ngrams( &self, message: &str, ) -> Result<(Vec<String>, Vec<String>)>

Preprocess and extract both unigrams and bigrams.

Convenience method that combines preprocessing with n-gram extraction. Useful for feature extraction in ML models.

Arguments

• message - Raw commit message

Returns

• Ok((Vec<String>, Vec<String>)) - (unigrams, bigrams)
• Err - If preprocessing fails

Examples

use organizational_intelligence_plugin::nlp::CommitMessageProcessor;

let processor = CommitMessageProcessor::new();
let (unigrams, bigrams) = processor.preprocess_with_ngrams("fix: memory leak defect").unwrap();
assert!(unigrams.contains(&"memori".to_string())); // Stemmed "memory"
assert!(unigrams.contains(&"leak".to_string()));
assert!(!bigrams.is_empty()); // Should have bigrams

Trait Implementations

impl Clone for CommitMessageProcessor

fn clone(&self) -> CommitMessageProcessor

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CommitMessageProcessor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CommitMessageProcessor

fn default() -> Self

Returns the “default value” for a type.