Crate mlmorph 

mlmorph

A Rust implementation of the Malayalam Morphological Analyzer using Finite State Transducer technology.

Overview

mlmorph is a Rust port of the mlmorph Malayalam morphological analyzer and generator. It provides fast and efficient morphological analysis and generation for Malayalam text using Finite State Transducers (FST) built with the Stuttgart Finite State Toolkit (SFST).

This library can:

• Analyze Malayalam words to identify their morphological structure
• Generate word forms from morphological descriptions
• Detect foreign words in Malayalam text
• Normalize Malayalam text using standard transformations

Features

• Fast Performance: Rust implementation provides excellent performance
• Morphological Analysis: Break down Malayalam words into morphemes and POS tags
• Word Generation: Generate word forms from morphological templates
• Foreign Word Detection: Identify non-Malayalam words in text
• Text Normalization: Standardize Malayalam text representations
• CLI Interface: Command-line tool for batch processing
• Library API: Easy-to-use Rust API for integration

Quick Start

Morphological Analysis

use mlmorph::Analyser;

let analyser = Analyser::new()?;

// Analyze a Malayalam word
let results = analyser.analyse("കേരളത്തിന്റെ", true, true)?;

for (analysis, weight) in results {
    println!("Analysis: {} (weight: {})", analysis, weight);
}

Word Generation

use mlmorph::Generator;

let generator = Generator::new()?;

// Generate word forms from morphological description
let results = generator.generate("കേരളം<np><genitive>", true)?;

for (word, weight) in results {
    println!("Generated: {} (weight: {})", word, weight);
}

Foreign Word Detection

use mlmorph::check_foreign_word;

let word = "computer";
let is_foreign = check_foreign_word(word);

if is_foreign == 1 {
    println!("{} is a foreign word", word);
} else {
    println!("{} is a Malayalam word", word);
}

Text Normalization

use mlmorph::normalize;

let text = "ണ്‍";
let normalized = normalize(text);
assert_eq!(normalized, "ൺ");

Performance

The Rust implementation provides significant performance improvements over the Python version:

• Analysis: ~10x faster than Python implementation
• Generation: ~8x faster than Python implementation
• Memory: Lower memory footprint
• Concurrency: Safe for concurrent use across threads

Requirements

• SFST Data: The library requires the compiled Malayalam FST file (data/malayalam.a)
• Rust: Version 1.70 or higher

Re-exports

pub use analyser::Analyser;

pub use foreign_word_detector::check_foreign_word;

pub use generator::Generator;

pub use normalizer::normalize;

Modules

analyser

foreign_word_detector

generator

normalizer

Structs

Morpheme

Represents a single morpheme in the morphological analysis.

ParsedAnalysis

Represents the complete parsed analysis of a word.

Functions

create_sfst

Type Aliases

AnalysisResult

Result type for morphological analysis operations.

GenerationResult

Result type for word generation operations.