Crate rustam 

rustam

Rustam is a full Persian NLP pipeline for Rust: normalization, tokenization, stemming, lemmatization, conjugation, informal normalization, corpus readers, spell correction, and embedding backends.

1. Normalization — character translation, diacritics removal, spacing fixes
2. Informal normalization — colloquial-to-formal Persian conversion
3. Sentence tokenization — boundary detection using punctuation patterns
4. Word tokenization — splits words, joins multi-part verbs
5. Stemming — rule-based suffix stripping
6. Lemmatization — dictionary lookup with full conjugation table
7. Conjugation — generates every tense/mood/voice form for a verb
8. Token splitting — compound token decomposition using the lexicon
9. Corpus readers — Bijankhan, Arman and more
10. Embedding traits — WordEmbedding / SentenceEmbedding interface

Quick-start

use rustam::{Normalizer, SentenceTokenizer, WordTokenizer, Stemmer, Lemmatizer};

let text = "اِعلاممممم کَرد : « زمین لرزه ای به بُزرگیِ 6 دهم ریشتر ...»";

// Normalize
let norm = Normalizer::new();
let normalized = norm.normalize(text);

// Sentence split
let sent_tok = SentenceTokenizer::new();
let sentences = sent_tok.tokenize(&normalized);

// Word tokenize
let word_tok = WordTokenizer::new();
let tokens = word_tok.tokenize(&normalized);

// Stem
let stemmer = Stemmer::new();
assert_eq!(stemmer.stem("کتاب‌ها"), "کتاب");

// Lemmatize
let lem = Lemmatizer::new();
assert_eq!(lem.lemmatize("می‌روم", ""), "رفت#رو");

Re-exports

pub use conjugation::Conjugation;

pub use data::past_roots;

pub use data::present_roots;

pub use data::stopwords_list;

pub use data::verbs_list;

pub use data::words_list;

pub use error::Error;

pub use error::Result;

pub use informal::InformalLemmatizer;

pub use informal::InformalNormalizer;

pub use lemmatizer::Lemmatizer;

pub use normalizer::Normalizer;

pub use normalizer::NormalizerConfig;

pub use sentence_tokenizer::sent_tokenize;

pub use sentence_tokenizer::SentenceTokenizer;

pub use spell::SpellCorrector;

pub use stemmer::Stemmer;

pub use token_splitter::TokenSplitter;

pub use types::ChunkedSentence;

pub use types::ChunkedToken;

pub use types::IobTag;

pub use types::Sentence;

pub use types::Tag;

pub use types::TaggedSentence;

pub use types::TaggedToken;

pub use types::Token;

pub use types::VerbRoots;

pub use types::WordEntry;

pub use word_tokenizer::word_tokenize;

pub use word_tokenizer::WordTokenizer;

pub use word_tokenizer::WordTokenizerConfig;

Modules

conjugation

Persian verb conjugation — generates every tense/mood/voice form.

constants

Compile-time constants: suffix lists, punctuation sets.

data

Embedded data files (lexicon, verbs, stopwords) compiled into the binary.

error

Library-wide error type and Result alias.

informal

Informal (colloquial) Persian normalization and lemmatization. Informal-to-formal Persian normalization.

lemmatizer

Dictionary-based lemmatizer with conjugation fallback.

normalizer

Multi-step text normalizer (diacritics, spacing, Unicode).

sentence_tokenizer

Sentence boundary tokenizer for Persian text.

spell

Persian spell corrector based on lexicon frequency. Persian spell corrector.

stemmer

Rule-based suffix-stripping stemmer.

token_splitter

Compound token splitter using the word lexicon. Splits a compound token into valid sub-word pairs using the lexicon.

translate

Arabic–Persian word translation table.

types

Core type aliases (Token, TaggedSentence, etc.).

word_tokenizer

Word tokenizer with optional verb-part joining.