Rishe (ریشه): fast Persian text processing for Python, built in Rust

pip install rishe gives you Persian normalization, tokenization, sentence splitting, lemmatization, ezafe detection and informal → formal conversion from one Rust core: no models to download, prebuilt wheels for Linux, macOS and Windows. The core is also available as a Rust crate.

Normalization is ~70× faster than Hazm and Shekar; tokenization is on par with them. See docs/BENCHMARKS.md for the full numbers, including where Rishe loses.
Pipeline.pipe() processes batches in parallel (rayon) with the GIL released.

Status: alpha (0.1). Rishe is a fast preprocessing layer, not a replacement for Hazm or Shekar: there is no POS tagging, NER, parsing, spell checking or embeddings.

Install

pip install rishe        # Python >= 3.9

or, from Rust (Cargo.toml): rishe = "0.1"

Quick start

from rishe import Pipeline

pipe = Pipeline()
doc = pipe("اون داره تو خونه کتاب میخونه، چیه؟")
doc.tokens       # ['آن', 'دارد', 'تو', 'خانه', 'کتاب', 'می‌خواند', '،', 'چیست', '؟']
doc.lemmas       # ['آن', 'داشت', 'تو', 'خانه', 'کتاب', 'خواند', '،', 'چیست', '؟']
doc.formal_text  # 'آن دارد تو خانه کتاب می‌خواند، چیست؟'
doc.sentences, doc.normalized_text, doc.ezafe_markers

pipe.word_tokenize("سلام، دنیا!")   # ['سلام', '،', 'دنیا', '!']
pipe.lemmatize("می‌روم")             # 'رفت'  (past-stem lemmas, UD style)
pipe.pipe(texts)                     # parallel batch -> list[ProcessedDoc]

use rishe::Pipeline;

let pipe = Pipeline::create(true, true, true); // unify digits, strip diacritics, fix ZWNJ
let doc = pipe.process("کتاب‌های خوب را به دوستانم می‌دهم.");
println!("{:?}", doc.tokens);

Known limitations

Pipeline.__call__ also rewrites colloquial words to formal ones, so doc.tokens can differ from the surface text. Use word_tokenize for plain tokenization.
Informal → formal conversion is lexicon- and rule-based. Common forms (اون, خونه, داره, چیه, میخوام) work; many suffixed or rarer forms (خونه‌ام, کتابه) are left unchanged.
Lemmas follow the Universal Dependencies convention (verbs lemmatize to the past stem). Accuracy on UD is ~0.90 (Seraji) and ~0.88 (PerDT).
Ezafe detection is rule-based and has not been evaluated.

Building from source

cargo test
maturin develop --release     # Python extension into the current venv
python scripts/benchmark.py   # needs: hazm shekar conllu

scripts/extract_lexicons.py regenerates data/*.tsv from Hazm's data files; build.rs compiles them into FSTs.

License