# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Goal
`rustam` is a full Persian NLP pipeline for Rust: normalization, tokenization, stemming, lemmatization, conjugation, corpus readers, and pluggable embedding/tagging interfaces.
## Commands
All commands run from this directory. There is no Cargo workspace — each sibling crate is independent.
```sh
cargo build # compile
cargo test # all tests
cargo test <test_name> # single test (e.g. cargo test stemmer_strips_ha_plural)
cargo test -- --nocapture # show println! output
cargo run --example pipeline # run the pipeline demo
cargo clippy -- -D warnings # lint
cargo doc --open # build & open docs
```
For sibling crates (`crfrs`, `crftag`, `nerrs`, `arc-eager`), run the same commands from their respective directories under `C:\Users\0xKrov\phd\`.
## Multi-crate Ecosystem
The project is split across five independent crates (no workspace):
| `rustam` | Core NLP pipeline — this repo |
| `crfrs` | Generic structured-perceptron CRF + Viterbi — the ML backbone |
| `crftag` | CRF-based POS tagger and chunker, built on `crfrs` |
| `nerrs` | CRF-based NER (BIO tagset), built on `crfrs` |
| `arc-eager` | Arc-eager transition-based dependency parser |
`crftag`, `nerrs`, and `arc-eager` appear as `[dev-dependencies]` in `Cargo.toml` so integration tests can cover the full pipeline. They are **not** runtime dependencies of `rustam` itself.
## Architecture
### Data embedding (no runtime I/O for core features)
All lexicon files (`data/*.dat`) are compiled into the binary via `include_str!` in `src/data.rs`. Parsers in `data.rs` turn them into `HashMap<String, WordEntry>` (words), `Vec<VerbRoots>` (verbs), and plain `Vec<String>` (stopwords/abbreviations). Corpus readers (`src/corpus/`) are the exception — they read user-supplied files from disk at runtime.
### Public trait interfaces (`src/api.rs`, `src/embedding.rs`, `src/corpus/mod.rs`)
The library exposes four NLP-pipeline traits:
- `Normalizer` — `normalize(&self, text: &str) -> String`
- `Tokenizer` — `tokenize(&self, text: &str) -> Vec<Token>`
- `Lemmatizer` — `lemmatize(&self, word: &str, pos: &str) -> String`
- `Tagger` — `tag(&self, tokens: &Sentence) -> TaggedSentence`
And two embedding traits in `src/embedding.rs`:
- `WordEmbedding` — `embed_word`, `similarity`, `most_similar`, `load`
- `SentenceEmbedding` — `embed`, `embed_str`, `similarity_str`, `load`
And `CorpusReader` in `src/corpus/mod.rs` — `sents()`, `words()`, `tagged_words()`.
**The concrete structs (`Normalizer`, `WordTokenizer`, `Lemmatizer`, etc.) shadow the trait names at the crate root.** The traits are re-exported with aliases: `use rustam::{Normalizer as NormalizerTrait, ...}`.
### Normalizer (`src/normalizer.rs`)
Runs up to eight sequential steps controlled by `NormalizerConfig` (all on by default): character translation -> Persian style -> Persian numbers -> diacritics removal -> spacing correction -> Unicode ligature replacement -> special-char removal -> repeated-char reduction -> mi/nami separation. All regex patterns are compiled once via `once_cell::sync::Lazy`. `correct_spacing` reuses the `WordTokenizer` instance stored at construction time — do not construct a new one per call.
### WordTokenizer (`src/word_tokenizer.rs`)
Splits on whitespace/punctuation, then optionally joins multi-part verbs using `_` as separator. Controlled by `WordTokenizerConfig { join_verb_parts: bool, .. }`.
### Lemmatizer (`src/lemmatizer.rs`)
Dictionary lookup first; falls back to stripping verb conjugation suffixes using the embedded verb table. Verbs are keyed as `past#present` (e.g., `"رفت#رو"`). Accepts an optional POS hint as the second argument; pass `""` when unknown.
### Conjugation (`src/conjugation.rs`)
Stateless zero-size struct. Takes a past or present root and generates all tense/mood/voice forms. `get_all("past#present")` returns the full set used by `Lemmatizer`.
### Corpus readers (`src/corpus/`)
Each reader in `src/corpus/` wraps a file path or directory. They implement `CorpusReader` and yield `TaggedSentence` iterators lazily. Corpus files must be obtained separately (not bundled). Notable encoding: `PeykareReader` handles Windows-1256.
### Key type aliases (`src/types.rs`)
```rust
Token = String
Tag = String
TaggedToken = (Token, Tag)
Sentence = Vec<Token>
TaggedSentence = Vec<TaggedToken>
IobTag = String
ChunkedToken = (Token, Tag, IobTag)
ChunkedSentence = Vec<ChunkedToken>
```
`WordEntry { frequency: u64, pos_tags: Vec<String> }` and `VerbRoots { past, present }` are the two non-alias structs.
## Key conventions
- `fancy-regex` is used instead of the standard `regex` crate to support lookbehind assertions required for Persian patterns.
- Verb lemmas always use `past#present` format separated by `#`.
- ZWNJ (U+200C) is the standard Persian non-breaking zero-width joiner; it appears inside verb joins and in affixed forms.
- Embedding and tagging backends (`WordEmbedding`, `SentenceEmbedding`, `Tagger`) are traits — concrete backends live in `crftag`, `nerrs`, `arc-eager`, or user crates.
- Optional Cargo features: `pos`, `ner`, `dep-parsing`, `hf-hub`, `full`.