# Akana (Turkish NLP Toolkit)
[](https://pypi.org/project/akana/)
[](LICENSE-MIT)
[](https://www.rust-lang.org)
[](https://www.python.org)
**Akana** (named after *Ak Ana*, the primordial creator goddess in Turkic mythology) is a modern, blazingly fast Turkish Natural Language Processing toolkit written in **Rust** with seamless **Python bindings via PyO3** and native hardware SIMD acceleration via **StringZilla**.
Repository: [https://github.com/altaidevorg/akana](https://github.com/altaidevorg/akana)
---
## Key Features
- **Phonology & Orthography Engine**:
- Turkish alphabet characteristics and locale-aware casing (`ı` $\leftrightarrow$ `I`, `i` $\leftrightarrow$ `İ`).
- Major (2-way `A/E`) and Minor (4-way `I/İ/U/Ü`) Vowel Harmony validation.
- Consonant softening / mutation ($p \rightarrow b$, $ç \rightarrow c$, $t \rightarrow d$, $k \rightarrow \check{g}/g$).
- Epenthetic vowel drop (*burun* $\rightarrow$ *burnu*, *akıl* $\rightarrow$ *aklı*).
- Consonant doubling / gemination (*hak* $\rightarrow$ *hakkı*, *his* $\rightarrow$ *hissi*).
- **Tokenization & Sentence Segmentation**:
- Zero-allocation, rule-based Turkish tokenizer handling proper nouns with apostrophes (`İstanbul'da`), abbreviations (`Prof.`, `Dr.`, `vb.`), currencies, URLs, emails, hashtags, dates, and times.
- Sentence Boundary Detector with Turkish quotation and abbreviation lookahead.
- **Normalization & Spell Checking**:
- **Asciifier** & **De-asciifier** for Turkish diacritics restoration.
- **SIMD Spell Checker**: Accelerated with **StringZilla** hardware instructions for ultra-fast Levenshtein / edit distance candidate scoring.
- **Informal Text Normalizer**: Spoken Turkish colloquialisms reduction (`yapcam` $\rightarrow$ `yapacağım`, `geliyom` $\rightarrow$ `geliyorum`, `noldu` $\rightarrow$ `ne oldu`) and letter elongation deduping (`çooook` $\rightarrow$ `çok`).
- **Dual-Engine Morphology Suite**:
1. **Standard Morphology (`akana.Morphology`)**:
- **66,491-Item Root Lexicon**: Zemberek's default dictionaries (TDK master dictionary, non-TDK words, proper nouns, person names and abbreviations), generated by `scripts/build_full_lexicon.py` with Zemberek's own POS and attribute rules, pronunciations and compound data, plus akana's supplements (`scripts/lexicon/akana-legal.dict`: HMK, TBK, AİHM, *tebliğname*, ...; given names and surnames of akana's PII lists that Zemberek lacks, `akana-names.dict`; misspellings read as nonstandard with the TDK lemma, `akana-spelling.dict`).
- A port of Zemberek's morphotactics: 287 states and 989 suffix transitions with morphophonemic templates and conditions, and stem alternations generated at load time (*kitab-ı*, *redd-i*, *burn-u*), so misspellings such as *kitapda*, *gelti* or *kitapı* get no analysis.
- Numbers, Roman numerals and abbreviations take the suffixes of their spoken form (`1923'te`, `%50'si`, `15.`, `XIX.`, `İİK'nın`, `TBMM'de`).
- Analyses are ranked by a corpus frequency prior estimated on court decisions mixed with the sentences of the UD Turkish treebanks, whose part-of-speech tags also split the words read in more than one part of speech (`kararının` is *karar+P3sg+Gen*, `hakkı` the noun *hak*, `Ali` the name).
- **Compound Word Decomposer**: compounds the lexicon marks or productive patterns on lexicon roots (`denizaltı` $\rightarrow$ `deniz + altı`, `başbakanlar` $\rightarrow$ `baş + bakanlar`); *mahkeme* is not *mah + keme*.
- Morphological Generator on the same graph (`generate("kitap", ["Noun", "A3sg", "P1sg", "Dat"])` $\rightarrow$ `"kitabıma"`); a generated word always analyses back to the requested tags.
- Context-aware Disambiguator: the corpus prior plus bounded context rules (genitive–possessive agreement, postposition case, light verbs, determiners).
2. **Syntactic Expressive Morphology (`akana.SyntacticMorphology`)** *(Google FSMNLP 2019 format)*:
- **Inflectional Groups (IG)**: the standard analyzer's readings, split where Zemberek's morphotactics derive (participles, converbs, verbal nouns, voice, *-Abil*, *-lIk*, *-lI*, *-sIz*, *-ki*, the copula), each group with its POS and Universal Dependencies features as UD Turkish writes them (`Case`, `Number`, `Person`, `Number[psor]`, `Person[psor]`, `Tense`, `Aspect`, `Evident`, `Mood=GenPot`, `Polarity`, `Voice`, `VerbForm`).
- **Zero-Derivation Elimination**: the lexicon lists cross-categorized roots (*güzel* as adjective, adverb and noun), and an adjective inflected as a noun (*güzeller*) is a noun; only the copula (*evde-yim*, *güzel-di*) opens a group of its own.
- Same lexicon, ranking and coverage as the standard analyzer.
- **Modern & Classic Turkish Readability Suite**:
- **Kalyoncu (2025) Formula Suite**: Multi-regression equations (Formulas 1–4, $R^2$ up to 0.99) with embedded 4,600-word familiarity lexicon and exact grade-level mapping (*3. Sınıf Öncesi* to *Lisansüstü*).
- **Classical Formulas**: Ateşman (1997), Çetinkaya-Uzun (2010), and Bezirci-Yılmaz (2010).
- **Turkish AI Writing Style Auditor & Humanizer Engine**:
- Detects LLM writing signatures: punctuation anomalies (excessive em-dashes, semicolons with conjunctions, colons), predicate tense repetition (`-mektedir` ratio), rhythm monotony ($CV = \sigma / \mu$), bureaucratic connectors, translationese calques, tricolon lists, and hypophora questions.
- Actionable prompt generation across 5 registers (*hukuki-idari, akademik-kurumsal, analitik-gazetecilik, deneme-blog, edebi-yaratıcı*).
- Trained AI-text detector in every audit report (`report.ai_detection`, next to the rule-based `ai_score`, which counts style findings): calibrated probability that the text is AI-written, from 500 gradient-boosted trees over 90 signals (content statistics, mined AI phrases such as *"Elbette"*, *"önemli bir rol oynar"*, token-transition scores over Akana's own embedding tokenizer, and a morphology profile from Akana's analyzer: parts of speech, cases, tenses, voice, participles), with a Turkish "why AI" report. On a held-out OpenAI evaluation set it flags about 96 of 100 AI texts long enough to judge (65+ words), at 1% false alarms on separate human texts outside academic writing. Not proof of authorship; poems get no decision, academic text a warning, shorter or highly repetitive texts are not judged.
- For texts the detector judges AI, `humanize_prompt` starts with the signals that made the text look AI.
- **Syntax & Universal Dependencies**:
- Rule-based dependency parser with Universal Dependencies relations and CoNLL-U output. On the UD Turkish test splits it attaches 55.9 % of words to the right head (UAS) and 41.9 % with the right relation too (LAS), where attaching every word to the next word that is not punctuation gets 51.8 % UAS (`scripts/parser_ud_eval.py`): a margin of four points, and on Atis (42.8 against 51.3) none. Trained parsers do much better: use it for rough structure.
- **Speech Front End (Phonemiser)**:
- `akana.phonemize(text)` turns Turkish text into phoneme and prosody tokens for a TTS model: phoneme, stress level, word start, length mark and the clause's punctuation, in microseconds and in process. Every word says where its stress and phonemes came from (`phonemize_detailed`). No eSpeak NG code or data.
- `akana.verbalize(text)` writes out what a screen reader meets, keeping the span of each stretch it reads: numbers with their written suffixes re-harmonised (`₺850'ye` sekiz yüz elli liraya), ordinals, Roman numerals, dates, times, money, percentages, units, telephone numbers, abbreviations, initialisms, URLs, e-mail, symbols and foreign words.
- Profile `tts-v1` (the default) gives word stress from the phonology literature and Akana's morphology, a prominence hierarchy (nucleus, phrase accents, deaccented heads, function words, post-nuclear words, clitics), pauses where a speaker pauses, and long vowels and geminates. Where it and eSpeak put a word's stress on different syllables, the teacher TTS's pitch peak is on `tts-v1`'s syllable 50% of the time and on eSpeak's 30% ([report](docs/phonemizer-tts-v1.md)). Its output on plain text is frozen.
- Profile `tts-v2` follows the consensus of the Turkish phonology literature, with every rule's sources cited ([report](docs/phonemizer-tts-v2.md)):
- **Segments** (acoustic multi-speaker studies and grammars): tense vowels; [æ] before coda sonorants; /ɾ/ as a tap, devoiced before a pause; palatal k and g.
- **Stress:** Sezer's rule for place names, pre-stressing *-en* adverbs and *-CA*, first-element compounds and intensives.
- **Also:** every verbaliser rule, and IPA symbols.
- About 85 µs a sentence, some 33,000× faster than real time.
- **High-Level NLP Primitives**:
- Turkish Syllabification & Hyphenation.
- Number to Words Converter (Cardinals, Ordinals, Currency).
- Named Entity Recognition (PER, LOC, ORG, DATE, MONEY, PERCENT).
- Keyword Extraction (Turkish RAKE) & Extractive Summarization (TextRank).
- **Turkish Text Chunking Suite for RAG & LLMs (Chonkie-Inspired)**:
- **`SemanticChunker`**: Segments long Turkish documents at natural topic shifts using embedded 2-bit TurboQuant vector similarity drops.
- **Adaptive Threshold Modes**: Direct `similarity`, `percentile` (default: 0.75), `standard_deviation` ($\mu - k\sigma$), `interquartile` ($Q_1 - k \cdot IQR$), and `auto`.
- **Similarity Smoothing**: Built-in 5-point Savitzky-Golay and Moving Average digital filters to suppress high-frequency local noise in sentence transitions.
- **`SDPMChunker` (Semantic Double-Pass Merge)**: Two-pass hierarchical merge algorithm creating dense, coherent chunks with high information packing for vector databases.
- **`SentenceChunker`**: Rule-based sentence chunking respecting Turkish grammatical boundaries with configurable token counts and overlaps.
- **High Performance**: **>50,000–1,000,000 words/sec** on CPU with exact UTF-8 character and byte offset preservation.
- **Embedded Turkish Sentence Embeddings (TurboQuant 2-Bit Model2Vec)**:
- **Ultra-lightweight (2.5 MB)**: Built-in 2-bit quantized static sentence embedding model distilled from BGE-M3 (`altaidevorg/turkish-bge-m3-model2vec-turboquant-2bit`).
- **256-dimensional** vector embeddings with **92.19%** STSb-TR benchmark accuracy.
- **High Throughput**: **>20,000 sentences/sec** on CPU with zero deep learning runtime or PyTorch dependencies.
- Built-in cosine similarity and batch embedding support.
- **Grammatical Error Detection & Correction (GED/GEC)**:
- **Rule-based and precision first**: 33 rules in 17 error categories: clitics (`de/da`, `ki`, `mi`), consonant assimilation (*kitapda* $\rightarrow$ *kitapta*), vowel drop (*şehire* $\rightarrow$ *şehre*), over-narrowing (*başlıyan* $\rightarrow$ *başlayan*), number and proper-noun apostrophes (*1923'de* $\rightarrow$ *1923'te*, *Ankara'lı* $\rightarrow$ *Ankaralı*), split *-ebil* (*ola bilir* $\rightarrow$ *olabilir*), indefinite determiners (*bir çok* $\rightarrow$ *birçok*), reduplications (*elele* $\rightarrow$ *el ele*) and more. 26 rules are on by default: the ones that leave correct, edited Turkish alone. The other 7 are experimental and run only when asked for. It does not cover all 25 GECTurk error types (see the [benchmark](#grammar-checker-precision-and-recall-gecturk-court-decisions-wikipedia)).
- **Selectable rules**: every rule has a stable id (`de-da-after-verb`, `mi-merged`, ...); choose them per checker or per call (`rules=`, `enable=`, `disable=`, `experimental=True` in Python; `RuleSet` in Rust; `--rules/--enable/--disable/--experimental` in the CLI). `akana.grammar_rules()` and `akana grammar --list-rules` list them.
- **Diagnostics**: Turkish and English explanations, character offsets, the rule id, and the rule's measured precision as `confidence`.
- **Turkish PII Masking & Bi-Directional Restoration (KVKK-Aligned)**:
- **Learned PII Tagger** (Rust and CLI, `pii-tagger` feature): two Turkish token classifiers, ELECTRA-small (fast mode) and BERTurk averaged with it (accurate mode), run on the CPU in pure Rust with [RTen](https://github.com/robertknight/rten). They find persons, usernames, addresses, phones, e-mails, personal URLs, identifiers, secrets and personal dates. The models are downloaded on first use from [`altaidevorg/akana-pii-tagger`](https://huggingface.co/altaidevorg/akana-pii-tagger) (CC BY-SA 4.0), checked and cached. Each mode has a strict operating point, calibrated so that on average at most 4% of a text's personal characters stay in clear. On held-out real Turkish text the accurate mode masks 93% of personal values completely, the rule engine 34% ([benchmark](#turkish-pii-detection-benchmark-held-out-sets)).
- **Rule Engine with 50+ Turkish PII Categories** (the default in Python, Rust and the CLI: `pii_mask`, `TurkishPiiEngine`): recognition of personal identity identifiers: TCKN (Turkish Republic ID with Modulo-10/11 algorithmic validation), VKN (Tax ID), IBAN (MOD-97 checksum; IBAN, TCKN and VKN shapes after their trigger words are masked even when the checksum fails), Credit Card (Luhn check), Turkish and international person names and surnames (curated lists plus 4,592 given names from Zemberek; names that are also words, such as *Deniz* or *Metin*, need a title, a role, a surname or a capital in mid-sentence), GSM and landline phones, Addresses, Passports, Driving Licenses, Vehicle Plates, VIN, IMEI, MAC, IP Addresses, Port numbers, Age/Age ranges, Blood types, Health conditions, Passwords, API tokens (OpenAI, Anthropic `sk-ant`, GitLab `glpat-`, GitHub PAT, HuggingFace `hf_`, Groq `gsk_`, npm `npm_`, Stripe, SendGrid, JWT, Bearer, PEM, Webhooks, Database URLs), and OTP codes.
- **Private Date & URL Detection**: Tells private dates (birth dates, appointments, billing due dates) from public ones (calendar holidays, campaign end dates) by the phrases around them, and recognizes private URLs that carry authentication or session tokens.
- **Bi-Directional Restoration with Morphological Vowel Harmony**: Automatically tracks an in-memory or serializable mapping dictionary to restore masked entities losslessly, adapting case and possessive suffixes in downstream text according to Turkish 2-way and 4-way vowel harmony (e.g. `{{AD_1}}'a` $\rightarrow$ `Ahmet Yılmaz'a`, `{{AD_1}}'e` $\rightarrow$ `Mehmet Demir'e`). Supports conversational session continuity via `mask_with_vault` to persist identifiers across multi-turn dialogues.
- **Multiple Masking Modes**:
- `placeholder`: Standard structured tokens (`{{AD_1}}`, `{{TCKN_1}}`, `{{IBAN_1}}`) designed for clean downstream processing and 100% reversible restoration.
- `surrogate`: Statistically and morphologically valid Turkish synthetic replacements (e.g., valid surrogate TCKNs, realistic Turkish names, valid surrogate IBANs).
- `tag`: Standard semantic entity tags (`[AD]`, `[TCKN]`, `[IBAN]`).
- `anonymize`: Category label redaction replacing sensitive data with uppercase entity tags (`[AD]`, `[TCKN]`, `[SIFRE]`, `[IBAN]`).
- **Smart Non-PII Filtering**: Preserves corporate/functional emails (`info@`, `destek@`, `satis@`) and public dates while filtering individual personal data. Optionally keeps the names of well-known public figures (`preserve_public_figures=True`: *Atatürk*, *Nazım Hikmet*, *Fatih Sultan Mehmet*), only when the whole name is such a phrase.
- **Informal Addresses**: Abbreviated, lower-case and all-caps addresses as people type them (*sedefli mh bayır sk. bina 89 D: 22*), with the district and province around them.
- **Optional Model2Vec Context Scoring** (`use_embeddings=True`): Akana's built-in 256-dim Turkish embeddings decide ambiguous names with no other evidence, and find health conditions missing from the dictionary before a clinical trigger (*fibromiyalji tanısı*, not *futbol hastasıyım*). `scripts/experiment_pii_embeddings.py` measures where this helps and where trigger words do better.
- **High Performance**: **>4,500 docs/sec** on short messages (about 0.2 ms per 90-character document) on one CPU core.
- **High-Performance Architecture**:
- Pure Rust core with zero JVM dependency.
- Python package via `pyo3` and `maturin` (compatible with `uv`).
- Command Line Interface (CLI) for shell workflows.
---
## Performance Benchmarks (Akana vs Zeyrek / Zemberek & StringZilla SIMD)
Tested on real Turkish text corpora and 10,500 morphological queries (`benchmarks/`). The
Zeyrek column is from the 0.8.0 run of `benchmarks/benchmark_vs_zeyrek.py` and was not
re-measured (Zeyrek is not installed here); Akana's morphology rows are the median of three
runs of the same script on 0.9.0, from Python, on a laptop CPU.
| Benchmark Metric | Zeyrek (Python Zemberek Port) | Akana (Rust + StringZilla SIMD) | Performance / Throughput |
| :--- | :--- | :--- | :--- |
| **Root Lexicon** | 66,821 entries (Zemberek's six dictionaries) | **66,491 items** | The same dictionaries, merged, cleaned and supplemented (court terms, abbreviations) |
| **Startup / Lexicon Init** | `2,733.8 ms` (~2.7s) | **`114.8 ms`** | **~24x faster** |
| **Morphological Parse (10.5k words)** | `55,349.7 ms` (55.3s) | **`178.7 ms`** | **~310x faster** (`58,760 words/sec`; 42k–63k over three runs) |
| **Tokenization (Zero-Allocation)** | ~230 words/sec | **`949,991 tokens/sec`** | **>4,000x faster** (<21 ms for 19.5k tokens) |
| **Informal Normalization** | N/A | **`36,222 words/sec`** | **High Throughput** (Zero-Regex Suffix Matching) |
| **AI Writing Style Audit** | N/A | **`44,000 words/sec`** | **StringZilla SIMD** rules + trained AI-text detector with its morphology profile (10.2k words of one repeated paragraph in about 230 ms, 39k–46k words/sec over three runs, with `benchmarks/benchmark_stringzilla_optimizations.py`; the rules alone run at ~120k words/sec in the same script on the same machine) |
| **Named Entity Recognition (NER)** | N/A | **`1.1–1.3 MB/sec`** | trained tagger (averaged perceptron, Viterbi) for persons, places and organisations, patterns for dates, amounts and streets: 1 MB of RagTurk news, words cached per thread (the benchmark script's repeated sentence: ~3.6 MB/s). WikiNER test F1: PER 0.76, LOC 0.71, ORG 0.54 (`scripts/ner_eval.py`) |
| **Grammar Checking (GED/GEC)** | ~22–55 sent/s (Neural) | **`~7,000 sent/s`** | default rules; the 20,769 GECTurk test sentences in 2.7–3.0 s from Python (`tests/benchmark_real_data.py`, three runs on a laptop CPU). With the experimental agreement rules, which run the disambiguator: ~1,650 sent/s |
| **Turkish Embeddings** | 79 sent/s (BGE-M3) | **`20,013 sent/s`** | **253x faster** (2.5 MB 2-bit TurboQuant) |
| **Sentence Chunking** | N/A | **`1,120,000 words/sec`** | **Zero-Allocation Slicing** (<3 ms / doc) |
| **Semantic Chunking (TurboQuant)** | ~80 words/sec (Neural) | **`62,500 words/sec`** | **~780x faster** (~40 ms for 200 sents) |
| **Turkish PII Masking & Restoration** | N/A (Standard Regex / Spacy) | **`3,570 docs/sec`** | **Ultra-Fast KVKK Pipeline** (<0.28 ms / doc, 50+ entity types) |
| **Learned PII Tagger (RTen)** | N/A | **`60–90 ms` per 2 KB** | fast mode (ELECTRA-small) on four laptop CPU threads; the accurate mode (with BERTurk) takes about 0.8 s per 2 KB |
| **Hardware Acceleration** | Pure Python loops | **StringZilla AVX-512 / AVX2 / NEON** | **Native SIMD Instructions** |
### Morphological Analysis on Held-Out Treebanks (`scripts/ud_eval.py`)
The analyzer and the disambiguator on UD Turkish data that trains nothing akana ships: the test
splits of the six treebanks whose train and dev sentences feed the ranking prior, and all of
IMST (CC BY-NC-SA, used for evaluation only). BOUN's test split is broken down by genre.
*Coverage*: the word has a reading. *Class*: the disambiguated reading's part of speech is UD's
(a participle or verbal noun counts as a verb or as the class it derives; names, punctuation
and symbols left out). *Twins*: the same, on the words whose readings fall in two classes or
more, one of them UD's. *Lemma*: the reading's lemma or root equals UD's lemma; UD writes a
verb's stem and some derivations differently, so this is agreement, not accuracy.
| set | words | coverage % | class % | twins | twins % | lemma % |
|---|---|---|---|---|---|---|
| Atis | 4784 | 98.9 | 89.4 | 1265 | 89.3 | 86.4 |
| BOUN biographical | 1797 | 99.4 | 88.9 | 641 | 90.0 | 92.6 |
| BOUN essays | 2192 | 99.0 | 85.4 | 844 | 83.2 | 89.6 |
| BOUN instructional | 1636 | 98.9 | 86.8 | 666 | 86.6 | 92.9 |
| BOUN news | 2541 | 98.4 | 88.4 | 742 | 83.2 | 90.8 |
| BOUN popular culture | 1821 | 99.0 | 86.5 | 721 | 82.5 | 92.0 |
| FrameNet | 1198 | 99.7 | 93.2 | 518 | 90.3 | 92.5 |
| IMST | 45998 | 99.2 | 86.3 | 18092 | 85.7 | 92.6 |
| Kenet | 15135 | 99.5 | 88.9 | 6291 | 85.7 | 90.4 |
| Penn | 8471 | 96.4 | 89.8 | 2844 | 87.9 | 90.7 |
| Tourism | 7915 | 99.8 | 96.1 | 3757 | 95.7 | 95.2 |
| all | 93488 | 99.0 | 88.2 | 36381 | 87.0 | 91.9 |
On the court-decision and RagTurk samples of the morphology sweep (3.6 M and 163 k tokens,
`examples/morph_sweep.rs`), 0.49 % and 7.6 % of tokens have no reading (0.36 % and 2.3 % of
lower-case words: the rest are names, foreign words and codes), with 2.2 and 1.9 readings per
token.
### Turkish Text Chunking Performance Benchmark (`scripts/benchmark_chunking.py`)
Evaluated on realistic multi-domain Turkish documents (50 paragraphs, ~200 sentences, ~2,500 words):
| Strategy | Algorithm | Latency per Doc | Throughput (Words/Sec) | Throughput (Docs/Sec) | Key Benefit |
| :--- | :--- | :---: | :---: | :---: | :--- |
| **`SentenceChunker`** | Rule-Based Sentence Packing | **2.23 ms** | **1,121,076 words/s** | **448.4 docs/s** | Ultra-low latency, token overlaps |
| **`SemanticChunker`** | 2-Bit Embedding Cosine Valleys | **39.85 ms** | **62,735 words/s** | **25.1 docs/s** | Natural topic boundary detection |
| **`SDPMChunker`** | Semantic Double-Pass Merge | **44.10 ms** | **56,689 words/s** | **22.7 docs/s** | Optimal chunk density & cohesion |
* 🚀 **High Throughput on CPU**: `SemanticChunker` processes **>62,000 words per second** on a single CPU core without PyTorch, CUDA, or heavy ONNX dependencies.
* 📦 **Zero External Infrastructure**: Built-in 2.5 MB TurboQuant embeddings execute entirely in-process with minimal memory footprint.
### Grammar Checker Precision and Recall (GECTurk, court decisions, Wikipedia)
A grammar checker that rewrites correct text does more harm than good, so each rule is measured on **correct, edited Turkish** as well as on errors:
- **Clean text**: 7,500 sentences from Turkish court decisions (CC0; 1,500 of them are committed as `crates/akana-core/tests/data/grammar_clean_court.txt`) and 4,452 Wikipedia sentences from the RagTurk articles (measured locally, not redistributed).
- **Errors**: the GGLab/GECTurk test set (20,769 sentences, 25 synthetic error types) and its 300 human-written movie-review sentences. A finding counts as right when it overlaps a token GECTurk labels as an error.
A rule is **on by default** only if it raises a false alarm on at most 1 in 1,000 clean sentences and, where GECTurk has its error type, at least 80 % of its GECTurk findings are on a labelled error. GECTurk labels only the errors it injected. For a rule whose errors it does not inject, such as proper-noun apostrophes or *peşpeşe*, every finding in the measured text was read and judged. The other rules are **experimental**: off unless asked for. `cargo test -p akana-core` runs the default rules over the committed clean sentences (`tests/grammar_precision_gate.rs`) and fails if any rule fires on more than 1 in 1,000 of them, or if `correct` changes more than 3.
**Correct text changed by `correct()`** (default rules; before = every rule as it was before the rule ids):
| Text | Sentences | Before | After |
| :--- | ---: | ---: | ---: |
| Court decisions, committed gate corpus | 1,500 | 16.5 % | **0.0 %** |
| Court decisions, local | 6,000 | 18.0 % | **0.8 %**, all real errors in the source (*yada*, *her hangi*, *İstekde*, *2003'de*) |
| Wikipedia (RagTurk) | 4,452 | 13.0 % | **0.2 %**, all but one real errors (*Milano'lu*, *senaristdir*, *6.600'dan*) |
**GECTurk** (token-level error detection, `python tests/benchmark_real_data.py`):
| Split | Rules | Precision | Recall | F0.5 | Error-free sentences flagged |
| :--- | :--- | ---: | ---: | ---: | ---: |
| Test set (20,769) | before | 79.5 % | 38.1 % | 65.3 % | 8.0 % |
| Test set (20,769) | default | **97.9 %** | 37.7 % | **74.2 %** | **0.8 %** |
| Test set (20,769) | all (`--experimental`) | 96.4 % | 37.6 % | 73.4 % | 1.5 % |
| Movie reviews (300) | before | 63.8 % | 53.8 % | 61.5 % | 21.3 % |
| Movie reviews (300) | default | **98.1 %** | 55.2 % | **84.9 %** | **1.3 %** |
Recall by GECTurk error type (default rules): the checker catches most *yada* (69/69), *te/ta* (16/16), lexicalised *ki* (344/344), indefinite determiners (1,176/1,224), *-ebil* (1,946/2,376), capitalisation (362/372) and colloquial forms (114/123), and fewer of the split locatives (1,270/11,120), merged *de/da* (798/1,934), merged *mi* (503/871) and over-narrowing (1,032/1,928). Earlier versions of this README claimed all 25 GECTurk categories; the checker has 18 error categories of its own (17 with a rule), and they do not map onto GECTurk's one to one. Nine GECTurk types have no rule and next to no recall: *de/da* harmony (0/13), loan-cluster epenthesis (0/59), merged reduplication such as *zamanzaman* (0/63), wrong softening (1/61), merged compound verbs (0/62), *-hane/-name* (0/16), circumflex (0/135), abbreviations (0/47) and one unidentified type (0/2); split compound verbs get 10/146. The GECTurk paper's neural models are scored with its own correction metric, not comparable to these numbers.
**Every rule** (clean text: findings per 1,000 sentences, before → after; \* = every finding after the change was read and is a real error in the source; GECTurk precision by its labels, before → after, with the number of findings):
| Rule | Default | Court (7,500) | Wikipedia (4,452) | GECTurk precision | Confidence |
| :--- | :---: | ---: | ---: | ---: | ---: |
| `capitalization` | on | 0 → 0 | 0 → 0 | 89 % (419) → 89 % (409) | 0.90 |
| `mi-merged` | on | 13.1 → **0** | 7.4 → **0** | 88 % (504) → 100 % (196) | 0.99 |
| `mi-merged-person` | on | 0 → 0 | 0 → 0 | 97 % (63) → 98 % (42) | 0.95 |
| `mi-harmony` | on | 0 → 0 | 0.2 → **0** | 0 % (1) → 0 % (1); judged 1/1 right | 0.67 |
| `de-da-hardened` | on | 0.1 → 0.1\* | 25.8 → **0** | 22 % (73) → 89 % (18) | 0.88 |
| `de-da-merged-fixed` | on | 0.7 → 0.7\* | 0 → 0 | 99 % (664) → 100 % (537) | 0.99 |
| `de-da-after-verb` | on | 96.8 → **0.3**\* | 50.3 → **0.4** (one typo split wrongly) | 36 % (619) → 99.7 % (369) | 0.99 |
| `locative-split` | on | 1.9 → **0** | 0.9 → **0** | 99 % (913) → 100 % (635) | 0.99 |
| `ki-lexicalized-split` | on | 0.1 → 0.1\* | 0 → 0 | 98 % (175) → 98 % (175) | 0.98 |
| `ki-merged-fixed` | on | 0.5 → 0.5\* | 0 → 0 | 100 % (84) → 100 % (84) | 0.99 |
| `ki-after-verb` | on | 17.1 → **0** | 7.2 → **0** | 36 % (311) → 100 % (78) | 0.99 |
| `ordinal-dot-apostrophe` | on | 0 → 0 | 0 → 0 | no findings | 0.5 |
| `number-suffix` | on | 0.1 → 0.1\* | 0 → 0.2\* | 0 % (5) → 0 % (5); judged 8/8 right | 0.90 |
| `apostrophe-derivational` | on | 4.0 → **0.1**\* | 34.6 → **1.3**\* | 21 % (604) → 4 % (53); judged 62/62 right | 0.98 |
| `apostrophe-plural-name` | experimental | (in the above) → 0 | (in the above) → 1.6 | 10 % (21); judged 19/28 right | 0.67 |
| `reduplication` | on | 0 → 0 | 0 → 0 | 0 % (8) → 0 % (8); judged 8/8 right | 0.90 |
| `ebil-split` | on | 10.8 → **0** | 1.6 → **0** | 89 % (1,125) → 100 % (939) | 0.99 |
| `indefinite-compound` | on | 3.1 → 2.9\* | 0.7 → **0** | 94 % (631) → 96 % (612) | 0.96 |
| `compound-verb-merged` | on | 0 → 0 | 0 → 0 | 0 % (4) → 0 % (4); judged 4/4 right | 0.83 |
| `compound-verb-split` | on | 0 → 0 | 0 → 0 | 100 % (5) → 100 % (5) | 0.86 |
| `consonant-assimilation` | on | 8.5 → **0.7**\* | 0.4 → 0.2\* | 86 % (143) → 96 % (49) | 0.94 |
| `consonant-softening` | on | 0 → 0 | 0 → 0 | 90 % (20) → 100 % (15) | 0.94 |
| `vowel-drop` | on | 31.6 → **0** | 4.9 → **0** | 35 % (456) → 97 % (78) | 0.96 |
| `yor-narrowing` | on | 0 → 0 | 0 → 0 | no findings | 0.5 |
| `over-narrowing` | on | 1.3 → **0.9**\* | 0.7 → **0** | 99 % (1,057) → 100 % (953) | 0.99 |
| `colloquial-form` | on | 0.4 → 0.1\* | 0 → 0 | 91 % (150) → 91 % (147) | 0.91 |
| `tautology` | on | 0.3 → **0** | 0 → 0 | 0 % (4) → no findings | 0.5 |
| `plural-harmony` | experimental | 0.1 → 0.1 | 1.1 → 0 | 0 % (3) → 0 % (2); judged 1/2 right | 0.5 |
| `quantity-plural` | experimental | 0.3 → 0.3 | 1.8 → 0.4 | 4 % (97) → 0 % (5) | 0.18 |
| `genitive-possessive` | experimental | 0 → 0 | 0 → 0 | 6 % (70) → 0 % (2) | 0.25 |
| `ne-ne-polarity` | experimental | 0.3 → 0 | 0.7 → 0.2 | 25 % (12) → 0 % (1) | 0.2 |
| `negation-concord` | experimental | 1.7 → 1.2 | 0.7 → 0.2 | 5 % (66) → 3 % (36) | 0.05 |
| `inanimate-plural-verb` | experimental | 1.2 → 1.5 | 1.1 → 1.3 | 5 % (62) → 3 % (65) | 0.04 |
Before, all rules ran together; after, each rule is measured alone. `confidence` is (right + 1) / (findings + 2) over a rule's GECTurk findings (by the labels where GECTurk has the error type, judged otherwise), or 0.5 for a rule that never fired. To reproduce: `scripts/build_grammar_clean_corpus.py --local DIR` writes the measurement sets, and `cargo run --release -p akana-core --example grammar_rule_report -- --clean court=... --clean ragturk=... --gec gecturk=...` prints the per-rule numbers (`--context RULE` prints each finding in its sentence).
### Turkish Sentence Embedding Benchmark (Model2Vec TurboQuant 2-Bit)
Evaluated on standard Turkish Semantic Textual Similarity Benchmark (STSb) test sets:
| Model | Vocab | Size | Compression | STSb (gorkem) | STSb (emrecan) | Speed | Speedup |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| BGE-M3 (Teacher) | 250,002 | ~2,200 MB | 1x | **96.35%** | **79.57%** | 79 s/s | 1.0x |
| Our Model | **39,655** | **2.50 MB** | **880x** | 92.19% | 63.53% | **20,013 s/s** | **253x** |
* ⚡ **880x Model Compression:** Compressed from **~2,200 MB** to **2.50 MB** embedded directly into the binary with zero runtime dependencies.
* 🚀 **253x Speedup:** Delivers **20,013 sentences/sec** on CPU with high retention of semantic quality against the teacher model.
### Ultra-Low-Memory Turkish RAG Retrieval Benchmark (RagTurk & Turkuaz-RAG)
Evaluated on **`metunlp/ragturk`** (Turkish agglutinative morphology failure modes, [arXiv:2602.03652](https://arxiv.org/abs/2602.03652); first 300 articles of `formal_5k`: 2,732 chunks, 600 questions) and **`eneSadi/turkuaz-rag`** (multi-context QA; first 500 samples: 1,000 news texts, 500 questions) with compiled **USearch** (`unum-cloud/usearch`) metrics. Recall@k counts a question as answered when any of its gold passages is in the top k.
| Retrieval Engine | Bytes / Vector | RAM Footprint (1M Docs) | Turkuaz Recall@10 | Turkuaz MRR@10 | RagTurk Recall@10 | RagTurk MRR@10 | Query Latency (RagTurk) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **Dense FP32 (256-dim, exact cosine)** | 1,024 B | 1,024 MB | 75.4% | 0.383 | 93.2% | 0.772 | 1,095 µs |
| **Akana Dense 2-Bit (TurboQuant)** | **64 B** | **`64 MB` (16x smaller)** | 73.8% | 0.375 | 91.2% | 0.738 | 481 µs |
| **Akana Dense 1-Bit (Binary Hamming)** | **32 B** | **`32 MB` (32x smaller)** | 67.0% | 0.329 | 86.3% | 0.662 | 250 µs |
| **Akana Morphological BM25 (inverted index)** | Inverted | ~116 MB | **`87.6%`** | **`0.473`** | **`98.0%`** | **`0.880`** | 2,471 µs |
| **Akana Single-Buffer Hybrid (128B)** | **128 B** | **`128 MB` (8x smaller)** | 83.6% | 0.444 | 94.8% | 0.789 | 1,453 µs |
| **Akana Dual-Index RRF (2-Bit + BM25)** | 64 B + Inverted | ~180 MB | 83.2% | 0.440 | 97.8% | 0.837 | 3,975 µs |
| **Akana Dual-Index RRF (1-Bit + BM25)** | 32 B + Inverted | ~148 MB | 81.8% | 0.418 | 97.7% | 0.812 | 3,942 µs |
Latency is per query and end to end (query embedding + search) in a single Python process on a laptop CPU; the BM25 inverted index is the pure-Python one in the benchmark script, so the BM25 and RRF rows are dominated by Python. The FP32 row is exact brute-force cosine over `akana.embed` vectors.
> **Reproducibility:** `python scripts/benchmark_rag_real_data.py --dataset all --readme` prints this table (defaults: 300 RagTurk articles, 500 Turkuaz samples; Turkuaz is streamed once and then read from `.cache/rag_benchmarks/turkuaz.jsonl`, so later runs work offline). Every quality figure is deterministic: two runs give the same reciprocal rank for every question. See also the runnable workflows in [`examples/11_ultra_low_memory_retrieval_usearch.py`](examples/11_ultra_low_memory_retrieval_usearch.py).
> **Correction (0.8.0):** earlier versions of this table (and of the benchmark script, example and README snippets) built USearch indexes as `Index(ndim=nbytes, dtype="u8")` and added numpy `uint8` arrays. USearch reads `uint8` arrays as packed bits and unpacks them into such an index, so it kept only the first `nbytes` bits of each vector as 0/1 bytes: 1/8 of each dense vector, and none of the hybrid's sparse half. Every USearch row of the old table measured that loss (for example 36.7% RagTurk Recall@10 for the hybrid, and 1-bit appearing better than 2-bit). Indexes must store the packed bytes verbatim: `Index(ndim=8 * nbytes, dtype="b1", metric=...)`, which also passes `nbytes` to the compiled metric. Any index built the old way has to be rebuilt. After `Index.load()` or `Index.view()`, set `index.metric` to the compiled metric again: USearch restores the metric from the file header, which cannot hold a function pointer (a reloaded `b1` index crashes on search without it; a reloaded `u8` index silently used USearch's own cosine).
#### Key Empirical Findings
1. **2-bit TurboQuant keeps almost all of FP32's quality**: 0.738 vs 0.772 MRR@10 on RagTurk and 0.375 vs 0.383 on Turkuaz, at 1/16 of the memory. **1-bit** halves the memory again and costs another 0.05–0.08 MRR@10.
2. **Morphological BM25 is the strongest single retriever on both sets** (0.880 MRR@10 on RagTurk, 0.473 on Turkuaz; RagTurk questions share most of their vocabulary with their passages). Fusing it with 2-bit dense through RRF matches its RagTurk Recall@10 (97.8% vs 98.0%) but not its MRR@10 (0.837), and trails it on Turkuaz.
3. **The 128-byte single-buffer hybrid is one memory-mappable index with no inverted index.** It adds +0.05 (RagTurk) and +0.07 (Turkuaz) MRR@10 over 2-bit dense alone but stays below BM25: its sparse half holds the 16 heaviest terms of a passage (a RagTurk chunk has a median of 39 terms), weighted by repetition and part of speech. Function words are no terms (`ve`, `için`, `bu`, `bir`, `tarafından`), and only the twenty lemmas most frequent in both court decisions and general text (*almak*, *vermek*, *yer* ...) carry an IDF weight, because a full static IDF cost the hybrid 0.01–0.04 MRR@10 by trading a passage's repeated topic words for words it mentions once.
4. **Hybrid vectors depend on the morphology build.** 0.9.0's analyzer and sparse encoder change every `sparse_embed` vector and the sparse half of every hybrid vector built with 0.8.0 (dense halves are unchanged): rebuild such indexes. From 0.9.0 on, store `akana.encoder_info()` with an index and check it with `akana.check_encoder()` when you open it (see *Index compatibility* below); a release that changes vectors within the same sparse spec reports how many and keeps a stale index within 0.005 MRR@10 of a rebuilt one.
5. **Embedding cost:** on one core of a laptop CPU, the 128-byte hybrid embeds 170–240 court-decision chunks of about 214 words per second, the sparse half alone 200–220 and the dense half alone 1,600–2,100 (`scripts/benchmark_embedding_cost.py`, three runs; the machine's load moves these by a third). `embed_batch_hybrid` splits a batch of 512 or more texts across the CPU cores.
#### Index compatibility
An index stores what built it, and checks it when it is opened:
```python
import akana
info = akana.encoder_info() # save this dict next to the index when you build it
# {'akana_version': '0.11.0', 'unicode_version': ..., 'sparse_spec': '1.0', 'hybrid_layout': 1,
# 'dense_model': ..., 'analysis_version': '1.0', 'analysis_fingerprint': ...,
# 'sparse_fingerprint': ..., 'dense_fingerprint': ..., 'fingerprint': ..., 'resources': {...}}
status = akana.check_encoder(stored_info, "hybrid") # when you open it: the dict, or its JSON text
```
The second argument is what the index holds: `"sparse"` (`sparse_embed` vectors, or a BM25 index of their terms), `"dense"` (`embed` and the packed 2-bit and 1-bit vectors) or `"hybrid"` (the default). Each is judged by what its vectors depend on, so a new dense model asks dense and hybrid indexes to rebuild but leaves sparse ones alone.
- `"identical"`: this library was built from the same inputs as that kind of vector needs (the resources, the source of its encoders, akana's own `Cargo.lock`, the Unicode version) and produces exactly the vectors the index holds.
- `"compatible"`: the same sparse spec major (sparse vectors), the same dense model, tokenizer and weights (dense vectors), both and the hybrid layout (hybrid vectors). Some vectors differ, as code or a dependency changed: releases that change them say how many, and keep at most 5 % of documents' vectors changed and a stale index's MRR@10 within 0.005 of a rebuilt one, so no rebuild is needed.
- `"rebuild"`: term ids, term definitions, the dense model or the vector layout changed, or the stored record is not one `encoder_info()` wrote. Re-embed the documents.
An index built with 0.8.0 or earlier stores no encoder info and must be rebuilt; `CHANGELOG.md`
lists what else changed since 0.8.0 and how to migrate.
`Cargo.lock` and `rust-toolchain.toml` pin what the vectors depend on (the Unicode tables of the Rust standard library included), and ranking inside the analyzer and disambiguator is done in integers. The published wheels, and the sdist built into a wheel, are built with that toolchain and lock file and tested against the golden vectors on Linux, macOS and Windows before release. The fingerprints also cover the Unicode version and a probe set's vectors, so a build with another toolchain reports another fingerprint when either differs; a build without akana's own lock file (from crates.io, or as another project's dependency, whose resolved versions it cannot see) always reports one of its own: `check_encoder` then says `"compatible"`, not `"identical"`.
The hybrid metric's alpha (`set_hybrid_metric_alpha`, `get_usearch_metric_pointer_hybrid(alpha)`) is one value for the whole process: setting it changes the metric of every open index that uses it. It must be a number in [0, 1] (outside it is clamped; NaN raises `ValueError`).
#### Comparison with RAGTurk (`arXiv:2602.03652`) & Heavy Neural Pipelines
| Pipeline | Model Weights | Vector RAM (1M Docs) | Query Latency | Turkish Recall@10 | Computational Requirement |
| :--- | :---: | :---: | :---: | :---: | :--- |
| **BGE-M3 (Teacher)** | 2,200 MB | 4,096 MB (4.1 GB) | ~45 ms | ~85–90% | Heavy GPU / PyTorch |
| **RAGTurk Baseline (EmbeddingGemma)** | ~600 MB | 1,024 MB (1.0 GB) | ~15 ms | ~70–75% | Neural embedder, drops on morphology |
| **RAGTurk "Production-Friendly" (MiniLM Reranker)** | ~730 MB | 1,024 MB (1.0 GB) | 40–80 ms | ~90.1% | Requires cross-encoder over all candidate pairs |
| **RAGTurk "Max Accuracy" (HyDE + LLM)** | Multi-GB | 1,024 MB (1.0 GB) | 1,500–4,000 ms | ~92.0% | Heavy LLM query generation ($3.7\times$ tokens) |
| **Akana Dual-Index RRF (2-Bit + Morph BM25)** | **`2.50 MB`** | **`~180 MB`** | **`2.9 ms`** | **`97.7%`** (RagTurk, 600 questions) | **Zero LLMs, zero GPUs. Runs entirely on CPU** |
*The neural rows are as reported for their own setups; the Akana row is measured on the 300-article RagTurk subset above, so the two are not a like-for-like comparison.*
*In Section 6.1 of the RAGTurk paper, the authors explicitly highlight the need to: "(iv) study Turkish morphology-aware retrieval features (e.g., lemma-aware sparse retrieval, morphological analyzers)." Akana directly solves this challenge natively in Rust.*
### Turkish PII Detection Benchmark (Held-Out Sets)
Both PII detectors, scored with `scripts/pii_eval/run_all.py` on held-out sets that no rule or
model was written, tuned or chosen on. A value counts as found only when the predicted span is
the gold span (edge punctuation and an apostrophe suffix aside), and as masked only when every
character of it is covered. *Rules* is the default engine (`pii_mask`, `TurkishPiiEngine`); the
tagger's modes are those of `akana pii-tag` (see [CLI Usage](#cli-usage)).
| Held-out set | Measure | Rules | Tagger, fast | Tagger, accurate | Fast, strict | Accurate, strict |
| :--- | :--- | ---: | ---: | ---: | ---: | ---: |
| **Real Turkish text** (1,617 units, 900 personal values) | values fully masked | 33.6% | 91.3% | 93.4% | 96.3% | **96.6%** |
| | exact F1 | 0.452 | 0.876 | **0.926** | 0.785 | 0.870 |
| | personal characters left in clear | 62.6% | 7.0% | 5.3% | **2.4%** | 2.6% |
| | false alarms per 10k characters of PII-free text | 1.97 | 4.27 | **1.31** | 12.02 | 6.63 |
| **yusuf-said** test part (3,537 synthetic texts) | exact F1 | 0.537 | 0.988 | **0.995** | 0.977 | 0.992 |
| | PII-free texts flagged (898) | 52.4% | 5.8% | **2.3%** | 13.3% | 5.9% |
| **Turkish WikiNER** test, person names (654) | fully masked | 26.3% | 93.0% | 94.3% | **97.9%** | 96.8% |
*Real Turkish text*: court decisions, Wikipedia, news, tweets, product reviews, voice-assistant
requests, chatbot prompts, social-media comments and maths word problems, held out by document,
1,149 of the units without personal data; its gold labels come from independent labellings by
Claude models, adjudicated and reviewed (`scripts/pii_data`). *yusuf-said*:
[`yusuf-said/turkish-privacy-filter-dataset`](https://huggingface.co/datasets/yusuf-said/turkish-privacy-filter-dataset),
a fixed 70% of its rows. The tagger misses one release gate of the PII audit by one date: its fast
and accurate modes fully mask 51 of the 57 personal dates in real text (the strict modes 56),
where 52 are needed. It is weakest on lower-case names in tweets and comments, and on bare numbers
in counting contexts, which it can read as identifiers (below).
**On the sets the rule engine was developed against.** The rules were written against Saturday
Labs' rows (19 strings from them appear verbatim in the code and tests) and tuned on BTX24's
test split and Belgin's training split, so their scores there are not held-out figures. Span
recall (a value counts as found when at least half of it is masked), measured with
`scripts/benchmark_pii_datasets.py` from the local Hugging Face cache:
| Dataset | Values | Recall | Over-masked characters | Notes |
| :--- | ---: | :---: | :---: | :--- |
| **Saturday Labs** (`cagrigungor/turkish-pii-masking-benchmark`, 1,000 rows) | 2,096 | **94.5%** | 20% | Values aligned from the expected masked output; 100% on VKN, phone, blood type, passwords, IP and MAC addresses |
| **Belgin** (`negentropi/belgin-pii-dataset`, all splits) | 1,851 | **74.0%** | 2.1% | Private dates 292/292, addresses 99.4%; **0/300** public dates (holidays, campaigns) masked |
| **BTX24** (`BTX24/turkish-privacy-pii-ner`, test) | 10,393 | **89.6%** | 1.2% | IBAN 99.6%, phone 99.9%, email 97.4%, person 92.3% |
| **Throughput** | | **`~4,600 docs/sec`** | | Short messages (about 90 characters), one CPU core, no neural network |
The previous version of this table (94.1% on 200 Saturday Labs samples) had no script behind it;
on the same three datasets the 0.6.0 engine reaches 85.6%, 50.9% and 58.4%.
The tagger learnt from BTX24's and Belgin's training splits, which share templates with their
test splits (exact F1 1.000 on both). On Saturday Labs, scored on the values its labels cover, it
reaches exact F1 0.896 (fast) and 0.929 (accurate) against the rules' 0.936, but flags 37% of the
200 PII-free rows, where the rules flag 5.5%: bare numbers in counting contexts (a fleet size, a
password length) read as identifiers.
* 🛡️ **KVKK Compliance**: Algorithmic validation for Turkish IDs (TCKN, VKN, IBAN, Driving License, Plate) prevents false positives while capturing authentic identifiers.
* 🔄 **100% Reversible Restoration**: Restores anonymized text losslessly with Turkish morphological vowel harmony adjustment on inflected suffixes.
---
## Python Quickstart
### Installation
```bash
# Using uv
uv pip install akana
# Using pip
pip install akana
```
### Usage in Python
```python
import akana
# 1. Advanced Grammatical Error Correction (GEC) & Diagnostics
text = "Ali de geldi, Veli te geldi. Evi terketmek zorunda kaldı ve 1923'de kurulan cumhuriyeti andık."
# Direct correction
corrected = akana.correct_grammar(text)
print("Corrected:", corrected)
# -> "Ali de geldi, Veli de geldi. Evi terk etmek zorunda kaldı ve 1923'te kurulan cumhuriyeti andık."
# Detailed diagnostic findings (offsets are characters; confidence is the rule's measured precision)
res = akana.check_grammar(text)
for f in res.findings:
print(f"[{f.rule}] '{f.original_text}' -> '{f.replacement}' ({f.confidence:.2f}) | {f.message_tr}")
# Rule selection: the default rules are the ones measured precise on correct text; the
# experimental ones run only when asked for. Every call and GrammarChecker take the same arguments.
for r in akana.grammar_rules():
print(r["id"], r["default"], r["confidence"], r["summary"])
akana.correct_grammar("Pazardan üç elmalar aldım.") # unchanged: quantity-plural is experimental
akana.correct_grammar("Pazardan üç elmalar aldım.", experimental=True) # -> "Pazardan üç elma aldım."
akana.check_grammar(text, disable=["capitalization"], enable=["quantity-plural"])
checker = akana.GrammarChecker(rules=["de-da-hardened", "number-suffix"])
checker.enable_rule("mi-merged")
# 2. Standard Morphological Analysis (Zemberek-Compatible Format)
morph = akana.Morphology()
parses = morph.analyze("kitabıma")
for parse in parses:
print(parse["lemma"], parse["primary_pos"], parse["morphemes"])
print(parse["analysis"])
# -> kitap Noun ['Noun', 'P1sg', 'Dat']
# -> [kitap:Noun] kitab:Noun+A3sg+ım:P1sg+a:Dat (Zemberek's format: every morpheme and its surface)
# `lemma` is the folded, lower-case key that sparse terms use; `display_lemma` spells it as
# the dictionary does, and `analysis` shows that spelling, as Zemberek prints it. `pos` is
# the POS after derivation: a participle reads as a noun and as an adjective with the same
# akana tags, and `pos` and `analysis` tell them apart.
p = morph.analyze("Ankara'dan")[0]
print(p["lemma"], p["display_lemma"], p["analysis"])
# -> ankara Ankara [Ankara:Noun,Prop] ankara:Noun+A3sg+dan:Abl
print([(p["pos"], p["analysis"]) for p in morph.analyze("olduğu")][:2])
# -> [('Noun', '[olmak:Verb] ol:Verb|duğ:PastPart→Noun+A3sg+u:P3sg'),
# ('Adj', '[olmak:Verb] ol:Verb|duğ:PastPart→Adj+u:P3sg')]
# 3. Google-Style Syntactic Expressive Morphology (Inflectional Groups & UD)
syn_parses = akana.syntactic_analyze("geldiğimizde")
for p in syn_parses:
print(p.formatted)
# Output: (gel[VB]+[Polarity=Pos])([NOMP]-PastNom+[Case=Loc]+[Number=Sing]+[Number[psor]=Plur]+[Person=3]+[Person[psor]=1]+[VerbForm=Vnoun])+[Proper=False]
for ig in p.inflectional_groups:
print(f" • IG [{ig.pos}] Deriv: {ig.derivation} -> {ig.features}")
# 4. Morphological Generation
surface = morph.generate("kitap", ["Noun", "A3sg", "P1sg", "Dat"])
print(surface) # -> kitabıma
# 5. Spell Checking with StringZilla SIMD
spell = akana.SpellChecker()
print("Is 'kitap' correct?", spell.is_correct("kitap"))
suggestions = spell.suggest("ktap", max_distance=2, max_suggestions=3)
print("Suggestions for 'ktap':", [s["word"] for s in suggestions])
# 6. De-asciification & Normalization
print(akana.deasciify("turkce nlp cok hizli calisiyor"))
# -> türkçe nlp çok hızlı çalışıyor
print(akana.normalize_informal("nooldu ya yapcam dedim"))
# -> ne oldu ya yapacağım dedim
# 7. Compound Word Decomposition
compounds = akana.decompose_compound("denizaltı")
print(compounds)
# -> [{'surface': 'denizaltı', 'part1': 'deniz', 'part2': 'altı', ...}]
# 8. Modern Turkish Readability Analysis (Kalyoncu 2025 & Classic)
report = akana.analyze_readability("Küçük çocuk bahçede neşeyle koşuyordu.")
print(f"Kalyoncu F1: {report.kalyoncu_formula1.score} ({report.kalyoncu_formula1.grade_level})")
print(f"Ateşman: {report.atesman.score} ({report.atesman.grade_level})")
# 9. Turkish AI Writing Style Auditor & Actionable Humanizer Prompt
audit = akana.audit_ai_style("Yapay zeka teknolojileri, modern dünyada kritik bir rol oynamaktadır. Bu bağlamda —özellikle veri alanında— hayati önem taşımaktadır.")
print(f"Style score: {audit.ai_score}/100 ({audit.verdict})") # rule-based style findings
prompt = akana.humanize_prompt("Bu doğrultuda hayati önem taşımaktadır.", register="blog")
print(prompt)
# Trained AI-text detector inside the audit report (a probability, not proof; needs 65+ words)
story = ( # Turkish Wikisource, public domain
"Nasreddin Hoca'nın bir oğlu varmış. Ne denirse tam tersini yaparmış. Bir gün Nasreddin Hoca "
"ile oğlu Akşehir'e gidip oradan iki çuval tuz almışlar. Eşeğe yükleyip köylerine dönmek için "
"yola koyulmuşlar. Köylerinin yakınındaki dereye geldiklerinde, Nasreddin Hoca, derenin en sığ "
"yerinden karşı tarafa geçmiş. Oğlu da eşeği sudan geçirmeye çalışıyormuş. Ama eşeğin "
"sırtındaki çuvallardan biri, neredeyse suya değmek üzereymiş. Hoca, oğlunun ters huyunu "
"bildiği için seslenmiş: Sevgili oğlum. Çuvallardan biri suya değecek. Biraz daha asıl da, "
"iyice suya gömülsün. Ne var ki çocuğun o gün uysallığı üzerindeymiş. Tutup, babasının "
"dediğini yapmış. Neticede iki çuval birden suyun içine düşmüş."
)
detection = akana.audit_ai_style(story).ai_detection
print(detection["verdict"], detection["ai_probability"]) # -> human 0.0029… (ai / uncertain / human / too_short / out_of_scope)
print(detection["report"]) # Turkish "why AI" report
# 10. High-Level Turkish NLP Suite
# Syllabification & Hyphenation
print(akana.syllabify("Türkçe")) # -> ['Türk', 'çe']
print(akana.hyphenate("bilgisayar")) # -> 'bil-gi-sa-yar'
# Number to Words Converter
print(akana.number_to_words(1923)) # -> 'bin dokuz yüz yirmi üç'
print(akana.currency_to_words(1250.50, "TL")) # -> 'bin iki yüz elli lira elli kuruş'
# Named Entity Recognition (NER)
entities = akana.extract_entities("Prof. Dr. Ahmet Yılmaz 16 Ağustos 2026 tarihinde 500 TL ödeme yaptı.")
for e in entities:
print(f"[{e.label}] {e.text}")
# Keyword Extraction (Turkish RAKE) & Extractive Summarization (TextRank)
keywords = akana.extract_keywords("Doğal dil işleme ve morfolojik analiz...", top_k=5)
summary = akana.summarize("Uzun metin...", max_sentences=2)
# 11. TurboQuant 2-Bit Turkish Sentence Embeddings & Semantic Similarity
vec = akana.embed("Türkiye'nin başkenti Ankara'dır.")
print(f"Vector dim: {len(vec)}") # -> 256
# Semantic cosine similarity
score = akana.similarity("ev", "evler")
print(f"Similarity: {score:.4f}") # -> ~0.9130
# Batch embedding
vecs = akana.embed_batch(["Merhaba dünya", "Hava bugün çok güzel"])
print(f"Batch size: {len(vecs)}") # -> 2
# 12. Ultra-Low-Memory Packed Embeddings & USearch Integration (1M Docs in 64 MB RAM)
from usearch.index import Index, CompiledMetric, MetricKind, MetricSignature
import numpy as np
# A. 2-Bit TurboQuant Packed Vectors (64 bytes per document)
vec_2bit = akana.embed_packed_2bit("Türkçe doğal dil işleme kütüphanesi")
print(f"Packed 2-bit bytes: {len(vec_2bit)}") # -> 64 bytes!
# Create zero-copy USearch index with Akana's pre-compiled C-ABI metric
metric_2bit = CompiledMetric(
pointer=akana.get_usearch_metric_pointer_2bit(),
kind=MetricKind.Cos,
signature=MetricSignature.ArrayArraySize,
)
# USearch reads uint8 arrays as packed bits: a b1 index of 8 * 64 dims stores the 64 bytes as given
# (a u8 index of 64 dims would keep only their first 64 bits)
index = Index(ndim=64 * 8, metric=metric_2bit, dtype="b1")
index.add(0, np.frombuffer(vec_2bit, dtype=np.uint8))
# B. 1-Bit Binary Packed Vectors (32 bytes per document - 32 MB for 1M docs)
vec_1bit = akana.embed_packed_1bit("Hızlı vektör araması")
metric_1bit = CompiledMetric(
pointer=akana.get_usearch_metric_pointer_1bit(),
kind=MetricKind.Hamming,
signature=MetricSignature.ArrayArraySize,
)
index_1bit = Index(ndim=32 * 8, metric=metric_1bit, dtype="b1")
index_1bit.add(0, np.frombuffer(vec_1bit, dtype=np.uint8))
# C. Morphological BM25 Sparse Embeddings (Agglutinative roots, lemmas & compounds)
sparse = akana.sparse_embed("Kitaplarımı çalışma masasına bıraktım.")
print(sparse["terms"]) # -> ['kitap', 'masa', 'çalışmak', 'bırakmak']: one lemma per word, in context
print(sparse["indices"]) # -> 32-bit term ids (ascending) for Qdrant / Milvus / Inverted Index
print(sparse["values"]) # -> BM25-saturated occurrence counts x POS weight x function-lemma IDF
# D. Reciprocal Rank Fusion (RRF) for Hybrid Retrieval
fused_results = akana.reciprocal_rank_fusion(
dense_results=[(101, 0.95), (102, 0.88)],
sparse_results=[(102, 14.2), (103, 11.0)],
k=60.0
)
# 13. Turkish Text Chunking for RAG & LLMs (Chonkie-Inspired)
# A. SemanticChunker: Splits at natural topic shifts using TurboQuant embeddings
semantic_chunker = akana.SemanticChunker(
chunk_size=512,
threshold_mode="percentile", # "percentile", "similarity", "stdev", "iqr", "auto"
threshold_value=0.75,
min_chunk_size=20,
min_sentences_per_chunk=1,
use_smoothing=True,
)
chunks = semantic_chunker(
"Kuantum mekaniği atom altı parçacıkları inceler... "
"Fenerbahçe dün akşam derbide galip geldi... "
"Geleneksel Türk mutfağında baklava çok meşhurdur..."
)
for chunk in chunks:
print(f"Chunk ({chunk.token_count} tokens, chars {chunk.start_index}:{chunk.end_index}): {chunk.text}")
print("Sentences:", chunk.sentences)
# B. SentenceChunker: Fast sentence packing with token constraints and overlaps
sentence_chunker = akana.SentenceChunker(chunk_size=256, chunk_overlap=30)
sentence_chunks = sentence_chunker("Metin 1... Metin 2...")
# C. SDPMChunker: Semantic Double-Pass Merge for dense, cohesive chunks
sdpm_chunker = akana.SDPMChunker(
chunk_size=512,
threshold_mode="percentile",
threshold_value=0.75,
merge_threshold=0.65,
)
sdpm_chunks = sdpm_chunker("Uzun doküman...")
# D. Dict / JSON serialization
chunk_dict = chunks[0].to_dict()
print(chunk_dict["text"], chunk_dict["start_index"], chunk_dict["end_index"])
# 13. Turkish PII Masking & Bi-Directional Restoration (KVKK-Aligned)
raw_text = (
"Müşterimiz Ahmet Yılmaz, 10000000146 nolu TCKN ve TR33 0006 1005 1978 6457 8413 26 "
"nolu IBAN ile 12.05.1988 doğumludur. Şifresi: Limon004_."
)
# A. Reversible Structured Placeholder Masking
masked = akana.pii_mask(raw_text, mode="placeholder")
print("Masked:", masked["masked_text"])
# -> "Müşterimiz {{AD_1}}, {{TCKN_1}} nolu TCKN ve {{IBAN_1}} nolu IBAN ile {{DOGUM_TARIHI_1}} doğumludur. Şifresi: {{SIFRE_1}}."
print("Mapping:", masked["mapping"])
# B. Lossless Restoration with Turkish Vowel Harmony
restored = akana.pii_restore(masked["masked_text"], masked["mapping"])
assert restored == raw_text
# Downstream inflected text restoration (vowel harmony preserved automatically):
inflected_text = "Ödeme {{AD_1}}'a yapılmış ve {{AD_1}}'ın dosyası onaylanmıştır."
print(akana.pii_restore(inflected_text, masked["mapping"]))
# -> "Ödeme Ahmet Yılmaz'a yapılmış ve Ahmet Yılmaz'ın dosyası onaylanmıştır."
# C. Realistic Turkish Synthetic Surrogates Mode
surrogate = akana.pii_mask(raw_text, mode="surrogate")
print("Surrogate:", surrogate["masked_text"])
# -> "Müşterimiz Can Demir, 52381940562 nolu TCKN ve TR92 0006 1000 ... doğumludur..."
# D. Keep the names of well-known public figures, mask everyone else
engine = akana.TurkishPiiEngine(preserve_public_figures=True)
print(engine.mask("Nazım Hikmet'i en çok Ahmet Yılmaz okurdu.")["masked_text"])
# -> "Nazım Hikmet'i en çok {{AD_1}} okurdu."
# E. The learned PII tagger runs in Rust and the CLI (`akana pii-tag`, see CLI Usage);
# examples/12_learned_pii_tagger.py drives it from Python and compares it with the rules.
```
### Offsets, threads and errors
- **Offsets are character offsets.** `text[start:end]` is the span for named entities (`start`/`end`), grammar findings (`start_offset`/`end_offset`), style findings (`start`/`end`), PII entities and chunks (`start_index`/`end_index`).
- **Calls release the GIL**, so a thread pool (a FastAPI server, a `ThreadPoolExecutor` ingestion job) runs them in parallel. `embed_batch`, `embed_batch_hybrid` and the packed batch functions also split a batch of 512 or more texts across the CPU cores themselves.
- **Objects can be configured while threads use them.** `Morphology.load_dictionary_*`, `SpellChecker.add_word`, `GrammarChecker.enable_rule`/`disable_rule` and `TurkishPiiEngine.enable_type`/`disable_type` wait for the calls in progress on that object, and the calls after them see the change. A `PiiVault` that threads share serves one `mask` at a time.
- **Engines are built once per process.** The lexicon and the embedding weights are loaded on first use and shared by every object, so the module-level functions (`analyze`, `stem`, `audit_ai_style`, `chunk_*`, `pii_mask`, ...) and constructors are cheap to call repeatedly.
- **An internal error raises `RuntimeError`** (not PyO3's `PanicException`, which `except Exception` does not catch).
---
## CLI Usage
The `akana` CLI supports direct text arguments or reading from file via `-f, --file`:
```bash
# Turkish PII Masking & Restoration (KVKK-Aligned)
akana pii-mask "Ahmet Yılmaz 10000000146 nolu TCKN ile başvurdu." --mode placeholder
akana pii-mask "Ali Kaya Kadıköy şubesine geldi." --mode surrogate
akana pii-mask -f confidential.txt --mode placeholder --json
akana pii-restore "Ödeme {{AD_1}}'a yapıldı." --mapping mapping.json
# Learned PII tagger (build the CLI with it: cargo install --path crates/akana-cli --features pii-tagger)
# spans are byte offsets; the models are downloaded on first use (AKANA_CACHE_DIR, AKANA_OFFLINE=1)
akana pii-tag "Ayşe Yılmaz 12 Mart 2024'te 0532 123 45 67 numaralı telefondan aradı."
akana pii-tag --accurate --strict -f letter.txt
akana pii-tag --jsonl texts.jsonl # {"text": ...} per line in, {"spans": [[start, end, label], ...]} per line out
akana pii-models --accurate # download both models now: 57 MB, and 443 MB more for the accurate mode
# Grammar checking (default rules; --experimental, --rules, --enable, --disable to choose)
akana grammar "Ali de geldi, Veli te geldi."
akana grammar --correct -f article.txt
akana grammar --experimental --disable capitalization --json "Pazardan üç elmalar aldım."
akana grammar --list-rules
# AI style auditing
akana ai-audit "Bu bağlamda kritik bir rol oynamaktadır."
akana ai-audit -f article.txt
# Generate humanizer rewrite prompt
akana humanize-prompt "Bu doğrultuda hayati önem taşımaktadır." --register blog
# Syntactic morphological analysis (Google FSMNLP format)
akana syntactic-analyze "geldiğimizde"
# Standard morphological analysis
akana analyze "evlerimizde"
# Readability analysis
akana readability "Küçük çocuk bahçede neşeyle koşuyordu."
# Syllabification & Number conversion
akana syllabify "bilgisayar"
akana number 1923
# De-asciification & Normalization
akana deasciify "turkce nlp"
akana normalize "yapcam"
# Universal Dependencies Parsing
akana parse "Ali güzel kitabı okudu."
# Turkish Sentence Embeddings & Similarity
akana embed "Türkiye'nin başkenti Ankara'dır."
akana similarity "ev" "evler"
# Text Chunking (Semantic, Sentence, SDPM)
akana chunk --strategy semantic --chunk-size 512 --mode percentile -t 0.75 -f document.txt
akana chunk --strategy sentence --chunk-size 256 --overlap 30 -f document.txt
akana chunk --strategy sdpm --chunk-size 512 --json -f document.txt
```
---
## Rust Crate Usage (`akana-core`)
Add to `Cargo.toml`:
```toml
[dependencies]
akana-core = { version = "0.10", default-features = true }
```
```rust
use akana_core::chunking::{SemanticChunker, SentenceChunker, SDPMChunker, ThresholdMode};
use akana_core::grammar::{GrammarRule, RuleSet, TurkishGrammarChecker};
use akana_core::morphology::TurkishMorphology;
use akana_core::syntactic_morphology::TurkishSyntacticMorphology;
use akana_core::embeddings::TurkishEmbeddings;
use akana_core::pii::{TurkishPiiEngine, PiiMode};
use akana_core::phonology::to_turkish_lower;
fn main() {
let lower = to_turkish_lower("İSTANBUL");
println!("Lower: {}", lower);
// 1. Semantic Chunking with TurboQuant Embeddings
let chunker = SemanticChunker::new(512, ThresholdMode::Percentile(0.75));
let chunks = chunker.chunk("Kuantum mekaniği... Fenerbahçe dün akşam...");
for chunk in chunks {
println!("Chunk ({} tokens, {}-{}): {}", chunk.token_count, chunk.start_index, chunk.end_index, chunk.text);
}
// 2. Turkish Sentence Embeddings (TurboQuant 2-Bit)
let embeddings = TurkishEmbeddings::new();
let vec = embeddings.embed("Türkiye'nin başkenti Ankara'dır.");
println!("Embedding dim: {}", vec.len()); // 256
let sim = embeddings.similarity("ev", "evler");
println!("Similarity: {:.4}", sim); // 0.9130
// 3. Grammatical Error Correction & Diagnostics (the default rules)
let grammar_checker = TurkishGrammarChecker::new();
let res = grammar_checker.check("Ali de geldi, Veli te geldi. Cumhuriyet 1923'de kuruldu.");
println!("Corrected: {}", res.corrected);
for f in &res.findings {
println!("[{}] '{}' -> '{}' ({:.2})", f.rule, f.original_text, f.replacement, f.confidence);
}
// experimental rules on request, by id or all of them
let all = RuleSet::default().with_experimental();
println!("{}", grammar_checker.correct_with_rules("Pazardan üç elmalar aldık.", all));
let mut only_clitics = TurkishGrammarChecker::with_rules(RuleSet::empty());
only_clitics.enable_rule(GrammarRule::DeDaHardened);
// 4. Standard Morphology
let morph = TurkishMorphology::new();
let parses = morph.analyze("kitabım");
for p in parses {
println!("{}", p.formatted);
}
// 5. Syntactic Expressive Morphology (Inflectional Groups)
let syn_morph = TurkishSyntacticMorphology::new();
let syn_parses = syn_morph.analyze("geldiğimizde");
for p in syn_parses {
println!("{}", p.formatted);
}
// 6. Turkish PII Masking & Bi-Directional Restoration (KVKK-Aligned)
let pii_engine = TurkishPiiEngine::new();
let pii_res = pii_engine.mask(
"Ahmet Yılmaz 10000000146 nolu TCKN ile TR33 0006 1005 1978 6457 8413 26 nolu hesaptan işlem yaptı.",
PiiMode::Placeholder,
);
println!("Masked: {}", pii_res.masked_text);
// Lossless restoration with automatic vowel harmony adjustment
let restored = pii_engine.restore_with_mapping(
&format!("İşlem {}'a ait hesaba başarıyla yansıtıldı.", "{{AD_1}}"),
&pii_res.mapping,
);
println!("Restored: {}", restored);
}
```
The learned PII tagger is behind a feature. With `pii-download` it downloads its models on first
use, checks their SHA-256 and caches them (`AKANA_CACHE_DIR`; `AKANA_OFFLINE=1` turns downloads
off; `pii-tagger` alone runs model directories you already have, `PiiTagger::load`):
```toml
akana-core = { version = "0.10", features = ["pii-download"] }
```
```rust
use akana_core::pii::tagger::{Mode, PiiTagger, TaggerOptions};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Mode::Accurate averages BERTurk with the small model: better, about five times slower
let tagger = PiiTagger::from_hub(Mode::Fast, TaggerOptions::default())?;
let text = "Ayşe Yılmaz 12 Mart 2024'te 0532 123 45 67 numaralı telefondan aradı.";
for span in tagger.tag(text)? {
println!("{} {:?}", span.label, &text[span.start..span.end]); // byte offsets
}
// the strict operating point: more person, username and date tags, fewer values left in clear
let strict = PiiTagger::from_hub(Mode::Fast, TaggerOptions::strict(Mode::Fast.strict_bias()))?;
println!("{:?}", strict.tag("yarın mehmetle görüşeceğim")?);
Ok(())
}
```
---
## Developer Guide & Publishing
For local development setup, testing, running benchmarks, compiling native wheels, and publishing releases to PyPI & crates.io, see the [Developer & Maintainer Guide](developer-guide.md).
---
## Acknowledgements & Academic Citations
Akana builds upon decades of pioneering linguistic and natural language processing research in Turkish. We gratefully acknowledge and credit:
- **Kemal Oflazer**: Foundational two-level Turkish morphological analysis (1994) and the Inflectional Group (IG) representation (2003) for Turkish dependency syntax.
- **Ahmet A. Akın & The Zemberek Team**: Open-source Turkish morphology, phonotactics, and extensive root vocabulary database.
- **Oğuzhan Güngör & Zeyrek Contributors**: The pure-Python Zemberek port that inspired modern open Turkish NLP tooling.
- **Koç University GGLab (Duygu Ataman & Co-authors)**: *"GECTurk: Grammatical Error Correction and Detection Dataset for Turkish"* (arXiv:2309.11346), providing the 25-category Turkish grammatical error taxonomy and benchmark datasets.
- **Google Research (Adnan Öztürel, Tolga Kayadelen, Işın Demirşahin)**: *"A Syntactically Expressive Morphological Analyzer for Turkish"* (FSMNLP 2019), introducing zero-derivation elimination and two-level inflectional group FSTs.
- **Mustafa Kalyoncu & Co-authors (2025)**: Development of modern Turkish readability formulas (Formulas 1–4) and the empirical 4,600-word grade-level familiarity corpus.
- **Ender Ateşman (1997), Çetinkaya-Uzun (2010), Bezirci-Yılmaz (2010)**: Classical readability research for the Turkish education system.
- **Ash Vardanian & Unum Cloud**: **StringZilla**, providing hardware-accelerated SIMD vector search and edit distance algorithms.
- **Bushra Beg (Turkce-Humanizer)**: Research into Turkish AI writing style signatures, calques, and stylistic heuristics.
- **Stefan Schweter (dbmdz)**: BERTurk and the Turkish ELECTRA models the learned PII tagger is fine-tuned from.
- **Robert Knight**: **RTen**, the pure-Rust runtime the learned PII tagger runs on.
---
## License
Licensed under either of:
- [Apache License, Version 2.0](LICENSE-APACHE)
- [MIT License](LICENSE-MIT)
at your option.
The learned PII tagger's models are not part of this repository: akana downloads them from
[`altaidevorg/akana-pii-tagger`](https://huggingface.co/altaidevorg/akana-pii-tagger), where
they are released under CC BY-SA 4.0, since they learnt from text under that licence; the model
card lists their training sources.