Obadh Engine
Obadh is a deterministic Roman-to-Bangla transliteration engine for a larger Bangla typing system. The base layer is deliberately rule-based: a Roman input sequence maps to Bengali because a documented rule says so, not because a dictionary guessed the word.
This project is an Avro successor in ambition, but the deterministic core is not an Avro clone and not a word-by-word compatibility table. Users type deliberately according to Obadh's own Roman rule contract. Correction, suggestion, ranking, personalization, and neural context models live above that core.
Live playground: https://sayom.me/obadh_engine/
Quick Start
init.sh initializes the direct data submodules, resolves Git LFS objects, and
installs web dependencies:
| Path | Data repo |
|---|---|
data/autocorrect |
nsssayom/obadh_autocorrect_dataset |
data/autosuggest |
nsssayom/obadh_autosuggest_dataset |
Native prerequisites that are not installed by init.sh:
Common commands:
Crate Features
The crates.io package is the Rust SDK. It includes source code, CLI/WASM source behind explicit features, tests, and the small deterministic rule fixtures. It does not bundle large runtime model artifacts.
# Native Rust SDK: deterministic core, autocorrect/autosuggest runtimes,
# personal autosuggest, and native model handoff helpers.
= "0.5"
# CLI tools and artifact builders.
= { = "0.5", = ["cli"] }
# Browser playground bindings.
= { = "0.5", = ["wasm"] }
The default feature set is empty. Native downstreams, including the planned iOS
wrapper, do not pay for clap, ZIP/EPUB tooling, wasm-bindgen, or browser
APIs unless they opt into those features.
Runtime Shape
flowchart LR
R[Roman input] --> O[Deterministic Obadh core]
O --> B[Bengali baseline]
B --> C[Active-word autocorrect FST]
C --> W[Correction candidates]
T[Committed Bengali text] --> N[Next-word ngram autosuggest]
N --> S[Suggestion candidates]
Core Contract
The deterministic transliterator must remain dictionary-free.
- No whole-word correction table in the core.
- No hidden compatibility aliases just because another keyboard accepted them.
- No ML or corpus dependency on the transliteration hot path.
- Rule aliases need an Obadh-specific phonetic, orthographic, or ergonomic reason.
- Spelling correction and ranking belong in layers above the core.
Representative deliberate signals:
| Roman Signal | Bengali Rule Intent |
|---|---|
o |
inherent অ / lowercase cluster separator |
a / A |
visible আ / া, including before clusters |
I, U, O |
long ঈ / ঊ and ও |
aY / AY |
অ্যা / ্যা, e.g. aYp -> অ্যাপ |
ng, M, Ng |
anusvara / explicit anusvara escape / velar nasal |
ngg, nggh |
ঙ্গ / ঙ্ঘ shorthand |
jNG, jn, gg |
জ্ঞ paths |
NGj, nj, nJ |
ঞ্জ paths |
rr + cluster |
reph over a valid cluster |
rZy / rZY |
non-conjunct ZWNJ-separated র্য form |
y, w |
য-ফলা / ব-ফলা markers in declared clusters |
,, |
explicit hasant / conjunct boundary command |
t</code> / <code>T |
খণ্ড ত / ৎ |
^, :, ., $ |
chandrabindu, visarga, danda, taka sign |
Rule sources live under data/rules/ and are checked by tests.
Autocorrect
Autocorrect is an active-word layer above the deterministic core. Obadh first produces a Bengali baseline. The autocorrect layer then retrieves valid lexicon candidates from compact FST artifacts and ranks them through bounded, explainable channels.
Current runtime channels:
- exact Obadh baseline lookup
- bounded Obadh-aware Roman repair, such as missing lowercase
oseparators - Bangla weighted edit lookup over the FST
- narrow vowel-length and nasal-mark rescue channels
- exact-stem suffix completion
- curated English-loanword exact and bounded fuzzy lookup
- bounded prefix completion
Runtime code does not parse CSV, TSV, EPUB, JSON, or large heap-resident trie structures. Native tools can memory-map the FST; WASM loads the same compact artifact bytes.
Inspect shipped artifacts:
Probe the production FST path:
The autocorrect dataset repo contains the auditable lexicon TSVs and runtime
FSTs. Build commands and corpus policy are documented in
data/autocorrect/README.md after ./init.sh.
Next-Word Autosuggest
Autosuggest is the next-word layer above committed Bengali text. It does not run while a Roman token is active, does not transliterate Roman input, and does not replace active-word autocorrect.
The production path starts with a bounded n-gram candidate generator that retrieves likely next Bengali words through suffix backoff. The current browser runtime uses a compact c16 fourgram profile plus the bounded personal overlay. Native integrations can use the wider c64 retriever plus the packaged GRU256 generator: the model emits a top-128 vocabulary proposal set, scores the static c64 pool, and a fixed scored-union policy merges both streams after a Bengali word is committed. The neural path is not part of Roman keystroke transliteration.
Runtime integrations can layer PersonalAutosuggest above the static artifact
for on-device learning. It stores only bounded token-id transitions, keeps the
static corpus artifact unchanged, and can merge personal candidates with the
model using caller-owned scratch buffers. The personal layer can also round-trip
through a compact binary snapshot for local persistence.
The WASM playground exercises this path: committed Bengali words update the
bounded personal model, and the ribbon can surface personal next-word
suggestions without modifying the static artifact.
Personal autosuggest has two separate lifetimes. The live session context is short-lived: it is the recent words in the current editor flow and should be cleared at editor/session boundaries. The personal dictionary is longer-lived: it is a bounded local overlay that survives app restarts only if the host platform exports and saves its snapshot, then imports it again during startup. Obadh owns the compact snapshot format, vocabulary-fingerprint validation, bounded learning rules, and merge behavior. Downstream keyboards own the storage policy: where the snapshot lives, when it is saved, when it is cleared, and whether a user-facing privacy/reset control is exposed. A missing or fingerprint-mismatched snapshot must be treated as an empty personal dictionary; the static autosuggest artifact is never rewritten by user learning.
The playground follows the same contract as a platform integration. It stores
the exported personal snapshot in browser localStorage, keyed by the
autosuggest vocabulary fingerprint, reloads it when the WASM autosuggest module
starts, and drops it if import validation fails. Native integrations should do
the equivalent with platform-local storage: load once when the keyboard/editor
session is initialized, call commit_token_id/commit_token as Bengali words
are accepted, export the snapshot on a debounce or lifecycle event, and import
it before serving suggestions in the next session.
The n-gram artifact is a fixed-width binary file designed for mmap/native and
byte-buffer/WASM loading. It stores vocabulary text, sorted token lookup rows,
bounded context rows, and candidate records in one portable blob. Native
integrations should keep resolved token IDs on the typing hot path and use
suggest_ids_for_context_into; materialize candidate text only for the few
words that will actually be shown. The text APIs remain available for tools,
tests, and slower integration points.
Sentence starts and explicit sentence/editor boundaries use the artifact's
learned <bos> row; unknown in-sentence tokens deliberately fall back without
pretending to be a boundary.
Build a local smoke artifact from the current corpus:
For full corpus artifact generation, use the SQLite backend. It is slower than the in-memory smoke path, but it keeps counting disk-backed, streams the final artifact sections, and preserves exact counts:
Use --compact-count-records for size-only profile experiments,
--score-mode kneser-ney for single-discount Kneser-Ney, and
--score-mode modified-kneser-ney for modified-discount Kneser-Ney profile
research. The SQLite export path streams final artifact sections and uses
cached lower-order lookups for Kneser-Ney, so full-corpus scored exports do not
materialize the entire n-gram table in Python heap. The checked-in mobile
artifact currently uses compact count records because it keeps the visible
top-5 rate tied with the smaller c5 profile while expanding the hidden pool and
shrinking the model versus scored c16 profiles.
When tuning profile parameters, keep the SQLite count DB and re-export without recounting the corpus:
SQLite count DBs are keyed by vocabulary token IDs and now record the vocabulary
fingerprint and context order used during counting. Reuse is only valid with the
same vocab artifact; changing vocab.tsv requires rebuilding the count DB.
Use repeatable --source-weight source=integer flags for corpus-prior
experiments. The weights are applied during counting, remain integer-valued in
the artifact, and are recorded in the manifest. Source weighting is useful for
audits, but it is not currently part of the shipped profile because EPUB-heavy
split profiles hurt the visible top-5 band.
Evaluate and benchmark an artifact:
The evaluator reports both product-facing all-target rates and diagnostic
in-vocabulary rates. Use top*_all_targets for profile comparisons because
unknown vocabulary targets count as misses; use top* and
skipped_unknown_ratio to diagnose whether the vocabulary cap is hiding useful
words. When --top-k is larger than the visible suggestion band, the evaluator
also reports topN_to_topK_headroom* fields. Those are the candidates a future
reranker could promote without changing the retrieval artifact.
Current full-corpus candidate profiles from the 161.6M-token corpus, measured on the same replay probe:
| Profile | Artifact | Context rows | Replay top-1 | Replay top-5 | Native context lookup |
|---|---|---|---|---|---|
| trigram c5 min8 | 16.69 MB |
466,003 |
11.61% |
23.89% |
~0.11 us |
| fourgram c5 b8/t8/f14 | 21.47 MB |
618,590 |
11.98% |
24.03% |
~0.12 us |
| fourgram c5 b8/t8/f14 scored | 26.34 MB |
618,590 |
12.58% |
24.97% |
~0.14 us |
| fourgram c5 b8/t8/f14 Kneser-Ney | 26.34 MB |
618,590 |
12.42% |
24.69% |
~0.18 us |
| fourgram c16 b8/t8/f14 | 25.20 MB |
618,590 |
11.98% |
24.03% |
~0.185 us |
| fourgram c16 b8/t8/f14 scored | 31.93 MB |
618,590 |
12.58% |
24.97% |
~0.14 us |
| fourgram c16 b8/t8/f14 Kneser-Ney | 31.93 MB |
618,590 |
12.42% |
24.69% |
~0.19 us |
| fourgram c5 min8 | 27.13 MB |
796,737 |
12.13% |
24.12% |
~0.12 us |
The replay metric is measured on corpus rows also used for counting, so it is a profile-comparison signal rather than a held-out product accuracy claim.
Held-out profile probe, trained on the first 100,000 sentences per source and
evaluated on the next 25,000 sentences per source:
| Profile | Split artifact | Held-out top-1 | Held-out top-5 | Held-out top-10 | MRR |
|---|---|---|---|---|---|
| trigram c5 min8 count/backoff | 1.79 MB |
8.12% |
16.64% |
18.22% |
11.43% |
| trigram c5 min8 Kneser-Ney | 1.92 MB |
8.12% |
16.80% |
18.38% |
11.51% |
| fourgram c5 min4 count/backoff | 2.69 MB |
8.97% |
18.28% |
19.91% |
12.73% |
| fourgram c5 min4 scored backoff | 3.01 MB |
8.97% |
18.28% |
19.91% |
12.73% |
| fourgram c5 min4 Kneser-Ney | 3.01 MB |
8.95% |
18.39% |
20.20% |
12.79% |
| fourgram c16 min4 count/backoff | 2.89 MB |
8.97% |
18.28% |
22.87% |
13.25% |
| fourgram c16 min4 Kneser-Ney | 3.31 MB |
8.95% |
18.39% |
22.96% |
13.30% |
| fourgram c16 min4, EPUB weight 2 | 3.60 MB |
8.75% |
18.21% |
22.72% |
13.02% |
The five-slot UI currently uses the full-corpus fourgram b8/t8/f14 compact
count/backoff profile with a wider hidden candidate pool. The visible top-5
rate ties the c5 compact profile, but top-10 all-target recall improves from
25.72% to 29.67% while the artifact stays smaller than the older scored c5
profile. The native c64 profile gives the neural generator a bounded static
pool without changing the browser artifact.
Neural packaging lives under tools/autosuggest/. A model is only packaged when
it improves the static c64 pool under held-out probes, passes source-balanced
EPUB/news/Wikipedia checks, and keeps the runtime bounded:
| Gate | Current result |
|---|---|
train_candidate_reranker.py |
candidate-feature GRU/Transformer trials were not strong enough to ship |
train_next_word_lm.py |
8.82M-parameter GRU256 over the 32k vocabulary; packaged as a top-128 generator plus c64 scorer head |
The packaged production path keeps static n-gram retrieval and slot one stable,
then lets the GRU proposal stream compete for the remaining visible slots. The
runtime contract is fixed: uint32 token IDs inside Obadh, int64 ONNX inputs,
int32 Core ML inputs, float32 scores, 16 context IDs, 128 generated token
IDs, and 64 static c64 candidate IDs.
Export and package that deployment-shaped generator:
validate-generator checks both the loaded n-gram compatibility and the
referenced package asset sizes/hashes. verify_generator_package additionally
replays the packager in --check mode and enforces the release-mode Rust
handoff latency/heap budgets. Use --asset-root when the manifest paths are
mounted under an app bundle or staging directory.
Current fixed-batch generator gate:
| Model | File | top-1 all | top-5 all | top-10 all | Local runtime |
|---|---|---|---|---|---|
| static c64 pool | 29.49 MB |
16.84% |
31.34% |
37.99% |
~0.83 us u32 candidate input |
| scored-union GRU256 | 17.67 MB Core ML, 18.49 MB INT8 ONNX |
16.84% |
32.89% |
39.42% |
~14.38 us release personal-aware Rust handoff, ~459 us Core ML graph |
The selected scored-union profile improves top-5 and MRR on EPUB, news, and
Wikipedia held-out slices. It is packaged in data/autosuggest/models/neural;
the final hardware gate remains measurement inside real keyboard-extension
constraints on Apple devices.
The checked-in runtime artifact is the mobile n-gram profile:
| Item | Value |
|---|---|
| model family | bounded n-gram LM |
| task | next Bengali word prediction |
| artifact version | 3 compact fourgram format |
| artifact context | newest 3 committed Bengali tokens |
| runtime context buffer | newest 3 committed Bengali tokens |
| short context behavior | <bos> at sentence start, then bigram/unigram backoff |
| vocabulary | 32,768 tokens |
| unigram fallback | top 4,096 tokens |
| bigram rows | 32,533 |
| trigram rows | 433,470 |
| fourgram rows | 152,587 |
| candidate rows | 1,679,620 |
| candidate record | 8 bytes, token + count |
| max candidates per context | 16 hidden, UI usually requests 5 |
| min count | bigram/trigram 8, fourgram 14 |
| artifact bytes | 25,195,978 |
| eval top-5 all targets | 24.03% |
| eval top-10 all targets | 29.67% |
| eval MRR all targets | 17.17% |
| native runtime | mmap binary artifact plus AutosuggestContext |
| playground runtime | Obadh WASM parser over a binary artifact |
Keyboard integrations should keep an AutosuggestContext as words are
committed and call suggest_ids_for_context_into with a reused candidate-ID
buffer. For personalized suggestions, keep an AutosuggestSession per editor
surface and call suggest_ids when the platform can stay token-ID-first.
Commit resolved vocabulary IDs through commit_token_id on the hot path, use
commit_token only when text still needs lookup, and reuse the session-owned
personal/model/output buffers. Persist personalization with
write_personal_snapshot_into when the platform can reuse a save buffer, and
restore it with import_personal_snapshot so model compatibility and token IDs
are validated before the session mutates. The personal store grows lazily, so
empty editor sessions do not reserve the full history cap.
The personal overlay remains capped to its compact two-token snapshot format.
Use estimated_heap_bytes and personal_snapshot_len for current session
resource use; use heap_limit_bytes and personal_snapshot_limit_bytes for
conservative caps under the current personal-store and candidate-count
configuration.
The text-based APIs remain useful for tools and tests, but they intentionally
include token parsing and lookup overhead that a keyboard does not need on each
suggestion request. Sentence-ending punctuation (।, ॥, ., !, ?, and
ellipsis) clears the recent context so the next sentence backs off cleanly
instead of inheriting the previous sentence's last words. Native editor
integrations can also call AutosuggestContext::push_boundary() on explicit
line or paragraph breaks.
Corpus snapshot:
| Source | Documents | Sentences | Tokens |
|---|---|---|---|
| curated EPUB | 13 |
159,068 |
1,472,288 |
| Bangla Wikipedia | 169,736 |
4,297,804 |
54,560,642 |
| Bangla newspaper | 408,471 |
8,887,488 |
105,605,338 |
| total | 578,220 |
13,344,360 |
161,638,268 |
The vocabulary is built with min_frequency = 3, covers 148,611,832 corpus
tokens, and reaches 91.94% token coverage.
Mobile artifact replay-probe snapshot:
| Metric | Value |
|---|---|
| top-1 | 11.98% |
| top-3 | 20.08% |
| top-5 | 24.03% |
| top-10 | 29.67% |
| MRR | 17.17% |
These numbers are for reproducibility and regression checks only. Replay is measured against corpus rows used for counting, so it is not a held-out product accuracy claim.
Data Policy
The main repo owns source code, docs, tests, the playground, and generated GitHub Pages assets. Heavy data lives in data-only submodules mounted at the same paths used by the tools:
data/autocorrect -> obadh_autocorrect_dataset
data/autosuggest -> obadh_autosuggest_dataset
Those dataset repos may use Git LFS for corpora, TSVs, FSTs, and binary model
artifacts. They must not contain training/runtime code. GitHub Pages runtime
files under docs/ are real bytes, not LFS pointers, because Pages branch
deploys do not serve LFS objects as ordinary static assets.
Manual submodule recovery:
Data Artifacts
Runtime data is packaged outside the crates.io tarball. The crate stays small and auditable; data repos carry the large LFS artifacts:
| Runtime | Required artifacts |
|---|---|
| deterministic core | none |
| autocorrect | data/autocorrect/models/bn.fst |
| autocorrect loanwords | data/autocorrect/models/en_bn_loanwords.fst |
| browser autosuggest | data/autosuggest/models/ngram/autosuggest-ngram.bin |
| native autosuggest | data/autosuggest/models/ngram/autosuggest-ngram-c64.bin |
| neural next-word package | generator manifest plus Core ML or ONNX model |
The main repo pins exact data commits through submodules. A fresh checkout
should run ./init.sh, which initializes both submodules, resolves LFS files,
and verifies the required runtime artifacts are present. Future data updates
should land in the data repo first, then the main repo should update the
submodule pointer and validation docs in the same engine release commit.
For iOS, obadh-ios should bundle only the runtime artifacts it uses:
autocorrect FSTs, the c64 n-gram, the generator manifest, and a compiled Core
ML model. Corpora, raw TSVs, training checkpoints, and builder outputs should
not ship inside the keyboard target.
Web / WASM Usage
import init from './js/obadh_engine.js';
await ;
const engine = ;
console.log;
// আজি এ প্রভাতে রবির কর
Strict transliteration returns the original text unchanged when unsupported
characters are present. Use transliterate_lenient only when the caller
deliberately wants unsupported characters removed before transliteration.
For WASM autosuggest, suggestSession and suggestSessionCandidates use the
current live session context. commitTokenId, commitToken, and
commitUnknown advance that context and update the bounded personal overlay
when the committed word is eligible. exportPersonalSnapshot returns the bytes
that a host should persist for this user; importPersonalSnapshot restores
those bytes and validates that they belong to the loaded vocabulary.
clearSession clears only the recent context. clearPersonal clears the
learned local dictionary.
Local playground workflow:
./build.sh dev runs the Tailwind watcher plus the same lightweight www/
server. The playground server is intentionally outside the Rust crate dependency
graph.
Rust Library Usage
use ObadhEngine;
let engine = new;
let bangla = engine.transliterate;
assert_eq!;
For editor integrations, reuse buffers where possible:
use ;
let engine = new;
let mut units: = Vecnew;
engine.tokenize_phonetic_into;
engine.tokenize_phonetic_into;
Current Metrics
| Check | Result |
|---|---|
| transliteration sample average | 0.002815 ms |
| sample iterations | 100,000 |
| Bangla FST entries | 845,461 |
| Bangla FST bytes | 8,847,897 |
| English loanword keys | 1,776 |
| English loanword FST bytes | 89,427 |
| optimized WASM | about 280 KB |
| autosuggest ngram artifact | 25,195,978 bytes |
| autosuggest artifact fingerprint | 381a5b7821e7c187 |
| autosuggest c64 candidate artifact | 29,486,274 bytes |
| autosuggest c64 fingerprint | b311c36a29c4579b |
| autosuggest INT8 generator | 18,492,708 bytes |
| autosuggest Core ML generator package | 17,668,804 bytes |
| autosuggest vocab | 1,058,854 bytes |
| autosuggest native model load sample | ~1.10-2.16 ms |
| autosuggest native context lookup sample | ~0.185 us |
| autosuggest c64 candidate-input sample | ~0.83 us |
| autosuggest generator scored-union handoff | ~14.38 us release, personal-aware |
| autosuggest Core ML generator sample | ~459 us |
| autosuggest native session lookup sample | ~0.188 us |
| autosuggest native text context lookup sample | ~0.463 us |
Autocorrect CLI process timings are not reported as keyboard latency because process startup dominates those measurements. Keyboard-time performance should be measured inside loaded runtimes.
Project Layout
src/engine/ deterministic tokenizer/transliterator
src/definitions/ compiled rule tables
src/autocorrect/ FST candidate generation and ranking primitives
src/autosuggest/ ngram runtime, personal overlay, neural handoff
src/wasm/ WebAssembly bindings
src/bin/ CLI binaries
data/rules/ documented deterministic rule sources
data/autocorrect/ direct data submodule: lexicon TSVs and FSTs
data/autosuggest/ direct data submodule: corpus, vocab, models
tools/autocorrect/ corpus and loanword data utilities
tools/autosuggest/ sentence corpus, vocab, and ngram model utilities
www/ playground source
docs/ generated GitHub Pages distribution
tests/ regression suite
benches/ Criterion hot-path benchmarks
docs/ is generated by ./build.sh dist. Do not edit generated CSS, WASM, or
copied distribution files directly.
Release Checklist
For a tagged release, bump the Cargo/npm versions together, rebuild docs/,
commit source plus generated artifacts, push, publish the crate, then tag the
exact published commit.