Obadh Engine

Obadh is a deterministic Roman-to-Bangla transliteration engine and runtime SDK for Bangla typing systems. The core transliterator is deliberately rule-based: Roman input becomes Bengali because an Obadh rule says so, not because a dictionary or model guessed a word.

Obadh is an Avro successor in ambition, but not an Avro clone. The deterministic layer has its own deliberate Roman contract. Autocorrection, next-word autosuggestion, personalization, and neural context models live above that core as separate runtime layers.

Live playground: https://sayom.me/obadh_engine/

Index

• Install
• SDK Shape
• Architecture
• Deterministic Core
• Autocorrect
• Autosuggest
• Runtime Data
• WASM Playground
• Rust Usage
• Performance Snapshot
• Project Layout
• Release Checklist

Install

Use the Rust library crate for native integrations:

obadh_engine = "0.6.0"

The default feature set is empty. Native downstreams, including iOS wrappers, do not pay for CLI tooling or browser/WASM dependencies.

Optional features:

# CLI tools and artifact builders.
obadh_engine = { version = "0.6.0", features = ["cli"] }

# Browser/WASM bindings for the playground.
obadh_engine = { version = "0.6.0", features = ["wasm"] }

Repository setup for development:

git clone https://github.com/nsssayom/obadh_engine.git
cd obadh_engine
./init.sh

init.sh initializes the data submodules, resolves Git LFS objects, verifies required runtime artifacts, and installs playground dependencies.

Native prerequisites not installed by init.sh:

rustup toolchain install 1.89.0
brew install wasm-pack binaryen

Common developer commands:

cargo run --features cli --bin obadh -- 'aji e probhate robir kor'
cargo test
cargo test --features cli
./build.sh dev
./build.sh dist

SDK Shape

obadh_engine is a Rust library crate. Its default public API is the native SDK surface:

• deterministic transliteration and tokenization
• active-word autocorrect primitives
• next-word autosuggest runtime
• bounded personal autosuggest snapshots
• native scorer/generator handoff helpers for Core ML/ONNX next-word models
• validated open-vocabulary generated-word candidates for native generators

The crate ships source code, tests, and small deterministic rule fixtures. It does not bundle large runtime model artifacts. Those artifacts are versioned in data-only repositories and pinned by this repo as submodules.

Feature policy:

Feature	Purpose	Pulls in
default	native SDK surface	no CLI or browser dependencies
cli	command-line tools and corpus/artifact builders	clap, ZIP/EPUB helpers
wasm	browser playground bindings	wasm-bindgen, web-sys

Architecture

flowchart LR
  R[Roman input] --> D[Deterministic Obadh core]
  D --> B[Bengali baseline]
  B --> C[Autocorrect FST]
  C --> W[Active-word candidates]
  T[Committed Bengali text] --> N[Next-word n-gram runtime]
  N --> P[Personal overlay]
  N --> G[Neural handoff]
  G --> O[Validated open-vocab text]
  P --> S[Suggestion ribbon]
  G --> S
  O --> S

The typing path is split intentionally:

1. The deterministic core converts the active Roman token to a Bengali baseline.
2. Autocorrect searches compact Bengali/loanword FST artifacts for active-word alternatives.
3. Once a Bengali word is committed, autosuggest uses committed Bengali context to produce next-word candidates.
4. The personal overlay adjusts next-word suggestions locally without mutating global model artifacts.
5. Native platforms can pass fixed buffers to Core ML/ONNX and return model scores, token IDs, or generated Bengali word candidates to Rust.
6. Rust validates, deduplicates, scores, and merges all suggestion channels.

Deterministic Core

The core is dictionary-free and must stay that way.

• No whole-word compatibility table in transliteration.
• No hidden aliases just because another keyboard accepts them.
• No ML or corpus dependency on the Roman-to-Bangla hot path.
• Rule aliases need an Obadh-specific phonetic, orthographic, or ergonomic reason.
• Spelling correction and ranking belong above the deterministic layer.

Representative deliberate signals:

Roman Signal	Bengali Rule Intent
o	inherent অ / lowercase cluster separator
a / A	visible আ / া, including before clusters
I, U, O	long ঈ / ঊ and ও
aY / AY	অ্যা / ্যা, e.g. aYp -> অ্যাপ
ng, M, Ng	anusvar / explicit anusvar escape / velar nasal
ngg, nggh	ঙ্গ / ঙ্ঘ shorthand
jNG, jn, gg	জ্ঞ paths
NGj, nj, nJ	ঞ্জ paths
rr + cluster	reph over a valid cluster
rZy / rZY	non-conjunct ZWNJ-separated র‌্য form
y / Y	য-ফলা after a consonant base (productive): ply -> প্ল্য, plYan -> প্ল্যান; standalone য়
w	ব-ফলা after a consonant (kw -> ক্ব); standalone ওয় glide (waTar -> ওয়াটার)
q, qq	ক (qaf): iraq -> ইরাক; qq -> চন্দ্রবিন্দু ঁ, resolved ahead of q by longest match (baqq -> বাঁ)
x	ক্স: box -> বক্স, fix -> ফিক্স
,,	explicit hasant / conjunct boundary command
t``	খণ্ড ত / ৎ
T``	খণ্ড ত / ৎ
^	chandrabindu
:	bisarga
.	danda, while decimal periods stay ASCII periods
$	taka sign

Rule sources live under data/rules/ and are checked by tests.

Autocorrect

Autocorrect is an active-word layer above the deterministic core. Obadh first produces a Bengali baseline. The autocorrect runtime then retrieves valid lexicon candidates from compact FST artifacts and ranks them through bounded, explainable channels.

AutocorrectEngine::decide returns an AutocorrectDecision { input, candidates, replacement }. The replacement field is the auto-insert gate: replacement.is_some() means a correction is confident enough to apply without asking (a lexicon edit that beats the baseline by a margin — never a prefix or skeleton guess). suggest is a convenience wrapper that returns only candidates and discards this gate, so downstreams gating auto-insert should call decide. Note that a request built from autocorrect_request(roman) carries the roman buffer, which suppresses auto-replacement by default — set AutocorrectConfig::auto_replace_roman_input for true auto-insert.

Runtime channels:

• exact deterministic baseline lookup
• Obadh-aware Roman repair, such as missing lowercase o separators
• weighted Bengali edit lookup over the FST
• narrow vowel-length and nasal-mark rescue paths
• exact-stem suffix completion
• curated English-loanword exact and bounded fuzzy lookup
• bounded prefix completion

Runtime code does not parse CSV, TSV, EPUB, JSON, or heap-resident tries. Native tools can memory-map the FST; WASM loads the same compact bytes.

Inspect artifacts:

cargo run --release --features cli --bin obadh-autocorrect -- inspect-fst-lexicon \
  --input data/autocorrect/models/bn.fst --pretty

cargo run --release --features cli --bin obadh-autocorrect -- inspect-loanword-lexicon \
  --input data/autocorrect/models/en_bn_loanwords.fst --pretty

Probe the production FST path:

cargo run --release --features cli --bin obadh-autocorrect -- suggest-fst \
  --lexicon data/autocorrect/models/bn.fst \
  --loanwords data/autocorrect/models/en_bn_loanwords.fst \
  --input sushil \
  --max-distance 2 \
  --max-candidates 512 \
  --max-prefix-candidates 24 \
  --response-candidates 8 \
  --pretty

Autosuggest

Autosuggest is the next-word layer above committed Bengali text. It does not run while a Roman token is active and does not replace active-word autocorrect.

The static runtime is a compact n-gram candidate generator with suffix backoff. The browser playground uses the compact c16 artifact. Native integrations can use the c64 candidate artifact plus an optional scorer or generator model.

The neural path is intentionally bounded and platform-runtime agnostic. A scorer model ranks a static candidate pool. A generator model can return known-vocab token IDs and, in the native SDK, open-vocabulary Bengali word candidates. Rust performs validation, deduplication, weighting, and final merge. The model does not replace lexicon retrieval and does not run on every Roman keystroke.

Open-vocabulary candidates are not dictionary-bound, but they are not unchecked free-form text either. The SDK accepts generated Bengali words only after a cheap validator confirms script, word shape, length, mark order, repetition, and confidence policy. Accepted generated text can be committed immediately and later learned by the personal overlay as local OOV text.

Personal autosuggest has two lifetimes:

• Session context: recent committed words in the current editor flow; clear at editor/session boundaries.
• Personal dictionary: a bounded local overlay; persists only if the host exports and stores the compact snapshot.

Obadh owns the snapshot format, vocabulary-fingerprint validation, bounded learning rules, and merge behavior. Downstream keyboards own storage policy, privacy controls, and lifecycle timing. A missing or fingerprint-mismatched snapshot must be treated as an empty personal dictionary.

For keyboard integrations:

• keep an AutosuggestContext as words are committed
• resolve vocabulary IDs once and prefer token-ID APIs on the hot path
• call suggest_ids_for_context_into with reused buffers
• use AutosuggestSession when personal overlay behavior is needed
• use AutosuggestScorerSession for a cheaper candidate-ranking model
• use AutosuggestGeneratorSession for known-token or open-vocab generation
• pass generated text through accept_open_vocab_text_outputs
• read final native candidates from the unified open-vocab merge path
• persist with write_personal_snapshot_into
• restore with import_personal_snapshot
• call push_boundary() on sentence/editor boundaries

Validate the packaged generator:

cargo run --release --features cli --bin obadh-autosuggest -- validate-generator \
  --model data/autosuggest/models/ngram/autosuggest-ngram-c64.bin \
  --manifest data/autosuggest/models/neural/autosuggest-generator-gru256-topk128-c64-balanced.manifest.json \
  --pretty

Benchmark an n-gram artifact:

cargo run --release --features cli --bin obadh-autosuggest -- bench \
  --model data/autosuggest/models/ngram/autosuggest-ngram.bin \
  --context 'আমি আজ' \
  --context 'বাংলাদেশের মানুষ' \
  --mode context \
  --iterations 200000 \
  --pretty

Runtime Data

Large runtime data is not published inside the crates.io tarball. The crate stays small and auditable; data-only repositories carry corpora, TSVs, FSTs, n-gram artifacts, and neural packages.

Path	Data repo
data/autocorrect	nsssayom/obadh_autocorrect_dataset
data/autosuggest	nsssayom/obadh_autosuggest_dataset

Manual recovery:

git submodule update --init --recursive -- data/autocorrect data/autosuggest
git -C data/autocorrect lfs pull
git -C data/autosuggest lfs pull

Runtime artifact map:

Runtime	Required artifacts
deterministic core	none
autocorrect	data/autocorrect/models/bn.fst
autocorrect loanwords	data/autocorrect/models/en_bn_loanwords.fst
browser autosuggest	data/autosuggest/models/ngram/autosuggest-ngram.bin
native autosuggest	data/autosuggest/models/ngram/autosuggest-ngram-c64.bin
neural next-word package	generator manifest plus Core ML or ONNX model

Fresh source checkouts should use ./init.sh. Runtime applications should pin an engine crate version and a compatible data commit/tag, then bundle only the artifacts needed by that target.

For iOS, the downstream obadh-ios package should bundle autocorrect FSTs, the c64 n-gram, the generator manifest, and a compiled Core ML model. Corpora, raw TSVs, training checkpoints, and builder outputs should not ship inside the keyboard extension.

Corpus snapshot used by the current autosuggest package:

Source	Documents	Sentences	Tokens
curated EPUB	13	159,068	1,472,288
Bangla Wikipedia	169,736	4,297,804	54,560,642
Bangla newspaper	408,471	8,887,488	105,605,338
total	578,220	13,344,360	161,638,268

The autosuggest vocabulary uses 32,768 tokens, covers 148,611,832 corpus tokens, and reaches 91.94% token coverage.

WASM Playground

The playground is a browser testing surface for the same core runtime. Build it with the explicit wasm feature:

./build.sh wasm
npm --prefix www run serve

./build.sh dev runs the Tailwind watcher plus the lightweight www/ server. The dev server is an npm-only playground tool and is outside the Rust crate dependency graph.

WASM usage:

import init, { ObadhaWasm } from './js/obadh_engine.js';

await init();
const engine = new ObadhaWasm();

console.log(engine.transliterate('aji e probhate robir kor'));
// আজি এ প্রভাতে রবির কর

WASM autosuggest exposes the same session/personal concepts through commitTokenId, commitToken, commitUnknown, suggestSession, suggestSessionCandidates, exportPersonalSnapshot, and importPersonalSnapshot.

Rust Usage

Basic transliteration:

use obadh_engine::ObadhEngine;

let engine = ObadhEngine::new();
let bangla = engine.transliterate("aji e probhate robir kor");

assert_eq!(bangla, "আজি এ প্রভাতে রবির কর");

Reusable editor buffers:

use obadh_engine::{ObadhEngine, PhoneticUnit};

let engine = ObadhEngine::new();
let mut units: Vec<PhoneticUnit> = Vec::new();

engine.tokenize_phonetic_into("rrkSh", &mut units);
engine.tokenize_phonetic_into("praNer", &mut units);

Strict transliteration returns the original text unchanged when unsupported characters are present. Use transliterate_lenient only when the caller deliberately wants unsupported characters removed before transliteration.

Performance Snapshot

Check	Result
transliteration sample average	0.002815 ms
Bangla FST entries	845,461
Bangla FST bytes	8,847,897
English loanword keys	1,776
English loanword FST bytes	89,427
optimized WASM	about 280 KB
autosuggest n-gram artifact	25,195,978 bytes
autosuggest c64 candidate artifact	29,486,274 bytes
autosuggest INT8 generator	18,492,708 bytes
autosuggest Core ML generator package	17,668,804 bytes
autosuggest native context lookup sample	~0.185 us
autosuggest c64 candidate-input sample	~0.83 us
autosuggest generator scored-union handoff	~14.38 us release, personal-aware
autosuggest Core ML generator sample	~459 us

Autosuggest package quality snapshot:

Path	top-1 all	top-5 all	top-10 all
static c64 pool	16.84%	31.34%	37.99%
scored-union GRU256	16.84%	32.89%	39.42%

Replay and held-out metrics are regression signals for runtime packaging, not claims of final keyboard product accuracy. Keyboard-time performance should be measured inside loaded platform runtimes, not from CLI process timings.

Project Layout

src/engine/                 deterministic tokenizer/transliterator
src/definitions/            compiled rule tables
src/autocorrect/            FST candidate generation and ranking primitives
src/autosuggest/            n-gram runtime, personal overlay, neural handoff
src/wasm/                   WebAssembly bindings
src/bin/                    CLI binaries
data/rules/                 documented deterministic rule sources
data/autocorrect/           data submodule: lexicon TSVs and FSTs
data/autosuggest/           data submodule: corpus, vocab, models
tools/autocorrect/          corpus and loanword data utilities
tools/autosuggest/          sentence corpus, vocab, and model utilities
www/                        playground source
docs/                       generated GitHub Pages distribution
tests/                      regression suite
benches/                    Criterion hot-path benchmarks

docs/ is generated by ./build.sh dist. Do not edit generated CSS, WASM, or copied distribution files directly.

Release Checklist

cargo test
cargo test --features cli
cargo check --target wasm32-unknown-unknown --no-default-features --features wasm --lib
cargo bench --bench hot_path --no-run
cargo publish --dry-run
./build.sh dist
git status --short

For a tagged release, bump the Cargo/npm versions together, rebuild docs/, commit source plus generated artifacts, push, publish the crate, then tag the exact published commit.