languagedetect
Small, dependency-light language detection engine in Rust. no_std-compatible
runtime (library/), plus a training tool (creator/) that builds compact
models.
- Crate:
edgevane_langdetect(lib nameedgevanelang) - Formats:
EVLD(single language DB),EVLC(combined multi-language DB) - Signals: char uni/bi/trigrams, token uni/bigrams + TF-IDF, prefixes/suffixes, char position, class transitions, word/sentence length distributions
- Unicode folding in
normalize.rs, FNV-1a hashing inhash.rs - C API (
evld_load/evld_classify/evld_free) for embedding
Layout
library/— runtime + trainer engine (Engine::ingest,finish,encode_combined,LangDb,CombinedDb,classify,classify_top, C API)creator/— CLI that trains.evldper language and packs.evlc
Use (Rust)
use ;
// combined.chunks / combined.dbs depending on model.rs API
let scores = classify;
let top = &scores;
println!;
Train a single language:
use ;
use BTreeMap;
let mut eng = new;
eng.ingest;
eng.ingest;
let idf = new; // real IDF comes from multi-language DF pass
let evld = finish;
Use (C)
void *h = ;
EvldScore out;
int n = ;
;
Build / test
# no_std check
Model format
Header (256 B): magic EVLD/EVLC, version, lang code, flags
(FLAG_LOW_RESOURCE), doc/char/token counts, BM25 params, section table with
per-section CRC32. Sections: char n-grams, token n-grams + IDF, affixes,
CharPos, ClassTrans, length histograms. See library/src/format.rs.
License
MIT — see library/LICENSE-MIT.