Whatlang

Natural language detection in Rust.

Features

• Support more than 50 languages
• 100% written in Rust
• No external dependencies
• Super fast
• Recognizes not only a language, but also a script (Latin, Cyrillic, etc)

Get started

The library is still in active development. Here is the short example how to use it:

Add to you Cargo.toml:

[dependencies]

whatlang = "*"

In you program:

extern crate whatlang;

use whatlang::{detect_lang, Lang, Script, Query};

fn main() {
    let text = "Guten Abend, meine Damen und Herren!";
    let query = Query::new(text);
    let result = detect_lang(query).unwrap();
    assert_eq!(result.lang, Lang::Deu);
    assert_eq!(result.lang.to_code(), "deu");
    assert_eq!(result.script, Script::Latin);
}

Blacklist

Your can blacklist undesired languages, passing a vector. In the example blow English and Spanish will be ignored:

let list = [Lang::Eng, Lang::Spa];
let query = Query::new(&text).blacklist(&list);

Whitelist

In similar way, you can whitelist specified languages. In this example, the library will recognize only Esperanto and Russian. Note, if it detects a script that is different from Latin(Esperanto) or Cyrillic(Russian), e.g. Greek, it will return None.

let list = [Lang::Epo, Lang::Rus];
let query = Query::new(&text).whitelist(&list);

Roadmap

• Support 100 most popular languages
• Allow to specify blacklist for Query
• Allow to specify whitelist for Query
• Improve README example
• Create demo application
• Provide some metrics about reliability in Result struct
• Write doc for public structurs and functions
• Tune performance
• Support syntax sugar:

  // Get result
  let result = whatlang::new(&text).detect();

  // Same with blacklist/whitelist, getting directly Lang
  let lang = whatlang::new(&text).blacklist(LIST).detect_lang();

Supported languages

License

MIT

Acknowledgments

• Thanks Franc JS for trigrams dataset.

Contributors

• greyblake Potapov Sergey - creator, maintainer.