rust-bert
Rust-native state-of-the-art Natural Language Processing models and pipelines. Port of Hugging Face's Transformers library, using the tch-rs crate and pre-processing from rust-tokenizers. Supports multi-threaded tokenization and GPU inference. This repository exposes the model base architecture, task-specific heads (see below) and ready-to-use pipelines. Benchmarks are available at the end of this document.
Get started with tasks including question answering, named entity recognition, translation, summarization, text generation, conversational agents and more in just a few lines of code:
let qa_model = new?;
let question = String from;
let context = String from;
let answers = qa_model.predict;
Output:
[Answer { score: 0.9976, start: 13, end: 21, answer: "Amsterdam" }]
The tasks currently supported include:
- Translation
- Summarization
- Multi-turn dialogue
- Zero-shot classification
- Sentiment Analysis
- Named Entity Recognition
- Part of Speech tagging
- Question-Answering
- Language Generation.
Sequence classification | Token classification | Question answering | Text Generation | Summarization | Translation | Masked LM | |
---|---|---|---|---|---|---|---|
DistilBERT | ✅ | ✅ | ✅ | ✅ | |||
MobileBERT | ✅ | ✅ | ✅ | ✅ | |||
DeBERTa | ✅ | ✅ | ✅ | ✅ | |||
DeBERTa (v2) | ✅ | ✅ | ✅ | ✅ | |||
FNet | ✅ | ✅ | ✅ | ✅ | |||
BERT | ✅ | ✅ | ✅ | ✅ | |||
RoBERTa | ✅ | ✅ | ✅ | ✅ | |||
GPT | ✅ | ||||||
GPT2 | ✅ | ||||||
GPT-Neo | ✅ | ||||||
BART | ✅ | ✅ | ✅ | ||||
Marian | ✅ | ||||||
MBart | ✅ | ✅ | |||||
M2M100 | ✅ | ||||||
Electra | ✅ | ✅ | |||||
ALBERT | ✅ | ✅ | ✅ | ✅ | |||
T5 | ✅ | ✅ | ✅ | ||||
XLNet | ✅ | ✅ | ✅ | ✅ | ✅ | ||
Reformer | ✅ | ✅ | ✅ | ✅ | |||
ProphetNet | ✅ | ✅ | |||||
Longformer | ✅ | ✅ | ✅ | ✅ | |||
Pegasus | ✅ |
Getting started
This library relies on the tch crate for bindings to the C++ Libtorch API. The libtorch library is required can be downloaded either automatically or manually. The following provides a reference on how to set-up your environment to use these bindings, please refer to the tch for detailed information or support.
Furthermore, this library relies on a cache folder for downloading pre-trained models.
This cache location defaults to ~/.cache/.rustbert
, but can be changed by setting the RUSTBERT_CACHE
environment variable. Note that the language models used by this library are in the order of the 100s of MBs to GBs.
Manual installation (recommended)
- Download
libtorch
from https://pytorch.org/get-started/locally/. This package requiresv1.11.0
: if this version is no longer available on the "get started" page, the file should be accessible by modifying the target link, for examplehttps://download.pytorch.org/libtorch/cu113/libtorch-shared-with-deps-1.11.0%2Bcu113.zip
for a Linux version with CUDA11. - Extract the library to a location of your choice
- Set the following environment variables
Linux:
Windows
$Env:LIBTORCH = "X:\path\to\libtorch"
$Env:Path += ";X:\path\to\libtorch\lib"
Automatic installation
Alternatively, you can let the build
script automatically download the libtorch
library for you.
The CPU version of libtorch will be downloaded by default. To download a CUDA version, please set the environment variable TORCH_CUDA_VERSION
to cu113
.
Note that the libtorch library is large (order of several GBs for the CUDA-enabled version) and the first build may therefore take several minutes to complete.
Ready-to-use pipelines
Based on Hugging Face's pipelines, ready to use end-to-end NLP pipelines are available as part of this crate. The following capabilities are currently available:
Disclaimer The contributors of this repository are not responsible for any generation from the 3rd party utilization of the pretrained systems proposed herein.
Extractive question answering from a given question and context. DistilBERT model fine-tuned on SQuAD (Stanford Question Answering Dataset)
let qa_model = new?;
let question = String from;
let context = String from;
let answers = qa_model.predict;
Output:
[Answer { score: 0.9976, start: 13, end: 21, answer: "Amsterdam" }]
Translation pipeline supporting a broad range of source and target languages. Leverages two main architectures for translation tasks:
- Marian-based models, for specific source/target combinations
- M2M100 models allowing for direct translation between 100 languages (at a higher computational cost and lower performance for some selected languages)
Marian-based pretrained models for the following language pairs are readily available in the library - but the user can import any Pytorch-based model for predictions
- English <-> French
- English <-> Spanish
- English <-> Portuguese
- English <-> Italian
- English <-> Catalan
- English <-> German
- English <-> Russian
- English <-> Chinese
- English <-> Dutch
- English <-> Swedish
- English <-> Arabic
- English <-> Hebrew
- English <-> Hindi
- French <-> German
For languages not supported by the proposed pretrained Marian models, the user can leverage a M2M100 model supporting direct translation between 100 languages (without intermediate English translation) The full list of supported languages is available in the crate documentation
use ;
Output:
Il s'agit d'une phrase à traduire
Abstractive summarization using a pretrained BART model.
let summarization_model = new?;
let input = ;
let output = summarization_model.summarize;
(example from: WikiNews)
Output:
"Scientists have found water vapour on K2-18b, a planet 110 light-years from Earth.
This is the first such discovery in a planet in its star's habitable zone.
The planet is not too hot and not too cold for liquid water to exist."
Conversation model based on Microsoft's DialoGPT. This pipeline allows the generation of single or multi-turn conversations between a human and a model. The DialoGPT's page states that
The human evaluation results indicate that the response generated from DialoGPT is comparable to human response quality under a single-turn conversation Turing test. (DialoGPT repository)
The model uses a ConversationManager
to keep track of active conversations and generate responses to them.
use ;
let conversation_model = new;
let mut conversation_manager = new;
let conversation_id = conversation_manager.create;
let output = conversation_model.generate_responses;
Example output:
"The Big Lebowski."
Generate language based on a prompt. GPT2 and GPT available as base models. Include techniques such as beam search, top-k and nucleus sampling, temperature setting and repetition penalty. Supports batch generation of sentences from several prompts. Sequences will be left-padded with the model's padding token if present, the unknown token otherwise. This may impact the results, it is recommended to submit prompts of similar length for best results
let model = new?;
let input_context_1 = "The dog";
let input_context_2 = "The cat was";
let generate_options = GenerateOptions ;
let output = model.generate;
Example output:
[
"The dog's owners, however, did not want to be named. According to the lawsuit, the animal's owner, a 29-year"
"The dog has always been part of the family. \"He was always going to be my dog and he was always looking out for me"
"The dog has been able to stay in the home for more than three months now. \"It's a very good dog. She's"
"The cat was discovered earlier this month in the home of a relative of the deceased. The cat\'s owner, who wished to remain anonymous,"
"The cat was pulled from the street by two-year-old Jazmine.\"I didn't know what to do,\" she said"
"The cat was attacked by two stray dogs and was taken to a hospital. Two other cats were also injured in the attack and are being treated."
]
Performs zero-shot classification on input sentences with provided labels using a model fine-tuned for Natural Language Inference.
let sequence_classification_model = new?;
let input_sentence = "Who are you voting for in 2020?";
let input_sequence_2 = "The prime minister has announced a stimulus package which was widely criticized by the opposition.";
let candidate_labels = &;
let output = sequence_classification_model.predict_multilabel;
Output:
[
[ Label { "politics", score: 0.972 }, Label { "public health", score: 0.032 }, Label {"economics", score: 0.006 }, Label {"sports", score: 0.004 } ],
[ Label { "politics", score: 0.975 }, Label { "public health", score: 0.0818 }, Label {"economics", score: 0.852 }, Label {"sports", score: 0.001 } ],
]
Predicts the binary sentiment for a sentence. DistilBERT model fine-tuned on SST-2.
let sentiment_classifier = new?;
let input = ;
let output = sentiment_classifier.predict;
(Example courtesy of IMDb)
Output:
[
Sentiment { polarity: Positive, score: 0.9981985493795946 },
Sentiment { polarity: Negative, score: 0.9927982091903687 },
Sentiment { polarity: Positive, score: 0.9997248985164333 }
]
Extracts entities (Person, Location, Organization, Miscellaneous) from text. BERT cased large model fine-tuned on CoNNL03, contributed by the MDZ Digital Library team at the Bavarian State Library. Models are currently available for English, German, Spanish and Dutch.
let ner_model = new?;
let input = ;
let output = ner_model.predict;
Output:
[
[
Entity { word: "Amy", score: 0.9986, label: "I-PER" }
Entity { word: "Paris", score: 0.9985, label: "I-LOC" }
],
[
Entity { word: "Paris", score: 0.9988, label: "I-LOC" }
Entity { word: "France", score: 0.9993, label: "I-LOC" }
]
]
Extracts Part of Speech tags (Noun, Verb, Adjective...) from text.
let pos_model = new?;
let input = ;
let output = pos_model.predict;
Output:
[
Entity { word: "My", score: 0.1560, label: "PRP" }
Entity { word: "name", score: 0.6565, label: "NN" }
Entity { word: "is", score: 0.3697, label: "VBZ" }
Entity { word: "Bob", score: 0.7460, label: "NNP" }
]
Benchmarks
For simple pipelines (sequence classification, tokens classification, question answering) the performance between Python and Rust is expected to be comparable. This is because the most expensive part of these pipeline is the language model itself, sharing a common implementation in the Torch backend. The End-to-end NLP Pipelines in Rust provides a benchmarks section covering all pipelines.
For text generation tasks (summarization, translation, conversation, free text generation), significant benefits can be expected (up to 2 to 4 times faster processing depending on the input and application). The article Accelerating text generation with Rust focuses on these text generation applications and provides more details on the performance comparison to Python.
Loading pretrained and custom model weights
The base model and task-specific heads are also available for users looking to expose their own transformer based models.
Examples on how to prepare the date using a native tokenizers Rust library are available in ./examples
for BERT, DistilBERT, RoBERTa, GPT, GPT2 and BART.
Note that when importing models from Pytorch, the convention for parameters naming needs to be aligned with the Rust schema. Loading of the pre-trained weights will fail if any of the model parameters weights cannot be found in the weight files.
If this quality check is to be skipped, an alternative method load_partial
can be invoked from the variables store.
Pretrained models are available on Hugging face's model hub and can be loaded using RemoteResources
defined in this library.
A conversion utility script is included in ./utils
to convert Pytorch weights to a set of weights compatible with this library. This script requires Python and torch
to be set-up, and can be used as follows:
python ./utils/convert_model.py path/to/pytorch_model.bin
where path/to/pytorch_model.bin
is the location of the original Pytorch weights.
Citation
If you use rust-bert
for your work, please cite End-to-end NLP Pipelines in Rust:
Acknowledgements
Thank you to Hugging Face for hosting a set of weights compatible with this Rust library. The list of ready-to-use pretrained models is listed at https://huggingface.co/models?filter=rust.