Struct rust_bert::pipelines::token_classification::TokenClassificationConfig

pub struct TokenClassificationConfig {
    pub model_type: ModelType,
    pub model_resource: Resource,
    pub config_resource: Resource,
    pub vocab_resource: Resource,
    pub merges_resource: Option<Resource>,
    pub lower_case: bool,
    pub strip_accents: Option<bool>,
    pub add_prefix_space: Option<bool>,
    pub device: Device,
    pub label_aggregation_function: LabelAggregationOption,
}

Configuration for TokenClassificationModel

Contains information regarding the model to load and device to place the model on.

Fields

model_type: ModelType

Model type

model_resource: Resource

Model weights resource (default: pretrained BERT model on CoNLL)

config_resource: Resource

Config resource (default: pretrained BERT model on CoNLL)

vocab_resource: Resource

Vocab resource (default: pretrained BERT model on CoNLL)

merges_resource: Option<Resource>

Merges resource (default: pretrained BERT model on CoNLL)

lower_case: bool

Automatically lower case all input upon tokenization (assumes a lower-cased model)

strip_accents: Option<bool>

Flag indicating if the tokenizer should strip accents (normalization). Only used for BERT / ALBERT models

add_prefix_space: Option<bool>

Flag indicating if the tokenizer should add a white space before each tokenized input (needed for some Roberta models)

device: Device

Device to place the model on (default: CUDA/GPU when available)

label_aggregation_function: LabelAggregationOption

Sub-tokens aggregation method (default: LabelAggregationOption::First)

Implementations

impl TokenClassificationConfig

pub fn new(
    model_type: ModelType,
    model_resource: Resource,
    config_resource: Resource,
    vocab_resource: Resource,
    merges_resource: Option<Resource>,
    lower_case: bool,
    strip_accents: impl Into<Option<bool>>,
    add_prefix_space: impl Into<Option<bool>>,
    label_aggregation_function: LabelAggregationOption
) -> TokenClassificationConfig

Instantiate a new token classification configuration of the supplied type.

Arguments

• model_type - ModelType indicating the model type to load (must match with the actual data to be loaded!)
• model - The Resource pointing to the model to load (e.g. model.ot)
• config - The `Resource' pointing to the model configuration to load (e.g. config.json)
• vocab - The `Resource' pointing to the tokenizers' vocabulary to load (e.g. vocab.txt/vocab.json)
• vocab - An optional Resource tuple (Option<Resource>) pointing to the tokenizers' merge file to load (e.g. merges.txt), needed only for Roberta.
• lower_case - A `bool' indicating whether the tokenizer should lower case all input (in case of a lower-cased model)

Trait Implementations

impl Default for TokenClassificationConfig

fn default() -> TokenClassificationConfig

Provides a default CoNLL-2003 NER model (English)