DistilBERT

Represents models based on DistilBERT.

Value

Does return a new object of this class.

References

Asnh, V., Debut, L., Chaumond, J. & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. doi:10.48550/arXiv.1910.01108

See also

Other Base Model: BaseModelAlbert, BaseModelBert, BaseModelDebertaV2, BaseModelEuroBert, BaseModelFunnel, BaseModelMPNet, BaseModelModernBert, BaseModelRoberta, BaseModelRobertaXML, BaseModelXmod

Super classes

AIFEMaster -> AIFEBaseModel -> BaseModelCore -> BaseModelDistilBERT

Methods

Public methods

• BaseModelDistilBERT$configure()

• BaseModelDistilBERT$get_n_layers()

• BaseModelDistilBERT$get_final_size()

• BaseModelDistilBERT$clone()

Inherited methods

• AIFEMaster$get_all_fields()
• AIFEMaster$get_documentation_license()
• AIFEMaster$get_ml_framework()
• AIFEMaster$get_model_config()
• AIFEMaster$get_model_description()
• AIFEMaster$get_model_info()
• AIFEMaster$get_model_license()
• AIFEMaster$get_package_versions()
• AIFEMaster$get_private()
• AIFEMaster$get_publication_info()
• AIFEMaster$get_sustainability_data()
• AIFEMaster$is_configured()
• AIFEMaster$is_trained()
• AIFEMaster$set_documentation_license()
• AIFEMaster$set_model_description()
• AIFEMaster$set_model_license()
• BaseModelCore$calc_flops_architecture_based()
• BaseModelCore$count_parameter()
• BaseModelCore$create_from_hf()
• BaseModelCore$estimate_sustainability_inference_fill_mask()
• BaseModelCore$fill_mask()
• BaseModelCore$get_flops_estimates()
• BaseModelCore$get_max_seq_len()
• BaseModelCore$get_model()
• BaseModelCore$get_model_type()
• BaseModelCore$get_special_tokens()
• BaseModelCore$get_tokenizer_statistics()
• BaseModelCore$load_from_disk()
• BaseModelCore$plot_training_history()
• BaseModelCore$print()
• BaseModelCore$save()
• BaseModelCore$set_publication_info()
• BaseModelCore$train()

---

BaseModelDistilBERT$configure()

Configures a new object of this class. Please ensure that your chosen configuration comply with the following guidelines:

• hidden_dim is a multiple of n_heads

Usage

BaseModelDistilBERT$configure(
  tokenizer,
  max_position_embeddings = 512L,
  dim = 768L,
  n_layers = 12L,
  n_heads = 12L,
  hidden_dim = 3072L,
  activation = "GELU",
  dropout = 0.1,
  attention_dropout = 0.1
)

Arguments

tokenizer

TokenizerBase Tokenizer for the model.

max_position_embeddings

int Number of maximum position embeddings. This parameter also determines the maximum length of a sequence which can be processed with the model. Allowed values:

\(10 <= x <= 4048\)

dim

int Number of neurons in each layer. This parameter determines the dimensionality of the resulting text embedding. Allowed values:

\(1 <= x <= 2048\)

n_layers

int Number of hidden layers. Allowed values:

\(1 <= x \)

n_heads

int determining the number of attention heads for a self-attention layer. Only relevant if attention_type='MultiHead' Allowed values:

\(0 <= x \)

hidden_dim

int determining the size of the projection layer within a each transformer encoder. Allowed values:

\(1 <= x \)

activation

string Name of the activation function. Allowed values:

• 'GELU'

• 'relu'

• 'silu'

• 'gelu_new'

dropout

double Ratio of dropout. Allowed values:

\(0 <= x <= 0.6\)

attention_dropout

double Ratio of dropout for attention probabilities. Allowed values:

\(0 <= x <= 0.6\)

Returns

Does nothing return.

---

BaseModelDistilBERT$get_n_layers()

Number of layers.

Usage

BaseModelDistilBERT$get_n_layers()

Returns

Returns an int describing the number of layers available for embedding.

---

BaseModelDistilBERT$get_final_size()

Size of the final layer.

Usage

BaseModelDistilBERT$get_final_size()

Returns

Returns an int describing the number of dimensions of the last hidden layer.

---

BaseModelDistilBERT$clone()

The objects of this class are cloneable with this method.

Usage

BaseModelDistilBERT$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.