Xmod-Transformer

Represents models based on Xmod

Xmod models support different languages. Before starting to work with this model set the correct language with the method set_default_language. You receive a list with all supported languages by calling get_supported_languages.

Value

Does return a new object of this class.

References

Pfeiffer, J., Goyal, N., Lin, X. V., Li, X., Cross, J., Riedel, S., & Artetxe, M. (2022). Lifting the Curse of Multilinguality by Pre-training Modular Transformers. arXiv. doi:10.48550/ARXIV.2205.06266

See also

Other Base Model: BaseModelAlbert, BaseModelBert, BaseModelDebertaV2, BaseModelDistilBERT, BaseModelEuroBert, BaseModelFunnel, BaseModelMPNet, BaseModelModernBert, BaseModelRoberta, BaseModelRobertaXML

Super classes

AIFEMaster -> AIFEBaseModel -> BaseModelCore -> BaseModelXmod

Methods

Public methods

• BaseModelXmod$configure()

• BaseModelXmod$get_supported_languages()

• BaseModelXmod$set_default_language()

• BaseModelXmod$get_default_language()

• BaseModelXmod$print()

• BaseModelXmod$clone()

Inherited methods

• AIFEMaster$get_all_fields()
• AIFEMaster$get_documentation_license()
• AIFEMaster$get_ml_framework()
• AIFEMaster$get_model_config()
• AIFEMaster$get_model_description()
• AIFEMaster$get_model_info()
• AIFEMaster$get_model_license()
• AIFEMaster$get_package_versions()
• AIFEMaster$get_private()
• AIFEMaster$get_publication_info()
• AIFEMaster$get_sustainability_data()
• AIFEMaster$is_configured()
• AIFEMaster$is_trained()
• AIFEMaster$set_documentation_license()
• AIFEMaster$set_model_description()
• AIFEMaster$set_model_license()
• BaseModelCore$calc_flops_architecture_based()
• BaseModelCore$count_parameter()
• BaseModelCore$create_from_hf()
• BaseModelCore$estimate_sustainability_inference_fill_mask()
• BaseModelCore$fill_mask()
• BaseModelCore$get_final_size()
• BaseModelCore$get_flops_estimates()
• BaseModelCore$get_max_seq_len()
• BaseModelCore$get_model()
• BaseModelCore$get_model_type()
• BaseModelCore$get_n_layers()
• BaseModelCore$get_special_tokens()
• BaseModelCore$get_tokenizer_statistics()
• BaseModelCore$load_from_disk()
• BaseModelCore$plot_training_history()
• BaseModelCore$save()
• BaseModelCore$set_publication_info()
• BaseModelCore$train()

---

BaseModelXmod$configure()

Configures a new object of this class. Please ensure that your chosen configuration comply with the following guidelines:

• hidden_size is a multiple of num_attention_heads.

Usage

BaseModelXmod$configure(
  tokenizer,
  languages = c("eng", "deu"),
  default_language = "deu",
  max_position_embeddings = 512L,
  hidden_size = 768L,
  num_hidden_layers = 12L,
  num_attention_heads = 12L,
  intermediate_size = 3072L,
  hidden_act = "GELU",
  hidden_dropout_prob = 0.1,
  attention_probs_dropout_prob = 0.1
)

Arguments

tokenizer

TokenizerBase Tokenizer for the model.

languages

string vector of strings representing the Language codes. Allowed values:

any

default_language

string string representing the language code of the default language. Allowed values:

any

max_position_embeddings

int Number of maximum position embeddings. This parameter also determines the maximum length of a sequence which can be processed with the model. Allowed values:

\(10 <= x <= 4048\)

hidden_size

int Number of neurons in each layer. This parameter determines the dimensionality of the resulting text embedding. Allowed values:

\(1 <= x <= 2048\)

num_hidden_layers

int Number of hidden layers. Allowed values:

\(1 <= x \)

num_attention_heads

int determining the number of attention heads for a self-attention layer. Only relevant if attention_type='MultiHead' Allowed values:

\(0 <= x \)

intermediate_size

int determining the size of the projection layer within a each transformer encoder. Allowed values:

\(1 <= x \)

hidden_act

string Name of the activation function. Allowed values:

• 'GELU'

• 'relu'

• 'silu'

• 'gelu_new'

hidden_dropout_prob

double Ratio of dropout. Allowed values:

\(0 <= x <= 0.6\)

attention_probs_dropout_prob

double Ratio of dropout for attention probabilities. Allowed values:

\(0 <= x <= 0.6\)

Returns

Does nothing return.

---

BaseModelXmod$get_supported_languages()

Get the supported languages of the model.

Usage

BaseModelXmod$get_supported_languages()

Returns

Returns a vector of strings that represent the supported language codes.

---

BaseModelXmod$set_default_language()

Set the default language of the model.

Usage

BaseModelXmod$set_default_language(language_code)

Arguments

language_code

string Language code to use as default.

Returns

Does nothing return.

---

BaseModelXmod$get_default_language()

Get the default language of the model.

Usage

BaseModelXmod$get_default_language()

Returns

Returns a string representing the default language.

---

BaseModelXmod$print()

Print method for classifiers.

Usage

BaseModelXmod$print()

Returns

Prints a short description of the object.

---

BaseModelXmod$clone()

The objects of this class are cloneable with this method.

Usage

BaseModelXmod$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.