ModernBert

Represents models based on Modern Bert.

Value

Does return a new object of this class.

References

Warner, B., Chaffin, A., Clavie, B., Weller, O., Hallstroem, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J. & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. doi:10.48550/arXiv.2412.13663

See also

Other Base Model: BaseModelAlbert, BaseModelBert, BaseModelDebertaV2, BaseModelDistilBERT, BaseModelEuroBert, BaseModelFunnel, BaseModelMPNet, BaseModelRoberta, BaseModelRobertaXML, BaseModelXmod

Super classes

AIFEMaster -> AIFEBaseModel -> BaseModelCore -> BaseModelModernBert

Methods

Public methods

• BaseModelModernBert$configure()

• BaseModelModernBert$clone()

Inherited methods

• AIFEMaster$get_all_fields()
• AIFEMaster$get_documentation_license()
• AIFEMaster$get_ml_framework()
• AIFEMaster$get_model_config()
• AIFEMaster$get_model_description()
• AIFEMaster$get_model_info()
• AIFEMaster$get_model_license()
• AIFEMaster$get_package_versions()
• AIFEMaster$get_private()
• AIFEMaster$get_publication_info()
• AIFEMaster$get_sustainability_data()
• AIFEMaster$is_configured()
• AIFEMaster$is_trained()
• AIFEMaster$set_documentation_license()
• AIFEMaster$set_model_description()
• AIFEMaster$set_model_license()
• BaseModelCore$calc_flops_architecture_based()
• BaseModelCore$count_parameter()
• BaseModelCore$create_from_hf()
• BaseModelCore$estimate_sustainability_inference_fill_mask()
• BaseModelCore$fill_mask()
• BaseModelCore$get_final_size()
• BaseModelCore$get_flops_estimates()
• BaseModelCore$get_max_seq_len()
• BaseModelCore$get_model()
• BaseModelCore$get_model_type()
• BaseModelCore$get_n_layers()
• BaseModelCore$get_special_tokens()
• BaseModelCore$get_tokenizer_statistics()
• BaseModelCore$load_from_disk()
• BaseModelCore$plot_training_history()
• BaseModelCore$print()
• BaseModelCore$save()
• BaseModelCore$set_publication_info()
• BaseModelCore$train()

---

BaseModelModernBert$configure()

Configures a new object of this class. Please ensure that your chosen configuration comply with the following guidelines:

• hidden_size is a multiple of num_attention_heads.

• hidden_size/num_attention_heads must be a multiple of 2.

• global_attn_every_n_layers is equal or smaller as num_hidden_layers.

Usage

BaseModelModernBert$configure(
  tokenizer,
  max_position_embeddings = 512L,
  hidden_size = 768L,
  num_hidden_layers = 12L,
  num_attention_heads = 12L,
  global_attn_every_n_layers = 3L,
  intermediate_size = 3072L,
  hidden_activation = "GELU",
  embedding_dropout = 0.1,
  mlp_dropout = 0.1,
  attention_dropout = 0.1
)

Arguments

tokenizer

TokenizerBase Tokenizer for the model.

max_position_embeddings

int Number of maximum position embeddings. This parameter also determines the maximum length of a sequence which can be processed with the model. Allowed values:

\(10 <= x <= 4048\)

hidden_size

int Number of neurons in each layer. This parameter determines the dimensionality of the resulting text embedding. Allowed values:

\(1 <= x <= 2048\)

num_hidden_layers

int Number of hidden layers. Allowed values:

\(1 <= x \)

num_attention_heads

int determining the number of attention heads for a self-attention layer. Only relevant if attention_type='MultiHead' Allowed values:

\(0 <= x \)

global_attn_every_n_layers

int Number determining to use a global attention every x-th layer. Allowed values:

\(2 <= x <= 36\)

intermediate_size

int determining the size of the projection layer within a each transformer encoder. Allowed values:

\(1 <= x \)

hidden_activation

string Name of the activation function. Allowed values:

• 'GELU'

• 'relu'

• 'silu'

• 'gelu_new'

embedding_dropout

double Dropout chance for the embeddings. Allowed values:

\(0 <= x <= 0.6\)

mlp_dropout

double Dropout rate for the mlp layer. Allowed values:

\(0 <= x <= 0.6\)

attention_dropout

double Ratio of dropout for attention probabilities. Allowed values:

\(0 <= x <= 0.6\)

Returns

Does nothing return.

---

BaseModelModernBert$clone()

The objects of this class are cloneable with this method.

Usage

BaseModelModernBert$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.