A02 Supported Base Models

All models created with ‘aifeducation’ do not need ‘sentencepiece’ even if this requirment is displayed in the table. ‘aifeducation’ uses a WordPieceTokenizer for all newly created models.

BaseModel	Reference	Require Sentencepiece
BaseModelAlbert	Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). ALBERT; A Lite BERT for Self-supervised Learning of Language Representations. arXiv. doi: 10.48550/ARXIV.1909.11942	TRUE
BaseModelBert	Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North (pp. 4171–4186). Association for Computational Linguistics. doi: 10.18653/v1/N19-1423	FALSE
BaseModelDebertaV2	He, P., Liu, X., Gao, J. & Chen, W. (2020). DeBERTa: Decoding-enhanced BERT with Disentangled Attention. doi: 10.48550/arXiv.2006.03654	FALSE
BaseModelDistilBERT	Asnh, V., Debut, L., Chaumond, J. & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. doi: 10.48550/arXiv.1910.01108	FALSE
BaseModelEuroBert	Boizard, N., Gisserot-Boukhlef, H., Alves, D. M., Martins, A., Hammal, A., Corro, C., Hudelot, C., Malherbe, E., Malaboeuf, E., Jourdan, F., Hautreux, G., Alves, J., El-Haddad, K., Faysse, M., Peyrard, M., Guerreiro, N. M., Fernandes, P., Rei, R. & Colombo, P. (2025). EuroBERT: Scaling Multilingual Encoders for European Languages. doi: 10.48550/arXiv.2503.05500	FALSE
BaseModelFunnel	Dai, Z., Lai, G., Yang, Y. & Le, Q. V. (2020). Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing. doi: 10.48550/arXiv.2006.03236	FALSE
BaseModelModernBert	Warner, B., Chaffin, A., Clavie, B., Weller, O., Hallstroem, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J. & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. doi: 10.48550/arXiv.2412.13663	FALSE
BaseModelMPNet	Song,K., Tan, X., Qin, T., Lu, J. & Liu, T.-Y. (2020). MPNet: Masked and Permuted Pre-training for Language Understanding. doi: 10.48550/arXiv.2004.09297	FALSE
BaseModelRoberta	Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. doi: 10.48550/arXiv.1907.11692	FALSE
BaseModelRobertaXML	Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzman, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2019). Unsupervised Cross-lingual Representation Learning at Scale. doi: 10.48550/arXiv.1911.02116	TRUE
BaseModelXmod	Pfeiffer, J., Goyal, N., Lin, X. V., Li, X., Cross, J., Riedel, S., & Artetxe, M. (2022). Lifting the Curse of Multilinguality by Pre-training Modular Transformers. arXiv. doi: 0.48550/ARXIV.2205.06266	TRUE

Florian Berding, Yuliia Tykhonova, Julia Pargmann, Andreas Slopinski, Elisabeth Riebenbauer, Karin Rebmann