Tokenizer class names

This list contains tokenizer class names. Elements of the list are used in aife_transformer.load_tokenizer() to get a tokenizer for a transformer model. This list is not designed to be used directly.

Usage

.AIFETrTokenizer

Format

An object of class list of length 6.

See also

Other Transformers for developers: .AIFEModernBertTransformer, .AIFETrConfig, .AIFETrModel, .AIFETrModelMLM, .AIFETrObj, .aife_transformer.check_type(), aife_transformer.load_model(), aife_transformer.load_model_config(), aife_transformer.load_model_mlm(), aife_transformer.load_tokenizer()