Load a tokenizer

Loads a tokenizer for a transformer with the passed type.

Usage

aife_transformer.load_tokenizer(type, model_dir)

Arguments

type

string A type of the transformer. Allowed types are bert, roberta, funnel, longformer, mpnet, modernbert. See AIFETrType list.

model_dir

string Path to the directory where the original model is stored. Allowed values: any

Value

If success - a tokenizer, otherwise - an error (passed type is invalid).

See also

Other Transformers for developers: .AIFEModernBertTransformer, .AIFETrConfig, .AIFETrModel, .AIFETrModelMLM, .AIFETrObj, .AIFETrTokenizer, .aife_transformer.check_type(), aife_transformer.load_model(), aife_transformer.load_model_config(), aife_transformer.load_model_mlm()