HuggingFaceTokenizer

Abstract class for all tokenizers used with the 'transformers' library.

Value

Does return a new object of this class.

See also

Other Tokenizer: WordPieceTokenizer

Super classes

AIFEMaster -> TokenizerBase -> HuggingFaceTokenizer

Methods

Public methods

• HuggingFaceTokenizer$create_from_hf()

• HuggingFaceTokenizer$get_vocab_size()

• HuggingFaceTokenizer$clone()

Inherited methods

• AIFEMaster$get_all_fields()
• AIFEMaster$get_documentation_license()
• AIFEMaster$get_ml_framework()
• AIFEMaster$get_model_config()
• AIFEMaster$get_model_description()
• AIFEMaster$get_model_info()
• AIFEMaster$get_model_license()
• AIFEMaster$get_package_versions()
• AIFEMaster$get_private()
• AIFEMaster$get_publication_info()
• AIFEMaster$get_sustainability_data()
• AIFEMaster$is_configured()
• AIFEMaster$is_trained()
• AIFEMaster$set_documentation_license()
• AIFEMaster$set_model_description()
• AIFEMaster$set_model_license()
• AIFEMaster$set_publication_info()
• TokenizerBase$calc_quantiles()
• TokenizerBase$calculate_statistics()
• TokenizerBase$decode()
• TokenizerBase$encode()
• TokenizerBase$get_special_tokens()
• TokenizerBase$get_tokenizer()
• TokenizerBase$get_tokenizer_statistics()
• TokenizerBase$load_from_disk()
• TokenizerBase$n_special_tokens()
• TokenizerBase$print()
• TokenizerBase$save()

---

HuggingFaceTokenizer$create_from_hf()

Creates a tokenizer from a pretrained model

Usage

HuggingFaceTokenizer$create_from_hf(model_dir, tokenizer_type = NULL)

Arguments

model_dir

Path where the model is stored.

tokenizer_type

string Name of the tokenizer belonging to a base model (e.g. BertTokenizer). This parameter is only used if an error occurs during loading as fall back option.

Returns

Does return a new object of this class.

---

HuggingFaceTokenizer$get_vocab_size()

Vocabulary. size

Usage

HuggingFaceTokenizer$get_vocab_size()

Returns

Returns a int representing the size of the vocabulary.

---

HuggingFaceTokenizer$clone()

The objects of this class are cloneable with this method.

Usage

HuggingFaceTokenizer$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.