HuggingFaceTokenizer

Abstract class for all tokenizers used with the 'transformers' library.

Value

Does return a new object of this class.

See also

Other Tokenizer: WordPieceTokenizer

Super classes

aifeducation::AIFEMaster -> aifeducation::TokenizerBase -> HuggingFaceTokenizer

Methods

Public methods

• HuggingFaceTokenizer$create_from_hf()

• HuggingFaceTokenizer$clone()

Inherited methods

• aifeducation::AIFEMaster$get_all_fields()
• aifeducation::AIFEMaster$get_documentation_license()
• aifeducation::AIFEMaster$get_ml_framework()
• aifeducation::AIFEMaster$get_model_config()
• aifeducation::AIFEMaster$get_model_description()
• aifeducation::AIFEMaster$get_model_info()
• aifeducation::AIFEMaster$get_model_license()
• aifeducation::AIFEMaster$get_package_versions()
• aifeducation::AIFEMaster$get_private()
• aifeducation::AIFEMaster$get_publication_info()
• aifeducation::AIFEMaster$get_sustainability_data()
• aifeducation::AIFEMaster$is_configured()
• aifeducation::AIFEMaster$is_trained()
• aifeducation::AIFEMaster$set_documentation_license()
• aifeducation::AIFEMaster$set_model_description()
• aifeducation::AIFEMaster$set_model_license()
• aifeducation::AIFEMaster$set_publication_info()
• aifeducation::TokenizerBase$calculate_statistics()
• aifeducation::TokenizerBase$decode()
• aifeducation::TokenizerBase$encode()
• aifeducation::TokenizerBase$get_special_tokens()
• aifeducation::TokenizerBase$get_tokenizer()
• aifeducation::TokenizerBase$get_tokenizer_statistics()
• aifeducation::TokenizerBase$load_from_disk()
• aifeducation::TokenizerBase$n_special_tokens()
• aifeducation::TokenizerBase$save()

---

Method create_from_hf()

Creates a tokenizer from a pretrained model

Usage

HuggingFaceTokenizer$create_from_hf(model_dir)

Arguments

model_dir

Returns

Does return a new object of this class.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

HuggingFaceTokenizer$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.