Base class for tokenizers containing all methods shared by the sub-classes.
See also
Other R6 Classes for Developers:
AIFEBaseModel,
AIFEMaster,
BaseModelCore,
ClassifiersBasedOnTextEmbeddings,
DataManagerClassifier,
LargeDataSetBase,
ModelsBasedOnTextEmbeddings,
TEClassifiersBasedOnProtoNet,
TEClassifiersBasedOnRegular
Super class
AIFEMaster -> TokenizerBase
Methods
Inherited methods
AIFEMaster$get_all_fields()AIFEMaster$get_documentation_license()AIFEMaster$get_ml_framework()AIFEMaster$get_model_config()AIFEMaster$get_model_description()AIFEMaster$get_model_info()AIFEMaster$get_model_license()AIFEMaster$get_package_versions()AIFEMaster$get_private()AIFEMaster$get_publication_info()AIFEMaster$get_sustainability_data()AIFEMaster$is_configured()AIFEMaster$is_trained()AIFEMaster$set_documentation_license()AIFEMaster$set_model_description()AIFEMaster$set_model_license()AIFEMaster$set_publication_info()
TokenizerBase$save()
Method for saving a model on disk.
TokenizerBase$load_from_disk()
Loads an object from disk and updates the object to the current version of the package.
TokenizerBase$encode()
Method for encoding words of raw texts into integers.
Usage
TokenizerBase$encode(
raw_text,
token_overlap = 0L,
max_token_sequence_length = 512L,
n_chunks = 1L,
token_encodings_only = FALSE,
token_to_int = TRUE,
return_token_type_ids = TRUE,
trace = FALSE
)Arguments
raw_textvectorRaw text.token_overlapintNumber of tokens from the previous chunk that should be added at the beginng of the next chunk. Allowed values:\(0 <= x \)
max_token_sequence_lengthintMaximal number of tokens per chunk. Allowed values:\(20 <= x \)
n_chunksintMaximal number chunks. Allowed values:\(2 <= x \)
token_encodings_onlyboolTRUE: Returns alistcontaing only the tokens.FALSE: Returns alistcontaing a list for the tokens, the number of chunks, and the number potential number of chunks for each document/text.
token_to_intboolTRUE: Returns the tokens asintindex.FALSE: Returns the tokens asstrings.
return_token_type_idsboolIfTRUEadditionally returns the return_token_type_ids.traceboolTRUEif information about the estimation phase should be printed to the console.
TokenizerBase$decode()
Method for decoding a sequence of integers into tokens
TokenizerBase$calculate_statistics()
Method for calculating tokenizer statistics as suggested by Kaya and Tantuğ (2024).
Kaya, Y. B., & Tantuğ, A. C. (2024). Effect of tokenization granularity for Turkish large language models. Intelligent Systems with Applications, 21, 200335. <https://doi.org/10.1016/j.iswa.2024.200335>
Usage
TokenizerBase$calculate_statistics(
text_dataset,
statistics_max_tokens_length,
step = "creation"
)Arguments
text_datasetLargeDataSetForTextLargeDataSetForText Object storing textual data.statistics_max_tokens_lengthintMaximum sequence length for calculating the statistics. Allowed values:\(20 <= x <= 8192\)
stepstringdescribing the context of the estimation.
TokenizerBase$calc_quantiles()
Method for calculating the necessary number of chunks to cover a specific quantile of documents completely.
Tokenizer split raw texts into a sequences of tokens. Most neural nets can only process a given sequence length. Thus, long texts must be split into a sequence of sequences in order to represent the whole text. This method calculates how many sequences are necessary. The number of sequences of sequences is called chunks.
You can think of these chunks as dividing a long texts into several paragraphs.
Usage
TokenizerBase$calc_quantiles(
text_dataset,
batch_size = 32L,
seq_len_tokens = 512L,
token_overlap = 0L,
trace = TRUE
)Arguments
text_datasetLargeDataSetForTextLargeDataSetForText Object storing textual data.batch_sizeintSize of the batches for training. Allowed values:\(1 <= x \)
seq_len_tokensintLength of a sequence in tokens Allowed values:\(1 <= x \)
token_overlapintNumber of tokens from the previous chunk that should be added at the beginng of the next chunk. Allowed values:\(0 <= x \)
traceboolTRUEif information about the estimation phase should be printed to the console.