sgV7ddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m Z mZddlmZerddlmZej&eZdd iZd ZGd d eZd gZy) N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple) AddedTokenPreTrainedTokenizer)logging) TextInput vocab_fileztokenizer.modelu▁c `eZdZdZeZddgZ ddeee e fffd Z dZ dZ edZd Zd d d ee ffd ZdZdZdZdZddee d ee fdZddZ d deedeeeded eeffd Z ddeedeeed eefdZ d!deededed e fdZxZS)"GemmaTokenizera Construct a Gemma tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is no padding token in the original model. Args: vocab_file (`str`): Path to the vocabulary file. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. sp_model_kwargs (`Dict[str, Any]`, `Optional`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. use_default_system_prompt (`bool`, *optional*, defaults to `False`): Whether or not the default system prompt for Gemma should be used. spaces_between_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add spaces between special tokens. input_idsattention_masksp_model_kwargsc  |in||_t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}||_||_||_| |_tjdi|j|_ |jj|t |4d|||||||| | | d | y)NFT) normalizedspecial) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrclean_up_tokenization_spacesuse_default_system_promptspaces_between_special_tokens)r isinstancestrr rrrrspmSentencePieceProcessorsp_modelLoadsuper__init__)selfrrrrrrrrrrr kwargs __class__s _/var/www/html/venv/lib/python3.12/site-packages/transformers/models/gemma/tokenization_gemma.pyr)zGemmaTokenizer.__init__\s &5%> r.filename_prefixctjj|stj d|dytjj ||r|dzndt dz}tjj|jtjj|k7rBtjj|jrt|j||fStjj|jsCt|d5}|jj}|j|ddd|fS|fS#1swY|fSxYw)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directoryN-rVrwb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathrisfileropenr&r3write)r*save_directoryr^out_vocab_fileficontent_spiece_models r-save_vocabularyzGemmaTokenizer.save_vocabularys'ww}}^, LL,^,<zzzNTFFFFrs)NF)FF) __name__ __module__ __qualname____doc__rhvocab_files_namesmodel_input_namesrrr#rr)r5r:propertyr=rDrrHrLrPrTr]r rqrzintboolr~rr __classcell__)r,s@r-rr*sw,\*$&67 48%*"'&+( "$sCx.1( T C .. 0[0tCy080 !x}!X]^aXb!6 sx# 9# 3;DI3F# ko# c# LJN93;DI3F cH%*.3 898"8(, 8 8r.r)rbshutilrtypingrrrrrr sentencepiecer$tokenization_utilsr r utilsr tokenization_utils_baser get_loggerrrerhrr__all__r!r.r-rsb, BBA4   H %!#45`8(`8F  r.