sgW ddlZddlmZddlmZmZddlmZddlm Z ddl m Z m Z ddl mZed e rd d lmZndZe j"eZd d dZGdde Zy)N)copyfile)OptionalTuple) processors)PreTrainedTokenizerFast)is_sentencepiece_availablelogging)require_versionztokenizers>=0.13.3)GemmaTokenizerztokenizer.modelztokenizer.json) vocab_filetokenizer_fileceZdZdZeZeZdZddgZ dfd Z e de fdZ dZe d Ze d Zej"d Zej"d Zdd edeedeefdZddZxZS)GemmaTokenizerFastu Construct a Gemma tokenizer fast. Based on byte-level Byte-Pair-Encoding. This uses notably ByteFallback and no prefix space. Normalization is applied to replace `" "` with `"▁"` ```python >>> from transformers import GemmaTokenizerFast >>> tokenizer = GemmaTokenizerFast.from_pretrained("hf-internal-testing/dummy-gemma") >>> tokenizer.encode("Hello this is a test") [2, 4521, 736, 603, 476, 2121] ``` If you want to change the `bos_token` or the `eos_token`, make sure to specify them when initializing the model, or call `tokenizer.update_post_processor()` to make sure that the post-processing is correctly done (otherwise the values of the first token and final token of an encoded sequence will not be correct). For more details, checkout [post-processors] (https://huggingface.co/docs/tokenizers/api/post-processors) documentation. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .model extension) that contains the vocabulary necessary to instantiate a tokenizer. tokenizer_file (`str`, *optional*): [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The padding token add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. left input_idsattention_maskc t |d||||||||| d | ||_| |_|j ||_y)N) rrclean_up_tokenization_spaces unk_token bos_token eos_token pad_token add_bos_token add_eos_token)super__init___add_bos_token_add_eos_tokenupdate_post_processorr) selfrrrrrrrrrkwargs __class__s d/var/www/html/venv/lib/python3.12/site-packages/transformers/models/gemma/tokenization_gemma_fast.pyrzGemmaTokenizerFast.__init__Zsa  !))E''  ,+ ""$$returncp|jr)tjj|jSdS)NF)rospathisfiler#s r&can_save_slow_tokenizerz*GemmaTokenizerFast.can_save_slow_tokenizerxs$26//rww~~doo.LuLr'cP|j}|j}||jr td|j}|j }||j r td|jr|dzndd|j rd|zdznd}||jrd|zd zndd |j rd|zd znd}g}|jr|j||f|j r|j||ftj||| |j_ y) ze Updates the underlying post processor with the current `bos_token` and `eos_token`. Nz)add_bos_token = True but bos_token = Nonez)add_eos_token = True but eos_token = Nonez:0 z$A:0 z:0z:1z $B:1)singlepairspecial_tokens) r bos_token_idr ValueErrorr eos_token_idrappendrTemplateProcessing _tokenizerpost_processor)r#bosr5eosr7r2r3r4s r&r"z(GemmaTokenizerFast.update_post_processor}s+nn(( ;4--HI Inn(( ;4--HI I#'#5#5SY2>dUYUgUgCGDLmoBpqD,>,>3s74zzzTFr?)__name__ __module__ __qualname____doc__rOvocab_files_namesr slow_tokenizer_class padding_sidemodel_input_namesrpropertyboolr.r"rrsetterstrrrrRrW __classcell__)r%s@r&rr%s-^*)L$&67%*%<MMM 4####%%%% !c!HSM!]bcf]g!( r'r)r*shutilrtypingrr tokenizersrtokenization_utils_fastrutilsr r utils.versionsr tokenization_gemmar get_loggerrXrLrOrrr'r&rmsb "!>8-$%2N   H %#4HXYb0br'