sgP>dZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZerdd lmZdd lmZmZej4eZd d iZd ZGddeZy)z$Tokenization class for SigLIP model.N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple)import_protobuf)PreTrainedTokenizer) AddedToken) TextInput)loggingrequires_backends vocab_filez spiece.modelu▁c eZdZdZeZddgZ d#deee e fddffd Z dZ e d Zd Z d$d eed eeed edeeffd ZdeedeefdZ d%d eed eeedeefdZ d%d eed eeedeefdZdZdZde de fdZdddZd&dddee ffd Ze dZdZdZdZdZ d%d e d!ee de!e fd"Z"xZ#S)'SiglipTokenizera Construct a Siglip tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. model_max_length (`int`, *optional*, defaults to 64): The maximum length (in number of tokens) for model inputs. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. input_idsattention_maskNsp_model_kwargsreturnc t|dt|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}|in||_||_||_|j|_||_t |(d|||||j||d| y)NprotobufTF)rstriplstrip normalizedspecial) eos_token unk_token pad_tokenadditional_special_tokensrmodel_max_length do_lower_case) r isinstancestrr rr#rget_spm_processorsp_modelsuper__init__) selfrrrr r!rr"r#kwargs __class__s a/var/www/html/venv/lib/python3.12/site-packages/transformers/models/siglip/tokenization_siglip.pyr*zSiglipTokenizer.__init__Xs $ +)S) yduVZ [ )S) yduVZ [ )S) yduVZ [  &5%A )B-43D3D"D MM,T^^,<=++   1 122 2r/ct|jg}|t||zdgzSt||z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r)rYrU)r+rOrPeoss r.$create_token_type_ids_from_sequencesz4SiglipTokenizer.create_token_type_ids_from_sequencessP   !  {S()QC/ /;${2S89QC??r/cX|j|}||S|j|}||zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r\)r+rOrPs r. build_inputs_with_special_tokensz0SiglipTokenizer.build_inputs_with_special_tokenss;&22;?   66{CK, ,r/cD|jj}d|d<|S)Nr()__dict__copy)r+states r. __getstate__zSiglipTokenizer.__getstate__s# ""$ j r/c||_t|dsi|_tjdi|j|_|j j |jy)Nrr$)rchasattrrr2r3r(Loadr)r+ds r. __setstate__zSiglipTokenizer.__setstate__sO t./#%D 22JT5I5IJ  4??+r/textcj|jtjddtjS)N) translater& maketransstring punctuation)r+rls r.remove_punctuationz"SiglipTokenizer.remove_punctuation s$~~cmmBF4F4FGHHr/keep_punctuation_exact_stringc|r*|jfd|j|D}nj|}tjdd|}|j }|S)aReturns canonicalized `text` (puncuation removed). Args: text (`str`): String to be canonicalized. keep_punctuation_exact_string (`str`, *optional*): If provided, then this exact string is kept. For example providing '{}' will keep any occurrences of '{}' (but will still remove '{' and '}' that appear separately). c3@K|]}j|ywrC)rs).0partr+s r. z4SiglipTokenizer.canonicalize_text..s!626''-6sz\s+ )joinsplitrsresubstrip)r+rlrus` r.canonicalize_textz!SiglipTokenizer.canonicalize_textsc )0556:>**Eb:c6D**40Dvvfc4(zz| r/rc t|t|jtdzfi|}t |dkDr"|dtk(r|d|j vr|dd}|S)z8 Converts a string to a list of tokens. r{rSrN)r)tokenizeSPIECE_UNDERLINEreplacerUall_special_tokens)r+rladd_special_tokensr,tokensr-s r.rzSiglipTokenizer.tokenize&se!"2T\\BRTW5X"Xc\bc v;?vay,<<dNeNeAeABZF r/cpt|jjt|jSrC)rUr(encoder&rrEs r.unk_token_lengthz SiglipTokenizer.unk_token_length0s'4==''DNN(;<==r/c |j|d}|jj|t}|jj|j|zt}t ||j k\r||j dS|S)u* Returns a tokenized string. We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give `['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the `unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`. `self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`. Nrt)out_type)rr(rr&rrUr)r+rlr,rs r. _tokenizezSiglipTokenizer._tokenize5s%%d$%O%%dS%9%%dnnt&;c%J25f+AVAV2Vvd++-.b\bbr/c8|jj|S)z0Converts a token (str) in an id using the vocab.)r( piece_to_id)r+tokens r._convert_token_to_idz$SiglipTokenizer._convert_token_to_idJs}}((//r/c<|jj|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r( IdToPiece)r+indexrs r._convert_id_to_tokenz$SiglipTokenizer._convert_id_to_tokenOs ''. r/cg}d}d}|D]P}||jvr-|s|dz }||jj||zz }d}g}>|j|d}R||jj|z }|j S)z:Converts a sequence of tokens (string) in a single string.rnFr{T)rr(decodeappendr)r+rcurrent_sub_tokens out_stringprev_is_specialrs r.convert_tokens_to_stringz(SiglipTokenizer.convert_tokens_to_stringTs  (E///&#%Jdmm223EFNN "&%'""))%0"' ( dmm**+=>> !!r/save_directoryfilename_prefixctjj|stj d|dytjj ||r|dzndt dz}tjj|jtjj|k7rBtjj|jrt|j||fStjj|jsCt|d5}|jj}|j|ddd|fS|fS#1swY|fSxYw)NzVocabulary path (z) should be a directory-rnrwb)ospathisdirloggererrorr|VOCAB_FILES_NAMESabspathrisfilerr4r(serialized_model_protowrite)r+rrout_vocab_fileficontent_spiece_models r.save_vocabularyzSiglipTokenizer.save_vocabularyhs%ww}}^, LL,^,<zrNN@T)NFrC)F)$__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrrr&rr*r'propertyrFrNrintboolrTr\r_rarfrkrsrrrrrrrr r __classcell__)r-s@r.rr,s&P*$&67 "&48/ "$sCx.1/  / b ..sxO9O3;DI3FOkoO cO: 3c 3tCy 3JN@9@3;DI3F@ c@0JN-9-3;DI3F- c-6 ,IsIsIHL*[QUVYQZ>>c*0  "(!c!HSM!]bcf]g!r/r) rrr~rqrZshutilrtypingrrrrrr sentencepiecer2convert_slow_tokenizerr tokenization_utilsr tokenization_utils_baser rutilsrr get_loggerrrrrrr$r/r.rsl+ BB5514/   H %!>2K!)K!r/