sg`PdZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z mZmZmZmZddlmZej(eZdd d Zed Zd Zd ZdZGddZGddeZy)zTokenization classes for CLIP.N) lru_cache)ListOptionalTuple) AddedTokenPreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filec tttdtddztttdtddzztttdtddzz}|dd}d }td D]1}||vs|j||jd |z|dz }3|Dcgc] }t |}}t t ||Scc}w) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNr)listrangeordappendchrdictzip)bscsnbs ]/var/www/html/venv/lib/python3.12/site-packages/transformers/models/clip/tokenization_clip.pybytes_to_unicoder%%s U3s8SX\ *+d5TCIPQM3R.SSVZ[`adeiajloptluxyly[zV{{ AB A 4[ B; IIaL IIdQh  FA   Q#a& B  B  s C4cbt}|d}|ddD]}|j||f|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charchars r$ get_pairsr->sF EEQIQR 9d#$  LcTtjdd|}|j}|S)Nz\s+ )resubstrip)texts r$whitespace_cleanr5Ls$ 66&#t $D ::r?r@rArBs r$__init__zBasicTokenizer.__init__ts<  K*{+&<#* 0r.cJ|r$|jjt|n |j}|j|}|jr|j |}t jd|}t|}g}|D]}||vrY|jr0|j}|jdur/|j|}n|jr|j|}|j|j||tdj|}|S)aj Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. NFCFr0)r?unionr' _clean_textr@_tokenize_chinese_chars unicodedata normalizer9r>lowerrA_run_strip_accentsextend_run_split_on_puncjoin)rCr4r?unicode_normalized_text orig_tokens split_tokenstoken output_tokenss r$tokenizezBasicTokenizer.tokenizesCNd&&,,S-=>SWScSc %  & &//5D"-"7"7t"D)*AB    MEK'%%!KKME))6 $ 7 7 >'' 33E:E    7 7{ K L M,CHH\,BC r.ctjd|}g}|D].}tj|}|dk(r|j|0dj |S)z$Strips accents from a piece of text.NFDMn)rJrKcategoryrrP)rCr4outputr,cats r$rMz!BasicTokenizer._run_strip_accentss^$$UD1 D&&t,Cd{ MM$   wwvr.cv|jr|||vr|gSt|}d}d}g}|t|krb||}t|r|j |gd}n)|r|j gd}|dj ||dz }|t|krb|Dcgc]}dj |c}Scc}w)z&Splits punctuation on a piece of text.rTFrrZ)rBrlenr rrP) rCr4r?charsistart_new_wordr\r,xs r$rOz!BasicTokenizer._run_split_on_puncs$$)@T[EX6MT  #e*n8Dt$ tf%!%!MM"%!&r !!$' FA#e*n%++q +++sB6cg}|D]c}t|}|j|r4|jd|j||jdS|j|edj|S)z)Adds whitespace around any CJK character.r0rZ)r_is_chinese_charrrPrCr4r\r,cps r$rIz&BasicTokenizer._tokenize_chinese_charssm $DTB$$R( c" d# c" d# $wwvr.c|dk\r|dksF|dk\r|dks<|dk\r|dks2|dk\r|dks(|d k\r|d ks|d k\r|d ks|d k\r|dks |dk\r|dkryy)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTF)rCrhs r$rfzBasicTokenizer._is_chinese_charsr6\bFlf vg "-g "-g "-g "-f vg "-r.cg}|D]Q}t|}|dk(s|dk(s t|r$t|r|jdA|j|Sdj |S)zBPerforms invalid character removal and whitespace cleanup on text.rir0rZ)rr r rrPrgs r$rHzBasicTokenizer._clean_textsf $DTBQw",+d*;d# c" d# $wwvr.)TNTNTr=) __name__ __module__ __qualname____doc__rDrVrMrOrIrfrHrjr.r$r;r;]s<0# 1 $L ,, 0 r.r;c (eZdZdZeZddgZ dfd ZedZ dZ dde e de e e d e e fd Z dde e de e e d ed e e ffd Z dde e de e e d e e fd ZdZdZdZdZdZddede ed eefdZxZS) CLIPTokenizera Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc t|trt|ddn|}t|trt|ddn|}t|trt|ddn|} ddl} | j|_t|d5} tj| |_ ddd|jjD cic]\} } | |  c} } |_||_t#|_|j$jD cic]\} } | |  c} } |_t|d5} | j)j+j-dd d }dddDcgc]}t/|j-}}t1t3|t5t7||_d d d |_t=j>dt<j@|_!tE|d|||||d|y#t $r2t jdtdd|_ d|_YwxYw#1swYxYwcc} } wcc} } w#1swYxYwcc}w)NF)lstriprstriprzKftfy or spacy is not installed using custom BasicTokenizer instead of ftfy.)rArButf-8encoding ri<|startoftext|> <|endoftext|>)r{r|z[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)errors unk_token bos_token eos_token pad_tokenrj)$ isinstancestrrftfyfix_text ImportErrorloggerinfor;nlpopenjsonloadencoderitemsdecoderr}r% byte_encoder byte_decoderreadr3r7tuplerrrr` bpe_rankscacher1compile IGNORECASEpatsuperrD)rCrrr}r~rrrkwargsr vocab_handlekv merges_handle bpe_mergesmerge __class__s r$rDzCLIPTokenizer.__init__sJTT]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir  !  MMDM *w / 3<99\2DL 3)-););)=>A1>  ,..2.?.?.E.E.GHdaQTH + 0 [M&++-335;;DA!FYZJ [8BCueEKKM*C Cc*eC O.DEF):_] :: n MM       - ! KKe f%EERDH DM !  3 3>I [ [Cs<#HI I I11I. I(7I?II I%c,t|jSr=)r`rrCs r$ vocab_sizezCLIPTokenizer.vocab_sizeLs4<<  r.cBt|jfi|jSr=)rradded_tokens_encoderrs r$ get_vocabzCLIPTokenizer.get_vocabPsDLL>D$=$=>>r. token_ids_0 token_ids_1returncl|jg}|jg}|||z|zS||z|z|z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) bos_token_id eos_token_idrCrrrrs r$ build_inputs_with_special_tokensz.CLIPTokenizer.build_inputs_with_special_tokensSsU(&&' &&'  {*Y6 6;&2Y>LyXXr.already_has_special_tokensc|rt|||dS|dgdgt|zzdgzSdgdgt|zzdgzdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrr)rget_special_tokens_maskr`)rCrrrrs r$rz%CLIPTokenizer.get_special_tokens_maskns& &72'[]a3   31#K 001QC7 7sqcC ,,-3qc9aS3{CS=STXYWZZZr.c|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r)rrr`rs r$$create_token_type_ids_from_sequencesz2CLIPTokenizer.create_token_type_ids_from_sequencessp &&' &&'  y;.:;qcA A9{*Y6B[PS\\]ab`cccr.c|jvrj|St|dd|ddzfz}t|}|s|dzS t|fd}|jvrn|\}}g}d}|t |kr |j ||} |j||| | }|||k(r6|t |dz kr%||dz|k(r|j||z|dz }n|j|||dz }|t |krt|}|}t |dk(rn t|}dj|}|j|<|S#t$r|j||dYiwxYw) Nr_cNjj|tdS)Ninf)rgetfloat)pairrCs r$z#CLIPTokenizer.bpe..s1C1CD%PU,1Wr.keyrrr0) rrr-minrr`indexrN ValueErrorrrP) rCrTr)r*bigramfirstsecondnew_wordrbjs ` r$bpezCLIPTokenizer.bpes DJJ ::e$ $U3BZ E"I$6#88$6> !$WXFT^^+"ME6HAc$i- 5!,A OOD1I.A7e#CIM(9d1q5kV>SOOEFN3FAOODG,FAc$i- XHD4yA~!$9:xx~  5 +"OODH-sEE98E9cg}j+djjj|}n(t j|j }t jj|D]a}djfd|jdD}|jdj|jdDc|S)zTokenize a string.r0rZc3<K|]}j|ywr=)r).0r#rCs r$ z*CLIPTokenizer._tokenize..s!)*!!!$srwc3 K|]}|ywr=rj)r bpe_tokens r$rz*CLIPTokenizer._tokenize..sTIiTs ) rrPrrVr5rLr1findallrencoderNrr7)rCr4 bpe_tokensrTs` r$ _tokenizezCLIPTokenizer._tokenizes == 88DHH--d34D#DMM$$78>>@DZZ$/ UEGG.3ll7.CE   T%9N9Ns9ST T  U r.c|jj||jj|jS)z0Converts a token (str) in an id using the vocab.)rrr~)rCrTs r$_convert_token_to_idz"CLIPTokenizer._convert_token_to_ids,||t||'7'7'GHHr.c8|jj|S)z=Converts an index (integer) in a token (str) using the vocab.)rr)rCrs r$_convert_id_to_tokenz"CLIPTokenizer._convert_id_to_tokens||&&r.cdj|}t|Dcgc]}|j|c}}|jd|jj ddj }|Scc}w)z:Converts a sequence of tokens (string) in a single string.rZrw)r}rr0)rP bytearrayrdecoder}replacer3)rCr8r4c byte_arrays r$convert_tokens_to_stringz&CLIPTokenizer.convert_tokens_to_stringsiwwvdC 1 1! 4CD    =EEfcRXXZ  DsA5save_directoryfilename_prefixc |tjj|s%tj dj |ytjj ||r|dzndtdz}tjj ||r|dzndtdz}t|dd5}|jtj|jd d d d zdddd}t|dd5}|jdt|jjdD]X\}} || k7r&tj!dj || }|jdj |d z|dz }Z ddd||fS#1swYxYw#1swY||fSxYw)Nz*Vocabulary path ({}) should be a directory-rZrrwrwrxrTF)indent sort_keys ensure_asciirzrz#version: 0.2 c |dS)Nrrj)kvs r$rz/CLIPTokenizer.save_vocabulary..s Y[\]Y^r.rzqSaving vocabulary to {}: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r0r)ospathisdirrerrorformatrPVOCAB_FILES_NAMESrwriterdumpsrsortedrrwarning) rCrrr merge_filefrwriterr token_indexs r$save_vocabularyzCLIPTokenizer.save_vocabularysww}}^, LLELL^\ ] WW\\ o_s22QbcoQpp WW\\ o_s22QbcpQqq *cG 4 c GGDJJt||ATYZ]aa b c *cG 4  LL* ++1$..2F2F2HN^+_ ' KK'NNMMSVT^M_(E SXXj1D89   :%%! c c :%%s56F#BF/#F,/F;)rr|r{r|r|r=)NF)rlrmrnrorvocab_files_namesmodel_input_namesrDpropertyrrrintrrboolrrrrrrrrrr __classcell__)rs@r$rqrqs@2*$&67 !#!!/ b!!?JNY9Y3;DI3FY cY8sx[9[3;DI3F[ko[ c[:JNd9d3;DI3Fd cd.(T I'&c&HSM&]bcf]g&r.rq)rorrrJ functoolsrtypingrrrregexr1tokenization_utilsrr r r r utilsr get_loggerrlrrr%r-r5r9r;rqrjr.r$rs% ((oo   H %   0 ^^BF&'F&r.