sgdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZddlmZddlmZddlmZmZmZmZd d lmZd d lm Z d d l!m"Z"d d l#m$Z$d dl%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/d dl0m1Z1m2Z2m3Z3e3jhe5Z6dZ7dZ8dZ9dZ:dZ;e&dz Z&eeeedZy)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)AnyDictIterableListOptionalTupleUnion)Encoding) Tokenizer)Decoder) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer)convert_slow_tokenizer)convert_gguf_tokenizer)load_gguf_checkpoint)PreTrainedTokenizer) INIT_TOKENIZER_DOCSTRING AddedToken BatchEncodingPreTokenizedInputPreTokenizedInputPairPreTrainedTokenizerBaseSpecialTokensMixin TextInput TextInputPairTruncationStrategy)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )BPEUnigram WordLevel WordPiece)tokenizer_file vocab_filec)*eZdZUdZeZdZeed<fdZ e de fdZ e de fdZ e defdZdeeeffd Ze deeeffd Ze deeeffd Ze deeeffd Zdeeeffd ZdefdZe defdZe defdZ dEdedee dee de de de de de deeee fe!effdZ"de#ee$efde#ee!effdZ%dedefdZ&d edeefd!Z'dFd"e!e#eefdefd#Z(dFd$e defd%Z) dFd&e#ee!efd'e de#ee!effd(Z*dGd)ed$eed*e de!efd+Z+d,e,d-e-d.ed/ed0eed1ee f d2Z.de,j^e-j`dd3ddddddddddddfd4e#e!e1e!e2e!e3e!e4fd*e d,e,d-e-d.eed/ed5e d0eed1ee d6eedee dee de de de de de d7e de5f&d8Z6dde,j^e-j`dd3ddddddddddddfd)e#e1e3fd9ee#e1e3fd*e d,e,d-e-d.eed/ed5e d0eed1ee d6ee dee dee de de de de de d7e de5f(d:Z7de!edefd;Z8 dHdZ9 dId?e#ee:jvfd@eedAee dBeedeef dCZ< dJdDZ=xZ>S)KPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classc  |jdd}|jdd}|jdd}|jdd}|jdd}|jdi}|r||j td|tj|} n||st j |} n|r t|} n|lt|jd } | d d } | d } | d } t| | \} }|j| t|dkDr|j|ny|j"|dur|j|i|}t|} nK|s>|jd d|_ |jdg|_t|d} d}n td| |_||j|j d|_|jj$}|q|jj&d"i||j)d|d|j)d|d|j)d|d|j)d|dn|jj+|jj,}||jj.d"i||j)d|d|j)d|d|j)d|d|j)d|d|j)d|dt1|dd"i||j4|j_|j8Dchc]}t;t=|}}t?|jAd Dcgc]\}}t;t=||vr|}}}tC|jDjG|Dcgc] }tI|c}z}||jJDcgc]}||vs||vs |c}z }t|dkDrg}|jL}|D]p}tO|tPr|jRxstI||vn tI||v}tO|tHrtQ||!}n||_)|jU|r|r|jW|yyycc}wcc}}wcc}wcc}w)#Ntokenizer_object__slow_tokenizer gguf_filer( from_slowFadded_tokens_decoderzCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r)config model_type tokenizertokenizer_configradditional_special_tokensT) from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofc |dSNr)xs W/var/www/html/venv/lib/python3.12/site-packages/transformers/tokenization_utils_fast.pyz2PreTrainedTokenizerFast.__init__..s STUVSWkey)specialrG),popr, ValueErrorcopydeepcopy TokenizerFast from_filerrgetrupdatelenr)r7 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr2hashreprsorteditemslistadded_tokens_encoderkeysstrall_special_tokens_extendedall_special_tokens isinstancerrNappend add_tokens)selfargskwargsr.slow_tokenizerr0fast_tokenizer_filer1r2fast_tokenizer gguf_param architecturetokenizer_dictr6additional_kwargs _truncation_paddingtokenadded_tokens_decoder_hashindex tokens_to_addencodertokensspecial_tokens is_special __class__s rIrbz PreTrainedTokenizerFast.__init__bs!::&8$?$6=JJ{D1 $jj)94@JJ{E2 %zz*@"E /D4M4M4U0   '!]]+;   3[5L M OO ) ) +??**   *DOO * * 6X 6   k8K+@ A   18M3J K   nh{.C D   lHX,> ?   2H=Q4R S "6"040I0I-DHD]D]$^5T$u+%6$^!$^!'';'A'A'C X uDK (AA   t005578Ta;b5CJ;bb#?? 5PWCW\aiv\vE   }  !F!44N& %"%4]]Bc%jN&BU~5 eS)&ujAE$.EM e$ %' "%_ PreTrainedTokenizerFast.added_tokens_encoder.. dhijdkrKrLrgr2rhcontentrrvks rIrjz,PreTrainedTokenizerFast.added_tokens_encoders; *00I0I0O0O0QWk)lmA 1 mmmAc6|jjS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )rXget_added_tokens_decoderrs rIr2z,PreTrainedTokenizerFast.added_tokens_decoders7799rKct|jjdDcic]\}}|j|c}}Scc}}w)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. c |dSrFrGrs rIrJz9PreTrainedTokenizerFast.get_added_vocab..rrKrLrrs rIget_added_vocabz'PreTrainedTokenizerFast.get_added_vocabs;*00I0I0O0O0QWk)lmA 1 mmmrc:|jjdS)zD Size of the full vocabulary with the added tokens. Trrrs rI__len__zPreTrainedTokenizerFast.__len__s---EErKc|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rXrs rIbackend_tokenizerz)PreTrainedTokenizerFast.backend_tokenizer s rKc.|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rXdecoderrs rIrzPreTrainedTokenizerFast.decoders &&&rKFTencodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec J|d|jv}|d|jv}|r|j|g|jz} n|g} tt} | D]} | dj | j |r| dj | j |r| dj | j|r| dj | j|r| dj | j|s| dj t| j | | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) token_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingrC) model_input_names overflowingrrirpidstype_idsrroffsetsrW) rrrrrrrrrr encodings encoding_dictes rI_convert_encodingz)PreTrainedTokenizerFast._convert_encodings$( ! ($48N8N$N ! ($48N8N$N ! $)=)=)I! X%9%99I! I#D)  ;A + & - -aee 4$./66qzzB$./66q7G7GH)34;;A fc ";;FC CMSTE88?TTTsAr~cX|jj|}| |jS|Sr)rX token_to_id unk_token_id)rrr~rs rIrz;PreTrainedTokenizerFast._convert_token_to_id_with_added_vocYs,++E2 =$$ $ rKrcJ|jjt|Sr)rX id_to_tokenint)rrrs rI_convert_id_to_tokenz,PreTrainedTokenizerFast._convert_id_to_token_s**3u:66rK new_tokenscr|r|jj|S|jj|Sr)rXadd_special_tokensrq)rrrrs rI _add_tokensz#PreTrainedTokenizerFast._add_tokensbs/ ??55jA A))*55rKpairc8|jj|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rXnum_special_tokens_to_add)rrrs rIrz1PreTrainedTokenizerFast.num_special_tokens_to_addhs&88>>rKrskip_special_tokensct|tr|jj|Sg}|D]H}t|}|r||jvr|j |jj|J|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rorrXrall_special_idsrp)rrrrrrs rIconvert_ids_to_tokensz-PreTrainedTokenizerFast.convert_ids_to_tokens}su c3 ??..s3 3 >EJE"u0D0D'D MM$//55e< =  >  rKtextrc J|jd|||d|jS)N)r text_pairrrG) encode_plusr)rrrrrrts rItokenizez PreTrainedTokenizerFast.tokenizes,tkTTN`kdjkrrttrKpadding_strategyr=r9r<rDrBc|jj}|jj}|tjk(r|||jj na|||j |jd} |d} n | D cic]} | |j| d} } | | k7r|jjdi| |tjk(r||jjyy|tjk(r|nd} | ||n |j|j|j |j"|d} || k7r|jj$di| yycc} w)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r9r<r>r;)rCr;pad_idr?rArDrG)rXr[r_r DO_NOT_TRUNCATEr^valuer:rUr\r! DO_NOT_PAD no_padding MAX_LENGTHrB pad_token_idr?r@r`) rrrr=r9r<rDrBr|r}targetcurrentrrCs rIset_truncation_and_paddingz2PreTrainedTokenizerFast.set_truncation_and_paddingsZBoo00 ??** "4"D"D D&--/) /55!11 F"@FG11kooa66GG& 111;F; 99 9#**,$$47Q7Q#QZW[F -9-E\4K\K\++!^^#55&8 F6!...88"%HsErbatch_text_or_text_pairsis_split_into_wordsreturn_tensorsrcc  t|ttfstdt |d|j |||||| |j j|k7r||j _|j j|||}|Dcgc]}|j|| | | ||||}}i}|ddjD]'}|Dcgc]\}}||D]}|}}}}|||<)|Dcgc]\}}|D]}|}}}}| r2g}t|D]\}\}}||gt|dzz }||d<|dD]} |j| ||t||| Scc}wcc}}}wcc}}}w) Nz:batch_text_or_text_pairs has to be a list or a tuple (got ))rr=r9r<rDrB)ris_pretokenized)rrrrrrrrrroverflow_to_sample_mapping) tensor_type)rotupleri TypeErrortyperrXrd encode_batchrrk enumeraterW&_eventual_warn_about_too_long_sequencer)!rrrrrr=r9r<rrDrBrrrrrrrrrcrrtokens_and_encodingssanitized_tokensrMr_rstacksanitized_encodingsritoksrs! rI_batch_encode_plusz*PreTrainedTokenizerFast._batch_encode_pluss .2UDMBLTRjMkLllmn  ''- 3!1% (  ?? 0 04H H4HDOO 1OO00 $1/1 .&   " "!&;&;*C+E'=+ #   ('*1-224 *C&:NN74DINqQNQNEN$) S ! *1ESSWQdSqSqSS %)+ & )*> ? K 9D!*qcC[8I4J.JJ* K=W 9 :)+6 XI  7 7 :w W X-/BP^__I ,OSs E2E7 <E>rc |r||fgn|g}|j|fid|d|d|d|d|d|d| d| d | d | d | d |d |d|d|d|d||}| `|s^t|jDcic].\}}|t|dkDrt |dt r|dn|0c}}|j }|j|d|||Scc}}w)Nrrrr=r9r<rDrBrrrrrrrrrcrr)rrrhrWrorirr)rrrrrrr=r9r<rrDrBrrrrrrrrrcrt batched_inputbatched_outputrMrs rI _encode_plusz$PreTrainedTokenizerFast._encode_plus@sg.09$ *+tf 000  3  2 .  !4  "    2 & * #8 #8 '@ (B $: (! "# $"6' 0  !*C*'5&:&:&<"Uc%j1nE!Hd9S%(Y^^(( N 33N;4OQ[]des-3C c|jj%|jjj|Sdj|S)N )rrdecodejoin)rrrs rIconvert_tokens_to_stringz0PreTrainedTokenizerFast.convert_tokens_to_string}sJ%%--9  " " * * 1 1& 9 &! rK token_idsclean_up_tokenization_spacesc |jdd|_t|tr|g}|jj ||}||n |j }|r|j|}|S|S)Nuse_source_tokenizerF)r)rOrZrorrXrr clean_up_tokenization)rrr rr rtr clean_texts rI_decodezPreTrainedTokenizerFast._decodes~-3JJ7Mu,U) i %" I%%iEX%Y,7 )22 % (33D9J KrKsave_directory file_names legacy_formatfilename_prefixct|}|j|dur td|duxs|duxr|jduxr |j}|duxs|du}|rtj j ||r|dzndtz}|jjD cic]\}} | |jk\s|| } }} | rDt|dd 5} tj| d dd d z} | j| ddd|j|| } || z|fz}|rOtj j ||r|dzndt z}|j"j%|||fz}|Scc} }w#1swY~xYw)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-wzutf-8)r)indent sort_keys ensure_ascii )r)rlr,rProspathrADDED_TOKENS_FILErjrhropenjsondumpswritesave_vocabularyTOKENIZER_FILErsave)rrrrrr save_slow save_fastadded_tokens_filetokr added_vocabfout_str vocab_filesr(s rI_save_pretrainedz(PreTrainedTokenizerFast._save_pretraineds^,  $ $ ,$1F`  d " ;mt&; -))5 -,,  "T)C]e-C  " /3!6rUf f! 9=8Q8Q8W8W8Yv*#u]bfjfufu]u3:vKv+S7C%q"jjQ$]bcfjjGGGG$%..~._K#k15F4HHJ WW\\/3!6rUc cN  " " ' ' 7#~&77J!w%%s&E;>E;.FF c tj|jj}|j d}|j d} d} |dddk(ri|dd<g|dd<np|ddd k(r=|dd ]|dd } |dd| d } | | |vr|| } d |dd <| d gg|dd<n(|ddd vr i|dd<nt d|ddd|"d|dvr|dd|vr||dd|dd<t jtj|} g} |D]b}|j dd}|j dd}|ddd k7r|s5||d|vr ||d|d<| jtd'i|d|| j||dddk(rd|vr|dd |dd|d<|dddk(rd|vr|dd |dd|d<|ddd k(r| | |d<|d |dddk(s*|dddk(r@d|dvr9td|ddDr!tjj|d<t |dd}|d'|| d|}| j#|||| &tj| j}d| vr| dD]}| d|d}||Dcgc]}|j%||}}|| d|d<|D] }| j'|}|t d |Dcgc]}| j'|c}| d|d!<d"D]?}|| vs| |\}}| ||vr||}| j'|}| t d ||g| |<A| |d<t jtj|} |j(j+}t,j.j+}|j1d#|D]}t3||t3||}| ||vr||}|j4j%|d}t7|tr=t||j8|j:|j<|j>d$%||<|||<|j@}||j|tC|d kDr||d#<|jDd'd&| i|Scc}wcc}w)(uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrr$rmergesr%unk_idrg)r&r'z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenrNidrcontinuing_subword_prefixend_of_word_suffix pre_tokenizer ByteLevelSequence pretokenizersc3,K|] }|ddk(yw)rr<NrG).0 pretokenizers rI zBPreTrainedTokenizerFast.train_new_from_iterator..5s" V$ 3sinitial_alphabet)rr)rCtrainerrrzQAttempted to set a token in the post processor that does not exist in the mappingr)clssepr7T) single_wordlstriprstrip normalizedrNr.rG)#r"loadsrXto_strrOrPrSfrom_strr#rprextendanypre_tokenizers_fastr<alphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrUrrYrQrSPECIAL_TOKENS_ATTRIBUTESremovegetattr_special_tokens_maprorGrHrIrJr7rWr)rr text_iteratorrrCnew_special_tokensspecial_tokens_maprttokenizer_jsonr2r3r7r6r5r added_tokenrNr trainer_classrDtrained_tokenizer_jsonrMrr~token_id special_tokenspecial_tokens_listspecial_token_fullr7s rItrain_new_from_iteratorz/PreTrainedTokenizerFast.train_new_from_iteratorsDDOO$:$:$<=%)).9 '++,<= ' "6 *e 3/1N7 #G ,02N7 #H - G $V , 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J/1N7 #G ,Mn]dNeflNmMno>>   *~g66w' 48JJ3EnU\F]^iFj3kN7 #K 0!**4::n+EF ' =K!ooi6Gd+Ag&v.);G!-+i2HL^2^);K /#BB$2?$CO$T *=)F)F)O)O)QF% &01H1PQ _:n_X^_%%mFG%T  %%)ZZ 0@0@0B%C ">1)*:; vC+,<=cB8LF)5TZ![5"4"8"8"F![![FLN#34S9(C!'#,#8#8#?#+", s#ouCuejIDYDYZ_D`CuN#34S9%@ v"0 F  N2-m?( 2EtU#/ 'e 4 %1mGY6Y$6}$EM%)%=%=%A%A%%N"0*=$.%$6$B$B188188#5#@#@ $ %F5M%2F5M% 2(%)$B$B!  ) % , ,-? @ ( )A -2KF. /t~~CyCFCCq"\Dvs SS)NNFFFFT)F)NF)FN)NN)NNN)?__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesr,r__annotations__rbpropertyboolrrrrrrlrrrjrr2rrrSr DecoderFastr EncodingFastrr rrrr rrrrrrrrr!r rrrrrrrrrrr rrPathLiker0rc __classcell__)rs@rIr+r+Qs *04-4m(^GCGG A4S>A tCH~  nd38nnn:d3 ?&;::nc3hnFF = '''1504*/+0',#-(-( (~-( (~ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(^UE#x}2D,EU%PSUYZ]U^P^J_U  7#7(3-76d5j+A&B6]`6 ?d?s?,GLd3i(?C sDI~ 4uSu uRVumqrumvuI9)I90I9 I9  I9 %SM I9tnI9`$(,;,F,F2D2T2T$($),0'+(,0404*/+0',#%*+Y`"' OT-0$7H2I4PeKf f# Y` ! Y` * Y`0Y`SMY`Y`"Y`%SMY`tnY`! Y` (~Y` (~Y` $(!Y`"%)#Y`$!%%Y`&'Y`()Y`*#+Y`, -Y`|DH#',;,F,F2D2T2T$($),0'+)-0404*/+0',#%*);I001;E)->">?@;! ; * ; 0 ;SM;;";%SM;tn;!; (~; (~;$(; %)!;"!%#;$%;&';(#);, -;z tCy S %*-1 d3i("'+  8)-)- /c2;;.//#J/ ~ / "# / s /j rDrKr+)?rgrQr"r collectionsrtypingrrrrrr r tokenizers.pre_tokenizerspre_tokenizersrP tokenizersr rnr rStokenizers.decodersr rmtokenizers.trainersrrrrrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utilsrtokenization_utils_baserrrrrrrrrr utilsr!r"r# get_loggerrdloggerr&SPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEr rRrhr+rGrKrIrs  #DDD7/16^^:5=3   @?   H %"3/'( !! (6EXY,-m D5m D.m DrK