sgIddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZej"eZdZdd d Zgd gd d ZddddddddZGddeZy)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁zsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)__java__ __python__ __en_XX__)rrr__javascript____php____ruby____go__)basemultirrrrrrr)javapythonen_XX javascriptphprubygoc beZdZUdZeZddgZgZee e d<gZ ee e d< d,de e eefffd Zd Zd Zed Zed efd Zej,ded dfdZ d-dee de ee ded ee ffd Z d.dee de ee d ee fdZ d.dee de ee d ee fdZdede ede efdZdZded eefdZdZdZdZ d.d ed!e ed e!efd"Z" d/d#eeded$e eeded e#f fd% Z$d&Z%d'Z&d0d(Z'd)ed dfd*Z(d)ed efd+Z)xZ*S)1PLBartTokenizera Construct an PLBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Args: vocab_file (`str`): Path to the vocabulary file. src_lang (`str`, *optional*): A string representing the source language. tgt_lang (`str`, *optional*): A string representing the target language. bos_token (`str`, *optional*, defaults to `""`): The start of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The cls token, which is a special token used as the first token for all tasks. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token(`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masking tasks. This is only used in the `"base"` tokenizer type. For `"multi"` tokenizer, masking is never done for the downstream tasks. language_codes (`str`, *optional*, defaults to `"base"`): What language codes to use. Should be one of `"base"` or `"multi"`. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import PLBartTokenizer >>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-python-en_XX", src_lang="python", tgt_lang="en_XX") >>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])" >>> expected_translation_english = "Returns the maximum value of a b c." >>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc Ft|trt|ddn|}| in| |_|j | } |j | } t j d i|j|_|jjt|||_ | |_ t|j}ddddd|_ d|_ t|j|_t!|Dcic]"\}}||j|z|jz$c}}|_|j"j%Dcic]\}}|| c}}|_|jd k(rEt|jt|j"z|jz|jd <|jj)|j"|jj%Dcic]\}}|| c}}|_t-|j"j/}|$|j1|Dcgc] }||vs| c}|jd k(r>| |_|j2|j"|j2n |j2|_n)| | nd |_|j"|j2|_t7|pd |||||||| | | | ||j|d || |_|j=|j2ycc}}wcc}}wcc}}wcc}w)NTF)lstriprstriprr )rr) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokenlanguage_codesrsrc_langtgt_langadditional_special_tokensr&clean_up_tokenization_spaces) isinstancestrr r&!_convert_lang_code_special_formatspmSentencePieceProcessorsp_modelLoadrr8FAIRSEQ_LANGUAGE_CODESfairseq_tokens_to_idsfairseq_offsetlen sp_model_size enumeratelang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextend _src_langcur_lang_code_idsuper__init__r:set_src_lang_special_tokens)selfrr1r2r4r5r3r6r7r8rr9r:r&r;r<kwargsfairseq_language_codesicodekv_additional_special_tokenst __class__s a/var/www/html/venv/lib/python3.12/site-packages/transformers/models/plbart/tokenization_plbart.pyrVzPLBartTokenizer.__init__us(KUU_adJeZ 4Fku %4%+>> > 261E1E1K1K1MNA1N   & (36t}}3EDL`L`Ha3adhdwdw3wD & &x 0 ""))$*>*>?7;7Q7Q7W7W7Y%Ztq!ad%Z"%)$*>*>*C*C*E%F" $ 0 & - -5]qB\9\]    & (%DN8<8R$$T^^4X\XfXf  !*2)=X;DN$($8$8$HD !  !))&@ 00)E  $!  ((8]  O &[ ^s3'L  L, L6 LLc~|jj}d|d<|jj|d<|S)NrCsp_model_proto)__dict__copyrCserialized_model_proto)rXstates rb __getstate__zPLBartTokenizer.__getstate__s; ""$ j"&--"F"F"H c||_t|dsi|_tjdi|j|_|j j |jy)Nr&r=)rehasattrr&rArBrCLoadFromSerializedProtord)rXds rb __setstate__zPLBartTokenizer.__setstate__sQ t./#%D 22JT5I5IJ  --d.A.ABrjc|jdk(r;t|jt|jz|jzdzSt|jt|jz|jzS)Nrr*)r8rHrCrKrGrXs rb vocab_sizezPLBartTokenizer.vocab_sizesn   & (DMM"S)=)=%>>ATATTWXX t}}%D,@,@(AADDWDWW Wrjreturnc|jSN)rSrqs rbr9zPLBartTokenizer.src_langs ~~rj new_src_langcj|j|}||_|j|jyru)r@rSrW)rXrvs rbr9zPLBartTokenizer.src_langs+==lK % ((8rj token_ids_0 token_ids_1already_has_special_tokensc|rt|||dSdgt|jz}dgt|jz}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rxryrzr*r)rUget_special_tokens_maskrHr$r%)rXrxryrz prefix_ones suffix_onesras rbr|z'PLBartTokenizer.get_special_tokens_masks& &72'[]a3 cC 2 233 cC 2 233  1#K(8"89KG GqcC $445!s;?O9OPS^^^rjc|||j|z|jzS|j|z|z|jzS)ac Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An PLBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r$r%)rXrxrys rb build_inputs_with_special_tokensz0PLBartTokenizer.build_inputs_with_special_tokenssG,  %% 3d6H6HH H!!K/+=@R@RRRrjc|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. PLBart does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r) sep_token_id cls_token_idrH)rXrxrysepclss rb$create_token_type_ids_from_sequencesz4PLBartTokenizer.create_token_type_ids_from_sequences/sm"  !  !  s[(3./1#5 53$s*S0;>DEKKrjreturn_tensorsr9r:c || td|j||_|j||_||fd|d|}|j |j}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorr@r9r:convert_tokens_to_ids)rX raw_inputsrr9r: extra_kwargsinputs tgt_lang_ids rb_build_translation_inputsz)PLBartTokenizer._build_translation_inputsGsz  x/`a a>>xH >>xH jiT.i\hi00? (3$% rjct|jDcic]}|j||}}|j|j|Scc}wru)rangerrconvert_ids_to_tokensrNadded_tokens_encoder)rXr[vocabs rb get_vocabzPLBartTokenizer.get_vocabTsK;@;QRa++A.1RR T../ SsAtextcD|jj|tS)N)out_type)rCencoder?)rXrs rb _tokenizezPLBartTokenizer._tokenizeYs}}##D3#77rjc||jvr|j|S|jj|}|r||jzS|jS)z0Converts a token (str) in an id using the vocab.)rFrC PieceToIdrG unk_token_id)rXtokenspm_ids rb_convert_token_to_idz$PLBartTokenizer._convert_token_to_id\sU D.. .--e4 4((/06v+++L4;L;LLrjc||jvr|j|S|jj||jz S)z=Converts an index (integer) in a token (str) using the vocab.)rOrC IdToPiecerG)rXindexs rb_convert_id_to_tokenz$PLBartTokenizer._convert_id_to_tokenesA D.. .--e4 4}}&&ut/B/B'BCCrjcldj|jtdj}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)rXtokens out_strings rbconvert_tokens_to_stringz(PLBartTokenizer.convert_tokens_to_stringks,WWV_,,-=sCIIK rjsave_directoryfilename_prefixctjj|stj d|dytjj ||r|dzndt dz}tjj|jtjj|k7rBtjj|jrt|j||fStjj|jsCt|d5}|jj}|j|ddd|fS|fS#1swY|fSxYw)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenrCrgwrite)rXrrout_vocab_fileficontent_spiece_models rbsave_vocabularyzPLBartTokenizer.save_vocabularyps%ww}}^, LL,^,<>xH >>xH w,Y LVLLrjc8|j|jSru)rWr9rqs rb_switch_to_input_modez%PLBartTokenizer._switch_to_input_mode// >>rjc8|j|jSru)set_tgt_lang_special_tokensr:rqs rb_switch_to_target_modez&PLBartTokenizer._switch_to_target_moderrjc|j|}||j|nd|_g|_|j|j|jg|_y|jg|_y)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].Nr@rK cur_lang_coder$ eos_token_idr%)rXr9s rbrWz+PLBartTokenizer.set_src_lang_special_tokenssk99(C?G?ST11(;Y]    )"&"3"3T5G5G!HD "&"3"3!4D rjlangc|j|}||j|nd|_g|_|j|j|jg|_y|jg|_y)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].NrrXrs rbrz+PLBartTokenizer.set_tgt_lang_special_tokenssk55d;;?;KT11$7QU    )"&"3"3T5G5G!HD "&"3"3!4D rjcL|tjvr t|}|S|}|S)z;Convert Language Codes to format tokenizer uses if required)FAIRSEQ_LANGUAGE_CODES_MAPrQrs rbr@z1PLBartTokenizer._convert_lang_code_special_formats137;U;Z;Z;\3\)$/ cg rj)r,r.r.r,r/r-r0rNNNNNT)NFru)rNr)rsN)+__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr$rint__annotations__r%rrr?rrVriropropertyrrr9setterboolr|rrrrrrrrrrr rrrrWrr@ __classcell__)ras@rbr!r!1s; z*$&67!M49!!M49! 48"&%)!\9"$sCx.1\9| CXX#__9S9T99 sx_9_3;DI3F_ko_ c_>JNS9S3;DI3FS cS8JNL9L3;DI3FL cL0 *- 9A# RZ[^R_  8c8d3i8MD  !c!HSM!]bcf]g!( )- M9 M MDI& M  M  M??5 5 5 5ccrjr!)rshutilrtypingrrrrr sentencepiecerAtokenization_utilsr r r utilsr get_loggerrrrrrErr!r=rjrbrs  33PP   H %#