sg7ddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZej"eZdZdd iZgd ZGd d eZy) N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁ vocab_filezsentencepiece.bpe.model)ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc NeZdZUdZeZddgZgZee e d<gZ ee e d< d+de e eefffd Zd Zd Zed Zed efd Zej,ded dfdZ d,dee de ee ded ee ffd Z d-dee de ee d ee fdZ d-dee de ee d ee fdZdede ede efdZdZded eefdZdZdZdZ d-d ed!e ed e!efd"Z" d.d#eeded$e eeded e#f fd% Z$d&Z%d'Z&d/d(Z'd)ed dfd*Z(xZ)S)0MBartTokenizeruT Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc t|trt|ddn|}| in| |_t j d i|j|_|j jt|||_ddddd|_ d|_ t|j |_ ttDcic]"\}}||j|z|jz$c}}|_|jj!Dcic]\}}|| c}}|_t|j t|jz|jz|jd <|jj%|j|jj!Dcic]\}}|| c}}|_t)|jj+}| $|j-| Dcgc] }||vs| c}t/|`d |||||||d| | ||jd || | nd |_|j|j2|_| |_|j9|j2ycc}}wcc}}wcc}}wcc}w) NTF)lstrip normalizedrr )) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr.r) isinstancestrr r.spmSentencePieceProcessorsp_modelLoadrfairseq_tokens_to_idsfairseq_offsetlen sp_model_size enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langcur_lang_code_idrBset_src_lang_special_tokens)selfrr9r:r<r=r;r>r?r@rArBr.rCkwargsicodekv_additional_special_tokenst __class__s _/var/www/html/venv/lib/python3.12/site-packages/transformers/models/mbart/tokenization_mbart.pyrZzMBartTokenizer.__init__?sd&FPPZ\_E`Jz$5 Afp &5%+>> > 261E1E1K1K1MNA1N/24==/ACH\H\D]/]`d`s`s/s""8, ""))$*>*>?7;7Q7Q7W7W7Y%Ztq!ad%Z"%)$*>*>*C*C*E%F" $ 0 & - -5]qB\9\]   !&@ 00  &.%9w $ 4 4T^^ D   ((8G  O&[ ^s;'I6 I<% J/ J9Jc~|jj}d|d<|jj|d<|S)NrIsp_model_proto)__dict__copyrIserialized_model_proto)r^states rg __getstate__zMBartTokenizer.__getstate__s; ""$ j"&--"F"F"H c||_t|dsi|_tjdi|j|_|j j |jy)Nr.rD)rjhasattrr.rGrHrILoadFromSerializedProtori)r^ds rg __setstate__zMBartTokenizer.__setstate__sQ t./#%D 22JT5I5IJ  --d.A.ABrocxt|jt|jz|jzdzS)Nr2)rMrIrQrLr^s rg vocab_sizezMBartTokenizer.vocab_sizes24==!C(<(<$==@S@SSVWWWroreturnc|jSN)r[rvs rgrAzMBartTokenizer.src_langs ~~ro new_src_langcH||_|j|jyrz)r[r])r^r{s rgrAzMBartTokenizer.src_langs% ((8ro token_ids_0 token_ids_1already_has_special_tokensc|rt|||dSdgt|jz}dgt|jz}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r}r~rr2r)rYget_special_tokens_maskrMr,r-)r^r}r~r prefix_ones suffix_onesrfs rgrz&MBartTokenizer.get_special_tokens_masks& &72'[]a3 cC 2 233 cC 2 233  1#K(8"89KG GqcC $445!s;?O9OPS^^^roc|||j|z|jzS|j|z|z|jzS)ab Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r,r-)r^r}r~s rg build_inputs_with_special_tokensz/MBartTokenizer.build_inputs_with_special_tokenssG,  %% 3d6H6HH H!!K/+=@R@RRRroc|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r) sep_token_id cls_token_idrM)r^r}r~sepclss rg$create_token_type_ids_from_sequencesz3MBartTokenizer.create_token_type_ids_from_sequencessm$  !  !  s[(3./1#5 53$s*S0;>DEKKroreturn_tensorsrArBc v|| td||_||fd|d|}|j|}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrAconvert_tokens_to_ids)r^ raw_inputsrrArB extra_kwargsinputs tgt_lang_ids rg_build_translation_inputsz(MBartTokenizer._build_translation_inputssY  x/`a a  jiT.i\hi00: (3$% roct|jDcic]}|j||}}|j|j|Scc}wrz)rangerwconvert_ids_to_tokensrTadded_tokens_encoder)r^r`vocabs rg get_vocabzMBartTokenizer.get_vocabsK;@;QRa++A.1RR T../ SsAtextcD|jj|tS)N)out_type)rIencoderF)r^rs rg _tokenizezMBartTokenizer._tokenize s}}##D3#77roc||jvr|j|S|jj|}|r||jzS|jS)z0Converts a token (str) in an id using the vocab.)rKrI PieceToIdrL unk_token_id)r^tokenspm_ids rg_convert_token_to_idz#MBartTokenizer._convert_token_to_idsU D.. .--e4 4((/06v+++L4;L;LLroc||jvr|j|S|jj||jz S)z=Converts an index (integer) in a token (str) using the vocab.)rUrI IdToPiecerL)r^indexs rg_convert_id_to_tokenz#MBartTokenizer._convert_id_to_tokensA D.. .--e4 4}}&&ut/B/B'BCCrocldj|jtdj}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r^tokens out_strings rgconvert_tokens_to_stringz'MBartTokenizer.convert_tokens_to_strings,WWV_,,-=sCIIK rosave_directoryfilename_prefixctjj|stj d|dytjj ||r|dzndt dz}tjj|jtjj|k7rBtjj|jrt|j||fStjj|jsCt|d5}|jj}|j|ddd|fS|fS#1swY|fSxYw)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenrIrlwrite)r^rrout_vocab_fileficontent_spiece_models rgsave_vocabularyzMBartTokenizer.save_vocabulary$s%ww}}^, LL,^,<>roc8|j|jSrz)set_tgt_lang_special_tokensrBrvs rg_switch_to_target_modez%MBartTokenizer._switch_to_target_modeDrroct|j||_g|_|j|jg|_y)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrQ cur_lang_coder, eos_token_idr-)r^rAs rgr]z*MBartTokenizer.set_src_lang_special_tokensGs6!11(;"//1C1CDrolangct|j||_g|_|j|jg|_y)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r^rs rgrz*MBartTokenizer.set_tgt_lang_special_tokensMs6!11$7"//1C1CDro) r4r6r6r4r7r5r8NNNNN)NFrz)rNr")rxN)*__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr,rint__annotations__r-rrrFrrZrnrtpropertyrwrAsetterboolrrrrrrrrrrrr rrrr]r __classcell__)rfs@rgr)r)$s (*$&67!M49!!M49! 48"&L9"$sCx.1L9\ CXX#__9S9T99 sx_9_3;DI3F_ko_ c_>JNS9S3;DI3FS cS8JNL9L3;DI3FL cL2 *- 9A# RZ[^R_  8c8d3i8MD  !c!HSM!]bcf]g!( )- M9 M MDI& M  M  M??E EEEror))rshutilrtypingrrrrr sentencepiecerGtokenization_utilsr r r utilsr get_loggerrrrrrPr)rDrorgrs_  33PP   H %!#<={mE(mEro