sgJ-ddlZddlmZddlmZmZmZddlmZddl m Z m Z ddl m Z ddlmZmZerd d lmZndZej&eZd d d ZgdZGdde Zy)N)copyfile)ListOptionalTuple) processors) AddedToken BatchEncoding)PreTrainedTokenizerFast)is_sentencepiece_availablelogging)MBart50Tokenizerzsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)4ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNaf_ZAaz_AZbn_INfa_IRhe_ILhr_HRid_IDka_GEkm_KHmk_MKml_INmn_MNmr_INpl_PLps_AFpt_XXsv_SEsw_KEta_INte_INth_THtl_XXuk_UAur_PKxh_ZAgl_ESsl_SIc eZdZUdZeZddgZeZgZ e e e d<gZ e e e d< dfd Zedefd Zedefd Zej(d eddfd Z dd e e dee e de e fdZ d de ededee ededef fd ZdZdZdeddfdZdeddfdZdedeedeefdZddedeedeefdZxZ S)!MBart50TokenizerFastu Construct a "fast" MBART tokenizer for mBART-50 (backed by HuggingFace's *tokenizers* library). Based on [BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. src_lang (`str`, *optional*): A string representing the source language. tgt_lang (`str`, *optional*): A string representing the target language. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. Examples: ```python >>> from transformers import MBart50TokenizerFast >>> tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50", src_lang="en_XX", tgt_lang="ro_RO") >>> src_text = " UN Chief Says There Is No Military Solution in Syria" >>> tgt_text = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt") >>> # model(**model_inputs) should work ``` input_idsattention_mask prefix_tokens suffix_tokensNc t| trt| ddn| } | jdgxsg| d<| dxxtD cgc] } | | dvs | c} z cc<t ||f|||||||| | d | ||_tD cic]} | |j| c} |_ ||nd|_ ||_ |j|j|_ |j|jycc} wcc} w)NTF)lstriprstripadditional_special_tokens) src_langtgt_langr eos_token sep_token cls_token unk_token pad_token mask_tokenr) isinstancestrr getFAIRSEQ_LANGUAGE_CODESsuper__init__rconvert_tokens_to_idslang_code_to_id _src_langrQcur_lang_code_idset_src_lang_special_tokens)selfrrPrQrrRrSrTrUrVrWkwargscode lang_code __class__s h/var/www/html/venv/lib/python3.12/site-packages/transformers/models/mbart50/tokenization_mbart50_fast.pyr]zMBart50TokenizerFast.__init__]s-KUU_adJeZ 4Fku .4jj9TVX.Y._]_*+*+30 t6JeCf7fD0  +   )!  %Oe AJIt11)< < &.%9w  $ 4 4T^^ D ((870 ( s C8 C8 C=returncp|jr)tjj|jSdS)NF)rospathisfilercs rhcan_save_slow_tokenizerz,MBart50TokenizerFast.can_save_slow_tokenizers$26//rww~~doo.LuLc|jSN)r`rns rhrPzMBart50TokenizerFast.src_langs ~~rp new_src_langcH||_|j|jyrr)r`rb)rcrss rhrPzMBart50TokenizerFast.src_langs% ((8rp token_ids_0 token_ids_1c|||j|z|jzS|j|z|z|jzS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. The special tokens depend on calling set_lang. An MBART-50 sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `[src_lang_code] X [eos]` - `labels`: (for decoder) `[tgt_lang_code] X [eos]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens. )rJrK)rcrurvs rh build_inputs_with_special_tokensz5MBart50TokenizerFast.build_inputs_with_special_tokenssG0  %% 3d6H6HH H!!K/+=@R@RRRrp src_textsrP tgt_textsrQc B||_||_t| ||fi|Srr)rPrQr\prepare_seq2seq_batch)rcryrPrzrQrdrgs rhr|z*MBart50TokenizerFast.prepare_seq2seq_batchs*!   w,Y LVLLrpc8|j|jSrr)rbrPrns rh_switch_to_input_modez*MBart50TokenizerFast._switch_to_input_mode// >>rpc8|j|jSrr)set_tgt_lang_special_tokensrQrns rh_switch_to_target_modez+MBart50TokenizerFast._switch_to_target_moderrpc |j||_|jg|_|jg|_|j |j}|j |j}t j|dgz|z|ddgz|ztt||z|j|jz|j_ y)z]Reset the special tokens to the source lang setting. prefix=[src_lang_code] and suffix=[eos].$A$Bsinglepairspecial_tokensN r^rarJ eos_token_idrKconvert_ids_to_tokensrTemplateProcessinglistzip _tokenizerpost_processor)rcrPprefix_tokens_strsuffix_tokens_strs rhrbz0MBart50TokenizerFast.set_src_lang_special_tokens $ : :8 D"334"//0 66t7I7IJ 66t7I7IJ)3)F)F$v-0AA"dD\14EE$58I$I4K]K]`d`r`rKr st* &rpc |j||_|jg|_|jg|_|j |j}|j |j}t j|dgz|z|ddgz|ztt||z|j|jz|j_ y)zaReset the special tokens to the target language setting. prefix=[src_lang_code] and suffix=[eos].rrrNr)rcrQrrs rhrz0MBart50TokenizerFast.set_tgt_lang_special_tokensrrpreturn_tensorsc v|| td||_||fd|d|}|j|}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrPr^)rc raw_inputsrrPrQ extra_kwargsinputs tgt_lang_ids rh_build_translation_inputsz.MBart50TokenizerFast._build_translation_inputssY  x/`a a  jiT.i\hi00: (3$% rpsave_directoryfilename_prefixc|js tdtjj |st j d|dytjj||r|dzndtdz}tjj|jtjj|k7rt|j||fS)NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory-r) rorrkrlisdirloggererrorjoinVOCAB_FILES_NAMESabspathrr)rcrrout_vocab_files rhsave_vocabularyz$MBart50TokenizerFast.save_vocabularys++  ww}}^, LL,^,<rzzzzrr)rNr%)!__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrslow_tokenizer_classrJrint__annotations__rKr]propertyboolrorYrPsetterrrxr r|r~rrbrrrr __classcell__)rgs@rhrGrG)s* X*$&67+!M49!!M49!-9^MMM#__9S9T99 JNS9S3;DI3FS cS@ )- M9 M MDI& M  M  M??  C  D    C  D   *- 9A# RZ[^R_ !c!HSM!]bcf]g!rprG)rkshutilrtypingrrr tokenizersrtokenization_utilsr r tokenization_utils_fastr utilsr r tokenization_mbart50r get_loggerrrrr[rGrprhrsh  ((!;>86   H %#