sg*ddlZddlmZddlmZmZmZddlmZddl m Z m Z ddl m Z ddlmZmZerd d lmZndZej&eZd d d ZgdZGdde Zy)N)copyfile)ListOptionalTuple) processors) AddedToken BatchEncoding)PreTrainedTokenizerFast)is_sentencepiece_availablelogging)MBartTokenizerzsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc eZdZUdZeZddgZeZgZ e e e d<gZ e e e d< d fd Zedefd Zedefd Zej(d eddfd Z d!d e e dee e de e fdZ d!d e e dee e de e fdZdedeedeefdZ d"de ededee ededef fd ZdZdZd#dZdeddfdZd!dedeedeefdZ xZ!S)$MBartTokenizerFastuO Construct a "fast" MBART tokenizer (backed by HuggingFace's *tokenizers* library). Based on [BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models). This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizerFast >>> tokenizer = MBartTokenizerFast.from_pretrained( ... "facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO" ... ) >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNc  t| trt| ddn| } tj }| $|j | Dcgc] }||vs| c}t |d||||||||| | | |d | ||_tDcic]}||j|c}|_ | | nd|_ |j|j|_ | |_ |j|jycc}wcc}w)NTF)lstriprstrip) rr bos_token eos_token sep_token cls_token unk_token pad_token mask_tokensrc_langtgt_langadditional_special_tokensr) isinstancestrr FAIRSEQ_LANGUAGE_CODEScopyextendsuper__init__rconvert_tokens_to_idslang_code_to_id _src_lang cur_lang_coder<set_src_lang_special_tokens)selfrrr4r5r6r7r8r9r:r;r<r=kwargs_additional_special_tokenst lang_code __class__s d/var/www/html/venv/lib/python3.12/site-packages/transformers/models/mbart/tokenization_mbart_fast.pyrEzMBartTokenizerFast.__init__Is"KUU_adJeZ 4Fku %;%@%@%B" $ 0 & - -5]qB\9\]   !)!&@  %Nd AJIt11)< < &.%9w!77G   ((87^( s C6C6C;returncp|jr)tjj|jSdS)NF)rospathisfilerKs rQcan_save_slow_tokenizerz*MBartTokenizerFast.can_save_slow_tokenizer~s$26//rww~~doo.LuLc|jSN)rHrWs rQr;zMBartTokenizerFast.src_langs ~~rY new_src_langcH||_|j|jyr[)rHrJ)rKr\s rQr;zMBartTokenizerFast.src_langs% ((8rY token_ids_0 token_ids_1c|||j|z|jzS|j|z|z|jzS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. The special tokens depend on calling set_lang. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r/r0)rKr^r_s rQ build_inputs_with_special_tokensz3MBartTokenizerFast.build_inputs_with_special_tokenssG0  %% 3d6H6HH H!!K/+=@R@RRRrYc|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r) sep_token_id cls_token_idlen)rKr^r_sepclss rQ$create_token_type_ids_from_sequencesz7MBartTokenizerFast.create_token_type_ids_from_sequencessm$  !  !  s[(3./1#5 53$s*S0;>DEKKrYreturn_tensorsr;r<c v|| td||_||fd|d|}|j|}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensriforced_bos_token_id) ValueErrorr;rF)rK raw_inputsrir;r< extra_kwargsinputs tgt_lang_ids rQ_build_translation_inputsz,MBartTokenizerFast._build_translation_inputssY  x/`a a  jiT.i\hi00: (3$% rY src_texts tgt_textsc B||_||_t| ||fi|Sr[)r;r<rDprepare_seq2seq_batch)rKrsr;rtr<rLrPs rQrvz(MBartTokenizerFast.prepare_seq2seq_batchs*!   w,Y LVLLrYc8|j|jSr[)rJr;rWs rQ_switch_to_input_modez(MBartTokenizerFast._switch_to_input_mode// >>rYc8|j|jSr[)set_tgt_lang_special_tokensr<rWs rQ_switch_to_target_modez)MBartTokenizerFast._switch_to_target_moderyrYc |j||_g|_|j|jg|_|j |j}|j |j}t j|dgz|z|ddgz|ztt||z|j|jz|j_ y)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].$A$Bsinglepairspecial_tokensN rFrIr/ eos_token_idr0convert_ids_to_tokensrTemplateProcessinglistzip _tokenizerpost_processor)rKr;prefix_tokens_strsuffix_tokens_strs rQrJz.MBartTokenizerFast.set_src_lang_special_tokenss!77A"//1C1CD 66t7I7IJ 66t7I7IJ)3)F)F$v-0AA"dD\14EE$58I$I4K]K]`d`r`rKr st* &rYlangc |j||_g|_|j|jg|_|j |j}|j |j}t j|dgz|z|ddgz|ztt||z|j|jz|j_ y)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].r~rrNr)rKrrrs rQr{z.MBartTokenizerFast.set_tgt_lang_special_tokenss!77="//1C1CD 66t7I7IJ 66t7I7IJ)3)F)F$v-0AA"dD\14EE$58I$I4K]K]`d`r`rKr st* &rYsave_directoryfilename_prefixc|js tdtjj |st j d|dytjj||r|dzndtdz}tjj|jtjj|k7rt|j||fS)NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory.-r) rXrmrTrUisdirloggererrorjoinVOCAB_FILES_NAMESabspathrr)rKrrout_vocab_files rQsave_vocabularyz"MBartTokenizerFast.save_vocabularys++  ww}}^, LL,^,<rrzzzNNNr[)rNr%)rRN)"__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrslow_tokenizer_classr/rint__annotations__r0rEpropertyboolrXr@r;setterrrarhrrr rvrxr|rJr{rr __classcell__)rPs@rQr,r,*s .*$&67)!M49!!M49!"&39jMMM#__9S9T99 JNS9S3;DI3FS cS<JNL9L3;DI3FL cL2 *- 9A# RZ[^R_  )- M9 M MDI& M  M  M??        !c!HSM!]bcf]g!rYr,)rTshutilrtypingrrr tokenizersrtokenization_utilsr r tokenization_utils_fastr utilsr r tokenization_mbartr get_loggerrrrrAr,r>rYrQrsi  ((!;>82N   H %$=P`a{d!0d!rY