sgA&ddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZej(eZdd d d d d ZdZGddeZdede ee fdej6fdZdeddfdZdedee e ffdZy)N)Path)copyfile)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)loggingz source.spmz target.spmz vocab.jsonztarget_vocab.jsonztokenizer_config.json) source_spm target_spmvocabtarget_vocab_filetokenizer_config_fileu▁c eZdZdZeZddgZejdZ d)de e e e fddffd Zd Zd e de fd Zd Zd e fdZd e dee fdZdede fdZfdZfdZdee de fdZd*deefdZdZdZedefdZd*de de e dee fdZ de fdZ!dZ"dZ#de fd Z$d!e ddfd"Z%d#Z&d$Z' d+d%ed&e ed'e(deefd(Z)xZ*S),MarianTokenizeraB Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: source_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the source language. target_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the target language. source_lang (`str`, *optional*): A string representing the source language. target_lang (`str`, *optional*): A string representing the target language. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`List[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ``` input_idsattention_maskz>>.+< token must be in the vocabz>>z<<) source_lang target_lang unk_token eos_token pad_tokenmodel_max_lengthrrseparate_vocabs)rrexistsr load_jsonencoderstrKeyErrortarget_encoderitemsdecodersupported_language_codes startswithendswithrr spm_filesload_spm spm_source spm_target current_spmcurrent_encoder_setup_normalizersuper__init__)selfrrrrrrrrrrrr kwargskv __class__s a/var/www/html/venv/lib/python3.12/site-packages/transformers/models/marian/tokenization_marian.pyr5zMarianTokenizer.__init__ks &5% -=> >9~--- "+,=">D -1-@-@-F-F-HITQAqDIDL,.D )-1\\-?-?-ABTQAqDBDL>Bll2vall[_N`efeoeopteu12vD )&&$j1#:t/C/CD":t/C/CD??#||    ##- 00/+  )JC2vs1 G(, G.G4'G49G4c ddlm}||jj|_y#t t f$r tjdd|_YywxYw)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.c|SNr!)xs r;z3MarianTokenizer._setup_normalizer..sQ) sacremosesr=r normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)r6r=s r;r3z!MarianTokenizer._setup_normalizersM / 7#78H8H#I#S#SD ./ / MM@ A#.D  /s'*,AAr@c,|r|j|SdS)zHCover moses empty string edge case. They return empty list for '' input!)rE)r6r@s r;rDzMarianTokenizer.normalizes*+t##A&33rBch|jj||j|jSr?)r2getr)r6tokens r;_convert_token_to_idz$MarianTokenizer._convert_token_to_ids*##''t/C/CDNN/STTrBtextc|jj|}|r|jdgng}||jjd|fS)z6Remove language codes like >>fr<< before sentencepiecerrK)language_code_rematchgroupsub)r6rPrScodes r;remove_language_codez$MarianTokenizer.remove_language_codesJ%%++D1).ekk!n%BT**..r4888rBcv|j|\}}|jj|t}||zS)N)out_type)rWr1encoder%)r6rPrVpiecess r; _tokenizezMarianTokenizer._tokenizes;..t4 d!!(((<f}rBindexcN|jj||jS)z?Converts an index (integer) in a token (str) using the decoder.)r)rMr)r6r]s r;_convert_id_to_tokenz$MarianTokenizer._convert_id_to_tokens||t~~66rBc $t||fi|S)ao Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. )r4 batch_decode)r6 sequencesr7r:s r;razMarianTokenizer.batch_decodes*w#I888rBc $t||fi|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )r4decode)r6 token_idsr7r:s r;rdzMarianTokenizer.decodes0w~i2622rBtokenscL|jr |jn |j}g}d}|D]>}||jvr||j ||zdzz }g}.|j |@||j |z }|j td}|jS)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserK ) _decode_use_source_tokenizerr/r0all_special_tokens decode_piecesappendreplaceSPIECE_UNDERLINEstrip)r6rfsp_modelcurrent_sub_tokens out_stringrNs r;convert_tokens_to_stringz(MarianTokenizer.convert_tokens_to_strings&*&G&G4??T__  1E///h445GH5PSVVV %'""))%0  1 h,,-?@@ ''(8#> !!rBcL|||jgzS||z|jgzS)z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)r6 token_ids_0 token_ids_1s r; build_inputs_with_special_tokensz0MarianTokenizer.build_inputs_with_special_tokens s5  $"3"3!44 4[(D,=,=+>>>rBcH|j|_|j|_yr?)r/r1r$r2r6s r;_switch_to_input_modez%MarianTokenizer._switch_to_input_modes??#||rBcb|j|_|jr|j|_yyr?)r0r1r r'r2rzs r;_switch_to_target_modez&MarianTokenizer._switch_to_target_modes*??   #'#6#6D  rBc,t|jSr?)lenr$rzs r; vocab_sizezMarianTokenizer.vocab_sizes4<<  rBsave_directoryfilename_prefixcztjj|stj d|dyg}|j rtjj ||r|dzndtdz}tjj ||r|dzndtdz}t|j|t|j||j||j|nXtjj ||r|dzndtdz}t|j||j|ttdtdg|j|j|jgD]\}}} tjj ||r|dznd|z} tjj!|tjj!| k7r=tjj#|rt%|| |j| tjj#|rt'| d 5} | j)} | j+| ddd|j| t-|S#1swY(xYw) NzVocabulary path (z) should be a directory-rKrrrrwb)ospathisdirloggererrorr joinVOCAB_FILES_NAMES save_jsonr$r'rlzipr-r/r0abspathisfileropenserialized_model_protowritetuple) r6rr saved_filesout_src_vocab_fileout_tgt_vocab_fileout_vocab_filespm_save_filename spm_orig_path spm_model spm_save_pathficontent_spiece_models r;save_vocabularyzMarianTokenizer.save_vocabularysLww}}^, LL,^,<   1 2   1 2WW\\/3!6rUfgnUo oN dllN 3   ~ .;> | ,.? .M N NN __doo .<  2 7 }i GGLL/3!6rUf fMww}-1OOTVT[T[TbTbcpTq 6""=1WW^^M2-.3"+4+K+K+M(HH123""=1 2"[!! 33s ("J11J: c"|jSr?) get_src_vocabrzs r; get_vocabzMarianTokenizer.get_vocabJs!!##rBcBt|jfi|jSr?)dictr$added_tokens_encoderrzs r;rzMarianTokenizer.get_src_vocabMsDLL>D$=$=>>rBcBt|jfi|jSr?)rr'added_tokens_decoderrzs r; get_tgt_vocabzMarianTokenizer.get_tgt_vocabPsD''E4+D+DEErBc|jj}|jdDcic]}|dc}|Scc}w)N)r/r0r1rEr)__dict__copyupdate)r6stater8s r; __getstate__zMarianTokenizer.__getstate__Ss? ""$ q rQW r   ss =dc|_tdsi_fdjD\__j_jy)Nrc3JK|]}t|jywr?)r.r).0fr6s r; z/MarianTokenizer.__setstate__..as+fRSHQ8L8L,M+fs #)rhasattrrr-r/r0r1r3)r6rs` r; __setstate__zMarianTokenizer.__setstate__ZsQ t./#%D +fW[WeWe+f(??  rBcy)zJust EOSr!)r6argsr7s r;num_special_tokens_to_addz)MarianTokenizer.num_special_tokens_to_addesrBct|j}|j|j|Dcgc] }||vrdnd c}Scc}w)Nrr)setall_special_idsremove unk_token_id)r6seqrr@s r;_special_token_maskz#MarianTokenizer._special_token_maskisFd223t001:=>QQ/)q0>>>sArvrwalready_has_special_tokensc|r|j|S||j|dgzS|j||zdgzS)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.r)r)r6rvrwrs r;get_special_tokens_maskz'MarianTokenizer.get_special_tokens_masknsS &++K8 8  ++K8A3> >++K+,EF!L LrB) NNNzzziNFr?)NF)+__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrecompilerRrrr%rr5r3rDrOrWrr\intr_rardrsrxr{r}propertyrr rrrrrrrrboolr __classcell__)r:s@r;rr,s8 t*$&67!rzz(+48< "$sCx.1<  < |/43434U99 cd3i 7#7#79.34"tCy"S" ?QUVYQZ?,7 !C!!+"c+"HSM+"]bcf]g+"Z$4$?Fd !d !t !? in M M.6tn Mae M c MrBrrrrcRtjdi|}|j||S)Nr!) sentencepieceSentencePieceProcessorLoad)rrspms r;r.r.zs%  . . A ACHHTN JrBcvt|d5}tj||ddddy#1swYyxYw)Nw)indent)rjsondump)datarrs r;rrs2 dC%A $!$%%%s/8cpt|d5}tj|cdddS#1swYyxYw)Nr)rrload)rrs r;r#r#s- dCAyy|s,5)rrrrHpathlibrshutilrtypingrrrrr r rtokenization_utilsr utilsr get_loggerrrrrnrr%rr.rr#r!rBr;rs ::5   H % ,4  KM)KM\ 3c3hM<`<` %#%$% CE$*-rB