sg!KdZddlZddlZddlZddlZddlmZmZmZm Z ddl m Z ddl m Z e jeZddd d Zd Zd Zd ZGdde Zy)zTokenization classes for FSMT.N)DictListOptionalTuple)PreTrainedTokenizer)loggingzvocab-src.jsonzvocab-tgt.jsonz merges.txt)src_vocab_filetgt_vocab_file merges_filecbt}|d}|ddD]}|j||f|}|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars ]/var/www/html/venv/lib/python3.12/site-packages/transformers/models/fsmt/tokenization_fsmt.py get_pairsr$sF EEQIQR 9d#$  Lc*|jdd}tjdd|}|jdd}|jdd}|jdd}|jd d }|jd d }|jd d }|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jd d!}|jd"d#}|jd$d%}|jd&d'}|jd(d)}|jd*d+}|jd,d-}tjd.d|}|jd/d0}|jd1d2}|jd3d4}|jd5d6}|jd7d8}|jd9d:}|jd;d<}|jd=d>}|jd?d@}|S)Azz Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/replace-unicode-punctuation.perl u，,u。\s*z. u、u”"u“u∶:u：u？?u《u》u）)u！!u（(u；;u１1u」u「u０0u３3u２2u５5u６6u９9u７7u８8u４4u．\s*u～~u’'u…z...u━-u〈u【[u】]u％%)replaceresub)texts rreplace_unicode_punctr71sM <<s #D 66)T4 (D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D 66)T4 (D <<s #D <<s #D <<u %D <<s #D <<s #D <<s #D <<s #D <<s #D <<s #D Krcg}|D]:}tj|}|jdr*|j|<dj |S)zw Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/remove-non-printing-char.perl C) unicodedatacategory startswithappendjoin)r6outputrcats rremove_non_printing_charrB\sSF""4( >>#   d  776?rc eZdZdZeZddgZ d fd Zdee e ffdZ e de fdZ dZd Zd Zd Ze d Ze d ZdZdZdZd!dZdZdZdZ d"dee deee dee fdZ d#dee deee dedee ffd Z d"dee deee dee fdZ d"de dee de!e fdZ"dZ#dZ$xZ%S)$ FSMTTokenizera Construct an FAIRSEQ Transformer tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following: - Moses preprocessing and tokenization. - Normalizing all inputs text. - The arguments `special_tokens` and the function `set_special_tokens`, can be used to add additional symbols (like "__classify__") to a vocabulary. - The argument `langs` defines a pair of languages. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: langs (`List[str]`, *optional*): A list of two languages to translate from and to, for instance `["en", "ru"]`. src_vocab_file (`str`, *optional*): File containing the vocabulary for the source language. tgt_vocab_file (`st`, *optional*): File containing the vocabulary for the target language. merges_file (`str`, *optional*): File containing the merges. do_lower_case (`bool`, *optional*, defaults to `False`): Whether or not to lowercase the input when tokenizing. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc  ddl} | |_||_||_||_||_i|_i|_i|_ |rt|dk(r|\|_ |_ ntd|dt|d5} tj | |_dddt|d5} tj | }|j%Dcic]\}}|| c}}|_dddt|d5}|j)j+ddd }dddDcgc]}t-|j+dd }}t/t1|t3t||_i|_t9|td ||||||||| d | y#t$r tdwxYw#1swY$xYwcc}}w#1swYxYw#1swYxYwcc}w) NrnYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.zFarg `langs` needs to be a list of 2 langs, e.g. ['en', 'ru'], but got zw. Usually that means that tokenizer can't find a mapping for the given model path in and other maps of this tokenizer.utf-8encoding ) langsr r r do_lower_case unk_token bos_token sep_token pad_token) sacremoses ImportErrorsmr r r rPcache_moses_punct_normalizercache_moses_tokenizercache_moses_detokenizerlensrc_langtgt_lang ValueErroropenjsonloadencoderitemsdecoderreadsplittupledictziprange bpe_rankscachesuper__init__)selfrOr r r rPrQrRrSrTkwargsrVsrc_vocab_handletgt_vocab_handle tgt_vocabkv merges_handlemergesmerge __class__s rrozFSMTTokenizer.__init__s  ,,&*-/)%'"')$ SZ1_+0 (DM4=XY^X_`88  .7 3 77G99%56DL 7 .7 3 @7G "23I-6__->?TQAqD?DL @+ 0 ;M"'')//5cr:F ;8>?u% bq)*??c&%F *<=>   ))#'  M M  8 7 7@ @ @ ; ;?sGF ;F8+)G  G! G ?#G.#G# F58GG  GG returnc"|jSN) get_src_vocabrps r get_vocabzFSMTTokenizer.get_vocabs!!##rc|jSr})src_vocab_sizers r vocab_sizezFSMTTokenizer.vocab_sizes"""rc||jvr+|jj|}||j|<|j|j|SNlang)rYrXMosesPunctNormalizer normalize)rpr6rpunct_normalizers rmoses_punct_normzFSMTTokenizer.moses_punct_normsT t88 8#ww;;;F 6FD - -d 3006@@FFrc||jvr+|jj|}||j|<|j|j|dddS)NrTF)aggressive_dash_splits return_strescape)rZrXMosesTokenizertokenize)rpr6rmoses_tokenizers rmoses_tokenizezFSMTTokenizer.moses_tokenizesc t11 1"gg44$4?O/>D & &t ,))$/88 %9  rc||jvr+|jj|}||j|<|j|j|Sr)r[rXMosesDetokenizer detokenize)rptokensrmoses_detokenizers rmoses_detokenizezFSMTTokenizer.moses_detokenizesT t33 3 $ 8 8d 8 C 1BD ( ( .++D1<D$=$=>>rcBt|jfi|jSr})rireadded_tokens_decoderrs r get_tgt_vocabzFSMTTokenizer.get_tgt_vocabrrct|dd|ddzfz}|jvrj|St|}|s|dzS t|fd}|jvrn|\}}g}d}|t |kr |j ||} |j||| | }|||k(r6|t |dz kr%||dz|k(r|j||z|dz }n|j|||dz }|t |krt|}|}t |dk(rn t|}dj|}|d k(rd }|j|<|S#t$r|j||dYpwxYw) NrNcNjj|tdS)Ninf)rlgetfloat)pairrps rz#FSMTTokenizer.bpe.. s1C1CD%PU,1WrkeyrrrI z z ) rhrmrminrlr\indexextendr_r>r?) rptokenrrbigramfirstsecondnew_wordijs ` rbpezFSMTTokenizer.bpesU3BZ E"I$6#88 DJJ ::e$ $$6> !$WXFT^^+"ME6HAc$i- 5!,A OOD1I.A7e#CIM(9d1q5kV>SOOEFN3FAOODG,FAc$i- XHD4yA~!$9:xx~ : D  5 /"OODH-sE F?FcP|j}|jr|j}|r|j}n&|j ||}|j ||}g}|D]=}|s|j t|j|jd?|S)av Tokenize a string given language code using Moses. Details of tokenization: - [sacremoses](https://github.com/alvations/sacremoses): port of Moses - Install with `pip install sacremoses` Args: - lang: ISO language code (default = 'en') (string). Languages should belong of the model supported languages. However, we don't enforce it. - bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False) (bool). If True, we only apply BPE. Returns: List of tokens. rr) r]rPlowerrgrrrlistr)rpr6rbypass_tokenizer split_tokensrs r _tokenizezFSMTTokenizer._tokenizeBs*}}   ::)>s)C$DE Frc|jj||jj|jS)z0Converts a token (str) in an id using the vocab.)rcrrQ)rprs r_convert_token_to_idz"FSMTTokenizer._convert_token_to_idis,||t||'7'7'GHHrcN|jj||jS)z=Converts an index (integer) in a token (str) using the vocab.)rerrQ)rprs r_convert_id_to_tokenz"FSMTTokenizer._convert_id_to_tokenms||t~~66rc|Dcgc]$}|jddjdd&}}dj|j}|j||j}|Scc}w)z:Converts a sequence of tokens (string) in a single string.rr:r)r3r?rgrr^)rprtr6s rconvert_tokens_to_stringz&FSMTTokenizer.convert_tokens_to_stringqshDJJa!))C$,,VS9JJ&&($$VT]]; Ks)A- token_ids_0 token_ids_1c@|jg}|||zS||z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A FAIRSEQ Transformer sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_idrprrseps r build_inputs_with_special_tokensz.FSMTTokenizer.build_inputs_with_special_tokens{s:&  !  $ $S ;.44ralready_has_special_tokensc|rt|||dS|'dgt|zdgzdgt|zzdgzSdgt|zdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrr)rnget_special_tokens_maskr\)rprrrrzs rrz%FSMTTokenizer.get_special_tokens_masksz& &72'[]a3   "C#k**qc1aS3{;K5KLPQsR Rc+&&1#--rc|jg}|t||zdgzSt||zdgzt||zdgzzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ Transformer sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An FAIRSEQ_TRANSFORMER sequence pair mask has the following format: rr)rr\rs r$create_token_type_ids_from_sequencesz2FSMTTokenizer.create_token_type_ids_from_sequencess^4  !  {S()QC/ /;$%+c+2C.Ds.JJJrsave_directoryfilename_prefixc tjj|stj d|dytjj ||r|dzndt dz}tjj ||r|dzndt dz}tjj ||r|dzndt dz}t|dd 5}|jtj|jd d d dzdddt|dd 5}|jjDcic]\}}|| } }}|jtj| d d d dzdddd} t|dd 5} t|jjdD]M\} } | | k7rtj!d|d| } | jdj | dz| dz } O ddd|||fS#1swYxYwcc}}w#1swYxYw#1swY-xYw)NzVocabulary path (z) should be a directoryr-r:r r r wrJrKrITF)indent sort_keys ensure_asciirMrc |dS)NrrU)kvs rrz/FSMTTokenizer.save_vocabulary..s Y[\]Y^rrzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr)ospathisdirloggererrorr?VOCAB_FILES_NAMESr`writeradumpsrcrerdsortedrlwarning)rprrr r r frurvrtrwriter bpe_tokens token_indexs rsave_vocabularyzFSMTTokenizer.save_vocabularys/ww}}^, LL,^,<?$!QA?I? GGDJJydQVWZ^^ _ ` +sW 5 +1$..2F2F2HN^+_ ' KK'NN/ }=MM(E SXXj1D89   ~{::' c c@ ` `  s76H'H0 H*.H0A7H<H'*H00H9<IcD|jj}d|d<|S)NrX)__dict__copy)rpstates r __getstate__zFSMTTokenizer.__getstate__s" ""$d  rcZ||_ ddl}||_y#t$r tdwxYw)NrrH)rrVrWrX)rpdrVs r __setstate__zFSMTTokenizer.__setstate__s>     M  s*) NNNNFzzzz)enFr})NF)&__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrorstrintrpropertyrrrrrrrr~rrrrrrrrrboolrrrrrr __classcell__)rzs@rrDrDrs+Z*$&67@ F$4S>$#C##G  E  !!!!??*X%NI7JN5953;DI3F5 c56sx.9.3;DI3F.ko. c.:JNK9K3;DI3FK cKB";c";HSM";]bcf]g";H rrD)rrarr4r;typingrrrrtokenization_utilsrutilsr get_loggerrrrrr7rBrDrUrrrsh% ..5   H %'& (V ,T'Tr