sg<dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z mZddlmZej"eZdd iZGd d ZGd d eZy)z"Tokenization class for model MyT5.N) defaultdict)DictListOptionalTupleUnion) AddedTokenPreTrainedTokenizer)logging vocab_filezbyte_maps.jsonc eZdZdZdZdeeeeefffdZdeeee e effdedefdZ deeefd eeee e efffd Z d e ed ed e effd Z dde ed e efdZy ) ByteRewriteraZ Byte rewriter class for MyT5 tokenizer. This class is used to rewrite bytes using a hash tree. The hash tree is constructed from a set of rewriting rules. Args: rewriting_rules (`str` or `Dict[str, str]`): A path to a json file containing the rewriting rules or a dictionary containing the rewriting rules. z[LEAF]rewriting_rulesct|tr+t|d5}tj|}dddn't|t st dt||j||_ |jDcic]\}}|| }}}|j||_ y#1swYYxYwcc}}w)NrzDrewriting_rules should be either a path to json file or a dict, got ) isinstancestropenjsonloaddict ValueErrortypeconstruct_hash_tree hash_treeitemsreverse_hash_tree)selfrfkvreverse_rewriting_ruless ]/var/www/html/venv/lib/python3.12/site-packages/transformers/models/myt5/tokenization_myt5.py__init__zByteRewriter.__init__.s os +os+ /q"&))A, / /OT2VW[\kWlVmn 11/B4C4I4I4K"LDAq1a4"L"L!%!9!9:Q!R / /#MsB3 B?3B<rbyte_in_sequencebyte_out_sequencec|jd}|jd}|}|D]}||vri||<||}|||j<y)zL Add a leaf with the output byte sequence to the hash tree.  N)splitLEAF)rrr&r' byte_in_list byte_out_list tree_pointerbs r$add_leafzByteRewriter.add_leaf;sb(--c2 )//4   +A $"$ Q'?L + #0 TYYreturnctt}dtdDD]}|g|||j<|j D]\}}|j ||||S)zE Construct a hash tree for rewritten byte sequences. c3$K|]}|d yw)02xN).0xs r$ z3ByteRewriter.construct_hash_tree..Os1QsG*1s)rrranger+rr0)rrrr/ in_sequence out_sequences r$rz ByteRewriter.construct_hash_treeJss % 1eCj1 *A'(cIaL # **9)>)>)@ @ %K MM)[, ? @r1 byte_sequenceNc\|j}|D] }||vr||} y||jS)zW Search the hash tree and return the rewritten byte sequence if found. N)rr+)rr>r.r/s r$search_hash_treezByteRewriter.search_hash_treeWsA~~  AL +A    DII&&r1in_bytescZg}d}d}|t|kr|s |jn |j}t|t|D]?}||}||vr||}n||k(r|g} |}n$n"|j|vs/||j} |}A|j  |dz}|t|kr|S)a6 Rewrite a sequence of bytes using the hash tree. Args: in_bytes (`List[str]`): A list of bytes to be rewritten. reverse (`bool`): If True, decoding is performed with the reverse hash tree. Returns: `List[str]`: The rewritten byte sequence. r)lenrrr;r+extend) rrAreverse out_bytesb_startb_endr.jr/cur_leafs r$ rewrite_byteszByteRewriter.rewrite_bytesds H %184>>d>T>TL7CM2 QK $#/?L'\ !sHE99 ,+DII6HE    X &aiG!H %$r1)F)__name__ __module__ __qualname____doc__r+rrrr%rrr0rr@rLr6r1r$rr!s D Sc4S>.A(B S 0$sE$S /,B'B"C 0WZ 0or 0 4S> d3PUVZ\`ad\eVePfKfFg  'd3i 'E$S /"`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 125): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary like in ByT5 preprocessing see [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)). additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. input_idsattention_maskr2c |dkDr|t|Dcgc]}d|d }}nK|dkDrF|Dt|dkDr6tttd|} | |k7rt d|d|dt |t rt|dd n|}t |t rt|dd n|}t |t rt|dd n|}|||d |_t|j|_ d |_ tjt|d |_t|jd |_t|jd|_t% |Ld|||d|d|ycc}w)Nrz c.tdt|vS)Nextra_id)boolr)r8s r$z(MyT5Tokenizer.__init__..sDs1v9M4Nr1zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to MyT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)rrCr:r decompose_map merge_map) eos_token unk_token pad_token extra_idsadditional_special_tokensr6)r;rDsetfilterrrrr _added_tokens_decoderoffset_utf_vocab_sizerrr byte_mapsrdecompose_rewritermerge_rewritersuperr%) rr r`rarbrcrdkwargsi extra_tokens __class__s r$r%zMyT5Tokenizer.__init__s q=6>DI)DT(Uq:aS):(U %(U ]8DMfIgjkIks6*NPi#jklLy( &yk1RSlRmn(( HRR[]`GaJydCgp GQR[]`GaJydCgp GQR[]`GaJydCgp )2yY%O"$445 #4 C#89".t~~o/N"O*4>>++FG  &?    3)Vs E1c|jSN)ri)rs r$ vocab_sizezMyT5Tokenizer.vocab_sizes###r1ct|j|jzDcic]}|j||}}|j |j |Scc}wrs)r;rtrhconvert_ids_to_tokensupdateadded_tokens_encoder)rrovocabs r$ get_vocabzMyT5Tokenizer.get_vocabsW;@SWS^S^A^;_`a++A.1`` T../ asA token_ids_0 token_ids_1already_has_special_tokensc|rt|||dS|dgt|zdgzSdgt|zdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r{r|r}rrC)rmget_special_tokens_maskrD)rr{r|r}rqs r$rz%MyT5Tokenizer.get_special_tokens_masksy$ &72'[]a3   C#k**qc1 1c+&&1#-!s;7G1GHA3NNr1 token_idsct|dkDr7|d|jk(r%tjd|jd|S||jgzS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)rD eos_token_idwarningswarnr`)rrs r$_add_eos_if_not_presentz%MyT5Tokenizer._add_eos_if_not_presents] y>A )B-43D3D"D MM,T^^,<=++   1 122 2r1ct|jg}|t||zdgzSt||z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. MyT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r)rrD)rr{r|eoss r$$create_token_type_ids_from_sequencesz2MyT5Tokenizer.create_token_type_ids_from_sequencessP   !  {S()QC/ /;${2S89QC??r1cX|j|}||S|j|}||zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r)rr{r|s r$ build_inputs_with_special_tokensz.MyT5Tokenizer.build_inputs_with_special_tokenss;&22;?   66{CK, ,r1textc r|jdDcgc]}|d}}|j|}|Scc}w)zTake as input a string and return a list of strings (tokens) for words/sub-words. Represents tokens in two character hex formatutf-8r5)encodemorphological_encode)rrrnrotokenss r$ _tokenizezMyT5Tokenizer._tokenize6s@'+kk'&:;QsG*;;**62 s6 u:?H5"~ 3Hr1c(||jz d}|S)z=Converts an index (integer) in a token (str) using the vocab.r5)rh)rindexrs r$_convert_id_to_tokenz"MyT5Tokenizer._convert_id_to_tokenHs4;;&s+ r1indicescz|jj|d}|jj|d}|S)NFrF)rkrLrlrrs r$rz"MyT5Tokenizer.morphological_encodeMs=))777O%%33GU3Kr1cz|jj|d}|jj|d}|S)NTr)rlrLrkrs r$morphological_decodez"MyT5Tokenizer.morphological_decodeSs=%%33GT3J))777Nr1cd}g}|D]`}||jvr|j|j|0||jvr|j|P|j|b|j|}t |jj t |jz}|D].}||vr|t |dz }|t j|z }0|jdd}|S)z:Converts a sequence of tokens (string) in a single string.r1rignore)errors) added_tokens_decoderappendrxrrevaluesbytesfromhexdecode)rrbstring out_tokensr _added_tokensstrings r$convert_tokens_to_stringz&MyT5Tokenizer.convert_tokens_to_stringYs  )E111!!$";";E"BC$333!!%(!!%(  )..z: D55<<>?#dF_F_B``  0E %5005==//  0 9 r1save_directoryfilename_prefixcntjj|r2tjj||r|dzndtdz}n |r|dznd|z}t |dd5}|j tj|jdd ddd|fS#1swY|fSxYw) N-r wr)encodingr]F)indent ensure_ascii) ospathisdirjoinVOCAB_FILES_NAMESrwriterdumpsrj)rrrr writers r$save_vocabularyzMyT5Tokenizer.save_vocabularyps 77== (/3!6rUfgsUt tJ4C/C/n\J *cG 4 S LLDNN15Q R S} S}s ,2B))B4)zzz}N)r2N)NFrs)rMrNrOrPmodel_input_namesrvocab_files_namesr%propertyrtrzrrrrYrrrrrrrrrrrrr __classcell__)rqs@r$rRrRs4%&67) "&,  , \$$sxO9O3;DI3FOkoO cO8 3c 3tCy 3JN@9@3;DI3F@ c@0JN-9-3;DI3F- c-4cS  DI$s) DI$s) . c HSM ]bcf]g r1rR)rPrrr collectionsrtypingrrrrrtokenization_utilsr r utilsr get_loggerrMloggerrrrRr6r1r$rs`) #55A   H %"#34ccLr'rr1