sg' d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
m Z ddl m
Z
mZ e
rdd lmZ ndZ ej" e Zd
d d Z G d
de Zy)z Tokenization class for model T5. N)copyfile)ListOptionalTuple )PreTrainedTokenizerFast)is_sentencepiece_availablelogging ) T5Tokenizerz spiece.modelztokenizer.json)
vocab_filetokenizer_filec e Zd ZU dZeZddgZeZg Z e
e e d< d fd Z
edefd Zed Zdd ed
ee dee fd Z dd e
e d
ee
e de
e fdZ dd e
e d
ee
e de
e fdZd Zd Z xZS )T5TokenizerFasta`
Construct a "fast" T5 tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
add_prefix_space (`bool`, *optional*):
Whether or not the tokenizer should automatically add a prefix space
from_slow (`book`, *optional*, defaults to `False`):
Whether or not the tokenizer should be converted from a slow one. If `add_prefix_space` is set, this will be set to `True`.
input_idsattention_mask
prefix_tokensc |q|D
cg c] }
dt |
v s|
} }
t | dk r!|t | D cg c] } d| d
c} z
}nC|dkD r>|t | k7 r0t d| d| d t | D cg c] } d| d
} } | }|t j d d | d
< t
| d |||||||d | || _ || _ y c c}
w c c} w c c} w )
Nz
r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokenszXYou set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizersT from_slow)r
r eos_token unk_token pad_token extra_idsadditional_special_tokens )
strlenrange
ValueErrorlogger warning_oncesuper__init__r
_extra_ids)selfr
r r r r r r add_prefix_spacekwargsx extra_tokensi __class__s ^/var/www/html/venv/lib/python3.12/site-packages/transformers/models/t5/tokenization_t5_fast.pyr$ zT5TokenizerFast.__init__V s4 % 0'@[!LTWXYTZDZA[L[< 1$)yIY-ZA
1#Q.?-ZZ)Q 9L0A#A &yk1RSlRm n 8=Y7GH!j1 -HLH(4 % ' j
#'F;
!) &?
%#? \-Z Is CC
C
C returnc p | j r)t j j | j S dS )NF)r
ospathisfiler& s r- can_save_slow_tokenizerz'T5TokenizerFast.can_save_slow_tokenizer s$ 26//rww~~doo.LuL c | t j v rEt j | }|||k7 r|S |'t j d| d| d| d| d t |S )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning) pretrained_model_name_or_pathmax_model_lengthinit_max_model_lengthdeprecated_max_model_lengths r- !_eventually_correct_t5_max_lengthz1T5TokenizerFast._eventually_correct_t5_max_length s (O,Q,Q Q*9*O*OPm*n '$05JN^5^,,&.
34 5 66 734 5$$?#@ Ag
g " r5 save_directoryfilename_prefixc | j s t d t j j | st
j
d| d y t j j ||r|dz ndt d z }t j j | j t j j | k7 r.t | j | t
j d| |fS )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory- r
zCopy vocab file to )
r4 r r0 r1 isdirr! errorjoinVOCAB_FILES_NAMESabspathr
r info)r& r@ rA out_vocab_files r- save_vocabularyz T5TokenizerFast.save_vocabulary s ++
ww}}^, LL,^,< ? r5 token_ids_0 token_ids_1c || j gz }|| j |z S || j gz }| j |z |z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) eos_token_idr )r& rM rN s r- build_inputs_with_special_tokensz0T5TokenizerFast.build_inputs_with_special_tokens sW & "T%6%6$77 %% 3 3%):):(;;K%% 3kA Ar5 c t | j g}|t ||z dgz S t ||z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )rP r )r& rM rN eoss r- $create_token_type_ids_from_sequencesz4T5TokenizerFast.create_token_type_ids_from_sequences sP ! {S()QC/ /;${2S89QC??r5 c T t t t d | j S )Nc D t t j d| d uS )Nz)boolresearch)r) s r- z5T5TokenizerFast.get_sentinel_tokens.. s bii0A1&E!Fd!R r5 )listsetfilterr r3 s r- get_sentinel_tokensz#T5TokenizerFast.get_sentinel_tokens s& RTXTrTrs t
r5 c f | j D cg c] }| j | c}S c c}w N)r^ convert_tokens_to_ids)r& tokens r- get_sentinel_token_idsz&T5TokenizerFast.get_sentinel_token_ids s* ?C?W?W?YZe**51ZZZs .)NNzzzd NNr` ) __name__
__module__ __qualname____doc__rH vocab_files_namesmodel_input_namesr slow_tokenizer_classr r int__annotations__r$ propertyrW r4 staticmethodr? r r r rL rQ rT r^ rc
__classcell__)r, s @r- r r ) s% $L *$&67& !M49! "& -$^ M M M *!c !HSM !]bcf]g !* JNB 9B3;DI3FB
cB6 JN@ 9@3;DI3F@
c@,
[r5 r )rh r0 rX r8 shutilr typingr r r tokenization_utils_fastr utilsr r
tokenization_t5r
get_loggerre r! rH r r r5 r- rw sc ' ( ( > 8 ,K
H %#1EUV @[- @[r5