sgN d Z ddlZddlZddlZddlmZ ddlmZmZm Z m
Z
m Z m Z ddl
ZddlmZ ddlmZ ddlmZ erdd lmZ dd
lmZ ej0 e Zd d iZd
Z G d de Z y)z Tokenization class for model T5. N)copyfile)
TYPE_CHECKINGAnyDictListOptionalTuple )import_protobuf)PreTrainedTokenizer)
AddedToken) TextInput)logging
vocab_filez spiece.modelu ▁c
e Zd ZdZeZddgZ d#deee e
f ddf fd
Z d$dZ e
d Zed
Zd Z d%d ee d
eee dedee f fd
Zd Zd Zdee dee fdZ d&d ee d
eee dee fdZ d&d ee d
eee dee fdZd Zd Z dddee f fd Z ed Z d Z d Z d Z!d Z"d&d e d!ee de#e fd"Z$ xZ%S )' T5Tokenizera
Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be
retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids
method
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
legacy (`bool`, *optional*):
Whether or not the `legacy` behaviour of the tokenizer should be used. Legacy is before the merge of #24622
and #25224 which includes fixes to properly handle tokens that appear after special tokens. A simple
example:
- `legacy=True`:
```python
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-base", legacy=True)
>>> tokenizer.encode("Hello .")
[8774, 32099, 3, 5, 1]
```
- `legacy=False`:
```python
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-base", legacy=False)
>>> tokenizer.encode("Hello .") # the extra space `[3]` is no longer here
[8774, 32099, 5, 1]
```
Checkout the [pull request](https://github.com/huggingface/transformers/pull/24565) for more details.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskNsp_model_kwargsreturnc
t |t r
t |d n|}t |t r
t |d n|}t |t r
t |d n|}|i n|| _ || _ || _ t
j di | j | _ | j j | |q|D cg c] } dt | v s| } } t | dk r!|t | D
cg c] }
d|
d
c}
z
}nC|dkD r>|t | k7 r0t d| d| d t | D
cg c] }
d|
d
} }
| }i | _
t t | D ]@ }
t d|
dd
dddd
| j t | j dz
|z |
z
< B |%t j d | j d
d}|| _ | j% |
j' dd
| _ || _ || _ | | _ t+ | X d|||||| j || d|
y c c} w c c}
w c c}
w )NT)specialz
r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensF) single_wordlstriprstripr
normalizedz2You are using the default legacy behaviour of the a_ . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 from_slow) eos_token unk_token pad_token extra_idsadditional_special_tokensr legacyadd_prefix_space )
isinstancestrr
r r
_extra_idsspmSentencePieceProcessorsp_modelLoadlenrange
ValueError_added_tokens_decoderlogger warning_once __class__r% get_spm_processorpopr& super__init__)selfr r r! r" r# r$ r r% r& kwargsx extra_tokensir5 s Y/var/www/html/venv/lib/python3.12/site-packages/transformers/models/t5/tokenization_t5.pyr9 zT5Tokenizer.__init__ sy DT^^DT UJ J
F ..vzz+u/MN
$# 0
&? 0 0 -
I \-Z Is =H2H22
H77
H<c 4 t j di | j }| j s|r |j | j
|S t
| j
d 5 }|j }t d| j j d }|j j | }|j }d|_ |j j! | |j# }|j% | d d d |S # 1 sw Y |S xY w)NrbzThe new behaviour of z (with `self.legacy = False`)Fr' )r+ r, r r% r. r openreadr r5 __name__
ModelProto
FromStringNormalizerSpecadd_dummy_prefixnormalizer_spec MergeFromSerializeToStringLoadFromSerializedProto)r: r tokenizerfr- model_pb2modelrI s r? r6 z T5Tokenizer.get_spm_processor s ..F1E1EF ;;) NN4?? +
$//4
( 8AvvxH'*?@W@W?XXu(vwI ((33H=E ' 6 6 8O/4O , ! ! + +O < ..0H - -h 7 8 8 s
!B"D
Dc | t j v rEt j | }|||k7 r|S |'t j d| d| d| d| d t |S )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning) pretrained_model_name_or_pathmax_model_lengthinit_max_model_lengthdeprecated_max_model_lengths r? !_eventually_correct_t5_max_lengthz-T5Tokenizer._eventually_correct_t5_max_length s (K,M,M M*5*K*KLi*j '$05JN^5^,,&.
34 5 66 734 5$$?#@ Ag
g " c 6 | j j S N)r- get_piece_sizer: s r?
vocab_sizezT5Tokenizer.vocab_size s }}++--r[ c t | j D ci c] }| j | | }}|j | j |S c c}w r] )r0 r` convert_ids_to_tokensupdateadded_tokens_encoder)r: r> vocabs r? get_vocabzT5Tokenizer.get_vocab sK ;@ ;QRa++A.1RR
T../ Ss A token_ids_0 token_ids_1already_has_special_tokensc |rt | ||d S |dgt | z dgz S dgt | z dgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rg rh ri r r )r8 get_special_tokens_maskr/ )r: rg rh ri r5 s r? rk z#T5Tokenizer.get_special_tokens_mask sy $ &72 '[]a 3
C#k**qc1 1c+&&1#-!s;7G1GHA3NNr[ c T t t t d | j S )Nc D t t j d| d uS )Nz)boolresearch)r< s r? z1T5Tokenizer.get_sentinel_tokens.. s bii0A1&E!Fd!R r[ )listsetfilterr$ r_ s r? get_sentinel_tokensz T5Tokenizer.get_sentinel_tokens s& RTXTrTrs t
r[ c f | j D cg c] }| j | c}S c c}w r] )ru convert_tokens_to_idsr: tokens r? get_sentinel_token_idsz"T5Tokenizer.get_sentinel_token_ids s* ?C?W?W?YZe**51ZZZs . token_idsc t | dkD r7|d | j k( r%t j d| j d |S || j gz S )z.Do not add eos again if user already added it.r zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r/ eos_token_idrS rT r )r: r{ s r? _add_eos_if_not_presentz#T5Tokenizer._add_eos_if_not_present s] y>A )B-43D3D"D MM,T^^,< =+ +
1 1 22 2r[ c t | j g}|t ||z dgz S t ||z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )r~ r/ )r: rg rh eoss r? $create_token_type_ids_from_sequencesz0T5Tokenizer.create_token_type_ids_from_sequences) sP ! {S()QC/ /;${2S89QC??r[ c X | j | }||S | j | }||z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)r )r: rg rh s r? build_inputs_with_special_tokensz,T5Tokenizer.build_inputs_with_special_tokens? s; & 22;? 66{CK , ,r[ c D | j j }d |d< |S )Nr- )__dict__copy)r: states r? __getstate__zT5Tokenizer.__getstate__Y s#
""$ j r[ c || _ t | d si | _ t j di | j | _ | j
j
| j y )Nr r' )r hasattrr r+ r, r- r. r )r: ds r? __setstate__zT5Tokenizer.__setstate__^ sO
t ./#%D 22JT5I5IJ
4??+r[ textr c 2 | j st | dk( rt | |fi |S |j t
d }| j r t
|z }t | |fi |}t | dkD r"|d t
k( r|d | j v r|dd }|S )z
Converts a string to a list of tokens. If `self.legacy` is set to `False`, a prefix token is added unless the
first token is special.
r r N)r% r/ r8 tokenizereplaceSPIECE_UNDERLINEr& all_special_tokens)r: r r; tokensr5 s r? r zT5Tokenizer.tokenizeh s
;;#d)q.7#D3F3 3|| ,c2 #d*D!$1&1 v;?vay,< <dNeNeAeABZF
r[ c p t | j j t | j S r] )r/ r- encoder) r! r_ s r? unk_token_lengthz T5Tokenizer.unk_token_lengthz s% 4==''DNN(;<==r[ c 8 | j s|j t df s!| j j |t
S | j j | j |z t
}t | | j k\ r|| j d S |S )u(
Returns a tokenized string.
We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any
SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give
`['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the
`unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`.
`self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`.
r )out_typeN) r%
startswithr r- r r) r! r/ r )r: r r; r s r? _tokenizezT5Tokenizer._tokenize~ s ;;doo/?.E F==''s'; ; %%dnnt&;c%J25f+AVAV2Vvd++-.b\bbr[ c 8 | j j | S )z0Converts a token (str) in an id using the vocab.)r- piece_to_idrx s r? _convert_token_to_idz T5Tokenizer._convert_token_to_id s }}((//r[ c < | j j | }|S )z=Converts an index (integer) in a token (str) using the vocab.)r- IdToPiece)r: indexry s r? _convert_id_to_tokenz T5Tokenizer._convert_id_to_token s
''. r[ c r |d j t r| j r |d dd |d< g }d}d}|D ]P }|| j v r-|s|dz
}|| j j | |z z
}d}g }>|j
| d}R || j j | z
}|j S )z:Converts a sequence of tokens (string) in a single string.r r N Fr T)r r r& r r- decodeappendstrip)r: r current_sub_tokens
out_stringprev_is_specialry s r? convert_tokens_to_stringz$T5Tokenizer.convert_tokens_to_string s !9 0 1d6K6K q !"
F1I
(E///& #%Jdmm 2 23E F NN
"&%'""))%0"'
( dmm**+=>>
!!r[ save_directoryfilename_prefixc t j j | st j d| d y t j j ||r|dz ndt d z }t j j | j t j j | k7 rBt j j | j rt | j | |fS t j j | j sCt |d 5 }| j j }|j | d d d |fS |fS # 1 sw Y |fS xY w)NzVocabulary path (z) should be a directory-r r wb)ospathisdirr3 errorjoinVOCAB_FILES_NAMESabspathr isfiler rB r- serialized_model_protowrite)r: r r out_vocab_fileficontent_spiece_models r? save_vocabularyzT5Tokenizer.save_vocabulary s% ww}}^, LL,^,<zzd NNNT)F)NFr] )&rD
__module__ __qualname____doc__r vocab_files_namesmodel_input_namesr r r) r r9 r6 staticmethodrZ propertyr` rf r intrn rk ru rz r r r r r r r r r r r r r
__classcell__)r5 s @r? r r - s L\ *$&67
"&48 H
"$sCx.1H
H
V" * . . sxO 9O3;DI3FOkoO
cO8
[ 3c 3tCy 3 JN@ 9@3;DI3F@
c@. JN- 9-3;DI3F-
c-4
,[ tCy $ > >c$0
".!c !HSM !]bcf]g !r[ r ) r r ro rS shutilr typingr r r r r r
sentencepiecer+ convert_slow_tokenizerr tokenization_utilsr tokenization_utils_baser
r utilsr
get_loggerrD r3 r r r r' r[ r? r si ' B B 5 5 1 4
H %!>2
R!% R!r[